It may be all well and good touting the efficacy of your blockchain offering, but does it really stack up under scrutiny?
A new paper from the Journal of the British Blockchain Association (JBBA) has explored blockchain technologies on the basis of evidence-based practice (EBP) – with sceptical results.
Evidence-based blockchain (EBB) is defined by the journal as ‘conscientious, explicit and judicious decision making based on professional expertise and evidence from organisations, stakeholders and scientific research.’ Its application, the JBBA notes, is useful in various ways: defining a clear problem which blockchain technologies can solve; adding scientific evidence to improve processes; and better reporting of potentially challenging results.
This includes defining more evidential outcomes. Filtered evidence, the strongest suit, requires peer-reviewed analysis, original research, case studies or critical reviews published in academic, peer-reviewed journals. Presentations at scientific or academic conferences also counts. Unfiltered evidence, the next rung down, relates to blog posts, whitepapers and presentations at tech events.
The Centre for Evidence Based Blockchain (CEBB), part of the British Blockchain Association, analysed 417 blockchain projects and startups launched between 2016 and 2020. Alongside this, an Angel.co database of more than 4,800 companies was scrutinised.
Of the companies analysed, only 7% displayed filtered evidence when looking at a ‘clearly defined problem’. 62% had unfiltered evidence, with 31% – perhaps worryingly – offering no evidence. When it came to looking for critical reviews of existing solutions, on average filtered evidence dropped to 5.8%. For scrutinised findings of intervention – why the company’s proposed solution is different or better than others on the market – filtered dropped further, to 4.57%.
The journal gives genuine examples of sweeping, spurious statements. If you read a whitepaper which says something along the lines of ‘current processes are slow and inefficient’, ‘one of the biggest challenges faced by governments around the globe’, or ‘our blockchain solution will transform the way data is managed around the globe’, approach with extreme caution.
In total (above), almost half of blockchain firms assessed showed ‘no explicit evidence of the problem to be solved.’ One third did not have a comparison and intervention analysis, while only 2% demonstrated evidence of outcomes backed by critical, peer-reviewed information.
So why is this important, and what should happen from here? While the British Blockchain Association has solid credentials – albeit with a rather grand claim of being ‘the world’s most prestigious and the most influential voice on blockchain’ – it’s not surprising, given its advocacy of EBB, that this report would be on the critical side.
Yet there is plenty for blockchain companies to learn. The association rejects any idea that organisations should shy away from this practice because it is too academic, and recommends an ‘evidence assessment framework’ for all DLT projects. This involves three ideas:
- Governments, organisations and enterprises looking to invest in blockchain projects need to ensure uses are based on scientific evidence. “For every £100 spent on blockchain and distributed ledgers, we propose that at least £2 should be dedicated to making sure the other £98 actually works,” as the report puts it
- Organisations deploying blockchain should consider a ‘chief evidence officer’ role “to ensure that blockchain products, services and solutions are built on best available scientific evidence”
- But it shouldn’t stop there. Policymakers, C-suite executives and investors need to be equipped with the ‘fundamental skills’ of EBB, the report adds. “Any investments of time and resources into blockchain projects must be preceded by critical appraisal of the strengths and weaknesses of their project and its potential long-term impact,” it concludes
You can read the full paper here (pdf, no email required).
Interested in hearing more in person? Find out more at the Blockchain Expo World Series, Global, Europe and North America.