Scientists are facing a ‘reproducibility crisis’

Earlier this year the Reproducibility Project at the University of Virginia released the first sets of analysed data from their study on 29 cancer biology papers. The study is part of a wider project to improve reproducibility across scientific papers and publications by independently repeating the experiments.

Reproducibility refers to the ability of scientists to repeat an experiment and produce results supporting those published by following the method detailed.

The reproduced results released were from five experimental studies, of which only two experiments were deemed to be “substantially reproduced”. Two had “uninterpretable results” and one failed completely. Tim Errington, who manages the project, explained that this may be due to a lack of details regarding method in scientific papers rather than fraud in the published results.

The scientific journal Nature sent a reproducibility survey to over 1,500 researchers in 2016 in which 90 per cent of respondents said they believed there was a reproducibility crisis in science. Increasingly, papers are being found to be irreproducible. The same survey in Nature found that over two-thirds of respondents had tried and failed to reproduce results found by other scientists.

Those surveyed identified the main issues negatively impacting the reproducibility of papers were selective reporting and pressure to publish. Furthermore, 80 per cent thought that funding bodies should do more to improve the reproducibility of papers, as they often choose the most exciting and unexpected results to pull attention to the studies they sponsor.

There are differing opinions across the scientific community in regards to the Reproducibility Project. Some scientists worry that failure to reproduce results in their experiments may lead to loss of funding; if the repeat experiment involved mistakes or problems it cannot be restarted to prevent replication bias.

However, reproducibility is at the core of evidence-based science. Despite the potential of negative implications for researchers from the reproducibility project, it promotes skepticism in reviewing and publishing papers. The project is undoubtedly a step towards pulling the field out of the ‘crisis’ it is currently facing.