Reproducibility and the research-integrity environment

Reproducibility depends on research practices and on the institutions, incentives and group cultures in which research is conducted and published.

Reported failures to reproduce research

In a 2016 survey of 1,576 researchers, more than 70% had tried and failed to reproduce another scientist’s experiments and more than half had failed to reproduce their own. ¹ A 2016 editorial criticised most journals for preferring innovation and refusing replication studies and described replication conventions and vocabulary as underdeveloped. ¹ Researchers could report replication attempts on blogs, preprint servers or in journals that did not require novelty, including dedicated replication sections. ¹

A 2015 Lancet editorial argued that much of the literature, perhaps half, might be untrue, citing small samples, tiny effects, invalid exploratory analyses, conflicts of interest and fashionable questions of dubious importance. ²

Practices supporting reproducibility

A 2015 expert assessment identified six problems. ²

• Data dredging.
• Omission of null results.
• Underpowered studies.
• Errors.
• Underspecified methods.
• Weak experimental design.

²

It proposed seven possible strategies to improve reproducibility. ²

• Open data.
• Preregistration.
• Collaboration.
• Automation.
• Open methods.
• Postpublication review.
• Reporting guidelines.

²

A 2016 Nature editorial called for four responses to make replication more useful. ¹

• Researchers should open their file drawers.
• Authors should cooperate with reasonable primary-data requests, assume good intent, and write papers and keep records expecting that others will try to replicate their work.
• Funders and publishers should support tools that connect the literature.
• Nature would welcome and help disseminate studies testing the validity of important publications, including its own.

¹

National integrity measures

After international scandals, China’s Ministry of Science and Technology issued a 2006 decree on misconduct in government-funded research and established an Office of Research Integrity and an inter-agency Joint Committee in 2007. ³ The committee issued Opinions on Strengthening Research Integrity of our Country, while the ministries responsible for science and education, the Chinese Academy of Sciences and the China Association for Science and Technology organised ethics workshops. ³ Chinese representatives also joined international standard-setting through a global network of scientific academies and the third World Conference on Research Integrity. ³

Medical-writing workshops and culturally appropriate free social-media resources promoted ethical practices and international reporting standards. ³ A 2013 assessment stressed that publication-ethics problems were not unique to China and that improving publication quality required sustained cooperation across sectors. ³

A programme launched on 29 November 2012 by the China Association for Science and Technology and Ministry of Finance allocated ninety million yuan to improve journals’ quality, influence and international competitiveness. ³ Its first round planned annual grants of 500,000–2 million yuan for three years to each of thirty-five English-language Chinese journals. ³

A 2015 commentary from an Iranian perspective described cultural norms and financial constraints being used to justify guest authorship or manipulated results and speculated that international sanctions could encourage such rationalisation. ⁴

Research-group culture

Research-integrity specialists argued in 2018 that focusing only on individual misconduct can obscure environments that promote it; persistent frustration, unhappiness or low productivity can also signal deeper problems. ⁵ A survey described inadequate research training, poor personnel management as a major contributor to unhealthy laboratory culture, and a divide between senior and junior researchers. ⁵

At the Cancer Research UK Beatson Institute, an integrity adviser helped teams conduct research and collaborate, including developing practices to curate and preserve all data supporting a paper, replicates included. ⁵ The institute reported no retractions or serious publication problems over five years. ⁵

The TRAGEDIES mnemonic identifies pressures that can lead to poor decisions. ⁵

• Temptation.
• Rationalisation.
• Ambition.
• Group and authority pressure.
• Entitlement.
• Deception.
• Incrementalism.
• Embarrassment.
• Stupid systems.

⁵

Recognising these pressures and responding appropriately can protect a career and strengthen science. ⁵