Rigour, reproducibility and data-sharing requirements

A research proposal makes its rigour assessable and sets out how its data will be managed and shared. NIH requirements illustrate how these commitments enter the research strategy and a separate sharing plan.

NIH rigour requirements and review criteria

NIH announced its Initiative to Enhance Research Rigor and Reproducibility in October 2013 and implemented it on 25 January 2016 across its institutes and centres. ^{1, 2} The four topics described in 2017 were: ¹

1. Scientific premise: prior research forming the basis of the proposed question, considered in the Significance score.
2. Rigorous experimental design: robust, unbiased design, methodology, analysis, interpretation and reporting, considered in the Approach score.
3. Sex and other relevant biological variables: consideration in design, analysis and reporting for vertebrate animal and human studies, with strong justification from literature, preliminary data or other relevant considerations for studying one sex only.
4. Authentication of key biological or chemical resources whose quality could affect the data, such as cell lines, specialty chemicals, antibodies and other biologics.

¹

NIH revised its rigour and transparency application instructions and review criteria in late 2015 and updated them in 2018. ³ In 2021 reviewers assessed prior research, proposed work, biological variables and authentication. ^{4, 5, 6, 7}

Prior and proposed research

The scientific premise should look back at strengths and weaknesses of research across the field and explain how they inform the proposed approach; significance looks forward to the proposed research’s effect on the field and human health. ^{8, 9} Premise should not be confused with significance or merely with the applicant’s preliminary data. ⁹ An account limited to the applicant’s positive findings can bias reviewers’ understanding; competing viewpoints and other groups’ work should be included. ⁴

The 2021 checks for prior research ask: ⁵

• Does Significance refer to other groups’ work?
• Does it critically appraise the technical or intellectual rigour of the applicant’s and others’ prior work?
• Does it say whether that work produced consensus or controversy?

⁵

Approach should justify experiments against weaknesses in prior work, identifying a milestone that made the work possible or explaining how the proposed work attempts to overcome a controversy. ⁵ Experimental and statistical detail should show that the team can navigate the field’s pitfalls, with reporting guidance such as ARRIVE helping establish the necessary design detail. ⁵

The proposed-work checks ask: ⁶

• Does Approach explain how the plans fill gaps left by prior work?
• Is methodological detail sufficient to show knowledge of the field’s pitfalls and how to avoid them?
• Is there a statistical analysis plan, including power analysis where appropriate?

⁶

Biological variables and resource authentication

Biological variables include intrinsic factors such as sex, weight, age and genetic background, and extrinsic factors such as animal food source and housing. ⁶ The 2021 account expected experimental design and analysis to be justified against relevant variables, with reviewers considering sex in human or vertebrate-animal work and requiring strong justification for one-sex experiments. ⁶

The biological-variable checks ask: ⁷

• Does Approach consider sex in the design and analysis of human or vertebrate-animal work?
• Does it consider other variables, especially those prior research shows to be important?
• Are proposed analyses of relevant variables sufficiently powered for meaningful results?

⁷

In 2017 a university editing group suggested, where using both sexes throughout was financially difficult, validating key findings in both while conducting most work in one; its investigator and reviewer panel affirmed that approach. ⁹ Another suggested option was preliminary data from both sexes showing that a specific phenotype was not sex-dependent. ⁹

Unreliable or mislabelled reagents can undermine reproducibility, so the 2021 account called for a brief plan to validate key resources, including cell lines, antibodies, specialty chemicals and transgenic animals, with commercial validation included where applicable. ⁷

The authentication checks ask: ¹⁰

• Is there a separate authentication plan for key biological or chemical resources?
• Does it contain resource and reagent information rather than preliminary data or methods?
• Does it specify how and how often resources will be authenticated?

¹⁰

Headings and reviewer understanding

In 2017 the University of Arkansas for Medical Sciences editing group used “Scientific Premise for the Proposed Project” within Significance and “Robust and Unbiased Results” within Approach. ^{1, 9} Comments attached to these headings prompted critical assessment of prior research and explanation of robust design and relevant biological variables, with further variable-specific detail in methods where applicable. ¹

A campus survey focused on scientific premise and robust, unbiased results received 39 complete and nine partial responses from 160 investigators, a 30% response rate. ^{11, 12, 13} Among respondents, 68.75% received or expected NIH funding and 38% had served on a study section since implementation. ^{13, 14} Of 39 answering the familiarity question, 38.5% were very familiar, 56.4% somewhat familiar and 5.1% unfamiliar; among 18 study-section members, 61.1% were very familiar and 38.9% somewhat familiar. ^{15, 14} Among those 18, 55.6% disagreed or strongly disagreed that most applicants fully understood rigour and reproducibility. ^{16, 13}

Discipline selections allowed multiple responses: “other” accounted for 32.6%, cell biology 19.8% and neuroscience 10.9% of the reported discipline distribution. ^{8, 13} Neuroscience respondents all reported being very familiar; most cell-biology and “other” respondents were somewhat familiar, and only the “other” group reported unfamiliarity. ¹³ The editing group thought “other” largely represented nursing. ¹³

Among study-section members, 38.9% estimated that a quarter of applications used a distinct rigour section, while 27.8% estimated three quarters did. ^{17, 14} Across respondents, 59% found distinct headings helpful and 41% did not. ¹³ Of the 18 study-section members, 61% agreed and 28% strongly agreed with the retrospective-premise and prospective-significance definitions. ^{8, 9, 14}

Two reviewers’ comments emphasised identifying prior work’s strengths and weaknesses and assessing whether a balanced account justifies the proposed design. ⁸ A funded non-reviewer respondent found the requirements difficult and advocated reviews that help applicants improve as a culture of rigour develops. ⁸ The editing group retained its two headings, judging them broadly aligned with reviewer expectations, and identified workshops, investigator–reviewer panels and online material as educational routes, potentially targeted to less-familiar disciplines. ^{9, 18} These findings came from a small, single-campus sample potentially affected by recall bias and did not support definitive conclusions; similar local surveys were proposed as an inexpensive way to identify needs. ⁹

Data management and sharing policy

NIH’s 2003 policy required investigator-initiated applications seeking more than US$500,000 in direct costs annually, or applications covered by a specific funding announcement, to include a final-data sharing plan or justify not sharing, for example on privacy, third-party agreement or national-security grounds. ¹⁹ The 2014 Genomic Data Sharing policy extended expectations to genomic data. ¹⁹ In 2021 the updated policy released in 2020 was described as requiring a data management and sharing plan for all NIH applications from 2023, with FAIR — findable, accessible, interoperable and reusable — principles. ^{3, 19}

For that updated policy, the drafting checks covered data and repositories, preservation and access timelines, and limits on sharing. ^{19, 20, 21} Decisions on data preservation and sharing should have ethical, legal and technical justification, and existing FAIR-aligned repositories were strongly encouraged; an NIH repository was not always necessary. ¹⁹

Data types, standards and repositories

The data and repository checks ask: ²⁰

• Does the plan summarise the data and metadata to preserve and share?
• Does it name common standards or state that none apply?
• Does it identify appropriate NIH or third-party repositories?
• Does it state limits on the archive’s data types and affirm a designated NIH repository where the funding announcement requires one?

²⁰

Preservation and access timelines

Data and metadata should be preserved at least as required by repository, award and journal policies, and made accessible by publication or the end of the performance period, whichever comes first, unless the plan explains a justified exception. ²⁰

The preservation and access checks ask: ²¹

• Does the plan follow FAIR principles for identification, access and reuse?
• Does it explain how shared data and metadata will be findable and identifiable?
• Does it affirm an acceptable sharing timeline?
• Will specialised tools needed for access or reuse remain available as long as the data are shared, and how will they be accessed?

²¹

Limits and responsibility

NIH expected sharing to the maximum extent allowable, with ethical, policy or legal justification for limits on access, reuse or distribution of confidential, sensitive or proprietary material. ²¹

The limitation checks ask: ²²

• Is exclusion of confidential, sensitive or proprietary data or metadata reasonably justified?
• Are restrictions stated, such as requiring explicit approval for access, reuse or distribution?
• If there are no limits, does the plan say so?

²²

The plan also needs to state how it will be managed and monitored and who is responsible. ²²