Open and post-publication peer review
Peer review can be changed in several ways: replaced by open science, opened by publishing reviewers' reports and names, moved before results are known, or extended after publication by letting readers comment.
Replacing peer review with open science
Open science, which means making the results of scientific research freely available and using the power of social networking to replace peer review, has been put forward as a possible solution to complaints about the peer-review process. Some scientists feel that peer review is hidebound, expensive and elitist and should be replaced by open science.
One commentator in 2012 judged these criticisms probably reasonable and noted that peer-reviewed journals are expensive to run, but held that open science does not completely resolve the cost, because professional curation and preservation of data are themselves time-consuming and expensive. In the same view, the hidebound and elitist charges are a bit vague and probably mean two things. There can be a political side to getting published, in that who the authors are and which institutions they belong to can affect publication when the quality of the science alone should decide. Reviews are also sometimes insufficient, so that bad science gets published while good, innovative science is sometimes blocked because it contradicts existing dogma. Although improvements were being made, publishing research results should, in this view, be better, faster and cheaper and should take better advantage of electronic media.
In the commentator's view, the examples of open science put forward did not replace peer review. Online-only open-access journals do not eliminate peer review, so they are not truly open science. ResearchGate, an interactive website where scientists pose and answer each other's questions, was in 2012 great for sharing ideas but not a site for publishing research results or for reviewing them, and sharing ideas is not the same as an in-depth critique of study results. Using the internet for post-publication review of research, on the other hand, the commentator judged a great idea.
The commentator judged it not yet clear whether many people critiquing an article through a social network can reach the depth of review that two or three dedicated peer reviewers reach, and held that experience with Wikipedia shows that using social networking in place of true peer review carries risks of abuse and misinformation. The commentator concluded that, for the moment, peer review is the only system for the in-depth evaluation of research and its conclusions, and that open science seems a great way to improve information sharing and access to published research, but whether it is a good replacement for peer review remains to be seen. Changes are coming, in this view, but what they will look like is not yet obvious.
Publishing reviewers' reports and names
Open peer review, one possible way forward for peer review, gives the reader full transparency by making the names and affiliations of the reviewers and their comments available online. One argument for it, made in eLife in 2015, is that referee reports are helpful and hiding them is an enormous waste.
Elsevier planned a pilot through 2012 that would publish peer reviewers' comments alongside the articles of one of its journals on its platform. Reviewers were to be informed before their comments were published and given the option of having their name included, and only good-quality reviews were to be published. The hope was that this would attract better reviewers and improve the value of the articles, and if the pilot succeeded, the project was to be extended to other journals in the portfolio.
Evidence that open review improves the quality of reviewers' comments comes from a study that used an online game to compare open and closed peer review. When reviewers' anonymity was removed, they spent more time reviewing, their reviews were more accurate, and they formed significantly more cooperative interactions with authors, all of which could reduce the risk of errors in reviewing.
In 2009, Richard Smith, a former editor of the BMJ, held that peer review has the potential to become an open scientific discussion, prominently linked from the primary publication and available for any later reader to peruse.
Registered reports
The Center for Open Science defines the registered reports model as peer review before results are known, to align scientific values and practice. Registered Reports is a publishing format that emphasises the importance of the research question and the quality of the methodology by conducting peer review before data are collected. Review happens at two stages: once the study is designed, and again once the report is written. High-quality protocols are provisionally accepted for publication if the authors follow through with the registered methodology. A manuscript that survives the review before the study receives an in-principle acceptance that is not revoked because of the outcomes, but only for failings of quality assurance or of following through on the registered protocol, or for unresolvable problems of clarity or style in the reporting.
In 2019 the format was used by over 200 journals, and medical journals offering it alongside their other reviewed articles included BMC Medicine, BMJ Open Science and Cancer Medicine. In 2020, 242 journals used it.
The format eliminates a variety of questionable research practices, including low statistical power, selective reporting of results and publication bias, while allowing complete flexibility to report serendipitous findings. A comparison published as a preprint, not yet peer reviewed, set all 71 Registered Reports published in psychology by November 2018 against a random sample of 152 hypothesis-testing studies from the standard literature, found by searching 633 journals. For the first hypothesis reported in each paper, 96% of standard reports had positive results against 44% of Registered Reports, and the difference remained nearly as large, 96% against 50%, when direct replications were excluded. The authors read this large gap as suggesting that psychologists under-report negative results to an extent that threatens cumulative science.
Post-publication commenting
Post-publication comments have proved a fast and efficient way of raising concerns about potential problems in published papers, as three controversial reports showed: one claiming that a bacterial DNA sample contained arsenic instead of phosphorus, one claiming that RNA sequences contain widespread edits relative to the corresponding DNA sequences, and one on stimulus-triggered acquisition of pluripotency. The same 2015 argument in eLife encouraged post-publication debate and urged journals to take the lead in considering non-anonymous comments posted on electronic platforms.
In 2014, Faculty of 1000 was a web platform whose resources included post-publication reviews of biomedical research papers, a repository for conference posters and presentations, and a manuscript publishing platform that made peer reviewers' comments available to the public.
PubMed Commons was created by the US National Center for Biotechnology Information (NCBI) to let researchers post comments on abstracts within the PubMed database, as a kind of post-publication peer review system for indexed articles. According to NCBI, it was created in response to repeated requests from the scientific community for such a forum to be part of PubMed. It was intended as a centralised area for researchers to engage in constructive criticism and discussion of scientific issues within the articles found in PubMed. Before it, online post-publication review was spread out and disjointed. Some publishers experimented with allowing comments on their articles under their own rules: the New England Journal of Medicine let registered users comment on articles that are not indexed and only viewable on its website, and the BMJ allowed rapid responses, brief letters to the editor posted online almost immediately after editors deemed them appropriate, of which selected ones appeared in the weekly journal and were indexed in PubMed. Apart from these, there was no central site for commenting on a particular article, and reviews and comments appeared on blogs or Twitter.
Anyone could read the comments, but who could post, and under what name, was restricted. Posting was open only to recipients of NIH or Wellcome Trust grants and to authors of any article listed in PubMed, anonymous accounts were not permitted, and people had to post under their real first and last names. Commenters had to abide by guidelines, comments could be removed as inappropriate, and NCBI reserved the right to suspend or cancel the accounts of members who did not adhere to the guidelines. Both restrictions were the subject of great debate while the system was being established and remained so. Some worried that the posting restriction was too narrow, since it kept graduate students, patient advocates, science journalists and others in the health care and health information community from taking part. The group of scientists who beta-tested the system remained divided on anonymous comments, some fearing that junior researchers and other scientists would be reluctant to write critical comments or questions.
Although PubMed Commons was created for researchers to engage in discussion, authors of papers also used it to add supplemental research and material. According to the PubMed Commons blog, almost 1 in 5 comments were made by the authors themselves, and the most common author comments added study data or linked to free full text, additional materials, or updates related to the article. Members could rate comments for their helpfulness, and each comment had a permalink, which made it easy to share or cite. Comments were used to:
- annotate publications that misreport the identities of known cross-contaminated cell lines
- link to blog posts and news coverage that add information about a paper on research misconduct, or about a paper on peer review
- note that an article has been retracted and give background on the circumstances of the retraction
- provide supplemental information to one's own articles
- ask the author questions
PubMed Commons went live in October 2013 and was still in its pilot phase in 2014. As of February 2014 its team reported over 3,000 members and over 1,000 comments, against more than 23 million articles in PubMed, so even if it succeeded, it might still be a while before comments were regularly seen on retrieved PubMed citations. It was not yet clear how popular it would become, but many comments posted by then were of potential interest to medical communicators, such as a medical writer's comment on the disclosure of nonfinancial conflicts of interest. Other sites for online post-publication review existed, among them blogs and websites such as PubPeer and ResearchGate, and it remained to be seen whether having comments embedded in an already established product would outweigh any limitations of the commenting system and sway researchers away from competing methods of online review.