What peer review can and cannot catch
Peer review is advice to an editor, not a verdict on a manuscript, and what it can be trusted to catch is limited. Reviewers are swayed by an author's prestige and institution, by whether they agree with the conclusion, by positive results, by language, culture and gender, and by their own errors of reasoning. Reviewers and journals have interests of their own, and an author can nominate a reviewer who turns out to be a fabrication. Reviewers miss many of the errors a manuscript carries and are not good at detecting fraud, and the reforms proposed against these limits, from shared reviews to automation, have been slow to change an entrenched routine.
What peer review is for
Peer review is a way of getting advice from workers in the field that can help an editor make a good decision about accepting or rejecting a manuscript, and expecting it to be the whole process, or to detect fraud, leads to disappointment. It was not designed to detect fraud, is not asked to and is not very good at it, and even at what it is supposed to do its quality suffers from the limited time reviewers have for this unpaid work and from the difficulty of setting aside their own biases.
Peer review is accused of being subjective, slow, expensive, biased and poor at detecting errors, yet no better system has replaced it, and most researchers remain confident in it because it is shown to improve papers. A former editor of the BMJ considers it slow, expensive and wasteful, and one textbook author adds that nothing better is on offer, so medical writers have to cope with its inefficiencies and caprice.
From the randomised trials and observational studies of 2019, one health-care advocate drew the messages that peer review might sometimes act as academic matchmaking, increasing the chance of future collaboration; that reviewers may provide no line of defence against authors' conflicts of interest; and that editorial peer review may be increasing the acknowledgement of study limitations without reducing spin.
Bias in reviewers' judgements
Who an author is, where they work, what language they speak daily and who reviews the paper may profoundly influence whether it is accepted.
Prestige and agreement can bias reviews. In studies in which a manuscript's institution of origin was masked or misstated, the putative institution predicted the final publication recommendation. Journals that have experimented with masking found that it does not work very well: self-citations, lines of investigation and writing style may reveal authors to referees close to the field, and only perhaps a third of manuscripts can have their origin masked. In Ernst's experiment with a rehabilitation paper deliberately engineered with flaws and strengths, reviewers whose own publications agreed with its conclusion spotted more strengths and fewer weaknesses and gave more favourable publication scores.
Other experiments point to prestige, and to positive results. In 1982 Peters and Ceci resubmitted 12 articles already published in prestigious American psychology journals to the same journals, retyped under fictitious author names and institutions. Only three of 38 editors and reviewers detected the ruse, eight of the nine articles that went on to a decision were rejected, and 16 of 18 referees recommended against publication, often citing serious methodological flaws. Peters and Ceci considered the possibility that systematic bias produced the discrepant reviews, and named the authors' status and institutional affiliation as its most obvious candidate sources. Okike and colleagues found reviewers more likely to recommend acceptance when prestigious authors' names and institutions were visible than when they were redacted. Emerson and colleagues sent reviewers for two orthopaedic journals two versions of a fabricated, well-designed randomised trial that differed only in whether the principal end point was positive or showed no difference, and reviewers recommended the positive version for publication more often, 97.3% against 80.0%.
Acceptance rates for manuscripts from authors whose first language is not English have historically been lower than for native English speakers, though this depends on the medical area and varies over time. One textbook reports that a study by Coates and colleagues found the acceptance rate of authors whose mother tongue is not English generally a lot lower than that of native English speakers: at Cardiovascular Research the US/UK acceptance rate of 30.4% was higher than for all other countries, and the lowest acceptance rate, 9% for Italian authors, coincided with the highest error rate. The textbook's author takes the fundamental issue to be language problems rather than discrimination, and reports that the study's authors concluded that with articles of equal scientific merit a poorly written one is more likely to be rejected. A medical writer reports that a 2002 Italian survey of articles in the same journal, analysing a sample of 120 articles without knowledge of the authors' nationality and cross-checking, found a control group of American and British authors with almost the same acceptance rate, 30.4%, and almost the same overall error rate as the test group. In that survey no direct relationship was found between acceptance rate and number of language errors, although badly written articles clearly correlated with a high rejection rate.
An Iranian physician who was a visiting editor at the BMJ argued that what he calls the culture of compliments in Eastern countries bears on ethical conduct in medical publishing: in most Eastern countries extra respect is given to elders and teachers in accordance with traditional religious belief, so a reviewer may think it proper to overlook shortcomings in an article written by a former professor.
Gender is a measured source of difference too, though the mechanism is not settled. A study of more than 23,000 manuscripts submitted to six ecology and evolution journals between 2010 and 2015 found papers with female and male first authors equally likely to be sent for review, but those with female first authors received slightly worse review scores and were more likely to be rejected after review, varying by journal. Part of the difference tracked authorial roles: papers whose first author deferred corresponding authorship to a co-author, which women do more often, scored worse and fared worse editorially. A survey of more than 12,000 published papers found women reporting similar rates of prior rejection. The authors could not test the mechanisms but concluded that papers authored by women had lower acceptance rates in that field. By 2020 a columnist on peer-review research counted among what was already known that women's articles are rejected more often than men's.
Reviewers' errors of reasoning are a bias of their own, and one medical writer argues from her own experience that editors should review the reviewers' reviews. Shortly before DSM-5 appeared she wrote an article arguing that conversion disorder and somatization disorder should never have been included in DSM-III, because they rested on unfounded speculation about aetiology rather than a case definition of a clinical syndrome, against that edition's own guiding principle. A reviewer recommended rejection on the ground that conversion disorder is not an aetiological diagnosis, citing an article whose first paragraph said that it is, and the editor, shown the flaw, let the rejection stand. The article was published in Medical Hypotheses, and Allan Frances, who had chaired the committee that compiled DSM-IV, then argued on a blog that its point was true and important. A friend of hers in the social sciences had a paper rejected by a blinded reviewer who told her to read the earlier articles on the model it applied, which were her own, and again the editor backed the reviewer.
In her view, articles saying something unorthodox, unexpected, or politically or commercially inconvenient are particularly likely to be rejected for bad reasons, whatever their importance, though not everything rejected for bad reasons deserves publication. She blames human fallibility and an education that teaches facts and doctrines rather than critical reasoning, so that highly credentialed reviewers know their field's conventional wisdom but are poorly equipped to judge groundbreaking work, which she takes to underlie the phenomenon Thomas Kuhn described in The Structure of Scientific Revolutions, and cannot tell that their reasoning is weak, the Dunning-Kruger effect. Editors should therefore spend more time on peer review, not less, reviewing the reviewers' reviews and paying attention to authors' rebuttals. Handing the work to artificial intelligence, then being discussed as a way to speed review, would in her view amplify the problem, because an algorithm follows conventional wisdom and cannot have the "aha" moment.
Conflicts of interest
Conflicts of interest are a further risk, especially in focused research communities where the few people expert enough to review a paper may be racing towards the same discovery; one textbook author suspects that a reviewer might use information from a paper under review or criticise unfairly a paper that seems a few months ahead of their own.
Editorial revenue can create interests too. According to a former editor of the BMJ, some journals make millions of dollars from reprints of drug-company-sponsored trials: the Lancet makes more from such reprints than from subscriptions or straight advertising, the manufacturer bought a million dollars' worth of reprints of the VIGOR study published in the New England Journal of Medicine, and on a million dollars' worth of reprints the profit is about $700,000, maybe $800,000. He calls this a very stark conflict of interest for an editor who knows which trials will make that kind of money, more stark than many researchers experience. For a researcher whose report will not inspire a company to buy thousands of reprints, it is a handicap, and an example of publication bias.
Fabricated reviewers
Fabricated reviewer identities or addresses can turn a supposed independent review into an author's own endorsement: the author submits a paper with a list of suggested reviewers, complete with e-mail addresses, but the names are bogus and the messages go to the author, who submits favourable reviews of his own paper. In August 2015 Springer retracted 64 articles from 10 subscription journals after editorial checks spotted fake e-mail addresses and investigations uncovered fabricated review reports, and about 15% of article retractions have been attributed to this scam.
Asked to suggest reviewers, the scientist Hyung-in Moon supplied a list of fake reviewers, some with names close to or the same as real authorities, complete with email addresses for accounts he or his colleagues had set up and controlled, and so ended up reviewing his own articles; 35 of them were retracted. In 2015 a number of journals, such as DNA and Cell Biology, were insisting that all reviewers use institutional rather than personal email accounts for their reviews.
A journal that asks authors to nominate peer reviewers creates the opening for this fraud. In a case that saw SAGE retract 60 articles published in the Journal of Vibration and Control, an author had created multiple accounts with different email addresses in the publisher's online submission system and reviewed his own papers under those false identities. Editors ask for nominations because many genuinely struggle to find people willing to review, but the onus is still on the editor to verify the identities of the reviewers they enlist.
By 2016, fake referee profiles created by authors to deliver favourable reviews of their own work had produced over 300 retractions across affected publishers including Elsevier, Springer and SAGE. In one case the British Journal of Clinical Chemistry was duped by the authors of a meta-review of therapies for heart failure patients: the authors suggested two referees, both cardiologists at top US universities, and the journal contravened its own policy by using both of them and no others. The journal's editors saw nothing odd in the fact that both reviews arrived within a week, were very short and overwhelmingly positive, and displayed clumsy English not compatible with having been written by leading US-based academics; more understandably, perhaps, they did not notice that the email addresses given for both referees were non-institutional and therefore unverifiable. Catastrophic methodological errors were missed as well.
The deception might have gone unnoticed had a couple of readers not outlined the paper's many flaws in a scathing letter to the editor. The journal then traced one of the named referees, who claimed to know nothing about the review, and, receiving an inadequate explanation from the authors, issued a prompt retraction notice.
The journal's editors judged that additional measures to prevent similar misconduct would likely fail and would risk inconveniencing honest researchers, believed their existing procedures adequate if properly followed, considered themselves too busy to check study details and held that it was not their job in any case, and insisted that responsibility for assuring research quality lies with the institutions where the research is performed; a columnist reporting the case asked how much oversight such institutions actually have. The same journal did identify two simple measures: do not rely solely on authors' suggested referees, and insist that institutional email addresses be used for correspondence with them.
Errors reviewers miss
In a study reported by Fiona Godlee of the BMJ, 221 reviewers returned assessments of a manuscript seeded with eight substantive errors: 16% found none, the median found two, few found five, and none found more. This may not be as bad as it sounds, because a reviewer is likely to stop looking once enough errors have been found to make clear that the recommendation will be against publication, but even with that caveat studies like this suggest that a lot of serious errors slip through peer review. At one journal where two referees each graded a manuscript on a five-point scale, running from acceptance without revision to rejection without an opportunity to revise and resubmit, a several-month sample found 36% of pairs identical, another 26% one step apart, and 9% more than two apart. Widely discrepant pairs almost always had an explanation, most commonly that one reviewer noticed a problem the other missed or that the two noticed the same serious errors but differed in their optimism for repair. Because redundancy is a means of improving error detection, such discrepancy may be a sign that redundancy has succeeded.
Proposed reforms
Proposed improvements include disclosure of earlier journal submissions and sharing earlier reviews with the journal now considering a paper; an initial assessment by methods and statistics specialists, so that conflicted subject experts receive narrower questions; and systematic feedback to reviewers about the quality of their reviews. The case for sharing between journals is that separate journals had each received concerns about a fraudster before anyone held enough evidence to act.
An overview of the problem of peer review as perceived almost a decade earlier was still current in 2007, which speaks to the difficulty of effecting change in a deeply entrenched routine.
One former publishing director considers peer review the most immediate and pressing problem that AI could try to solve. Within scholarly publishing, AI was seen as a way of bringing speed, efficiency and effectiveness to increasingly challenged processes and of supporting open-science principles. Web of Science figures show annual growth since 2013 of 2.6% in published articles and 6.1% in submissions, and a survey found that an editor in 2017 needed on average 2.4 review invitations to obtain one report, up from 1.9 in 2013. So finding reviewers was projected to become much harder across disciplines, and journals would have to scale with submissions and with rising demands on the quality of review, which would demand significant advances in automated technologies, including algorithms for assigning reviewers and improving reviewer experience and retention. What he wants from AI is therefore more operational speed and efficiency, and in 2019 it was beginning to arrive in five areas: identifying new reviewers through broader searches; detecting plagiarism at the level of whole sentences and paragraphs rather than verbatim text; finding where authors fail to report information that would affect an accept or reject decision; spotting statistical errors that generate false conclusions; and detecting data fabrication. One real-world example, reported in Communications of the ACM, showed machine-learning and AI tools making inroads to automate and improve parts of the peer-review process, and much progress has been made in allocating papers or grant proposals to reviewers. With such applications, he hopes, publishing may be able to move on from a process that has remained relatively unchanged since the eighteenth century.