Question the research, not only the language
Editing a scientific text can question the research it reports as well as its language: whether the assumptions hold, whether the design and conduct of the study are documented, and what the results imply. That pass needs no medical knowledge, only knowledge of how research is done and the standards it should be reported to, and a single sentence worked through to the end shows how far a reader's apparent understanding can be from what the study found.
What analytical editing is and who can do it
Critical thinking is necessary to edit a scientific article, and in addition to questions about the language, an editor can question the assumptions, the documentation and the implications of the research, in a process Tom Lang calls analytical editing. Analytical editing is editing to ensure that research designs and activities are documented appropriately and explained adequately. It seeks to meet the needs of evidence-based medicine by making sure the evidence itself is completely and clearly reported. A text with unverified assumptions, missing documentation and unconsidered implications can lead readers into believing that they understand an article when they do not, which Lang calls the illusion of certainty.
Analytical editing does not require knowing medicine. It does require knowing how medical research is conducted, or at least what questions to ask about the research, as well as the standards to which this research should be documented. It requires knowledge of several topics: the principles of measurement; research designs and activities; statistical reporting and interpretation; sources of error, confounding and bias; and specialised forms of data display, such as receiver operating characteristic curves, Kaplan-Meier curves and life tables.
Analytical editing is a task often left to peer reviewers, and it can be done by trained editors. In conjunction with traditional substantive editing, it allows editors to increase their professionalism and the value they add for clients. The learning curve is not that steep, and substantive editors who can add analytical editing to their services can greatly increase both their professional skills and their value to authors.
Why the pass is needed at all
Science is based on writing. Only writing allows science to be recorded, evaluated and reproduced and enables it to be systematic, cumulative and public, the characteristics that distinguish science from authority, intuition and tradition as a way of establishing truth. Publication, the final stage of research, depends on writing, as does evidence-based medicine, which is literature-based medicine.
Yet at least in clinical medicine, the support physicians and researchers are given in preparing publications is often inadequate. It is often inadequate even in the United States, and especially in developing countries. Researchers are not expected to do their own literature searches and so are given access to librarians; they are not expected to do their own data analysis and so are given access to statisticians; they are not expected to render their own graphs and drawings and so are given access to medical illustrators. But they are expected to do their own writing, to communicate technical information accurately, completely, clearly and economically, in words and images, without specific training and often without the support of professional medical writers and editors. So it should be no surprise that a large portion of the scientific literature is not immediately, accurately and completely understandable.
Lang counts among the most important lessons of almost 40 years of editing that the certainty a reader believes they have about understanding even a simple, straightforward sentence is often illusory, and that the sense of certainty is so strong that the reader does not think to question what the sentence means. Only on closer examination does the illusion become apparent. Such sentences are found in most scientific articles, so these illusions are also a certainty in the scientific literature.
One sentence worked through completely
The example sentence came from the results section of a poorly written abstract and read: one group of patients was significantly less depressed than the other. It seemed straightforward, but the more it was analysed, the more questions it raised. A close look at it raises questions about meaning, measurement, statistical analyses, presentation of data and interpretation of results.
The first question is the context of the sentence. Meaning is a product of message and context, in the same way that the meaning of a picture is a product of image and background: change the context, and the same message has a different meaning, as "the wall was built to scale" means something different to an architect than to a climber. Taken by itself, the sentence, the second in the results section of the abstract, could have described the patients at baseline, an incidental finding that might confound the results, or a result itself; given that the article was a study of a new antidepressant, it was probably the result of the study. So the context of every scientific article needs to be clear, to rule out the interpretations that different contexts make possible, and one function of a good introduction is to put the research in the proper context.
The next questions are who and what were studied. The participants were women outpatients with moderate-to-severe depression being treated at a university hospital, and the two groups were the treatment and control groups of the study, something the sentence could have said: patients in the treatment group were significantly less depressed than were patients in the control group. A reader also needs the patients' ages and diagnoses, how they were selected for the study (the sampling method and eligibility criteria), their other health conditions, and how the sample size was determined. Depression can be treated in several ways, so the treatment needs to be described in detail: for a drug, the generic name, manufacturer, dosage and route of administration, and perhaps the indications, possible side effects and the degree to which each group took the medication as planned. The rate of protocol adherence is usually higher in inpatient studies than in outpatient studies, for example.
How depression was measured comes next. All study variables must be defined in objective, measurable terms, so the reader needs to know whether the measurement was based on a physician's judgement, a self-report questionnaire or some other way. The text said that all patients completed the Beck Depression Inventory before and after treatment, a common, validated instrument for measuring depression, and this information was encouraging. Many authors do not say how they measured their variables, often because they assume their readers will know.
Then comes the type of comparison being made. In a study with two groups in which both pre- and post-treatment values are measured, two comparisons are possible: the within-group comparison looks at the changes between pre- and post-test values for each group, whereas the between-group comparison looks at the differences between groups at the beginning or end of the study. In a study like this one, both comparisons are likely. Because the sentence says that one group was less depressed than the other, it could refer to a between-group comparison, in which mean depression scores in one group were lower than in the other at the end of the study, presumably with similar baseline scores, or to a within-group comparison, in which the change in depression scores during the study was greater in one group than in the other, with baseline scores not necessarily similar.
The measurement scale has to be understood too. The Beck Depression Inventory runs from 0 to 63, with scores of 0 to 9 indicating no or minimal depression, 10 to 18 mild depression, 19 to 29 moderate depression and 30 to 63 severe depression. A figure accompanying those bands divides the scale differently, into 0 to 13 healthy, 14 to 19 mild, 20 to 28 moderate and 29 to 63 severe. Understanding the measurement means asking whether the scale is linear, so that a 3-point difference at one end means the same thing as a 3-point difference at the other; what the smallest clinically meaningful difference in scores is; and whether any threshold scores, when crossed, change the interpretation of the results. The editor does not have to know whether the scale is linear, only to ask the authors whether it is, and should not be surprised if they do not know. Knowing the scale also says something about the baseline values: since all patients completed the inventory before treatment, it is reasonable to conclude that all of them had scores of at least 20 at baseline, which one would hope the text confirms, and although the results were reported as means of the post-treatment scores, the mean baseline values of both groups would have been useful.
The authors reported that after the intervention the difference between the means of the treatment and control groups was 3 points, but they did not give the minimum clinically important difference for the inventory, which is 5 points. A difference, to be a difference, must make a difference.
The actual mean values of both groups exposed the illusion. A table showed a mean score of 5 in the treatment group and 8 in the control group, consistent with the 3-point difference between means and with the treatment group's 38% lower score. Both means are in the normal range, so describing the result as one group being less depressed than the other is incorrect and misleading. The scores also differ by less than 5 points, so the difference is not clinically important in the 2016 version of Lang's account and probably not clinically important in the 2013 version. The authors seemed to have based their interpretation solely on a significant P value, without considering the clinical implications of the results. Why they did so could be a most interesting question. The study compared the means of the two groups, and it would have helped to know how many patients in each group were no longer depressed by the end of the study.
The last question is whether the drug is likely to be generally effective, since determining its effectiveness was the purpose of the study and the authors' claim that one group was significantly less depressed than the other was supposed to mean that the drug was effective. They should have written something like: after treatment, 72% (38 of 53) of the treated patients and 49% (27 of 55) of the control patients scored 13 or below on the Beck Depression Inventory (95% CI for the 23% difference, 2% to 41%). The numbers in that sentence are made up. The 2016 version of the account sets the cut-off in that sentence at 9 or below rather than 13 or below. Given the small effect size, the lack of a confidence interval for it, the fact that the means of both groups were in the healthy range, and not knowing how many patients were no longer depressed at the end of the study, it does not seem reasonable to agree with the authors that the drug was effective. Nor can it be concluded that the drug was ineffective: the difference was statistically significant, if clinically irrelevant. The drug did reduce the mean of the treatment group from well above 20 to 5, which supports the claim of efficacy, but the mean in the control group may have been reduced to a similar degree. All that can be said is that the study was not well conducted, not well reported, or both. The analysis does leave several important questions to ask the author, and such questions form the basis of analytical editing.
What the pass costs and what it is worth
Not all sentences are as involved as the one worked through above, but many are and require analysis as detailed. Analytical editing can take time, as well as skill, training and experience. What makes good writing and editing valuable is that they reduce readers' time, effort and uncertainty about the meaning of a text without creating the illusion of clarity. Many scientific articles are poorly written and poorly edited: worldwide, authors are generally not skilled in communicating technical information in writing and do not receive adequate editorial support, and most journals provide only superficial copy-editing. This situation pretty much ensures that readers of the scientific literature will regularly encounter the illusion of certainty and must therefore be prepared to accept what Lang calls the certainty of illusion.
Checking the report against its reporting guideline
Where authors do not know that reporting guidelines exist, a medical writer can be the first peer reviewer of the study and help the authors understand what their study is worth. Using the guidelines properly helps medical writers increase both their professionalism and the professionalism authors perceive in them.
One long-practising publications professional reports that peer-reviewed journals increasingly make acceptance of a manuscript depend on how far it meets consensus guidelines for its preparation. He holds that using the consensus checklists should help to optimise first-journal acceptance rates, because many journals refer to them to judge the quality of submitted manuscripts and decide their disposition. To remain up to date with consensus guidance for manuscript preparation, he recommends following the website of the EQUATOR Network.