Reporting harms
A study report or manuscript makes the harms a study observed assessable beside its benefits. It states the event definitions it applies, says how events were looked for, coded and analysed, presents them without a reassuring filter, and accounts for why one trial's harms read differently from one document to the next.
Event definitions
An adverse event is any untoward medical occurrence in a patient or trial subject given a pharmaceutical product, and it need not have a causal relationship with the treatment. An adverse drug reaction is, before approval, any noxious and unintended response to a medicinal product related to any dose, and after approval, a noxious and unintended response at doses normally used for prophylaxis, diagnosis or therapy of disease or for modifying physiological function. Treatment-emergent adverse events first appear after treatment starts, or are present at baseline and worsen after it starts. An adverse event of special interest, serious or not, is one of scientific and medical concern specific to the sponsor's product or program, for which ongoing monitoring and rapid communication by the investigator to the sponsor can be appropriate.
A serious adverse event is any untoward medical occurrence that, at any dose, results in death, is life threatening, requires or prolongs hospitalization, results in persistent or significant disability or incapacity, or is a congenital anomaly or birth defect. Serious and severe are not synonymous: severe marks a step up in intensity from mild and moderate, while serious rests on an outcome or event that threatens the patient's survival or function. An unexpected adverse reaction is one whose nature or severity is not consistent with the information in the relevant source documents.
The CONSORT harms extension asks the methods to list the adverse events addressed, with a definition for each, attending where relevant to grading, expected versus unexpected events, reference to standardized and validated definitions, and description of new definitions.
Tolerability and safety are often confused but are not synonymous. In one manuscript guide's usage, tolerability typically covers adverse events and treatment-emergent adverse events, and safety covers serious adverse events together with laboratory, electrocardiographic and vital-sign changes.
Harms in each section of a report
The CONSORT extension for harms published in 2004 makes 10 recommendations that correspond to items of the original CONSORT checklist.
Title, abstract and introduction
If a study collected data on harms and benefits, its title or abstract should say so, and if a trial addressed both, its introduction should say so.
Methods
Adverse events should be elicited by open-ended questioning by the investigator or other appropriately trained trial personnel at each study visit.
One manuscript guide's example tolerability table gives the observation period in its title, and in its footnotes the MedDRA version and preferred terms, the frequency threshold for including an event, the order of presentation and the analysis population, the safety population being all randomized subjects who received at least one dose of study treatment. Where percentages of participants with adverse events sum to more than 100 because each participant could have more than one event, a footnote says so.
The methods should name the dictionary used to categorise adverse events, such as MedDRA version 17.1, and the level of categorisation, such as system organ class or preferred term.
Clarify how harms-related information was collected: the mode of data collection, timing, attribution methods, intensity of ascertainment, and any harms-related monitoring and stopping rules. Describe the plans for presenting and analysing harms, including coding, handling of recurrent events, timing issues, handling of continuous measures and any statistical analyses.
Results
Describe, for each arm, the participant withdrawals due to harms and those participants' experiences with the allocated treatment. Give the denominators for analyses of harms, and present the absolute risk per arm and per adverse-event type, grade and seriousness, with appropriate metrics for recurrent events and continuous variables whenever pertinent. Describe any subgroup and exploratory analyses of harms.
For laboratory parameters, report mean changes from baseline to the end of treatment or study together with any outliers, such as the number and percentage of patients with values at least five times the upper limit of normal; individual or group changes in electrocardiography and vital signs often go in shift tables. Safety can be covered in shift tables of changes in mean values by treatment group, in narrative text, or both. For a serious adverse event, report how it presented, the actions taken, the outcomes of those actions, and whether the investigators judged it related to treatment. Investigators typically make that judgment from the event's temporal pattern relative to drug administration, dosing or discontinuation, so it is often a judgment call.
Discussion
Provide a balanced discussion of benefits and harms, with emphasis on the study's limitations, its generalizability and other sources of information on harms.
Reassuring wording
“Well tolerated” is a vague, subjective judgment about a drug's adverse-reaction profile for which there are no commonly understood parameters, and the FDA has no standard for it; ICH E2F discourages such nonspecific terms in characterizing overall adverse-reaction experience because they can mislead. Instead, give the incidences of the most salient adverse events, usually in descending order of frequency and often in a table. Specific frequency ranges, such as adverse reactions occurring in fewer than 1 in 500, give more precise information about incidence.
A figure such as “only 1% of subjects experienced adverse events” likely underestimates the actual frequency and severity of untoward effects. Trial patients are often highly selected for favourable tolerability and adherence, with few comorbidities and concomitant medications, and are monitored for shorter intervals than in clinical practice and phase 4 pharmacovigilance, which seriously undermines fair balance in the editorialising “only”. A short trial in a small number of subjects, such as fewer than 200 followed for 12 weeks, that failed to identify safety issues is not evidence that there are none: absence of evidence is not evidence of absence.
The commonly used phrases no clinically significant adverse events and no unexpected adverse events lack clinical relevance, particularly for rare adverse events that may be important for agents used over a long period in large populations. More uniform reporting guidelines that clearly specify the type and format of adverse-event data to report would benefit everyone, and journals may need to revisit their manuscript length policies if they want that information in the main document. Authors should be made aware of the need to balance the strength of adverse-event claims against the features and limitations of the trial design.
The 2010 revision of CONSORT uses harms rather than safety, because safety can be falsely reassuring.
Poor reporting practices and recommendations
A conference talk on the 2010 revision's approach to harms listed these common poor reporting practices to avoid:
- a generic statement that the drug was generally well tolerated
- no separate data for each study arm
- sums of adverse events per arm without separate data for each type of event
- numbers of events without severity information
- means or medians reported without extreme values
- reporting the adverse events for which the difference between arms reached P < .05
- disregarding the timing of events, so that early and late toxicity are not distinguished
- failing to distinguish participants with one event from participants with several
- commenting on statistical significance without giving exact event counts
- not providing data on all randomly assigned participants
- reporting only adverse events observed at a certain frequency, such as above 3%
On that last practice, the speaker remarked that the lack of a significant difference is very illuminating, since it can tell a reader a lot about the patient population. One manuscript guide's checklist, by contrast, suggests a table of treatment-emergent adverse-event frequencies in the active and control groups, in total and by preferred term or system organ class, for events exceeding a prespecified percentage such as 3% or 5%.
Reporting guidelines for harms such as the CONSORT harms extension exist, but the communication of adverse-event data in publications remains suboptimal, and recommendations published in 2016 from a joint pharmaceutical-industry and journal-editor perspective aim to improve it.
The five recommendations to improve adverse-event reporting in publications are:
- Identify and communicate the most clinically relevant drug adverse event data as part of a comprehensive safety profile.
- Report timing, frequency, duration and other potentially relevant descriptors where clinically appropriate.
- Use statistical analysis for clinically relevant adverse events where appropriate.
- Avoid overly general text descriptions of adverse events, including in abstracts.
- Discuss adverse-event findings in the broader context of the available evidence, and maintain consistency of the data across different public reports.
They are intended to supplement the existing guidelines for reporting adverse-event data.
Discrepancies across a trial's documents
Little is known about how adverse events are collected and reported in clinical trials. In a 2016 document analysis, researchers at the Nordic Cochrane Centre analysed seven randomised placebo-controlled trials of an anti-obesity drug, together 4,225 participants, conducted in the US and Europe between 1992 and 1996 for a drug approved by the European Medicines Agency in 1998 and the subject of an FDA warning in 2011 about 13 cases of liver failure associated with it. They found important disparities in the reporting of adverse events between the protocols, the clinical study reports and the published papers, and judged that the reports of these trials seemed to have systematically understated the events.
None of the protocols or clinical study reports contained instructions for investigators on how to question participants about adverse events. All events were coded by the sponsor using a glossary the sponsor could update. Between 3% and 33% of the total number of investigator-reported events from the trials were reported in the publications, because of post-hoc filters, though six of the seven papers stated that all adverse events were recorded. In one trial the majority of patients had multiple episodes of the same event that were counted only once, which was not described in the clinical study report. Participants on the active drug had more days with adverse events than those on placebo, 22.7 against 14.9 days (P < 0.0001), and their events were more severe, and neither the clinical study report nor the published paper stated this.
The analysis was exploratory and restricted to one drug tested in the mid-1990s, so its results might not apply to newer drugs, because the standards of reporting clinical study reports and publications have improved since. Many drugs approved in that period were, however, still on the market in 2016. The study's authors highlight the need for detailed analysis plans for harms data.