Why clinical trial data is disclosed
Clinical trial data is disclosed because participants, funders, journal editors and assessors of evidence came to demand it, and because comparisons of published accounts with fuller reports showed what went missing when only the publication was read. The case rests on what reporting bias hides, on whether the regulators or the medical press should carry disclosure, and on what disclosure is expected to deliver against what it puts at risk.
Reporting bias
Publication bias and outcome reporting bias are two types of reporting bias, and they refer to missing data at two levels: the study level, through non-publication because a report was never submitted or was rejected, and the outcome level, through the selective non-reporting of outcomes within published studies. Evidence dating back several decades shows that reporting bias is a universal problem in medical research. Positive results of new drugs are published more rapidly and more often than negative or neutral ones, so the published literature may overestimate benefits while harms are underestimated.
Most of the data generated and information elicited from clinical trials were not available to the scientific and clinical communities, and clinicians often recommended treatments on incomplete information: not all trials were published, and not all outcomes collected during a trial were reported even when it was. Much of the evidence on how well drugs work is not available to patients and prescribers, which is the problem of publication bias. The evidence for a drug cannot be assessed if not all its trials are published, and the trials that are not published tend to be different from those that are. All who carry out a Cochrane meta-analysis are warned of the importance of identifying all relevant trials. Missing trials are a problem not just because they represent a loss of information, but in particular because the missing information may differ systematically from what is available.
Many attempts had been made to fix the problem, and most large pharmaceutical companies had committed to publishing all their trials. Even so, it would be overly optimistic to assume that every study is published, and until that is certain everyone needs to try harder to ensure more complete publication. Even if incomplete publication had been fixed, a mountain of past studies remained unpublished whose results were still relevant to medical practice, so the problem should not be considered solved until it has been solved retrospectively as well.
Regulators have access to huge amounts of data on the drugs they approve but do not publish them, which one critic of drug regulation holds to be unjustifiable. One writer sees no valid reason why regulators do not routinely make submission dossiers available on their websites, and holds that if they did, everyone could have far more confidence in the regulatory process and know far more about the drugs in use.
Clinical study reports should be disclosed to allow scrutiny, so that readers can see the omission of pre-specified data, data dredging, arbitrary changes to data collection and other sources of bias. Journal publications often do not adequately reflect the completeness of the data in clinical study reports. Research comparing the results reported in journal publications and study registries with those in clinical study reports found that the reports gave complete information on a considerably higher proportion of outcomes (86%) than publications and registries combined (39%).
Germany's health technology assessment agency, the Institute for Quality and Efficiency in Health Care (IQWiG), depends on access to all the relevant data on a topic to produce unbiased assessments. One of its 2012 assessments shows the level of detail that needs to be available for a meaningful assessment of a drug. The institute assessed whether linagliptin had an added benefit over glimepiride, a sulphonylurea, in patients with diabetes. The assessment was based on a study published in The Lancet in 2012, whose authors reported that hypoglycaemic episodes occurred in fewer patients in the linagliptin group, a result suggesting an added benefit. With the full clinical study report, the institute could read the intention-to-treat analysis of the time course of glycated haemoglobin, which the publication did not carry. That analysis showed a sharp fall in glycated haemoglobin in the glimepiride group, but not in the linagliptin group, in the first 12 weeks, probably because glimepiride was force-titrated towards a low blood glucose target while linagliptin was given at a fixed dose without such a target. The patient data listings showed that almost all hypoglycaemic events occurred during that 12-week titration period. The institute therefore concluded, against the publication, that the study gave no convincing evidence of an added benefit, because it could not distinguish the effect of the two treatment regimens, fixed dose against forced titration, from that of the two drugs. Relying on the publication alone might have led to inappropriate decisions on the use of linagliptin and on its reimbursement price in Germany.
The neuraminidase inhibitors
The case of the neuraminidase inhibitors is probably the best-known example of reporting bias leading to incorrect conclusions about a drug's effects. Oseltamivir, sold by Roche as Tamiflu, is an antiviral used to treat infections caused by viruses, particularly influenza. It came to prominence in the general media in May 2009, when the World Health Organization requested stockpiles to tackle what it termed a global pandemic of H1N1 swine flu, and its sales reached 3.2 billion Swiss francs that year. The drug's use in a pandemic had been evaluated in a 2003 meta-analysis of ten studies sponsored by Roche, eight of them unpublished.
Little was known about the measurable impact of reporting bias on the health care system until 2006, when the Cochrane Collaboration published a systematic review of the efficacy of the neuraminidase inhibitors oseltamivir and zanamivir in the prevention and treatment of influenza. The review concluded that the drugs were effective in reducing complications of influenza in otherwise healthy adults. In 2009 the reviewers became aware that the review rested on a single manufacturer-funded study using unpublished data, asked Roche for all data, and found that 60% of the patient data from the trials had never been published. The updated review showed insufficient evidence that the drugs reduced complications of influenza or hospitalisations. This raised the question whether stockpiling them for flu epidemics in many countries had been an appropriate use of public money, £424 million in the United Kingdom alone, and prompted a search for measures to aid decision-making when information is incomplete.
The Cochrane Collaboration and the BMJ had been at loggerheads with Roche over full disclosure of the drug's results for some time, and a fresh dispute was set off by an updated Cochrane review of oseltamivir and zanamivir published in the January 2012 issue of The Cochrane Library. The director of the German Cochrane Centre stated that over 50% of the results of oseltamivir studies had not been made public, making a proper evaluation nigh on impossible although the drug had been on the market since 1999, and Roche replied that it had given the Cochrane group access to 3,200 pages of very detailed information. Cochrane had pressed Roche for several years to release all study data, and although some information had come, Cochrane found the detail unsatisfactory, particularly the data relating to side effects. One published study ran to seven pages while its clinical study report ran to over 2,000, and Cochrane suspected that much information about that study had not been made public.
The US Food and Drug Administration and the European Medicines Agency reached different conclusions about oseltamivir. The director of the German Cochrane Centre found the difference worrying and questioned whether both authorities had been presented with the same information. Since the case of the neuraminidase inhibitors became public, both the discussion of transparency and the measures taken seem to have accelerated.
Pressure for disclosure
Transparency matters because of the pharmaceutical industry's reputation, which is roughly that of the tobacco industry: second to last, better only than the weapons industry's, so a plan was needed to raise it. The first reason for that plan was that only a small proportion of clinical trials used to be published when they finished. The failure was not the industry's alone, because most of the time academic researchers were even more inclined to publish only what supported their hypothesis.
Participants and patient organisations press for disclosure as well. People who took part in a study ask to know its results and do not want to be kept blind to any part of them, and patient organisations are increasing their pressure to be involved at every stage of research, from the design of a study to the disclosure of its results. Patients say that they are the ones who make it possible to create the data and the ones who will benefit from the outcomes, even when a study's result is not aligned with the researchers' expectations. Some of the more radical say "nothing about us without us", meaning that patients must be involved. There should be a circular, continuous feedback between the public volunteers who take part in studies and the companies and researchers, and that cycle is not yet working very well. Good publication practice was to be updated again to highlight the importance of the patient, and ICH was to be revised in the same way. Patients are becoming a crucial audience for everything a writer produces, and they require transparency.
In one regulatory strategist's judgement, true transparency, in the sense of every pharmaceutical company, physician, caregiver and patient having access to all the raw data produced and documented in every clinical trial, did not exist. In that view it was a goal the industry was evolving towards while focusing on maintaining patient privacy, industry credibility and the ability of pharmaceutical companies and other medical research facilities to remain viable and competitive. That evolution was the result of the empowered-patient movement, which had produced the Sunshine Act's disclosure of pharmaceutical companies' payments to physicians, the Centers for Medicare and Medicaid Services' disclosure of patient-level information on procedures, prescriptions and payments collected by hospitals, physicians and other providers, and the creation and expansion of ClinicalTrials.gov, where information on clinical trials is made public. Much of the raw data in clinical trials nevertheless remained unavailable. Companies faced continual pressure to reach complete transparency. Ben Goldacre had voiced strong criticism of the industry, describing its research practices as bad science, and had gained a wide audience through his books, public speaking and television appearances. Bioethics International, a nonprofit founded in 2005 to advance industry ethics and patient-centricity in medicine, had created the Good Pharma Scorecard, which rates companies on their transparency.
Transparency is asked for not only by regulators but by many of the audiences a writer serves, chiefly the International Committee of Medical Journal Editors, and it has to be systematic: isolated data dumps are not an answer, and sharing has to happen always. Initiatives such as BMJ Open and the EU Open Science Cloud should be a solution: all the data collected during clinical trials could be stored in the Cloud and analysed by anyone with the capability, who could then publish the results.
AMWA's code of ethics asks medical communicators to apply objectivity, scientific accuracy and rigour, and fair balance while conveying pertinent information in all media. One writer holds that this principle speaks to the continual effort to avoid bias in reporting research and clinical trial results, and that a writer can be objective, accurate and balanced only if there is transparency in the information reported.
Commitments and requirements to share trial data
In clinical research, data sharing is the practice of a research team making trial data available to individuals with whom it is not collaborating. Sharing increases the value of all clinical trial research by encouraging the use of data already collected. It reduces the potential for inaccurate or incomplete reporting of study outcomes that distorts the medical evidence, and ultimately ensures the reliability of the evidence base on which patients and physicians make clinical decisions. As data sharing initiatives grow, the scientific community will be able to adopt a more open approach to research. When trial data are made more widely available, science can function as a community, continually vetting, critiquing and building on each other's ideas.
One set of recommendations for reporting clinical research asks that all results, including negative or unfavourable ones, be made public in a timely fashion while avoiding redundancy. Many industry sponsors had committed to disclosing the results of all clinical studies through trial registries, and cross-referencing trial registries, results databases and all related publications tells the scientific community whether studies are completed or under way and discourages selective reporting. The recommendations ask for more complete protocol information as well. Some journals requested the protocol at submission and some published it online, but making protocols public raises issues that need further discussion, such as the protection of intellectual property, and because protocols are often amended as a trial progresses, public disclosure of several versions may cause confusion.
A number of major clinical research funders had adopted policies supporting or mandating data sharing, among them the US National Institutes of Health, the UK Medical Research Council and the Bill and Melinda Gates Foundation, while major regulators, most notably the European Medicines Agency, and several pharmaceutical and medical device companies were contemplating open data access policies. These moves should improve the transparency and availability of clinical trial data.
In January 2016 the International Committee of Medical Journal Editors proposed requirements on sharing clinical trial data, in an editorial in Annals of Internal Medicine. The committee stated its belief that there is an ethical obligation to responsibly share data generated by interventional clinical trials because participants have put themselves at risk, and that in a growing consensus many funders around the world, among them foundations, government agencies and industry, now mandate data sharing. It invited feedback on the proposals by 18 April 2016.
Regulators or the medical press as the route to disclosure
Among the failings Bad Pharma charges the industry with is failing to publish negative studies. The evidence-based medicine movement has railed against regulatory studies not always being published, and Bad Pharma proposes making it mandatory for studies to be published within 12 months of completion, in summary table form if academic publication has not occurred. Stephen Senn held that the appropriate solution to making trial results more widely available is to make the rigour of drug regulation available to all, rather than extending the use of the medical press, which he called a mediocre medium.
In Senn's comparison, regulators are professional, thorough and expert. In his account the FDA in particular has played an important role in promoting the study of many methodological issues affecting the analysis and interpretation of clinical trials, through its own staff and by encouraging and sometimes commissioning others, and bioequivalence, non-inferiority, multiplicity and missing data have all greatly benefited from regulatory input. He held that statisticians working for the pharmaceutical industry have also made important methodological contributions to drug development science, and that the ICH E9 guideline on statistical analysis is much superior to the alternatives the journals offer. The net result, in his view, is that the quality of review the regulator provides far exceeds that the journals provide, and the regulators see all the studies, or at least all the studies for any product seeking a licence.
Subgroup analyses of the kind Bad Pharma criticises are outlawed in regulatory submissions but scarcely policed by the journals. Much of Senn's own consulting for industry concerns designing watertight, pre-specified analyses to control the type I error rate, while in reviewing for the medical press he has never been provided with the statistical analysis plan.
Regulators are not the only ones who make decisions about medicines, however: reimbursers, physicians and patients do too, and journals provide a visible forum in which researchers exchange, discuss and disseminate results. In Senn's view peer review makes only a weak contribution to quality, but there is not much point lauding the superiority of studies that are not seen. The first place any independent meta-analyst looks for studies is the medical press, so he held it unacceptable that studies are seen only by the regulators. As Senn put it in 2000, no sponsor who refuses to provide end-users with trial data deserves to sell drugs.
Senn held that any system relying on academic publication is unworkable, principally because the medical press is not a single authority but a collection of competing interests, none of which can be made responsible for publishing any given paper, and that Bad Pharma underestimates the difficulties. In his view the journals cannot be part of the solution because they are part of the problem: as long as they are seen as the most prestigious route for disseminating results, it will be difficult to get all results out in a timely manner. His own proposal for openness is that sponsors self-publish the results of their trials, that they produce a publication plan as part of the regulatory submission, and that a licence to market be given only once the plan is fulfilled.
Any such plan leaves many difficult details to be worked out, in particular what level of detail should be provided and how confidentiality will be guaranteed in practice, and whether, to control data dredging, those who want data for an independent analysis should be required to pre-specify that analysis. Senn held that regarding either point as a symptom of resistance to progress by the pharmaceutical industry is unhelpful. He predicted mistakes with inadvertent disclosure of confidential data, and that neither the evidence-based medicine movement nor the Cochrane Collaboration would come to the industry's rescue when they happen. He asked how, unless pre-specification is addressed, the problem of missing analyses will be dealt with, whether every researcher requesting data must be required to publish the pre-specified analysis, and, if not, how selective analyses can be guarded against and how that will be policed.
Whether journals are biased against negative studies
Senn held that it is quite possible that journals are prejudiced in favour of positive studies. The experimental evidence, from studies in which positive and negative versions of the same paper were submitted to journals, seems in his reading to show quite strongly a bias in favour of positive studies.
A number of studies found that when submissions to journals are classified by whether the findings were positive or negative, the acceptance rate is similar, and Bad Pharma concludes from this that there is no editorial bias in accepting or rejecting. The observational paper the book deals with in most detail is one in which editors conclude that they are doing a good job. Senn objected that the inference implicitly assumes that the quality of the submitted studies is equal: if authors submit by estimated probability of acceptance, not bothering unless it is higher than some threshold, the acceptance probability might show no difference while the quality differs, with the negative studies submitted being of higher quality. Senn compared the inference to claiming that there is no bias against women in higher education because the same percentage of each sex applying for promotion to professor succeeds, which overlooks the higher qualifications of the women who apply. An explanation would be that women do not apply unless their qualifications are exemplary, because they know the system is biased against them.
Evidence that the negative studies submitted are of higher quality comes from a paper the book itself cites. Its title states that commercially funded and United States-based research is more likely to be published while good-quality studies with negative outcomes are not, and its abstract reports that studies with a negative outcome were of higher quality and included larger sample sizes, the first of these being the most significant finding in the article.
What disclosure delivers and what it puts at risk
The European Medicines Agency publishes clinical data to build public trust and confidence in the scientific data collected and in the decision-making by which it licenses medicines, to help avoid duplication of clinical trials, and perhaps to foster innovation, further research and development and the development of new medicines. Researchers and academics can gain access to clinical trial data for meta-analyses and systematic reviews and can reassess the data for new innovations, better medicines or better practice in medicine, which enhances scientific knowledge. Doctors and other health practitioners can practise better medicine with the information at hand, and patients get better health care, a benefit to public health in general. Applications that were not approved or were withdrawn are published as well as approved ones, so researchers can find out what did not work instead of repeating it, which avoids research waste. Publication is also meant to benefit patients, who are trying to understand what their treatment involves, whether it is the right one and whether other options exist. One practitioner, for whom the whole point of the work is keeping patients safer and making their lives better, holds that letting patients understand what is going on with their health is a huge step.
Against those benefits stand weaknesses and threats, of which one practitioner identifies two while allowing that there might be others. Disclosure puts protected patient data at risk, which can lead to patients being identified, and it puts commercially confidential information at risk, which lets competitors know what a company is doing and will lead to higher costs of medicines. Regulators and editors alike stress that patient privacy has to be guaranteed completely, and another crucial point is that the data could be analysed and disclosed in an inappropriate way, which could be highly risky.
As the sharing of raw, granular trial data approaches, patients may become identifiable. The European Medicines Agency had already issued transparency policies for full data disclosure, while efforts to protect patients from identification included blacking out names, using ranges and even creating synthetic data.