Communicating real-world evidence to a regulator
Presenting a study built on real-world data so that a regulator can judge it means knowing why regulators began accepting such evidence, framing the case against the evidence an approval needs, saying what the objectives and design sections have to cover, knowing the data source behind the study, and showing that the data are fit for the purpose with a checklist written for medical writers.
Why regulators turned to real-world evidence
As early as 1998 a historical control drawn from a registry supported the approval of Refludan.
One account of the regulatory interest in real-world evidence holds that it arose from demand and supply together, though which came first is hard to tell. On the demand side, public demand for radical innovation in drug development led to the 21st Century Cures Act in December 2016, which required the FDA to develop a plan for using real-world evidence, and the European Medicines Agency and Health Canada issued reports of their own. These initiatives made a ripe environment for industry and academia to invest in real-world evidence projects that support regulatory decisions, projects particularly appealing in oncology, where the pace of development and the small numbers of patients in subpopulations make classical randomised trials impractical. On the supply side, technological advances — digitised health records, devices capturing passive and active patient-reported outcomes, genetic profiling and next-generation sequencing, and the capacity to house and manage big and deep data — made data available at an unprecedented scale. Evolving analytical methods and advances in artificial intelligence and blockchain made it increasingly feasible to draw meaningful conclusions from real-world data, which often lack the traditional structure and rigour of trial data. As of early 2020, open questions remained about standards for the quality and reliability of data, and about how data are judged fit to meet the rigour and substantial-evidence standards regulators require; the FDA had issued procedural guidance on them but nothing comprehensive.
By mid-2020 the FDA and the European Medicines Agency were both encouraging the use of real-world data, increasingly collected through mobile devices and patient registries, as evidence of efficacy and safety.
During the COVID-19 pandemic, international regulators and experts from WHO and the European Commission acknowledged the value of real-world data from COVID-19 observational studies as a complement to trials, and the EU PAS Register and ClinicalTrials.gov were identified as suitable places to share and exchange information about such studies.
The evidence an approval needs
In December 2019 the FDA updated its 1998 guidance on providing clinical evidence of effectiveness for human drug and biological products, retitling it as guidance on demonstrating substantial evidence of effectiveness. Its preamble states that the evidentiary standard for effectiveness has not changed since 1998, but that the evolution of drug development and science requires flexibility in the amount and type of evidence, real-world evidence included, needed to meet it, chiefly for serious diseases without a treatment option, rare diseases, and treatments aimed at subsets of disease where patient numbers are relatively small. Because of that flexibility, the writer must understand the substantial evidence the approval needs and the company's strategy, so as to frame the data against those expectations and address any gaps, and must present the rationale for using real-world evidence and for why other types of study are not feasible or not preferred.
Reporting the study
A real-world evidence study should be reported according to the standard regulations and guidances, in the form its audiences are used to from traditional clinical investigations. A few parts deserve special attention, chiefly the objectives, the design methodology and the demonstration that the data are fit for purpose.
Objectives
In light of the FDA's framework, the objectives of a real-world evidence study can range from establishing a historical or synthetic comparator, through expanding an indication to new populations or disease stages or adding efficacy claims, to supporting a move from accelerated to full approval, meeting postmarketing commitments or answering safety signals. Regulators single out the potential to expand indications to the extremes of disease and patient characteristics, which trials often exclude. Because data sources vary so much in quality and fitness for purpose — some with substantial missingness, some richer in administrative than clinical detail, some specific to one geography — the objective statement should name the type of data source as well as the research question's population, products and comparators. Data sources also vary in how long their data have accumulated, which may affect how relevant the clinical practice at the time of capture is, so the objective statement should state the time and duration of the look-back. The FDA requires sponsors to say clearly when real-world evidence forms part of a package supporting a regulatory decision, but not necessarily when it was used to establish outcome assessments, assess feasibility or generate hypotheses.
Study design
Under the FDA's programme, data from electronic health records, claims and billing, product and disease registries, patient-generated data, and wearables and other devices can support many designs: some randomised ones, such as large simple trials and pragmatic trials; prospective or retrospective observational studies; and hybrids. Some common designs are these:
- Pragmatic trials: simple, large randomised trials, often used to generate data in populations with broader inclusion and exclusion criteria, where randomisation helps minimise confounders and channelling bias.
- External or synthetic control arms: comparators for a single-arm trial when a lack of equipoise rules out randomisation, which can be based on registries, chart reviews or disease history studies and designed as historical or contemporary controls.
- Observational studies: studies in which the investigator observes participants without manipulation or intervention, including cohort, case-control and cross-sectional studies.
- Registry studies: observational studies of long-term trends in a specific population, such as patients with a particular disease or exposed to a certain treatment.
- Decentralised trials: virtual real-world studies collecting data at home or at the point of care with digital tools such as wearables and mobile devices.
- New-user designs: designs that identify patients starting the treatment under study and exclude prevalent users, allowing an active or a non-user comparator.
As of early 2020 the FDA was commissioning duplication projects to test whether randomised trial results could be reproduced in real-world studies and the reverse, to learn which clinical questions such analyses can answer with confidence and with which designs and methods. That work had shown that avoidable design faults, such as immortal person-time and over-adjustment for causal intermediates, are common and can produce inconsistent results. Time spent scrutinising potential flaws, biases and confounders at the design stage is key to selecting the appropriate method to establish causality.
A real-world study should be designed to resemble an experiment: a prospectively defined protocol and analysis plan, appropriate governance and a predefined data access and review plan must be in place before any analysis, and a rationale for the design must be given. The FDA strongly recommends that sponsors discuss their plans and candidate designs with it early and use presubmission processes to ensure alignment with it. The design discussion also covers feasibility assessments and how the treatment effect is estimated, and the writer must describe how confounders will be identified and controlled. Without randomisation, other methods must validate the statistical assumptions, and because some designs are retrospective, those assumptions may need testing, with the validation methods described clearly.
The data source
Industry and academic investment led to large, curated data sets that could potentially be used to generate real-world evidence, and the writer should try to become familiar with some of them and their advantages and limitations. Academic, public-private and industry initiatives were evolving quickly and had led to networked databases, novel research platforms and collaborations to make better use of real-world data. The writer should become familiar with the specific source, network and platforms used to conduct the study in hand.
Fitness for purpose: the RIGOR checklist
A key consideration in whether real-world data and evidence are regulatory grade is showing that the data are fit for purpose. Several researchers have offered their opinions on what criteria make data fit for purpose, and the RIGOR checklist, written for medical writers, organises criteria found in FDA guidances and industry presentations under Relevance, Integrity, Generalizability, Organization, and Reliability and Validation.
- Relevance: the types of patients, the intervention of interest, the duration of follow-up, the dose and dose schedule, and the outcomes of interest.
- Integrity: informed consent, confidentiality, controlled access, correct data and transparency.
- Generalisability: geographic location, health-system specifics, the timing of the research and the impact of limitations on generalisability.
- Organisation: provenance, completeness, granularity, scalability and traceability.
- Reliability and validation: identifying bias — misclassification, selection bias, confounders — managing confounders, sensitivity analyses and validation.
Relevance
Under relevance, the writer must address why the data apply to the research question at hand. That means asking whether the data include enough patients with the appropriate disease severity and stage, whether patients receive the dose and dose schedule to be studied and are followed long enough to collect the outcome of interest, and whether that outcome is collected consistently and reliably and is serious enough that patients will visit a health care professional to report it. It also means asking whether the source includes data from the appropriate resources, such as hospitalisation and prescription drug data, whether the intervention is used as monotherapy or with other treatments, whether switching is an issue, and, for a safety outcome, whether the appropriate risk window is represented. These and many other elements help provide a compelling rationale for why the database was selected.
Integrity
Good clinical practice mandates appropriate informed consent, and although in large aggregated databases it is often hard to establish whether patients authorised a particular use of their data, the writer must try to find out. It is also important to describe the measures that protect the confidentiality of patient records and identifiable information, who has access to the data during the analysis, and whether a data review plan controls access to data and results. Transparency is another element of data integrity: whether record-keeping processes are in place, whether human-readable audit trails are accessible to inspectors, whether the protocol is registered and whether there are publication plans.
Generalisability
For the data to be fit for purpose, they must be generalisable to the population of intended use. The writer must consider whether clinical practice at the time and place of collection matches current practice and has not changed over the look-back period, whether a single health system's database carries centre- or system-specific practices that differ from clinical practice at large, and whether coverage policies or step-up rules preclude or institute particular interventions over others.
Organisation
The structure of the database and the provenance of its data should be described: how the data are organised, stored and archived; how they are integrated at provider level and moved into clinical and then analysis data sets; how they are exchanged between provider systems and vendors; and how they are aggregated. Poor interoperability between databases is another issue, particularly relevant for studies outside small populations that collect longitudinal data, and the writer should ask how it is managed and how far it is a challenge. Where missing data are an issue, the writer must describe their direction and extent and how to adjust for them. Electronic health records are often unstructured and inconsistent in format, which leaves data missing, and claims databases often include patients who opt in and out as their employment or residence changes, which raises how continuity of coverage is tracked and accounted for. Organisation also covers how granular the data are and how the lack of linkage between prescribing and dispensing is managed, and whether it matters. Not every such question has to be addressed, but the writer should ask them at the protocol stage and again at the report stage, to avoid surprises from health authority reviewers.
Reliability and validation
Without randomisation, the writer must describe all potential biases in drawing causal inferences, including confounding, measured and unmeasured, misclassification and selection bias, and the methods used to manage confounding, such as restriction, high-dimensional propensity score adjustment, inverse probability of treatment weighting and multivariable modelling. The writer should also describe the prespecified sensitivity analyses that test the robustness of the findings, showing the effect of unmeasured confounding and other sources of bias and how large the effect of residual confounding would have to be to change the results materially. FDA guidance asks that the outcomes of interest be validated, by establishing a clinically appropriate outcome definition and determining the positive predictive value of that definition. Where the outcome has been validated before, the writer should cite the literature and describe the validated algorithm in detail, with its population, database, performance characteristics and period of validation; where it has not, the outcome definition should be justified.