Observational study reports
An observational study report has to show a reader what randomisation would otherwise have secured: how people and measurements entered a cohort, case-control or cross-sectional study, a registry analysis or a study of claims and other routinely collected data, which reporting guideline frames that account, how the groups and their denominators were built, and how far the result can be read as more than an association.
What the report has to establish without randomisation
Publishing patient registry data follows the same rules as publishing any other scientific data, with clarity, conciseness, accuracy and precision as the milestones of quality, and complete information on the design and methodology lets readers evaluate the value and reliability of the results.
The objectives of the study or analysis must be specified, and the methods section seems to matter even more in a registry paper than in a trial report, because no strict protocol lies behind a patient registry, so at least in theory there is more room for data manipulation. Besides describing how the study was performed and how it addressed the objectives, the section aims to convince readers that the way the data were collected and analysed guarantees reliable, robust results.
Reporting guidelines for the design and the data
The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement was developed by a European group of methodologists, researchers and editors to improve the quality of reporting of observational studies. Its checklist has 22 items that an article should address, 18 of them common to cohort, case-control and cross-sectional studies and 4 specific to each of those three designs.
Publications based on observational studies often lack critical information or are unclear through insufficient reporting of potential confounding variables, of the methods used to identify cases and controls, and of eligibility criteria. A network of methodologists, researchers and journal editors met in 2004 to develop recommendations, and the STROBE statement was published in eight journals, with an explanation and elaboration article published simultaneously in three. It recommends the minimal information an accurate and complete article on any of the three main observational designs should include. It was the first guideline designed specifically for observational studies, and it can be applied to any study type, although many more specific guidelines now exist.
When the statement appeared, three journals gave free access to the full document, of about 30 pages, and to an introductory capsule article: Annals of Internal Medicine and PLoS Medicine in their issues of 16 October 2007, and Epidemiology in its November issue. The statement's website linked from the start to its published versions and to editorials and commentaries about it. By 2017 it also held the original English version, translations into eight other languages, the lists of journals that published the statement and the explanatory paper and of journals that refer to STROBE in their instructions for authors, and the members of the STROBE group.
STROBE is the main guideline for observational reports, including retrospective cohort analyses of administrative claims and electronic medical records, case series and patient registries, and it has been adapted to particular kinds of study, such as genetic association studies. A registry uses observational methodology, so its report should follow STROBE, which helps researchers report their work transparently and give enough information for a reader to assess its design, conduct and analysis and so judge its credibility.
The checklist provides guidance on reporting observational studies and none on designing or conducting them, and it is not designed as an instrument for evaluating their quality.
For studies of routinely collected health data, the RECORD statement was created to help authors write papers: an extension of STROBE with its own website and a 22-item checklist, produced by a collaborative process involving more than 100 international stakeholders, among them researchers, journal editors and data consumers. Such real-world data are routinely collected without a specific or a priori research question developed before the data are used for research, and their sources include registries, primary care databases and administrative data.
The STROBE checklist in sequence
The 22 items say what an article should include, section by section from title and abstract to discussion, and one of them covers funding.
- Title and abstract: indicate the study's design with a commonly used term, and give an informative and balanced summary of what was done and what was found.
- Introduction: explain the scientific background and rationale for the investigation, and state specific objectives, including any prespecified hypotheses.
- Methods: present key elements of the design early; describe the setting, locations and relevant dates, including the periods of recruitment, exposure, follow-up and data collection; give the eligibility criteria and the sources and methods of selection of participants, with the methods of follow-up in a cohort study and, in a case-control study, the sources and methods of case ascertainment and control selection and the rationale for the choice of cases and controls; for matched studies, give the matching criteria and the number of exposed and unexposed, or of controls per case; clearly define all outcomes, exposures, predictors, potential confounders and effect modifiers, with diagnostic criteria where applicable; for each variable of interest, give the sources of data and the methods of assessment, and the comparability of assessment methods where there is more than one group; describe any efforts to address potential sources of bias; explain how the study size was arrived at; explain how quantitative variables were handled and, where applicable, which groupings were chosen and why; and describe all statistical methods, including those used to control for confounding, to examine subgroups and interactions and to address missing data, where applicable how loss to follow-up, the matching of cases and controls or the sampling strategy was handled, and any sensitivity analyses.
- Results: report the numbers of individuals at each stage with reasons for non-participation, and consider a flow diagram; give the characteristics of the participants with their exposures and potential confounders, the number with missing data for each variable, and for a cohort a summary of follow-up time; report the numbers of outcome events or summary measures, or for a case-control study the numbers in each exposure category; give unadjusted and, where applicable, confounder-adjusted estimates with their precision, making clear which confounders were adjusted for and why they were included, the category boundaries where continuous variables were categorised, and where relevant consider translating relative into absolute risk for a meaningful period; and report other analyses done, such as subgroup, interaction and sensitivity analyses.
- Discussion: summarise key results with reference to the study objectives; discuss limitations, taking account of sources of potential bias or imprecision and discussing both the direction and the magnitude of any potential bias; give a cautious overall interpretation considering objectives, limitations, multiplicity of analyses, results from similar studies and other relevant evidence; and discuss generalisability.
- Funding: give the source of funding and the funders' role for the present study and, where applicable, for the original study the article is based on.
Separate checklists are also available, and for some items the information has to be provided separately for cases and controls in a case-control study, or for exposed and unexposed groups in a cohort or cross-sectional one.
The extensions and related guidelines
More than 50 guidelines are available for reporting observational studies, as extensions of STROBE or related guidelines developed for other designs or specific topic areas. Case studies and series have CARE for case reports and PROCESS for case series in surgery, with further guidance for uncontrolled case series, plastic surgery series, colorectal tumour series, acupuncture series and audits, case studies of extracorporeal treatment in poisoning and organisational case studies. Surveys have CHERRIES for internet surveys, mERA for health interventions using mobile phones, and separate guidance for self-administered surveys of clinicians and for survey research generally. Genomics and genetics have STREGA for genetic association studies, STROBE-ME for molecular epidemiology, GRIPS for genetic risk prediction and a community standard for immunogenomic data. Epidemiology and routinely collected data have RECORD, GATHER for health estimates, STROND for neurological disorders and STROBE-nut for nutritional epidemiology, alongside guidance on the methodology of chronic kidney disease prevalence studies. Infectious disease has STROBE-NI for newborn infection, STROBE-AMS for antimicrobial resistance and stewardship, STROME-ID for molecular epidemiology, ORION for outbreak reports and intervention studies of nosocomial infection, and CONSISE ROSES-I for influenza seroepidemiology. Rheumatology has a EULAR extension of STROBE for longitudinal observational drug studies and EULAR points to consider for reporting safety data from biologics registers.
Imaging and markers have PRECISE and START for prostate magnetic resonance imaging studies, REMARK for tumour marker prognostic studies, GRAPH for psychiatry and heart rate variability, criteria for evaluating novel markers of cardiovascular risk, and neuroimaging standards for small vessel disease research. Surgery and illnesses have MARE for medical abortion efficacy, CLASSIC for intraoperative complications, and standards for reporting pressure results after glaucoma surgery and outcomes in metabolic and bariatric surgery; emergency medicine has CONFIDE for disaster medicine and standardised guidelines for emergency department syncope risk-stratification research. Pain and fatigue have ACTTION for the quality of pain intensity assessment reporting, a consensus on back pain definitions for prevalence studies, and minimum data elements for chronic fatigue syndrome research. Psychology and counselling have guidance on mixed research in counselling and on effect sizes in neuropsychology. Other guidelines cover violence risk assessment under RAGEE, menopause and ageing under STROMA, end-of-life care under MORECare, veterinary observational studies under STROBE-Vet, respondent-driven sampling under STROBE-RDS, healthcare simulation research under INSPIRE, adverse reactions to traditional Chinese medicine, recurrent venous thromboembolism after stopping treatment, endovascular repair of intracranial aneurysms, immunisation safety data for viscerotropic disease, the role of narrative in clinical research, preparing raw clinical data for publication, reporting participation in case-control studies, and instrumental variable methods in comparative safety and effectiveness research.
New guidelines for observational studies continue to be developed, so the best way to find the right one is the reporting-guideline search on the EQUATOR Network's website. Guidelines under development as of 2017 included an extension of STROBE for conference abstracts, recommendations for preparing protocols for observational studies under the name SPIROS, COHERE for observational epidemiology integrating data on humans, animals or vectors and their shared environments, GREEN for environmental epidemiology analyses, GESUQ for reporting the long-term impact of genocide and war on mental health, and guidance on the psychometric properties of patient-reported outcomes.
The introduction and the gap it addresses
As with other introductions, build to the problem, issue or gap the study addresses and follow with the study's aims; whether the introduction is longer and clinically comprehensive or shorter and problem-focused depends largely on the subject matter and the journal's author guidelines. A published observational study frequently has identifiable objections that a novel study design can overcome. In one comparison of two intravenous antibiotics for community-acquired pneumonia, a de novo propensity-score matching procedure found that previously reported differences in health resource utilisation had been biased away from the null by methodological issues in the earlier study.
The data source, its purpose and the population drawn from it
Real-world evidence studies often have no tidy clinical study report summarising their methods and results, so request the protocol, the data or statistical analysis plan and the table shells from the researchers, including the biostatistician, as early as possible. Devote most of the methods to describing the setting, such as the health plan, claims database, electronic medical record setting or registry, and the patient profiles given by the eligibility criteria. For a cohort study, name the type of database, such as administrative claims or electronic medical records, and the method of case ascertainment, such as ICD-10 or, in the UK, Read codes.
RECORD asks that the title or abstract specify the type of data used and, when possible, name the databases. It asks for the methods of selecting the study population, such as the codes or algorithms used to identify subjects, to be listed in detail or their absence explained, and for any validation studies of those codes or algorithms to be referenced, with detailed methods and results where the validation was done for this study and not published elsewhere. Where the study linked databases, it suggests a flow diagram or other display of the linkage process, with the number of individuals with linked data at each stage.
Always describe the general purpose of the registry and the specific aims of an analysis of a subset of its data separately; where the analysis uses all the data, state clearly that its purpose was the same as the registry's. One study of overactive bladder first described the General Practice Research Database as a longitudinal general practice database collecting data from a representative sample of UK practices, and then gave its own purpose, which was to estimate the incidence and prevalence of the symptoms and to analyse the use of anticholinergic and antispasmodic drugs and of healthcare resources.
Give the selection criteria for the registry and for the specific study in detail, because a reader needs to know both whether the analysed population reflected the registry and whether it was representative of the target patient population. The two can differ: a registry can represent its patient population well while the analysed cohort does not. Sometimes the analysis aims to include all the registry's patients and excludes only those with incomplete data for predefined variables. In an analysis of fracture risk in adults with growth hormone deficiency, untreated and treated, the authors referred to the already published Hypopituitary Control and Complications Study and its inclusion and exclusion criteria, and then named their own, which were age, sex, disease onset as adult or childhood, at least one follow-up visit after study entry, and treatment with growth hormone, including only patients with no missing information for any of those predefined variables.
Under the US FDA Modernization Act, an observational or real-world study of a marketed product should not violate its labeling in letter or spirit. For instance, the eligibility criteria of a claims analysis should not take in patients for whom the medication is contraindicated, populations in whom it has not been adequately studied, or off-label indications, dosages or routes of administration.
For a survey, describe the sampling method, interviewer training and blinding, and any translation or back-translation of the questionnaire with its validation.
Exposures, outcomes and group definitions
As in a trial, the endpoints or outcomes should be clearly defined, before retrieval of the patient data begins and before the analyses are performed. Sometimes researchers check the availability of the data before deciding which outcomes to include, using simple frequency tables of the numbers of patients with missing and non-missing data for given variables. The fracture-risk paper clearly defined all its outcomes, and especially how the fracture data were collected, how a fracture was defined and how fractures were categorised.
Describe very precisely the definitions used in the paper. One published analysis of InfCare HIV, a registry that is mainly Swedish with some Danish centres, asked whether patients who were suboptimally treated had any specific characteristics. It started from a clear definition of suboptimal treatment, in two parts: patients who did not receive treatment despite a clear clinical indication, and patients in whom there was no successful response to treatment, defined either by the number of copies of HIV RNA or by a frequent change in treatment because of virological failure.
Results may cover adherence, health-care utilisation and other clinical and subjective outcome measures, among which minimum clinically important differences matter for patient-reported outcomes. Resource-use outcomes may include hospitalisation and length of stay, relapse or rehospitalisation, use of durable medical equipment, office visits and medications prescribed, and indirect health-care cost data are not typically available or included. For a patient survey instrument, scale or subscale, state whether an increase or a decrease signifies improvement or worsening, and give its psychometric data, validation including any back-translation, normative values, sensitivity, specificity, c-statistic and predictive values, and minimum clinically important differences; these can be omitted for a journal read by specialists well acquainted with them.
Prescriptions filled, as claims record them, cannot necessarily be equated with medications taken as prescribed. Adherence is typically assessed as the medication possession ratio (MPR) and the proportion of days covered (PDC), which range from 0 to 100%, and acceptable adherence is often defined as an MPR above 0.80 or a PDC above 80%. The PDC may be less susceptible to overestimating adherence, and may be more reliable overall, than the MPR, because the PDC's numerator is the number of days covered in a given interval whereas the MPR's is the sum of the days' supply for all fills in that interval.
Cohorts, time windows and participant flow
In a retrospective cohort study, clearly define the cohorts or subcohorts being compared, the baseline or look-back period, the index date and the follow-up period. The index date often serves as the fulcrum for the entire study timeline and data analysis. The report states how many mutually exclusive cohorts or subcohorts there were. In parallel to a trial's CONSORT diagram, an observational report may include a flow diagram based largely on eligibility criteria informed by reimbursement codes, such as ICD-10 or Read codes.
Report the period during which the data were collected, preferably in calendar dates and at least in years, and the follow-up time. Because registries enrol continuously, at the database cut-off some patients have been followed for many years and others for a short time, so the median and the percentiles or range of follow-up are recommended over the mean, which can mislead. One comparison of aetiology groups in KIMS, the registry of adults with growth hormone deficiency, reported median follow-up of 6.7 and 5.8 years in two groups, with a range of 0 to 18 years. Studies with long-term follow-up should also report the number of patients per year of follow-up.
One registry researcher holds that a registry's participant flow diagram can resemble the CONSORT flow chart of a randomised trial but is constructed the other way round. A prospective randomised trial's description runs forward in time from screening, describing what happened to participants until the analysed population is reached. A registry analysis starts from the total number of patients in the registry, looks at how many meet the analysis's inclusion criteria, and retrieves the cohort that meets them. In the HIV registry analysis the diagram started from 530 patients, and the basic inclusion condition was 48 weeks of treatment, since virological failure was defined after that time; almost 8% did not meet it, and 10% of patients were excluded. Almost 11% had received no treatment, most of them because they had no clinical indication for it, which left four patients suboptimally treated on that part of the definition; almost 80% had been treated for longer than 48 weeks, and 27 of those met the no-improvement part. The analysis compared the patients with virological failure with those who responded to treatment, and the four untreated patients were described separately.
Privacy protections, practice standards and registration
The STROBE guidelines include a complete checklist of items to be addressed when reporting an observational study, such as study design, participants and results, but make no mention of ethical requirements or registration.
Where appropriate, the methods state that the data were anonymised, which HIPAA or other patient-privacy protections applied, and whether the study was conducted consistently with the International Society for Pharmacoepidemiology's guidelines for good pharmacoepidemiology practices. The same check extends to the protections of the Ethical, Legal and Social Implications research programme and the US Genetic Information Nondiscrimination Act of 2008.
If the study was registered, give its identifier. Registries for observational studies include the WHO Registry Network and the US AHRQ Registry of Registries.
Statistics as the substitute for randomisation
Because treatment allocation is not randomised, patient groups in an observational study may differ in systematic but not necessarily immediately evident ways, and one frequent result is confounding by severity: patients with more severe disease may be more likely to receive specialty or other more advanced care that optimises their outcomes. Observational studies may also be subject to bias, confounding or residual confounding on unmeasured variables, which the randomised design typically excludes.
Applying proper statistical methodology is crucial for the credibility of registry results. Simple descriptive studies need only simple descriptive statistics, but comparisons between treatment groups, prediction of treatment outcomes and mortality analysis need more advanced techniques, because treatments in a registry are not randomised and the groups may differ systematically. The analysis must consider that selection bias, the advanced methods it uses must be described precisely in the paper, and the analysis should account for all known or potential confounders. A KIGS registry paper comparing changes in body mass index during long-term growth hormone treatment across aetiology groups that already differed before treatment used advanced statistical methods and described them well.
In an observational study, state the strategies used to minimise, control or account for confounding or bias, such as analysis of covariance, matching, sensitivity analysis or stratification. Where propensity score matching is used, disclose its method, such as greedy matching. Mixed models may conserve patient data, so that a patient is not lost from the analysis for a single missing datum or a small number of missing observations. Intracorrelated longitudinal registry data require special statistical models such as generalised estimating equations, with the structure of the correlation matrix disclosed.
For comparative effectiveness research, the 2009 ISPOR good research practices for retrospective database analysis ask the background to discuss the suitability of the database employed. They ask the methods to give the study's goals and subquestions, the design and why it was chosen, the strengths and weaknesses of the data source and how the study groups were identified, how treatment effects and outcomes were measured and classification bias addressed, and the potential for confounding, by indication, by measured and unmeasured variables and over time, with how it was assessed and addressed. They ask the discussion to interpret the results taking account of confounding and imprecision, and to address generalisability and the results in the context of current evidence.
Results and their denominators
Results should be factual, structured according to the study objectives, giving exact data and avoiding interpretation, and STROBE supplies a comprehensive guide to presenting them. The level and type of missing data should be predicted at design time and assessed again while the data are cleaned and analysed. Different statistical approaches can be used depending on the extent and kind of missingness, whether completely at random, at random or not at random, and the approach needs to be described precisely in the statistical methods section. Good practice is also to report the number of observations each result rests on, as the KIMS table of baseline characteristics for the 2,589 patients in whom cardiovascular risk factors were analysed did with a column giving the number of non-missing values, which showed almost complete data for lipids and blood pressure and body composition data for only a proportion of patients.
In the HIV registry analysis, the baseline characteristics table resembles the baseline table of a randomised trial but belongs to the results, so the two groups are compared and p-values are given. The patients who failed treatment were younger and were younger when the infection was diagnosed. They had a more profound phenotype. More of them were of African origin.
The analysis also showed why treatment failed: most of the patients who failed treatment had not been adherent to it. No single reason for the non-adherence could be identified, and many reasons fell under other causes, which was not very informative. The treating physicians were nonetheless convinced that treatment and adherence could be improved in the majority of those patients, which was the pragmatic conclusion of the analysis. Of the four who received no treatment, three had refused it.
Having defined suboptimal treatment partly as a frequent change in treatment because of virological failure, the HIV registry analysis went on to compare the number of treatment changes between the failure group and those who responded, and reported that more patients in the failure group had changed treatment more than five times. One registry researcher judges that comparison not really logical and would rather not have shown it, because if change of treatment sits inside the definition, then there must be a difference.
Limitations and conclusions of association
The limitations section is particularly important in a registry paper, and should cover both the aspects specific to the particular analyses and the general issues inherent in this type of research. STROBE highlights the identification and discussion of bias, confounding and imprecision, and recommends relating the limitations discussed to other studies in terms of validity, generalisability and precision. The fracture-risk paper named a number of limitations and, wherever possible, explained the attempts made to minimise their impact on the credibility of the results: the authors recognised that patients on growth hormone differed from untreated patients, accounted for the identified confounders in the statistical analyses, and acknowledged in the discussion that residual confounding could still exist.
Potential biases include channeling, treatment by indication, selection and immortal time, with possible confounding on unmeasured variables. Claims analyses may hold lower-quality or limited patient-level data, and ICD-9-CM, ICD-10 and Read codes are intended for reimbursement rather than case ascertainment, so coding errors are possible. A cross-sectional study may not be suited to assessing a frequently waxing and waning disorder.
The baseline or look-back period is in many cases relatively short, on the order of 1 to 2 years, and may not capture patients treated many years before the index date. Patients identified in a short baseline who receive a new medication on the index date may have more severe disease than those whose disease was managed, or even cured, earlier by older medications, and channeling (allocation) bias may result when patients with more advanced disease are more likely to receive a new medication, spuriously associating its use with a previously unknown comorbidity. Most health claims database analyses require a patient to be enrolled for at least 1 year before and after the index date, and because continuity of care may be associated with higher adherence, MPR and PDC values may be inflated in some such analyses.
For routinely collected data, RECORD asks the discussion to address the implications of using data that were not created or collected to answer the specific research question, including misclassification bias, unmeasured confounding, missing data and changing eligibility over time.
The general rule is that studies performed with registry data do not prove causality and can only indicate associations between observations, so conclusions must be drawn with caution, taking into account the nature of the study and its potential sources of bias, confounding including residual and unknown confounding, and imprecision. The results of sensitivity analyses and of subgroup analyses may help make those conclusions balanced and reliable.
Observational studies may be associational in nature, unable to determine causality or its direction conclusively. Correlations between independent and dependent variables in regression analyses of observational data do not necessarily imply causation, and are most conservatively reported as associations between the variables rather than as effects of one on the other.
Access to codes and analysis materials
RECORD asks for a complete list of the codes and algorithms used to classify exposures, outcomes, confounders and effect modifiers, or an explanation where they cannot be reported. It also asks authors to say how to access any supplementary information, such as the study protocol, raw data or programming code.