Statistical analysis planning and reporting

A statistical analysis plan elaborates the analysis a trial's protocol describes and settles, before the data arrive, the question the analysis answers, the error rates and sample size it works to, and the methods that suit the data and the design. Good clinical practice sets requirements for the plan, for the statistical programming behind the analysis and for any departure from the plan. The analysis is then reported so that a reader can check what was done.

The plan and the protocol

ICH E9 describes the plan as a document that "contains a more technical and detailed elaboration of the principal features of the analysis described in the protocol, and includes detailed procedures for executing the statistical analysis of the primary and secondary variables and other data". ¹ The plan is not a stand-alone document but should be read in conjunction with the clinical trial protocol, which should be consistent with the principles of the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) statement. ¹

A list of 55 items and sub-items, prepared, tested and published by a group of experts, guides the preparation of a statistical analysis plan for a clinical trial. ¹ When the list was prepared, the experts' literature search and their contacts with funders and regulators identified no existing guidance on the content of the plan. ¹ The items fall under six sections: Title and Trial Registration; Introduction; Study Methods; Statistical Principles; Trial Population; and Analysis. ² The list's supplementary online content gives additional information and examples for each item. ² As of 2018, some journals, including JAMA, required the plan to be submitted along with the report of a clinical trial for use within the peer-review process. ^{1, 2}

Prespecifying the question and the analysis

Planning a statistical analysis is arguably more consequential than drafting a manuscript. ³ Although both processes are often iterative, the consequences of a poor statistical analysis plan, once codified as a study protocol, may be irrevocable. ³

The lack of a clearly formulated study hypothesis is a common and fundamental statistical mistake in the preparation of medical manuscripts. ⁴ The hypothesis is a quantitative formulation of the research question, and it dictates the research methodology, sample size planning, statistical analysis and reporting. ⁴ A well-written hypothesis sets out the framework for the rest of the study and ties together the flow of the manuscript. ⁴

Statistical analyses should not merely reflect planning and thoughtful analysis but culminate them, establishing and adhering to prespecified parameters and other widely recognised conventions. ⁵ After a planned test returns a statistically nonsignificant result, an analyst might be tempted to repeat the test while modifying various parameters until it gives a P value below 0.05. ^{3, 5} By the same token, the analyst should be just as prepared to vary the analysis when the first comparison gives a P value below 0.05, and then watch the initially significant P value fade. ⁵

Power and sample size

An internal logic should inform the plan and the protocol. ⁵ They typically specify a priori the outcome measures and the minimum clinically important differences sought between treatments or exposures, based on published literature. ⁵ They then set the level of confidence wanted in the findings, typically 95%, and project the number of patients needed to compare groups with a given statistical power, typically 80%, at a two-tailed α of 0.05. ⁵ Each of these limits is consistent with standard practice but also largely arbitrary and modifiable. ⁵

Computing a sample size typically requires two properties of the populations in similar studies, for example from the published literature: the effect size, which is the difference between group means, and the variance. ^{6, 7} In a comparison of lung function between people who recently quit smoking and never-smoker controls, controls drawn from the same pulmonology practice might attenuate the expected effect size, because the specialty-care setting might improve the controls' lung function more than expected. ⁶ Controls from a healthy community screening group would be expected to give a larger effect size and a smaller variance than controls from the pulmonology practice, which includes both relatively healthy people and people with marked deficits in lung function. ⁶ The larger effect size and smaller variance should give a greater chance of rejecting the null hypothesis with fewer controls. ⁶

A comparison of two antihypertensive agents shows the calculation for a continuous outcome with two groups of equal size, a two-sided α of 0.05 and a power of 0.80, whose conventional multipliers are 1.96 and 0.842. ⁶ If a difference in blood pressure of 15 mmHg is clinically meaningful, and prior studies show blood pressure in similar patients examined in similar ways to be normally distributed with a standard deviation of 20 mmHg, n = [2(1.96 + 0.842)² × 20²] / 15² = 27.9, so each group needs 28 participants. ⁶ With other parameters equal, a crossover study requires fewer participants than a parallel-group study, and it has intrinsically higher statistical power because each subject serves as his or her own control. ⁶

Methods that suit the data and the design

The selection of an appropriate statistical test depends on the character and distribution of the data. ⁸ The questions to ask in selecting one include whether the data are paired or unpaired and whether they are nominal, ordinal, interval or ratio, and, for tests of association, how many dependent variables there are, how many levels the independent variables have, and the nature and distribution of the dependent variables. ^{8, 9, 10}

The selection of a test or an overall analysis should be sound and unassailable from statistical, biological and clinical perspectives. ³ Categorising an inherently continuous outcome measure without providing boundaries for precision and statistical confidence is indefensible, and such a categorisation should also comport with prevailing biological theory. ³ Serum uric acid level, for example, is a continuous variable. ³ Suppose that investigators evaluating a new gout medication target, in an efficacy responder analysis, a mean serum uric acid level after treatment below 8 mg/dL. ³ Evidence suggests that serum uric acid levels below 6 mg/dL are associated with freedom from acute attacks of gout and a reduction of proinflammatory monosodium urate crystals in joints and soft tissues, including tophi, so the investigators set an overly conservative efficacy threshold. ³ Had they set a cut-point below 6 mg/dL, ideally with accompanying confidence or other precision limits, their analysis strategy would have been defensible. ³

Many components of a statistical analysis plan, as of a study protocol or a clinical study report, are the culmination of a rational, methodical scientific process that includes implicit value judgments. ¹¹ These judgments may inform the fundamentals of study design and conventional statistical tests as well as the most nuanced choices, such as the correlation matrix structure of a generalised estimating equation that controls for data correlated over time in a longitudinal observational study. ¹¹ Certain statistical issues warrant critical thinking, ideally during study design, to ensure fair balance: ¹¹

• Are observations independent? Independence is a common assumption of many statistical tests but may not be appropriate in certain settings. Blood pressure measured at 3 time points in 6 patients does not give 18 independent observations, and data from crossover studies are paired because each subject serves as his or her own control.
• Do the prespecified inferential tests match the likely character of the data: are the data continuous or categorical, limited by upper or lower bounds, and likely or unlikely to include outliers? A percentage cannot exceed 100 and a blood pressure cannot be below 0 mmHg, and such bounds may result in range restriction.
• How are the data distributed? Symmetrical data warrant parametric tests and asymmetrical data nonparametric tests, and some asymmetrical shapes, including Poisson and geometric, may require novel analyses.
• If the distribution is unknown from prior study reports, proactively specify an analysis that makes no assumptions about it, such as the Kolmogorov-Smirnov test, or graph the data or test their normality, for example with the Shapiro-Wilk test or the Microsoft Excel test of skew.
• What methods are proposed to adjust for any increased likelihood of spurious statistically significant associations because of multiple comparisons?
• What methods are used to control for potential bias and confounding, and residual confounding, on unmeasured variables, such as analysis of covariance, multivariate analysis, propensity score matching, sensitivity analysis and stratification?

¹¹

Good clinical practice requirements for the plan and its programming

The ICH E6(R3) section on the sponsor's statistical programming and data analysis concerns the documentation of the operational aspects of a trial's statistical activities and should be read in conjunction with ICH E9 on statistical principles for clinical trials and its E9(R1) addendum on estimands and sensitivity analysis, which give detailed guidance on statistical principles for clinical development, trial design, conduct, analysis and reporting. ¹² The sponsor should develop a statistical analysis plan that is consistent with the trial protocol and that details the approach to data analysis, unless that approach is sufficiently described in the protocol. ¹² The sponsor should ensure that appropriate and documented quality control of statistical programming and data analysis is implemented, for example for sample size calculations, analysis results for review by an independent data monitoring committee, outputs for the clinical trial report, and statistical or centralised monitoring. ¹² The sponsor should ensure the traceability of data transformations and derivations during data processing and analysis. ¹² The sponsor should ensure that the criteria for including or excluding trial participants from any analysis set are pre-defined, for example in the protocol or the statistical analysis plan, and the rationale for excluding any participant or particular data point should be clearly described and documented. ¹²

Deviations from the planned statistical analysis, and changes made to the data after the trial has been unblinded where applicable, should be clearly documented and justified and should occur only in exceptional circumstances, such as data discrepancies that must be resolved for the reliability of the trial results. ¹² Such data changes should be authorised by the investigator and reflected in an audit trail, and post-unblinding data changes and deviations from the planned statistical analyses should be reported in the clinical trial report. ¹² The sponsor should retain the statistical programming records that relate to the output contained or used in reports of the trial results, including the quality control and validation activities performed. ¹² Outputs should be traceable to the statistical software programs, dated and time stamped, protected against any changes, and subject to access controls that avoid inappropriate viewing of information that may introduce bias. ¹²

Reporting the analysis

Employing statistics is not sufficient: to be completely transparent and accountable, a report must also disclose the requirements, assumptions and other parameters of its tests, models and resulting data. ¹³ Failing this risks unsettling its readers, including journal referees, key opinion leaders, other clinicians, patients and patient advocacy groups, regulatory authorities, third-party payers and health policymakers. ¹³ In many cases a statistical assessment or model is chosen from among many other, perfectly suitable alternatives, and these choices are not free of value judgments. ¹³ A major challenge for medical writers and other researchers and communicators is not only to comprehend but to clearly disclose the statistical methods used to estimate outcomes. ¹⁴ All planned analyses and endpoints stated in the Methods must be reported in the Results, and any outcome data presented in the Results must be motivated in the Methods, including elements of the statistical analysis plan. ⁵

The Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines set out what the report of a primary analysis should contain: ¹⁵

• Describe the purpose of the analysis.
• Identify the variables used in the analysis and summarise each with descriptive statistics.
• When possible, identify the smallest difference considered to be clinically important.
• Describe fully the main methods for analysing the primary objectives of the study.
• Make clear which method was used for each analysis, rather than just listing in one place all the statistical methods used.
• Verify that the data conformed to the assumptions of the test used to analyse them. In particular, specify that skewed data were analysed with non-parametric tests, that paired data were analysed with paired tests, and that the underlying relationship analysed with linear regression models was linear.
• Indicate whether and how any allowance or adjustments were made for multiple comparisons.
• If relevant, report how any outlying data were treated in the analysis.
• Say whether tests were one- or two-tailed and justify the use of one-tailed tests.
• Report the alpha level, such as 0.05, that defines statistical significance.
• Name the statistical package or programme used in the analysis.

¹⁵

The numbers of patients or observations often vary considerably across populations and analyses, such as the safety, intention-to-treat and per-protocol populations, and information on certain variables is inevitably missing or incomplete, so percentages and other calculations rest on many different denominators. ³ These disparities often need to be footnoted in tables. ³