Comparing means and adjusting for covariates
The means of a continuous outcome are compared between groups by tests chosen for how the observations are related: a t test for two independent groups or for paired observations, and an analysis of variance whose overall test asks whether three or more groups differ. Such a test can be run in software, adjusting for a covariate changes the comparison an analysis of covariance makes, and designs with several factors, several outcomes or repeated measurements each extend the analysis of variance to their own question, the last also answered by mixed models.
Independent and paired t tests
Student's t test compares the means of a continuous variable between two groups, in one form for two separate samples and in a paired form for two datasets obtained from the same subjects, such as a value before and after treatment, while a comparison among three or more groups goes to analysis of variance, ANOVA. It is probably the most widely used test for comparing two small datasets, and in essence it determines the 95% confidence limits of the difference between the means of two samples.
For two independent groups, t is the difference between the sample means divided by the standard error of that difference, which can be computed in several ways. A textbook's worked example compares the body mass of 10 professional American football players, mean 103.9 kg (SD 17.4), with that of 10 Olympic marathon runners, mean 63.1 kg (SD 9.5), finds a standard error of the difference of 6.27, and concludes at p < 0.001 that in the study population the football players are significantly heavier.
A paired comparison takes the difference within each pair and analyzes those differences, keeping the sign of each so that an increase can be told from a decrease. The t value is the mean of the differences divided by its standard error, which is the standard deviation of the differences divided by the square root of the number of pairs, and the worked examples in both books, each of 10 pairs, read the critical value at 9 degrees of freedom.
One book's paired example is the diastolic blood pressure of 10 patients with essential hypertension before and after 4 weeks of treatment with a beta-blocker, where a mean difference of −12.5 mmHg, with a standard deviation of 12.3 and a standard error of 3.89, is judged a statistically significant decrease at p < 0.05. The other book's paired example is a randomized crossover trial in which each of 10 subjects receives an investigational antidiabetic medication and placebo in a randomized sequence, serving as his or her own control, with haemoglobin A1c as the outcome. Its mean values are 6.57% on active treatment and 7.58% on placebo, with a mean difference of 1.01 (SD 0.28), and the two distributions are significantly different at p = 9.55 × 10⁻⁶.
The analysis of variance
ANOVA sets the variation due to treatment effects against the variation due to random probability or biological variation, as the signal against the noise. In a one-way analysis of variance, the total sum of squares is partitioned into the sum of squares due to differences between the group means and the residual sum of squares due to differences within the groups. Dividing each by its degrees of freedom gives mean squares, and the ratio of the between-group to the within-group mean square is the F statistic. In a nutshell, ANOVA compares the individual variances of the subgroups with the variance of the whole group, and it is one of the most widely used statistical procedures for assessing multiple samples. ANOVA has assumptions similar to those of the t test: a normal distribution of results in each group, that is, normally distributed residuals, independent observations, and equal variances in each group, the last also known as homogeneity of variances or homoscedasticity.
One manual holds that ANOVA is a versatile test that has been applied in numerous experimental designs and enables comparisons of multiple samples, and that its F test limits statistical error and hence augments power. The same manual holds that ANOVA is less frequently associated with type 1 error than the t test.
In a worked example, three daily doses, 2, 3 and 4 mg, of an investigational erythropoietin-stimulating agent are given to three independent groups of women with anaemia associated with chronic kidney disease, with mean on-treatment haemoglobin levels of 11.47, 12.62 and 15.54 in groups of 20, 16 and 19. F is 134.4 against a critical value of 3.15 to 3.18 on 2 and 52 degrees of freedom, so the null hypothesis is rejected and the alternative accepted, that there is a statistically significant difference among the effects of the three doses.
One manual calls ANOVA robust to violations of its assumptions, naming independence, normality and equal variances. It records as a limitation of its own example that the variances were similar in the 2 mg and 3 mg groups but not in the 4 mg group, a disparity that might warrant further statistical testing, such as sensitivity analyses, or discussion in the manuscript.
Running the analysis in software
In Microsoft Excel, a one-way ANOVA is run by choosing ANOVA: Single Factor from the Data Analysis option, which may first need the Analysis ToolPak loaded as an add-in. The user specifies the input range of the group columns, the grouping, whether the first row holds labels, the significance level and the cell where the output goes, and the output is a summary of the groups followed by the ANOVA table.
Analysis of covariance
Analysis of covariance, ANCOVA, joins analysis of variance to the concepts and procedures of linear correlation and regression, and it can often be used to adjust associations between independent and dependent variables for the effects of one or more continuous variables, often termed covariates, codeviates or nuisance variables. One aim of ANCOVA is to increase statistical power by decreasing error variance when trial participants are studied under several independent experimental settings, and in nonrandomized studies it can be employed to offset biases in samples. Covariates commonly encountered in randomized controlled trials and their analyses include treatment centre, gender, and baseline values for age, disease severity and comorbidities, the last often expressed as a continuous variable through the Charlson Comorbidity Index. One manual lists, among the questions a Methods section should answer, what the major assumptions of any statistical model were, such as the covariates in an ANCOVA.
In a worked example, 20 overweight patients lose weight over 3 months on one of two diets, 10 on high-protein solid foodstuffs and 10 on isocaloric prepackaged nutrient drinks, and each patient's interest in carbohydrates, scored beforehand on a craving scale from 1, extreme cravings, to 30, no cravings, is the covariate. The mean weight lost is 30.8 lb (SD 7.3) in one group and 29.8 lb (SD 6.0) in the other, and a one-way ANOVA finds no significant difference, with F = 0.11 and p = 0.74; the mean craving scores, 13.2 (SD 5.8) and 18.0 (SD 7.5), do not differ significantly either, with p = 0.13. Once the sums of squares for weight lost are adjusted by removing the proportion of their variance that covaries with the craving scores, F is 9.6 on 1 and 17 degrees of freedom against a critical value of 8.4 at p = 0.01, so differences in weight lost significantly covary with differences in craving scores, and the adjusted mean values differ significantly although the original ones did not. The calculation keeps the covariance within the groups apart from the covariance in the total array of data, and it subtracts one further degree of freedom from the within-group term for the covariance removed, which is how 20 observations in 2 groups leave 17.
Factorial and multivariate designs
One assumption of a factorial ANOVA is that the design is balanced, with all combinations of the variables equal. One manual's example of a three-way design is six patients treated with one of two medications, each at one of three doses, over four time points, which it calls a 2 × 3 × 4 factorial ANOVA with 24 values for each patient and 144 in all.
Multivariate analysis of variance, MANOVA, extends ANOVA to two or more outcome measures or response variables.
Repeated measures and mixed models
Repeated-measures ANOVA may be appropriate in a longitudinal trial in which each patient is randomized to more than one treatment and has a value for each treatment and observation time, because it evaluates random variability in both nonrepeated and repeated factors. An example is a 12-month study of two oral antidiabetic drugs in 10 patients observed at four quarterly visits, asking whether mean changes in haemoglobin A1c differ across the treatments and over time in the same patient, where the analysis accounts for variability within each patient over time, between patients and between treatments. Its assumptions include what one manual writes as sphericity/compound symmetry, which it says requires equal correlations between repeated measures, irrespective of the time between observations, and equal variances in the endpoint at each observation, and to avoid violating it the manual advises randomizing treatment sequences and spacing observations so that the effect of a treatment at one time point does not carry over to the next. Its other assumptions are a normal distribution of the dependent variable at each level of the within-subject variables, with standard deviations that do not differ markedly between treatments or patients, and independent observations across patients drawn from randomly selected subjects.
The potential limitations of repeated-measures ANOVA are that it treats time as a categorical variable, so that all patients must be assessed at the same visits and over equal intervals, that it cannot be used for time-dependent covariates, that it may make missing data need imputing, and that its assumptions about correlational structure are restrictive. Most of these assumptions are not violated by the typical randomized controlled trial but may well be in patient registries and other cohort studies, where mixed models, with random and fixed effects, may have advantages over repeated-measures ANOVA in discriminating between-group differences in longitudinal studies with within-patient correlated data. Mixed models function satisfactorily even if observations are not evenly spaced over time, are effective whether a design is balanced or imbalanced, do not require sphericity and enable computational flexibility.