Regression and model reporting

A regression or other fitted model estimates an outcome from a set of predictors, and each of its numbers claims something specific: an odds ratio, a hazard ratio and a beta coefficient each describe a different kind of effect against a different reference. Reporting a model accurately means naming its family and its variables, allowing for observations that are not independent, justifying the variables it adjusts for, stating each parameter with its direction and its interval, and handling a continuous predictor without cutting it into categories.

Simple linear regression

Linear regression describes the relationship between two variables: an independent variable, also called the predictor or explanatory variable, that is believed to determine the value of the other, the dependent variable, response or outcome. ¹ Which variable is the cause and which the effect has to be clear, because the relationship runs in one direction only. ¹ As so often in statistics, the language makes a simple concept seem difficult: the word linear all but gives the method away, and the word regression causes the confusion. ¹

In a simple experiment, fifteen identical glasses of ice-cold water at 32°F are heated one at a time in a microwave oven set to 100 watts, each for a different number of seconds between 1 and 30, and each temperature is measured as the oven stops. ¹ The readings rise from 41°F after 1 second to 118°F after 30. ¹ Plotted, the points wobble around a straight line. ¹ A line drawn with a ruler to sit as close as possible to all of them is a compromise that best represents the data, and in a loose sense the points "regress" to it. ¹ Eyes and a ruler give several plausible best fits, so simple mathematical methods are used to find the line that actually fits best. ^{1, 2} Drawing a straight line is still a conceptual step, because the line is a model asserting that the relationship is linear: each value of X, the time in the oven, has exactly one corresponding value of Y, the water temperature. ^{2, 3} A line running from lower left to upper right says that the longer the time in the oven, the higher the temperature. ³

The line is written Y = a + (b × X), where a is the intercept, the value of Y when X is 0, and b is the slope, the steepness of the line; a slope of 0 gives a horizontal line. ⁴ Software such as Excel or GraphPad Prism calculates the line and its formula, here Y = 41.837 + (2.1788 × X). ^{5, 6} The formula gives values that were not measured: after 15 seconds, 41.837 + (2.1788 × 15) = 74.519°F. ⁷ It also allows hypothesis testing. ⁷

A comprehensive description of a linear regression shows two more things. ⁷ Because the fitted line is only an estimate from data that are neither perfect nor complete, the graph should show the 95% confidence interval for the regression line, which can be said to include the true regression line with 95% probability; the interval is wider at both ends, where there are fewer data points. ^{7, 8, 9} The reader also needs the goodness of fit, which many programs report as R², the coefficient of determination, running from 0, no fit, to 1, every point on the line. ^{7, 10, 11, 9} In the experiment, an R² of 0.948 means that the model explains 94.8% of the variance of the points around the line, an extremely good fit, and the remaining 5.2% comes from other factors such as measurement error and machine variation. ⁹

Its apparent simplicity lends linear regression to misinterpretation and abuse, so a linear regression has to be described very carefully. ⁹ The description should make clear what is thought to be the cause and what the effect, and there must be a plausible medical or biological rationale for the two variables to be in a cause-and-effect relationship; without one, the results are meaningless and arbitrary. ⁹ Simple linear regression considers only two variables, so other factors may be influencing their relationship unseen. ⁹ Mathematics always produces a regression line, and whether the line truly represents reality has to be judged from expertise and knowledge. ⁹

Linear regression rests on four assumptions: ¹²

• The linear model is correct. Many relationships are better described by a curve, and linear regression is then inappropriate.
• The scatter of points around the line is normally distributed. A plot of the distances from the line should look like a bell-shaped curve, and formal tests can also check it.
• The variability around the line, its standard deviation, is the same along the whole line.
• The points are independent: whether a point falls above or below the line is due to chance, and one point does not influence the position of another.

¹²

Variables and kinds of analysis

Regression predicts or estimates a dependent variable from the independent variables present, and it is an associational, not a causal, form of analysis. ¹³ A regression model can compare levels of a treatment response across two intervention or exposure groups, or across three or more, estimating the association of each level of a categorical variable against baseline through indicator variables. ¹³ Multiple regression separates the associations of several independent variables with one dependent variable, so that each variable can be analysed separately. ¹³

Multivariable analyses are among the central statistical methods of clinical trials. ¹⁴ A univariate analysis involves a single variable and is descriptive by nature, and it answers only descriptive questions in a single arm or cohort, such as the rate of responders to drug X or the mean survival time in patients treated with drug Y. ¹⁵ Bivariate and multivariable analyses probe the relationships between two, or more than two, variables. ¹⁵ They can test a previously defined hypothesis, such as the primary efficacy analysis of a confirmatory study, or explore the relationships among the collected variables, as in between-arm, subgroup and exploratory analyses, and they answer analytical questions in one or more cohorts. ¹⁵ Their variables are dependent, also called outcome or predicted variables, or independent, also called predictor or explanatory variables, and in some models the independent variables are further classified into factors and covariates. ¹⁶ A bivariate analysis has one independent and one dependent variable, and a multivariable analysis has one dependent variable and two or more independent variables. ^{16, 15}

A bivariate analysis is often called univariate to distinguish it from a multivariable analysis, and in that context the term univariate is correct and replaces bivariate. ¹⁵ A multivariable analysis is not a multivariate analysis, which assesses the relationships of several predictors with two or more dependent variables at the same time. ¹⁵ The two terms are often, and incorrectly, used interchangeably, so a multivariable analysis may be found described as multivariate. ^{15, 13} Regression follows the same distinction: multiple regression has only one dependent variable, while multivariate regression involves two or more main dependent variables and is less commonly used. ¹⁷ Simple linear regression includes a single independent variable, and multiple linear regression two or more. ¹³

Both dependent and independent variables can be either quantitative or categorical, and identifying which is essential to interpreting and reporting the results correctly. ¹⁸ Common quantitative outcomes are cholesterol levels, blood pressure and questionnaire scores, and common categorical outcomes are survival, response to treatment and the presence or absence of a specific event. ¹⁸ Common quantitative predictors are age, body mass index and baseline values of the outcome, and common categorical predictors are treatment arm, gender and baseline disease severity. ¹⁸ A categorical variable with only two categories is dichotomous. ¹⁸ The dependent variable is the one the study assesses. ¹⁸

To interpret a multivariable analysis correctly, the bivariate analyses between the variables in the model should be looked at first. ¹⁵ They show the raw relationships between the dependent and independent variables, from which the unadjusted associations can be quantified, and any correlations or associations among the independent variables, which may require changes to the model. ¹⁵

A multivariable analysis is needed in two cases. ¹⁸ Where there is one main independent variable of interest and the others are secondary, the analysis evaluates the relationship between that variable and the outcome after adjusting, or controlling, for other independent variables that may also be related to the outcome. ¹⁸ Such results read "Patients treated with drug A had significantly higher cholesterol levels at 6 months compared to patients treated with placebo, after adjustment for baseline cholesterol" or "Higher biomarker X levels were significantly associated with a higher response rate, independently of age and gender". ¹⁸ Where there are two or more main independent variables of interest, the model either explores which of them are independently associated with the outcome, keeping a significant P value despite the inclusion of the others, or predicts the outcome from independent variables known to be associated with it. ¹⁸ Exploratory models are commonly used to look for "causal relationships", but their results must always be interpreted with caution, because associations may be due to confounding factors that were not accounted for. ¹⁸ A result of such a model reads "In patients with disease Z, male gender and higher blood pressure were independently associated with higher mortality". ¹⁸ Predictive models are commonly used in oncology to establish prognostic factors that may be useful to select candidate patients for more aggressive therapies, and they can also predict response, compliance and quality of life. ¹⁸ A result of such a model reads "In patients with disease Z, the independent factors predicting response to drug X were tumour stage at diagnosis and baseline beta-2-microglobulin level. The model with these two variables correctly predicted the response in 65% of patients". ¹⁸

Model families

The general linear model is a broad term for a formulation that encompasses multiple regression, ANOVA, ANCOVA and MANOVA, and simple and multiple linear regression models are examples of it. ¹⁹ Generalized linear models extend linear regression to response variables whose errors are not normally distributed. ¹⁹ They relate the independent variables to the outcome through a link function, which is the log odds, or logit, in logistic regression and logarithmic in Poisson and Cox regression; multiple regression is the special case in which the link is the identity function. ¹⁹ Poisson models relate to infrequent events, including rare genetic disorders. ¹⁹ Conditional regression models, which include Cox regression and conditional logistic regression, are special generalized linear models that base their estimation on the distributions of exposures among participants within risk sets or case-control strata. ¹⁹

Linear regression assumes a continuous dependent variable, and logistic regression a categorical one, such as a binary one. ^{20, 13} Linear regression models are used when the association between the variables is known or likely to be linear. ¹³

Three of the most commonly used multivariable models differ in the type of their dependent variable. ^{17, 21} Multiple logistic regression determines how one dichotomous dependent variable varies with two or more independent variables, as in the probability of developing liver cancer over ten years, given weekly alcohol consumption at age 30 and gender. ¹⁷ Multiple Cox regression is similar but takes a dichotomous time-to-event dependent variable, such as time to death; it could potentially be used to determine whether a new treatment affects the probability of disease progression in patients with renal cell carcinoma of different clinical stages at baseline. ¹⁷ Multiple linear regression, multiple analysis of variance and analysis of covariance take a continuous dependent variable, one that can theoretically take any value in its range, as in whether a new treatment reduces the score for a disease index after adjusting for country and baseline score. ^{17, 21} These three models differ slightly from each other but can be considered equivalent from a medical writer's point of view. ¹⁷ Analysis of variance and covariance need at least one categorical independent variable. ²¹

Correlated observations

Repeated observations of the same participant over time, as in registry and other longitudinal data, are not entirely independent but intracorrelated within each individual. ¹⁹ A multivariate analysis, which evaluates how several outcomes vary together, can also handle data from repeated measures in a single person over time or from clustered settings with two or more persons in each cluster. ¹³

General linear models include fixed-effects, random-effects and mixed models. ²² Gender is a fixed effect, as are treatment group assignment in most studies and, typically, baseline age or age group in a study without a long follow-up, while each study participant, who can be viewed as a random sample of the study population, is a random effect. ²² Mixed models incorporate both fixed and random effects. ²² In a study of blood pressure among the enrollees of a health-care plan, a fixed-effects model might compare three angiotensin II receptor blockers on the plan's formulary, a random-effects model three randomly selected medications, and a mixed-effects model the formulary medications against the randomly selected ones. ²² Mixed models also need to account for interactions between fixed and random effects: where a relationship, such as the slope of a regression line, may differ in each participant, a random slope term would need to be introduced to account for that interaction. ²² Mixed-model repeated-measures analyses may be well suited to intracorrelated or clustered data, and they conserve data, because a subject's observations are not all lost on consent withdrawal or attrition. ²²

Generalized estimating equations extend the generalized linear model to correlated observations, which occur in longitudinal studies and cluster designs. ¹⁹ Linear mixed models suit correlated responses partly modelled by random effects, and generalized linear mixed models extend generalized linear models to include both fixed and random effects. ¹⁹ Typically, generalized estimating equations are used for population-averaged inferences, which characterise differences in the mean response across a total population and can inform formulary decisions for a hospital, clinic or health plan, while generalized linear mixed models are used for patient-specific inferences, conditional on each patient's random effect. ¹⁹ Generalized estimating equations include no probability model for the origin of the correlations, so they are acceptable when random effects and their variances are not of interest. ¹⁹ Generalized linear mixed models, a type of mixed or multilevel model, can help explain those origins by providing a probability model that considers random effects as a linear predictor. ¹⁹ Generalized estimating equations may require large samples to ensure adequate accuracy and are not robust to nonrandom missing data, while generalized linear mixed models are more demanding to construct and fit to data. ¹⁹ Against a fixed-effects model, generalized estimating equations offer greater computational efficiency and stability and allow a less restrictive correlational structure, while a mixed model gives its coefficients a subject-specific interpretation. ²² A report that uses generalized estimating equations should specify the structure of the correlation matrix: autoregressive, exchangeable, independent, M-dependent or unstructured. ²³

Continuous predictors and cutpoints

In medicine, continuous measurements such as age, weight and blood pressure are often used to assess risk, predict outcome or select a therapy. ²⁴ Background knowledge or the type of question can strongly influence how continuous variables are used, but the method for analysing them must often be selected. ^{24, 25} Continuous variables are commonly assumed to be linearly related to the outcome, which is often inappropriate. ²⁵ To avoid that assumption, cutpoints are often applied to categorise the variable, which implies a regression model with step functions. ²⁵ At first glance categorising seems to simplify the analysis and aid interpretation, but it discards information, raises the questions of how many cutpoints to use and where to place them, and creates biologically implausible step functions in which individuals above and below a cutpoint have different risks. ²⁵ Continuous predictors are very commonly converted into categories before the analysis, using a cut-off previously defined in the literature, because physicians find a comparison of one category with another much easier to interpret than a risk attached to a one-unit increase. ¹⁸ The conversion loses statistical power and risks not finding significant results. ¹⁸

The prognostic effect of age on recurrence-free survival in breast cancer, in publicly available data from a study by the German Breast Cancer Study Group, changes with the way age is categorised. ²⁵ An "optimal" cutpoint at 37 years produces a large difference between the survival curves, with a Cox hazard ratio for older patients of 0.54 (95% confidence interval 0.37 to 0.80). ²⁵ With the median of 53 years as the cutpoint the difference disappears, with a hazard ratio of 1.1 (95% confidence interval 0.88 to 1.39). ²⁵ Three groups formed on predefined menopausal cutpoints of 45 and 60 years again show small differences. ²⁵ Five ten-year age groups starting at 40 show recurrence-free survival slightly lower below 40 and negligible differences between the other groups, revealing that age is not linearly related to the outcome. ²⁵ So cutpoints can lead to different and inconsistent results, even when only one variable is considered. ²⁵

An alternative and more appropriate approach estimates the functional form of the continuous variable on the outcome, for example with spline-based approaches or fractional polynomials, which use the full information from the variable and have several advantages over cutpoints. ²⁵ In the breast cancer data, fractional polynomials clearly showed a strong non-linear effect: the relative risk of an event is high for patients of about 30, decreases rapidly with age, and from 40 onwards age has a negligible influence on recurrence-free survival. ²⁵ Because there is no widely accepted agreement about how to handle continuous variables, many analysts proceed with cutpoints, and introductory graduate-level courses often encourage them, so guidance that includes evidence of the advantages and disadvantages of competing strategies is needed. ²⁵

Different cutpoint strategies across individual studies complicate assessing the risk or prognostic effect of a continuous variable in a meta-analysis. ²⁶ By 1994, 19 different cutpoints had been used in the literature to categorise S-phase fraction as a prognostic factor in breast cancer. ²⁶ A meta-analysis comparing low against high values could be done but could not be interpreted, because a patient with a given value could belong to the low group in one study and the high group in another. ²⁶ In a review of p53 as a prognostic factor for bladder cancer, cut-off values for defining nuclear overexpression ranged between 1% and 75%, and the reviewers found it frustrating that a decade of research on p53 and bladder cancer had not placed them in a better position to draw conclusions relevant to the clinical management of patients. ²⁶ Forcing cutpoints to fit the data may therefore both lead to misleading conclusions and reduce the usefulness of the results for clinical decisions. ²⁶

In an observational study several factors can influence the outcome, so a multivariable analysis is needed, and besides investigating the functional form of a continuous variable the researcher must decide which other variables to include in the model. ²⁶

Confounders and covariates

Some independent variables enter the model because they are confounding variables, sometimes called covariates: factors related to both the dependent and the independent variables. ¹⁸ Unless the analysis is adjusted for them, the result may misrepresent the true relationships. ¹⁸ In many clinical trials the baseline value of a quantitative outcome, such as baseline blood pressure in a hypertension trial, is a potential confounder if randomisation has not fully balanced it between the arms, because it is also related to the outcome. ¹⁸ So the primary efficacy analysis should always include the baseline value of a quantitative outcome as a covariate. ¹⁸

No set of prespecified covariates is ever complete. ²⁷ In most randomised trials the covariates controlled for in an analysis of covariance include treatment and study centre, while patient age, duration of disease and degree of comorbidity, which are often clinically important, are controlled somewhat less often. ²⁷

Odds ratios, hazard ratios and beta coefficients

The independent variables define the subgroups of patients in which the outcome is compared, such as treatment arms, and what a model's parameter estimates depends on whether an independent variable is categorical or quantitative. ¹⁸ For a categorical variable, such as treatment arm or gender, the odds ratio, hazard ratio or beta coefficient always estimates the effect on the outcome of one or more categories against a reference category, such as placebo or female gender, which must be defined a priori. ¹⁸ For a quantitative variable, such as age, no subgroups are compared, and the parameter estimates the effect on the outcome of each one-unit increase, such as each 1 mg/dl increase in baseline cholesterol. ¹⁸ Where the model keeps the original continuous predictor, the writer can help interpretation by reporting the risk associated with, for example, a ten-unit increase. ¹⁸

An odds ratio gives the odds for a category against the reference category, or for a one-unit increase, as in "the odds of treatment failure were three times higher in men than in women". ²¹ A hazard ratio gives the instantaneous risk, or hazard, per unit time, usually one day, for the same comparisons. ^{21, 28} A new treatment with a hazard ratio for death of 0.5 against the existing gold standard means that the rate of death at any time during follow-up is twice as high with the gold standard as with the new treatment, and a hazard ratio of 1.0 would mean no difference in the rate of death between the two. ²⁸ A beta coefficient gives the size of the effect on the outcome in the outcome's own units, as in "systolic blood pressure was 3 mmHg higher in men than in women". ²¹ Multiple linear regression returns one beta coefficient for each quantitative independent variable and for each non-reference category of each categorical independent variable. ²⁹ For a continuous independent variable, the coefficient indicates how a one-unit change in it would affect the dependent variable if all other variables in the model were held constant, so its units are those of the dependent variable divided by those of the independent variable; for a categorical independent variable, its units are those of the dependent variable. ²⁹ Where beta coefficients are given for a logistic or Cox model, beta is the natural logarithm of the odds ratio or of the hazard ratio. ^{29, 21, 30} Odds ratios from a multiple logistic regression adjusted for confounding variables are called adjusted odds ratios. ³¹

Odds ratios, hazard ratios and beta coefficients are typically presented with confidence intervals. ^{31, 28, 32} For an odds ratio or a hazard ratio, a 95% interval that includes 1 prevents an inference of a significant difference between groups. ^{31, 20, 32} For a beta coefficient, an interval that excludes 0 means that the association between that independent variable and the dependent variable, after adjustment for covariates, is significant. ^{32, 29} The interval for a beta coefficient is sometimes replaced by the standard error and the P value. ³²

Stating an association

Whatever the model, good practice is to list all the factors taken into account in the multivariable analysis, including those discarded during modelling, and always to include the parameter that indicates the strength and direction of the association, the odds ratio, hazard ratio or beta coefficient, preferably with its 95% confidence interval, the P value for that variable, which differs from the overall P value for the model, or both. ³² A writer at all unsure of the direction of a particular association should ask a statistician. ³²

A common mistake is to leave out the direction of an association, as in "Biomarker X levels were significantly associated with the response rate", from which the reader cannot tell whether a higher response rate goes with high or low biomarker levels. ³³ Such an association would usually be read as positive if nothing indicates otherwise, but the direction should be stated clearly. ³³ The words "independently associated", "independent factor" and "independent predictor" imply that a multivariable model was used and that the relationship has been adjusted for at least one additional factor. ³³

The wording of a result differs with the direction of the association. ³² For odds ratios and hazard ratios, a value below 1 means the factor is negatively associated with, or protects against, the outcome, and the percentage decrease in the odds or the risk is (1 − the ratio) × 100, as in "Category X protected against mortality (adjusted OR=0.8, 95% CI=0.6 to 0.9 versus reference category Y)" or "Compared to Y, X was associated with a 20% reduction in the odds of death". ³² A value above 1 means the factor is positively associated with the outcome, and the percentage increase is (the ratio − 1) × 100, as in "Category X was a risk factor for mortality (adjusted HR=1.5, 95% CI=1.1 to 1.9 versus reference category Y)" or "Compared to Y, X was associated with a 50% increase in the risk of death". ³² When the percentage is 100 or more, the "number of times" construction is often used, as in "Patients with X had a risk of death approximately three times higher compared to those with Y (adjusted HR=3.2, 95% CI=2.1 to 4.9)". ³² An adjusted result from a logistic model reads "Compared to teetotallers, those who consumed 40+ units of alcohol per week at age 30 had higher odds of developing liver cancer (adjusted OR=1.68, 95% CI=1.25 to 2.13)". ³¹ One from a Cox model reads "Compared to the gold standard treatment, the new treatment was associated with a significantly lower rate of death (adjusted HR=0.5, 95% CI=0.25 to 0.75)". ²⁸

For beta coefficients, a value below 0 means the factor is negatively associated with the outcome and a value above 0 that it is positively associated, as in "A 1-unit increase in X was associated with a decrease of 3 mmHg in systolic blood pressure at 4 weeks (β=−3, 95% CI=−2.1 to −3.9)" or "A 1 mg/dl increase in X was associated with a 2-unit increase in quality of life score at 6 months (β=2, SE=0.3, p=0.025)". ³² The statistical significance of such results should be reported with the P value of each coefficient. ³² For a categorical factor, the coefficient and its P value show whether the category is associated with a greater outcome than the reference category: in a model of height at age 18, a positive coefficient for males relative to females with a P value below 0.05 would indicate that males are likely to be taller than females at age 18. ³² For continuous factors, a result reads "Higher birth length was associated with greater height at age 18 (β=1.2 cm/cm, 95% CI=0.93 to 1.49). Age at puberty onset was inversely associated with height at age 18 (β=−0.3 cm/year, 95% CI=−0.19 to −0.45)". ²⁹

Reporting the analysis

A methods section should give the major assumptions of any statistical model, such as the covariates in an analysis of covariance, say whether the analyses were based on multivariable or multivariate regression models, and name the populations that made up the analysis datasets. ²⁰ For odds ratios and other comparative relative-risk data, it should also say which group was prospectively designated the referent, the one whose odds ratio is 1.0. ²⁰

The SAMPL guidelines (Statistical Analyses and Methods in the Published Literature) recommend that a report of a regression analysis: ³⁴

• Describe the purpose of the analysis.
• Identify the variables used and summarise each with descriptive statistics.
• Confirm that the assumptions of the analysis were met, for example in linear regression by indicating whether an analysis of residuals confirmed the assumption of linearity.
• If relevant, report how any outlying values were treated.
• Report how any missing data were treated.
• For a simple or a multiple regression, report the regression equation.
• For a multiple regression, report the alpha level used in the univariate analysis, report whether the variables were assessed for collinearity and for interaction, and describe the variable selection process by which the final model was developed, such as forward-stepwise or best subset.
• Report the regression coefficients of each explanatory variable with their confidence intervals and P values, preferably in a table.
• Give a measure of the model's goodness of fit: the coefficient of determination for a simple regression and the coefficient of multiple determination for a multiple one.
• Specify whether and how the model was validated.
• For a primary comparison analysed with simple linear regression, consider a scatter plot showing the regression line and its confidence bounds, and do not extend the regression line, or the interpretation of the analysis, beyond the minimum and maximum values of the data.
• Name the statistical package used.

³⁴

For an analysis of variance or of covariance, the SAMPL guidelines recommend that a report: ³⁵

• Describe the purpose of the analysis.
• Identify the variables used and summarise each with descriptive statistics.
• Confirm that the assumptions of the analysis were met, for example by indicating whether an analysis of residuals confirmed the assumption of linearity.
• If relevant, report how any outlying data were treated.
• Report how any missing data were treated.
• Specify whether the explanatory variables were tested for interaction and, if so, how those interactions were treated.
• If appropriate, report in a table the P value for each explanatory variable, the test statistics and, where applicable, the degrees of freedom for the analysis.
• Name the statistical package used.

³⁵

Dimension reduction

Principal component analysis is a form of exploratory data analysis that originated with Karl Pearson. ³⁶ By an eigenvalue-based orthogonal transformation, it reduces a set of variables that may be correlated to a new set of linearly uncorrelated variables, the principal components. ³⁶ In one study, it reduced the responses of more than 4,000 people to questions on three indices of sexual function to three principal components of sexual satisfaction, which together accounted for more than 90% of the variance in the scores. ³⁶ Most of the factors were correlated with Spearman ρ values above 0.60, except the responses to one question of the International Index of Erectile Function, on the frequency of attempts at intercourse (ρ below 0.35), and the analysis suggested that this frequency construct represented a different dimension of sexual satisfaction. ³⁶ Analyses related to principal component analysis include cluster analysis, which seeks to group individuals into natural systems, and factor analysis, which may be applied to reduce the complexity of certain psychometric tests to a set of more basic constructs. ³⁶