Diagnostic and prediction performance
A diagnostic test and a prediction model are judged by several measures, each answering its own question. Sensitivity, specificity, predictive values and likelihood ratios say what a test result means for the presence of disease. Discrimination and calibration describe how well a model's predicted probabilities separate and match outcomes, and fairness metrics show whether a risk score performs equally across the patients it is applied to.
Sensitivity and specificity
Sensitivity, also called the true-positive rate, is the proportion of people with the condition who test positive: the number of true positives (TP) divided by the sum of true positives and false negatives (FN), TP/(TP+FN). It is the probability that a test result will be positive when a patient has the disease or other predicted condition. For example, a test with a sensitivity of 1.0 for tuberculosis would identify everyone with the disease, and one with a sensitivity of 0.5 would detect half of those infected. A test with high sensitivity has few false negatives.
Specificity, also called the true-negative rate, is the proportion of people without the condition who test negative: the number of true negatives (TN) divided by the sum of true negatives and false positives (FP), TN/(TN+FP). It is the probability that a test result will not be positive when a patient does not have the disease or other predicted condition. For example, a test with a specificity of 1.0 for tuberculosis would identify no one from the healthy group as sick. A test with high specificity has few false positives.
Predictive values
Positive predictive value is the proportion of patients with a positive test who have the disease, TP/(TP+FP), and negative predictive value is the proportion of patients with a negative test who do not have the disease, TN/(TN+FN). Predictive value is strongly affected by the prevalence of a disease, even when the sensitivity and specificity of the test stay constant.
Pretest and posttest probability
Pretest probability is the likelihood of a positive test result, determined before the result is known. If a low vitamin D level is found in 9% of a large population of asymptomatic 50-year-old women, 9% is the pretest probability of finding a low level in the next such woman tested.
Posttest probability is the proportion of patients with a particular test result who have the target disorder. In deciding whether to recommend a test, such as a magnetic resonance scan for a patient with back pain, knowing the posttest probability helps the clinician decide whether the result is likely to make a difference to the patient's treatment.
Likelihood ratios
The likelihood ratio compares the probability of a given test result in a patient with the disease with its probability in a patient without the disease, and in diagnostics it is a salient measure. It incorporates both the sensitivity and the specificity of the test, and it helps in assessing how likely a diagnostic test is to be useful by giving a direct estimate of how much a test result will change the odds of finding a disease. For a positive result it is sensitivity/(1−specificity), and for a negative result it is (1−sensitivity)/specificity. A likelihood ratio below 1 indicates a lower likelihood of disease, and one above 1 a higher likelihood. Tests with likelihood ratios below 0.2 or above 5.0 tend to be the most useful clinically.
Discrimination and calibration
The area under a receiver operating characteristic (ROC) curve, also called the c-statistic, measures the discriminative performance of a model, such as its ability to distinguish the presence from the absence of disease. The ROC curve plots sensitivity on the ordinate against 1−specificity on the abscissa. A c-statistic above 0.70 is generally considered acceptable, although such cut-points are somewhat arbitrary and continue to be debated.
A prognostic model is adequately calibrated when its projected probabilities, the expected frequencies of events, match the observed frequencies. The Hosmer–Lemeshow statistic determines the goodness of fit of prognostic models, such as risk prediction models, and especially of logistic regression models. It is computed from the expected events, the observed events, the number of observations and the predicted risk in each of G risk groups, such as deciles of predicted risk, and in general it asymptotically follows a χ² distribution with G−2 degrees of freedom.
Reporting a model's performance
The methods of a report on a predictive model should state its sensitivity, specificity and negative and positive predictive values, the area under its ROC curve, and how it was validated and tested for goodness of fit, for example by the Hosmer–Lemeshow test for a logistic regression model. For diagnostics and predictive models, a quality-control checklist for the statistical methods of a manuscript lists Bland–Altman plots, negative and positive predictive values, sensitivity and specificity, and the Hosmer–Lemeshow test of the goodness of fit of regression models, and it holds that the area under the ROC curve should exceed 0.70.
Fairness across subgroups
One developer of clinical risk assessment methods holds that fairness-related metrics can aid in designing the next generation of risk assessment tools, beyond assessing them only by traditional metrics such as prediction performance and calibration.
A study in which the developer's own research organisation took part assessed two widely used cardiology risk scores, CHARGE-AF for atrial fibrillation and the Pooled Cohort Equations for atherosclerotic cardiovascular disease, in three large independent datasets: the Explorys Life Sciences Dataset, Mass General Brigham and the UK Biobank. It used standard metrics such as discrimination, calibration and standard hazard ratios, and fairness-related metrics considering sex, race and age ranges. It found evidence of potentially unfair performance, with significant differences in fairness metrics for sex and race in both scores. The sensitivity of both scores was much lower for women than for men in the intermediate age groups, suggesting that the scores may miss more women at high risk of events and potentially worsen existing sex-related treatment gaps. In the developer's view, the findings underscore the importance of evaluating prognostic models across specific subpopulations to better understand the accuracy and potential unfairness of the prognostic information that drives clinical decisions at the point of care. The developer also holds that the study highlights the importance of assessing prognostic models with metrics specific to fairness and calls for continued evaluation of widely used risk scores to understand their impact on patient outcomes across subpopulations.
The MELD 3.0 score
The Model for End-Stage Liver Disease (MELD) score prioritises patients on the liver transplant waiting list, a higher score indicating greater illness severity and so greater urgency for transplant. By September 2023 the Organ Procurement and Transplantation Network (OPTN) Board had decided to replace MELD-Na (MELD with serum sodium) with the MELD 3.0 score for determining organ allocation priorities in the United States. Among the principles that guided its development, the creators of MELD 3.0 required every feature in it to be measurable in an objective fashion, generalisable, devoid of unnecessary volatility without biological significance, and reportable to the OPTN without undue burden. The developer of risk assessment methods expects the decision to have a significant impact on organ allocation in the United States, describing MELD 3.0 as a more refined and accurate way of determining allocation priorities that is expected to result in better outcomes for patients in need of liver transplants.
MELD 3.0 incorporates sex as a variable for two reasons: to mitigate a sex disparity in access to transplantation and to improve prediction performance. Including sex corrects for a sex disparity caused by creatinine and by differences in risk of death, among other factors. The primary objective of adding sex and of revising the creatinine coefficient was to improve fairness across the sexes. However, the developer notes that the publications on the score did not thoroughly discuss how to assess fairness quantitatively.
In the developer's view, it is crucial to assess the performance of MELD 3.0, as a score that accounts for fairness, with metrics specific to fairness, and if its performance on those metrics is found unsatisfactory, a revised version must be developed urgently. Standard metrics such as discrimination and calibration have been used to assess the performance of MELD 3.0, but the developer holds that, for risk assessment tools used in organ allocation, it is also important to use measures such as statistical parity difference, true positive rate difference and true negative rate difference to assess fairness within the context of patient characteristics such as sex, race and age. The developer also holds that incorporating sex into MELD 3.0 is a step towards improving access and fairness in organ allocation. In the same view, future versions of the MELD score must exhibit small to non-existent bias across all age ranges and characteristics such as sex and race.
Selecting predictors
The developer of risk assessment methods holds that developing an effective clinical risk assessment tool requires selecting patient characteristics that are potentially predictive, such as a subset of laboratory values, comorbidities, medications and genetic profiles, and that the selection should incorporate both practical experience and knowledge from the scientific literature. Unbiased, machine-learning-based feature selection techniques can help recommend which characteristics to include. Subpopulation-based feature selection is an iterative machine-learning-based technique that identifies the most important features for risk assessment in specific subgroups of patients, and the developer, one of its co-developers, reports that it proved superior to notable widely used feature-selection methods. The developer expects incorporating novel covariates that improve both performance and fairness to give clinicians more accurate and unbiased risk assessments. New versions of MELD are expected to rank patients on the liver allocation list more fairly once they incorporate novel features that are also adjusted to optimise fairness-related metrics.