Communicating an assessment's uncertainty

Uncertainty in a health technology assessment concerns the clinical evidence, the economic model and applicability to practice. Reporting it makes the limitations of the findings visible, including in the summaries on which decisions often depend.

The reporting problem

Uncertainty is unavoidable in health technology assessment (HTA) and can make decisions difficult. ^{1, 2} Medical research is imperfect: known and unknown factors affect results and how far they can be trusted, and acknowledging uncertainty can be uncomfortable for clinicians. ¹ Understanding research quality, limitations and real-world applicability is what elevates an HTA beyond its systematic review and economic analysis. ²

End users are often short of time and unlikely to read the complete technical documents of a full HTA. ² Assessors therefore need to summarise reports that can run to hundreds of pages accurately and succinctly, often using bullet points, tables and figures and deciding what belongs up front. ³ Condensing findings while highlighting important issues is harder for uncertainty than for the results themselves. ² Uncertainty may or may not influence a decision, but always needs consideration; omitting it risks oversimplifying the assessment. ³

Sources of uncertainty

Clinical evidence

Uncertainty in the assessment of safety, efficacy and effectiveness arises from the type and quality of the available evidence. ² Randomised evidence may be unavailable for the intervention, population or comparator of interest, and a new technology may have too few large or rigorous studies to establish efficacy and safety with certainty. ² A large evidence base can still provide low confidence: a Canadian assessment of implants for hearing loss included 20 systematic reviews, but most primary evidence came from small before-and-after case series. ²

Certainty can differ across outcomes, population subgroups and follow-up periods. ² Safety outcomes are particularly difficult because trials are often underpowered for rare but important adverse events; a World Health Organization review of human papillomavirus vaccine safety generally did not identify rare events despite including many large, well-designed trials. ² Without other safety information, such as large observational studies or real-world evidence from hospital or primary-care databases, gaps can remain for key outcomes. ² HTA evidence bases commonly combine study types and varying quality across studies and outcomes, making interpretation of uncertainty across the whole body of evidence challenging. ²

Economic models

Economic models combine multiple inputs with different levels of uncertainty. ² Estimates of health gain and health-state utilities, and extrapolation over time, can depend on assumptions. ² Scenario and sensitivity analyses typically explore uncertainty, for example by testing the smallest and largest plausible input estimates and their effects on results. ² These technical analyses can be difficult for non-economists to grasp, so the analyst needs to explain where uncertainty lies and how it affects the result. ²

Applicability to practice

Applicability concerns a technology's fit with existing clinical practice, infrastructure and policy. ⁴ Assessing it requires evidence about the population and setting of intended use, including clinical and demographic characteristics, disease spectrum and technology delivery. ⁴ Practical uncertainties include workforce training or accreditation, required equipment and whether existing facilities can accommodate it, access for everyone eligible, and possible uptake by people for whom the technology is not intended. ^{4, 5} Sometimes assessors can offer only their best guess about implementation; contextualising the evidence in this way makes HTA an essential decision-making tool. ⁵

Transparency and visual summaries

Transparent reporting documents all methods and their justification, including the choice of economic-model inputs and reasons for downgrading or upgrading evidence, so that end users can make informed decisions. ³ Comprehensive explanations rarely fit into bullet points, and reporting needs to balance adequate detail with useful, accurate summaries. ³

The evidence-rating tool GRADE uses solid and hollow circles to communicate certainty for individual outcomes, but GRADE does not provide an overall assessment of an intervention across all its possible benefits and harms. ³ HTA needs to communicate both desirable and undesirable outcomes and whether benefits outweigh harms in the proposed clinical setting. ³ In 2021 there was no standard visual method for showing uncertainty across an entire HTA's clinical, applicability and economic components, and further research was needed. ^{1, 3} Suggested approaches included a traffic-light system beside key results or different text colours or fonts for different certainty levels, potentially helpful in interpreting economic analyses. ³

Medical writers should consider carefully how abridged reports convey uncertainty, because assessments are often read only in summary. ³ Acknowledging uncertainty and describing how it was handled can add credibility rather than signify weakness. ³

Reporting checklist

The following reporting considerations, set out in 2021, provide a starting point rather than an exhaustive checklist. ³

1. Clinically important effects. Distinguish clinically important effects from statistically significant effects: statistical uncertainty can be unrelated to the intervention, whereas clinically important effects are more likely to be related to it.
2. Patient-relevant outcomes. Prioritise findings in patient-relevant outcomes when reporting safety and effectiveness, and direct outcomes over surrogate outcomes.
3. Compounding uncertainty in economic models. Consider how uncertainty in clinical and other model inputs affects results and may multiply when several uncertain inputs are combined. Where appropriate, best- and worst-case scenarios can help because they are easily understood.
4. No or insufficient evidence. State where there is no evidence, where evidence is insufficient and why, and where certainty differs across the evidence base, so that decision makers understand the limitations.
5. Recipients. Consider differences between the needs of policy makers or funders, for whom the HTA is designed, and consumers, particularly in language and statistics.
6. Visual representations. Present results visually where appropriate. There is no standard way to represent uncertainty; suggestions include traffic lights or a thermometer-style measure, cold for uncertain and warm for more certain.

⁵