Critical appraisal of evidence

Critical appraisal assesses whether research is relevant, valid and useful for the question at hand. It weighs strengths alongside weaknesses and examines whether the reported results support the conclusions drawn from them.

Purpose and scope

Critical appraisal systematically assesses relevance, validity and results, balancing a study’s benefits and strengths against its flaws and weaknesses. ¹

For a review, appraisal helps set a minimum study-quality threshold, explain differences between included studies and judge the strength of conclusions. ²

Initial screening asks: ²

• Why was the study done?
• What type of study is it?
• Is its design appropriate to the review topic?

²

Studies included in a review should ideally be critically appraised. ²

Medical writers need to evaluate articles they use as sources or cite; deliberate checks can improve selection and interpretation even without formal research-design training. ^{3, 4, 5} Effective use of clinical literature does not necessarily require a detailed technical critique. ⁶

Appraisal questions depend on the kind of research: ¹

Research	Core questions
Randomised trials, cohort studies and case-control studies	Is the study valid? What are its results? Will they help locally or in caring for the patients concerned?
Economic evaluation	Is it likely to be usable? How were costs and consequences assessed and compared? Will the results help local purchasing or patient care?
Qualitative research	Were key methods rigorous? Are the findings credible? How relevant are they to the intended organisation or practice?

¹

Checklists should be tailored to the study design and clinical problem, with the same important basic questions applied consistently to studies of a given type. ¹

Methodological strength and usefulness

Study quality depends on both methodological strength and usefulness. ^{3, 5} Strength concerns measures protecting internal validity and minimising bias: research design, study conduct, sample size, and reporting and analysis. ⁷ Design usually sets the initial evidence level, which the other factors may raise or lower. ⁷

Internal validity concerns findings within the study population; external validity concerns their applicability outside it. ¹ A well-designed, well-conducted study may still be inapplicable to a particular clinical or policy setting. ⁸

Tightly controlled treatment, monitoring, highly trained staff and selected patients can make randomised-trial results difficult to generalise to routine care, while expense may limit follow-up for survival and long-term safety outcomes. ⁸ Observational studies are more prone to bias and confounding than randomised trials, but the measures protecting a trial’s internal validity can undermine its external validity. ⁹

The way participants are selected also affects representativeness: random treatment assignment does not ensure that the overall sample was selected systematically. ⁸ Consecutive enrolment of eligible patients or random sampling from the eligible population can help achieve sufficient representation. ⁸

Conduct, follow-up and analysis

Knowledge of treatment assignment can bias subjective symptom reporting and outcome evaluation, and patients aware that they are not receiving the experimental treatment may be more likely to withdraw. ¹⁰ Appraisal of nonrandomised comparisons considers whether eligibility criteria and matching make groups comparable, and whether treatment settings and outcome-assessment timing or methods differ. ¹⁰ Comparison protocols should minimise treatment differences unrelated to the intervention: for example, supplemental corticosteroids in a shock-wave treatment study should be prohibited in both arms or allowed in both. ¹⁰

Follow-up must be long enough for the outcome to occur: a brief interval may suffice for short-term adverse reactions, a few months for fracture repair, and several years for a cholesterol-lowering treatment’s effect on cardiovascular events. ¹⁰ Longer follow-up increases the risk of losses, and differences in follow-up rates between groups suggest bias. ¹⁰ Even an uncontrolled case series can mislead if return for follow-up depends on whether patients did well or poorly; a reader can look for efforts to reduce losses or compensate through methods such as telephone interviews. ¹⁰

No sample size is universally adequate; larger samples reduce sampling error, and smaller expected differences require larger samples to reach statistical significance. ¹¹ A reported power calculation is a mark of planning, specifying sample size in advance around an expected effect that may be chosen for its clinical importance. ¹¹

Reporting should show appropriate analysis and enough data to judge the conclusions, including significance and variability of key results. ¹² Effect magnitude and discussion of clinical importance are also marks of good analysis: a statistically significant one-point improvement on a ten-point pain scale may not improve well-being or function. ¹³ A very large study can detect small differences that have no important consequences for care or policy. ¹⁴

A statistics instructor advises checking subgroup-analysis counts, sample size and the appropriateness of the population before celebrating a P value below 0.05, and leaving room for doubt rather than treating one publication as definitive. ^{15, 16}

Missing results also merit scrutiny: a statistician describes looking for an important secondary endpoint that is not mentioned and for reporting that seems inconsistent with the rest of a paper. ^{17, 18}

A statistician recounts tracing irreconcilable calculations to an incorrect formula in a 1963 publication. ¹⁹

Outcomes and clinical usefulness

Patient-centred outcomes can be more useful than intermediate measures: return to tennis is more meaningful than range of motion, and preventing heart attacks or strokes matters more ultimately than reducing blood pressure. ⁸ Quality-adjusted or healthy life-years are especially meaningful from a patient perspective, while cost-effectiveness may be useful to payers and policymakers. ⁸

The intervention’s stage of development affects usefulness: comparisons with another relatively new alternative may be more useful than placebo or standard-treatment comparisons, long-term safety may need study after effectiveness is established, and remaining questions about high-risk groups may matter most for an otherwise well-studied intervention. ⁸

Design-specific appraisal checklists

The following checklists give questions for different study types. ¹

Randomised controlled trials

• Did the trial address the clearly focused question of interest?

• Were participants randomly selected from a defined population?

• Was the assignment of subjects to intervention groups really random?

• Were participants and observers both blinded to their treatment (and if not, why not)?

• Were there any differences between the two groups in terms of selection bias or confounding variables that could explain the differences between them?

• Were the eligibility criteria specified?

• Apart from the experimental intervention, were the groups treated equally?

• Was outcome assessment blinded?

• What were the outcomes, were they valid and meaningful, and was the period over which they were measured valid in terms of the clinical context?

• How large was the treatment effect? What were the absolute and/or relative sizes of any effects seen?

• How precise was the estimate of the treatment effects?

• Were all the subjects who entered the trial properly accounted for at its conclusion?

• Was follow-up adequate (i.e. >80%)?

• Were patients analysed in the groups to which they were randomised?

• Was the power of the study calculated and, if so, did the study have adequate power to see an effect if there was one? (Many studies reporting 'non-significant' results do so because they had too few participants to ever realistically demonstrate a difference.)

• Can the results be applied to the work in question?

²⁰

Cohort studies

• Did the study address a clearly focused question?

• Was the study prospective or retrospective?

• Was the cohort recruited in an acceptable way, and was everyone included who should have been?

• Were the control and exposed groups comparable on all important confounding factors at the start of the study (e.g. age, sex, social class, smoking, health)?

• Was there adequate adjustment for the effects of these confounding variables?

• Were all the participants who entered the study properly accounted for?

• Was follow-up adequate (i.e. >80%)?

• Were participants analysed in the groups to which they were initially allocated?

• What outcomes were measured?

• Was outcome assessment blind to exposure status?

• Was exposure accurately measured to minimise bias?

• How large was the effect of the exposure?

• How precise was the estimate of the exposure effect?

• Was follow-up long enough for the outcomes to occur?

• Were drop-out rates and reasons for drop-out similar across the control and exposed groups?

• Were the study participants sufficiently similar to the population in question to make the results helpful?

²¹

Case-control studies

• Did the study address a clearly focused question?

• Were the cases and controls recruited in an acceptable way? Was there any selection bias? Were they randomly selected from the source population of the cases and controls?

• Was exposure accurately measured to minimise bias, i.e. assessed in the same way for both cases and controls?

• How was the response rate defined?

• What confounding factors were and were not accounted for in the study?

• What were the results?

• How precise was the estimate of risk?

• Was an appropriate statistical analysis used (matched or unmatched)?

• Can the results be applied to the local population, and do they fit with other evidence?

²²

Screening and diagnostic-test validation

For diagnostic, prognostic and screening methods, principles used to appraise treatment studies apply when the study measures clinical impact, such as changes in management or survival. ²³ Most such studies instead estimate sensitivity and specificity against a reference standard or surgical or pathological confirmation. ²³ A typical disease mix, a reasonable source of normal values and blinded interpretation improve validity; studies limited to associations with known disease or subjective image-quality ratings are less informative. ²³

• Was there an independent, blind comparison with a universally accepted standard control test or screening method?

• Did the patient sample include an appropriate spectrum of patients to whom the diagnostic test will be applied in clinical practice?

• Did the study contain enough cases to compare the new test with the control test reliably? Was a power calculation included?

• Were all people diagnosed with both tests, regardless of the results from either?

• Were the methods for performing the test described in sufficient detail to allow their replication?

• Are likelihood ratios for the test results presented, or data necessary for their calculation provided?

• Are the test's sensitivity and specificity adequate?

• Will the reproducibility of the test result and its interpretation be satisfactory in the setting in question?

• Are the results applicable to the patient?

• Will the results change current management?

• Will patients be better off as a result of the test?

²⁴

Qualitative studies

• Is there a clear aim for the project, not just an attempt to gather masses of data and then impose some order or structure on to it? Did the study ask how or why something was taking place?

• Is the choice of qualitative methodology used appropriate to the stated aim, and is it clearly discussed?

• Is there a clear and logical sampling strategy, i.e. does the study report clearly state who was and was not selected, and why, from where and how? Is justification given for the sample size chosen? Is there some discussion of the effect of drop-outs and non-responders? It may be more important to employ purposive sampling (where subjects are selected because of some characteristic) to gather the whole range of possible opinions and experiences, rather than random sampling to produce a representative study group.

• Is it clear how, where and why the data were collected, e.g. via interviews, focus groups, surveys using tapes, note taking, video, etc.? Was the setting appropriate? Were the methods used explicit, justifiable and validated?

• Was the relationship between investigators and participants examined critically throughout the study for any potential bias or influences? Was the investigator's potential influence on the results considered, for example with respect to the questions posed and the data used?

• Is there an in-depth description of how the data were analysed, for example if it was thematic (i.e. analysis that focuses on identifiable themes and patterns of living and/or behaviour). Is it clear how the themes were derived?

• Was the data analysis sufficiently rigorous? Did more than one researcher perform the analysis independently? Is there an explanation for how and why certain data were selected for presentation, and are sufficient data described to support the study's conclusions? Is there any discussion of contradictory data?

• What were the main findings of the research? Are they coherent, and do they address the research question?

• Are the results credible and consistent with the data? Have alternative explanations for the results been explored and discounted?

• Were the conclusions valid and consistent with the data and results?

• To what extent are the findings transferable to current practice and patients?

²⁵

Systematic reviews

• Did it address an important, clearly focused clinical question and is this clearly defined?

• Was a thorough search of appropriate databases carried out, and were potentially important sources explored?

• Was the methodological quality and validity of the trials assessed (by more than one independent assessor), and the trials weighted accordingly?

• Were the criteria used to select articles for inclusion and exclusion explicit and appropriate?

• Were assessments of the studies reproducible?

• Were the results similar from study to study?

• If the results of the review have been combined, was it reasonable to do so?

• What are the overall results of the review, and how precise were the results?

• How sensitive are the results to the way the review has been conducted?

• Is the numerical result clinically significant, i.e. relevant to the care of the patient?

²⁶

Economic evaluations

• Is there a well-defined question?

• Is there a comprehensive description of competing alternatives?

• What is known about the clinical effectiveness of the interventions being compared?

• Are the interventions sensible and workable in the settings where they are likely to be applied?

• Are all the important and relevant costs and health outcomes for each alternative identified?

• Are costs and outcomes measured accurately, valued as credible, and adjusted for different times?

• Is there an incremental, rather than an absolute, analysis of the consequences and costs?

• Has health status now been given precedence over health status in the distant future?

• Has an adequate sensitivity analysis been conducted to consider areas of uncertainty?

• Are the results useful in terms of purchasing for local people?

²⁷

Selection and source limitations

Writers have a responsibility to remain objective, exacting in research and alert to the implications of conclusions drawn from selected data. ²⁸

A bibliometric study of 2002–2013 European cancer papers illustrates a limit of counting publications: it covered published data without appraising paper quality or the funding amount behind each paper. ²⁹

For forwarded messages, a physician-columnist recommends checking both the sender and the originator, including when the content agrees with the reader’s own views. ^{30, 31} The sender’s known interests may help anticipate selection bias, because people often forward views they already favour. ³⁰

His procedure is: ³¹

1. Read the forwarded message.
2. Check who sent it.
3. Trace it as far as possible to the originator and investigate the individual or organisation.
4. In almost all cases, delete it rather than forward it.

³¹

He defines propaganda as material intended to persuade for or against something, including press releases promoting a person, action or institution. ³¹ He argues that persuasive material can contain truths, half-truths and lies, so agreement does not establish its quality and disagreement does not justify dismissing it without assessment. ³¹