Appraising and analysing the literature
Appraising the literature for a clinical evaluation means judging each paper the search returned for its relevance and its weight, and accounting for every paper left out, so that the counts of included and excluded papers reconcile with the search. Analysing it means extracting the data into tables and writing one analysis across the whole body of papers rather than study by study.
Appraising each article
Each article about the device under evaluation, or about an equivalent device, is appraised: its weighted contribution to the evaluation of clinical safety and performance is assessed in a methodological and documented way. MEDDEV 2.7/1 revision 4 gives no examples of appraisal methods but refers to the widely used Appendix D of the Global Harmonization Task Force's clinical evaluation guideline, which by June 2022 had become Appendix F of the International Medical Device Regulators Forum's updated guideline.
Screening for relevance
The search results should be imported into reference management software, such as EndNote, optionally alongside a spreadsheet, and managed there. Abstracts are screened first, to exclude those that are obviously irrelevant. Full-text articles are then obtained for the remaining abstracts and assessed for whether they directly demonstrate adequate clinical performance and clinical safety of the device, as pivotal data, or serve an indirect supporting role. Each full-text article is appraised according to an appraisal plan. Section 9.3.2.c of MEDDEV 2.7/1 revision 4 lists questions that help determine whether data are relevant, which in summary are:
- How far are the data representative of the device under evaluation?
- Which aspects do they cover?
- Are they relevant to the intended purpose of the device or to claims made about it?
- Where they bear on specific aspects of the intended purpose or claims, are they relevant to particular device models, user groups, medical indications, age groups, genders, severities of condition or time periods?
Weighting the evidence
Once an article is found relevant, its contribution to the evaluation is weighted. There is no single well-established method for weighting clinical data, and a method appropriate to the target device should be chosen, such as the Oxford Centre for Evidence-Based Medicine levels of evidence, which rank a systematic review of randomised trials highest. In practice, clinical evidence from systematic reviews may be available only for conditions with abundant published literature, such as heart valve replacement surgery, and most of the evidence will come from randomised controlled trials (level 2) and non-randomised controlled cohort studies (level 3).
Accounting for included and excluded papers
A reason is given for every excluded paper, whether it was excluded at abstract screening or at full-text appraisal. Reasons might include too little information about the study, as when safety or performance data cannot be extracted, too few patients, as in case reports, improper statistical methods and inadequate controls. The disposition of screened and appraised articles should be recorded, and a spreadsheet or another program, such as DistillerCER, is a convenient way to do so. The included and excluded papers can then be tallied, and together they must equal the number of articles the search identified.
The list of excluded papers, with the reasons for exclusion, is attached as an appendix to the clinical evaluation report. The included papers are presented in a bibliography that should be separate from the state-of-the-art bibliography. The full-text articles, as PDFs, are themselves part of the clinical evaluation and must be provided with the report.
What the analysis must establish
Under MEDDEV 2.7/1 revision 4, the goal of the analysis stage is to establish whether the appraised datasets collectively demonstrate compliance with each Essential Requirement bearing on the device's clinical performance and clinical safety when it is used according to its intended purpose. The analysis in turn contributes to the assessment of clinical safety and performance.
The analysis of the clinical data explains whether and how the information provides sufficient clinical evidence to demonstrate the clinical performance and clinical safety of the device under evaluation. It also describes the device's benefits and risks and explains the acceptability of the benefit-risk profile according to the state of the art. It should also look for consistency between the clinical data, the instructions for use, the risk management documentation and the state of the art, to identify gaps, discrepancies, residual risks and uncertainties or unanswered questions that should be evaluated further during post-market surveillance, including in post-market clinical follow-up studies. Such questions include rare complications, uncertainties regarding medium- and long-term performance, and safety under widespread use.
Data extraction tables
Data from the appraised literature are extracted into tables, summarised and analysed. Data extraction tables are a convenient way of presenting papers: they give an overview of the literature and make comparisons between papers easier, but they lack narrative detail. Because of their size they are usually presented as an appendix to the report, and they may be split into smaller tables to fit the width of an A4 page. The tables can be presented as follows:
- Study details, such as evidence weighting, study design, treatments or interventions, devices used and follow-up period
- Patient population, such as number of patients, demography and baseline disease characteristics
- Performance, such as endpoints as determined by the disease under study
- Safety, such as post-operative complications or adverse events, and deaths
Papers can be presented in groups, for example by study design, or simply listed alphabetically by author or by publication year.
Writing the analysis across the dataset
The data are analysed as a whole across the dataset, and the analysis is objective and critical. Analysing the data as a whole lets studies be compared and the comparisons summarised in the report. A combination of descriptive text and in-text tables presents the data and explains the outcome measures used. A narrative of each study is not required, but presenting important pivotal studies is helpful.