Systematic reviews and meta-analyses
A systematic review and a meta-analysis are planned, searched, appraised, combined and reported so that a reader can reproduce the selection of studies and judge what the pooled answer is worth.
Strengths and limits of meta-analysis
The chief advantage of a systematic review or meta-analysis is that pooling data from multiple trials increases statistical power, so hypotheses can be tested more readily; its two potential pitfalls are heterogeneity and publication bias. Meta-analysis is a statistical technique for combining the results of more than one study, and it is useful because a single study very rarely gives a definitive answer in medicine: whether an intervention works, or whether an environmental factor is associated with increased risk, frequently has to be judged from many studies. Combining studies reduces the risk of being misled by one study whose unusual result comes from a statistical fluke or bad design, and it gives more precise estimates of the size of an effect: several studies that were each underpowered and each failed to find a significant effect can, combined, confirm a statistically significant effect.
A meta-analysis also reduces a large quantity of information to a manageable size, and it is considerably cheaper and faster than a randomised trial, and it is particularly useful where a trial would be difficult or impossible, such as where it would be unethical or where the condition is so uncommon that recruiting enough participants would be difficult.
Unlike a randomised trial, a meta-analysis cannot give direct evidence of cause: it is an observational study, and its result depends heavily on which studies were included and how they were combined. It is therefore most appropriately used to support other studies' findings, to generate hypotheses and to help design future trials.
The question and the protocol
A systematic review commonly runs through these stages: formulating the review question and the methodology, defining a search strategy, describing study characteristics, assessing study quality and relevance by applying appraisal criteria, synthesising the findings to answer the original question, and communicating the findings to stakeholders. Cochrane, a not-for-profit global network that summarises research results to make scientific evidence readily available, is recognised as a leading organisation promoting evidence-based health decision making through researching and producing independent systematic reviews based on primary research, and its Cochrane Reviews are recognised internationally as the gold standard for high-quality, trusted information.
Before the search, the Cochrane Database of Systematic Reviews and other published reviews should be scanned to confirm that a similar review has not already been done; if one has, the objectives should build on, or further evaluate, its findings and conclusions. Considering every paper on a disease or a therapy is a fool's errand, so the load should be shared among a few committed researchers, and the team focused on a discrete scope translated into a manageable number of objectives.
For a systematic review to be meaningful, the research question, and how to answer it, has to be clear from the beginning. A conceptual framework lets several key components be considered at the outset; in reviews of medical treatments it is often a PICOT framework — population, intervention, comparison, outcomes and time — and enough time should be allowed to get the framework right, because only then are the inclusion criteria and a relevant search strategy properly developed. The equivalent of a protocol, stating the approach and the methods to be used, should be prepared before the review begins.
A protocol is a document that presents an explicit plan for a systematic review, detailing its rationale and its a priori methodological and analytical approach. The items PRISMA-P 2015 recommends for a protocol cover the rationale, the questions, the eligibility criteria, the information sources and search strategy, the selection and data-collection processes, the outcomes and their prioritisation, the assessment of risk of bias in individual studies, the synthesis, and any planned assessment of meta-biases, such as publication bias across studies or selective reporting within them. By those items, the protocol should identify itself as the protocol of a systematic review, and as an update of a previous review where it is one; name its authors and guarantor, its funding and the sponsor's role, its registration where applicable, and how amendments will be documented; and state what will be done if pooling is inappropriate and how the strength of the body of evidence will be assessed.
The PROSPERO register can be used both to register a review and to find out whether a similar one has been or is being performed. Like any other research, a meta-analysis needs a prespecified protocol to avoid selective analysis and reporting driven by interim results, and registering the protocol, for example in PROSPERO, is recommended to encourage authors to follow it. Registration may still be insufficient: in a study of 288 Cochrane reviews, all with registered protocols, the primary outcome differed from the protocol in 48 (17%), and results were more likely to be positive in the reviews whose outcome had changed.
Searching and screening
A review generates, manages and accounts for very large amounts of information, and both its information management and its search strategy need processes in place that allow transparency, accountability and repeatability.
Reporting checklists ask the report of the search to name the databases and other information sources, with their dates of coverage and the date of the last search; to present the full electronic search strategy for at least one database, with any limits used, such that it could be repeated; to state any restrictions, such as the years considered, publication status and language, with a rationale; and to describe approaches other than electronic searching, such as registers, contact with study authors, hand-searching and citation snowballing. The STARLITE checklist asks for the sampling strategy, whether comprehensive, attempting to identify all relevant studies, selective, identifying all relevant studies within specified limits, or purposive, sampling from specific disciplines, years or journals, and it counts a type of study given only as an umbrella category such as “qualitative studies”, without defining it, as partially reported. Functional limits, such as restricting to human studies or to English, are applied for logistic reasons and do not alter the topic conceptually, whereas inclusions and exclusions such as geographical location, setting or a specific focus of study are conceptual limits that mediate the scope of the topic. The range of years is reported in full, with its start and end dates and a justification for the period, and the electronic sources with the databases used and, optimally, the search platforms and vendors, to help replication.
Search results can be exported from reference-management software such as EndNote or Reference Manager, using its tab-delimited function, into a Microsoft Excel file shared with colleagues for review; it is often wise to have three independent reviewers judge each article's relevance and suitability for inclusion, one of them to break ties. A PRISMA flow diagram traces the search through its stages, from the citations identified, through the duplicates removed, the abstracts screened and excluded, and the full articles retrieved and excluded, to the articles included.
Inclusion criteria and search limits
Setting the inclusion criteria is an important process in a systematic review, and there are no hard and fast rules for them. The choices include whether to require a minimum sample size and what size, whether to include only placebo-controlled trials or also active comparators, whether to admit observational research alongside randomised trials, whether to set a minimum study duration, and whether to take all patients with cancer, all with advanced cancer or only those with confirmed metastatic disease. The possibilities are endless and there are no right answers, because the best choice depends on the circumstances.
A reviewer who knows the literature of an area well, as many systematic reviewers do, knows the important studies, and so knows, when setting the criteria, which of them a particular choice will exclude. A reviewer with an agenda can therefore cherry-pick subtly by choosing criteria that exclude the studies they dislike, so a review conducted thoroughly and scrupulously according to its criteria still does not guarantee that all relevant trials were included. The inclusion criteria of a review are always worth reading carefully, to judge how reasonable they are.
One manual on writing medical publications holds that limits must be imposed on the search, such as English-language articles in peer-reviewed journals from the previous 5 to 10 years, related to a limited number of key search terms and publication types. A journal article on the merits and limitations of meta-analyses holds that bias can be introduced in various ways when studies are selected, and that in general the more limited the selection, the likelier the bias, because potentially relevant articles may not be retrieved. By that article's account, restricting to English-language studies is convenient but may overlook important studies; it is common, and its effect probably varies by topic: one study of 18 meta-analyses found effect sizes only 2% higher when only English-language studies were analysed. Using a single database or search portal, by contrast, can have a major effect on the results, largely because databases query different journals: roughly a third of the journals indexed in MEDLINE are not indexed in EMBASE, and the reverse.
Appraising the included studies
Bias within the included studies limits a meta-analysis however well it is otherwise conducted: garbage in, garbage out. Meta-analyses generally assess within-study risk of bias with tools such as the Jadad score or the Cochrane Collaboration's domain-based tool, and excluding poor-quality studies, or analysing high- and low-quality studies separately, can improve the validity of the result. Unless censoring is employed, the character of a systematic review or meta-analysis and the strength of its conclusions will be consistent with the quality of the studies analysed. Systematic reviews and meta-analyses aim to reduce bias by methodological means, but inherent bias exists, and ideally the quality biases of the individual studies are assessed and reported.
An analysis carried out in 2013 of 118 systematic reviews published in four general medical journals and the Cochrane Library checked whether each applied procedures to counterbalance six forms of malpractice in its included studies: publication bias, by searching for unpublished trials; selective outcome reporting, by contacting the original studies' authors; duplicate publications; sponsors' conflicts of interest; authors' conflicts of interest; and the ethical approval of the included studies. Fifty-nine reviews (50%) applied three or more procedures, and eleven (9%) applied none. Seven suspected misconduct — data falsification in three, data manipulation in one, a discrepancy between a published trial and re-analysed data on a regulator's website in one, and selective outcome reporting in two — and five of those seven did not report it. Most authors recognised the risks of double counting participants across duplicate publications and of selective outcome reporting, while conflict of interest was in general underestimated.
A 2011 study found that meta-analyses rarely report who funded the trials they include. Its review of 29 meta-analyses from high-impact journals and the Cochrane Database, which together synthesised data from 509 randomised controlled trials, found that 318 of the trial reports disclosed a funding source, 219 disclosed funding wholly or partly from the pharmaceutical industry and 126 had at least one industry-employee author, yet only two of the 29 meta-analyses reported the funding sources of the trials they included. Of the six that assessed risk of bias, none considered funding as a possible source of it. The review's authors suggest that reports of meta-analyses disclose the funding sources and any author-industry financial ties of the trials analysed, and that this information be included when evaluating potential sources of bias.
Whether the studies can be combined
Heterogeneity occurs when study designs, populations and other factors are so disparate across the included studies that they are not necessarily all measuring the same treatment effect, and also when data from patient subgroups differ meaningfully from the findings in the overall population. Individual studies may measure different parameters, in different ways, in dissimilar patients, with disparities in populations, interventions, endpoint definitions and the direction or magnitude of effects; these considerations militate against pooling, and excessive methodological or statistical heterogeneity can undermine an attempt to pool.
When it makes sense to combine data is one of the most important decisions a meta-analyst makes, because combining a wide range of studies gives an apparently more precise estimate whose precision may be illusory. Where the efficacy of a treatment is investigated in different study populations, for example, an overall estimate can conceal that the treatment works very well in some patients and is harmful in others. A reader of a meta-analysis should therefore look at the detail of the individual studies and ask whether they are investigating the same thing, because if they are not, the overall estimate may be meaningless.
Deciding which studies are similar enough to pool is somewhat subjective and frequently difficult, and it needs a solid grasp of both the topic and research methods. Differences in populations or methods are called clinical or methodological heterogeneity respectively, as distinct from the statistical heterogeneity of results, and some clinical heterogeneity is desirable, because it makes the results more generalisable than those of individual studies, which often use strict criteria to evaluate homogeneous populations; a large degree of heterogeneity rules out combining all the studies in a single analysis but can support subgroup analyses.
About ten years before 2020, a systematic review to inform guidelines for treating elderly people with insulin identified some 3,000 published articles on insulin in the elderly, and screening left 13 eligible. The data from those 13 proved impossible to combine, because each had used different outcome measures, run for a different length of time, collected its data in a different way, and coded the drugs and the diseases differently. The team checked five times before accepting that no meta-analysis was possible, and decided to publish.
Combining results is a staged process: the preliminary stages of selecting the studies, extracting the data and describing the studies' key features in a well-defined and transparent way come before the combination itself, which uses one of two modes, configuration or aggregation. Which mode is appropriate depends broadly on how much the studies differ from each other, that is, on whether they are heterogeneous or homogeneous, and that in turn depends heavily on the type of question asked. Every review shares the common stages of synthesis, but their order depends on the type of synthesis pursued, and the methods of synthesis range from non-statistical to statistical. A medical writer will probably work with a statistician on this part of a review but needs to understand the concepts to write about them later; the methods are not simple, and some prior knowledge of them pays dividends.
Measuring and handling heterogeneity
Not all studies are expected to give exactly the same result, because of normal random variation; the question heterogeneity asks is whether they vary more than chance would explain. A simple statistical test can give a P value, a significant value showing significant heterogeneity, but its results are not straightforward to interpret, because the risk of both false positive and false negative conclusions is high. Cochran's Q is calculated by summing the squared deviation of each study's estimate from the overall estimate, weighting each study's contribution as in the overall analysis, by the inverse of its variance, and is compared with the χ² distribution with k − 1 degrees of freedom, for k studies, to give a P value.
The I² statistic proposed by Higgins and colleagues, of more recent vintage and increasingly used, expresses the share of the total variation across the studies that can be ascribed to heterogeneity rather than to chance. It runs from 0, where the studies are all identical and no heterogeneity is observed, up to a maximum of 100%, with higher values showing increasing heterogeneity, and it can be compared directly between meta-analyses with different types of outcome and different numbers of patients. By convention an I² of 75% is considered high heterogeneity, 50% moderate and 25% low, and values exceeding 75% are typically taken to show an unacceptable degree of heterogeneity that does not support pooling. Other tests for heterogeneity include the τ² test of between-study variance put forward by DerSimonian and Laird, Galbraith plots, meta-regression analysis and random-effects modelling. By gathering the results of many studies in one place, a meta-analysis lets the overall consistency of the results be observed, and that consistency is generally quantified by tests such as I², Tau or Cochran's Q, each of which generates a single value.
In general, as consistency falls, so does the validity of the pooled result. Where substantial statistical heterogeneity appears, the individual studies should be examined for patterns that may explain it, and where a specific group appears to show different results, subgroup analysis may be performed to confirm it. Where substantial heterogeneity is observed, it is reasonable to question the relevance of an overall estimate.
Fixed-effect and random-effects estimates are alternative ways of combining studies that rest on different assumptions. The fixed-effects method assumes there is no important heterogeneity and that all the studies are essentially measuring the same thing, so that any differences between their treatment-effect estimates are purely random variability; where heterogeneity is observed to be low, it gives a good summary. The random-effects method assumes heterogeneity is present, and that differences among studies are due partly to random variability and partly to differences in the true treatment effect each study is measuring; in practical terms, its main difference is that it gives more weight to small studies whose results differ from the average.
A random-effects result is difficult to interpret. It estimates the average effect, but that effect may depend on specific characteristics of the studies, and there is no guarantee that a real-life situation matches an average one; a situation may match some of the studies far better than others. If high-dose studies show greater treatment effects than low-dose ones, for example, the relevant estimate is not the average but the effect at the dose of interest. Inclusion criteria, treatment duration, concomitant medications and healthcare setting can affect treatment effects in subtler ways.
Where the studies are, or are likely to be, heterogeneous, there are four strategies: allow for the heterogeneity with a random-effects meta-analysis, which models between-study heterogeneity and incorporates it; decide not to pool the findings and write a more descriptive narrative review instead; use a fixed-effects model, which essentially ignores the heterogeneity but might be biased and give pooled estimates that are difficult to interpret; or explore and try to account for the causes of the heterogeneity, for example with a sensitivity analysis that includes or excludes the most divergent studies.
One way to handle heterogeneity is to identify its major cause and present separate estimates for the groups it defines. A systematic review and meta-analysis by Annane and colleagues of the effects of corticosteroids on overall mortality at 28 days in patients with severe sepsis and septic shock found no significant effect on mortality overall — relative risk 0.92, 95% confidence interval 0.75 to 1.14, P = 0.46 — alongside significant heterogeneity, with I² = 58% and P = 0.003. Dividing the studies into those using long courses of low-dose corticosteroids and those using short courses of high doses found a significant reduction in mortality in the long-course low-dose studies, relative risk 0.80 with an interval of 0.67 to 0.95 and P = 0.01, and no significant reduction in the short-course high-dose ones, so ignoring the heterogeneity would have meant missing the difference between the dosing regimens. The use of corticosteroids in sepsis is nonetheless complex and controversial, and the analysis is unlikely to be the last word: there is probably considerably more heterogeneity to unpick, including genetic features of the patient and the nature of the infecting organism. Although a meta-analysis can give more reliable results than a single study, even a meta-analysis is often not sufficient to settle a medical question once and for all.
The forest plot
The results of a meta-analysis are very commonly presented in a forest plot. In a typical example, a meta-analysis of the effects on coronary heart disease of increasing polyunsaturated fat in place of saturated fat, the plot shows for each study its name, the number of patients, the number of events, and how extensive the dietary change was, as the percentage of polyunsaturated fatty acid consumption in the control and intervention groups. The central blob of each line shows that study's estimated relative risk and the horizontal line its 95% confidence interval; the size of the blob shows how much weight the study carries, mainly a function of its number of patients, so the bigger the blob, the more the study contributes to the overall analysis. The same information appears in text form to the right of the graph. At the bottom is the overall estimate, again a relative risk with its confidence interval shown both ways; it is the important number to take away, though it may be hard to interpret where heterogeneity is significant, and the spread of the individual estimates is itself a way of assessing heterogeneity by eye. A meta-analysis typically gives a table of each study's effect estimate and confidence interval, generally combined with a forest plot. A 95% confidence interval for an odds ratio or a hazard ratio that overlaps 1.0 indicates that the estimate is not statistically significant.
A large trial can exert an excessive effect on the pooled result, and because a forest plot shows each study's size beside its result, inspecting it helps to detect this; a sensitivity analysis that re-runs the pooling without the large trial confirms whether it has undue influence.
Publication bias and small-study effects
A meta-analysis can never give a true summary of all the research done if some studies are excluded from it. Not all studies are published, and studies reporting negative results are less likely to be published than positive ones, so a meta-analysis that includes only positive trials and ignores negative ones gives an over-optimistic estimate of the true treatment effect. Publication bias may result from many peer-reviewed journals being more likely to publish trials with larger sample sizes, so that smaller trials must report greater effect sizes to be published, and from many being somewhat prejudiced against reports of studies with negative, null results.
Restricting a search to peer-reviewed journals introduces the possibility of publication bias, because studies with positive results tend to be preferentially published; that possibility is often estimated with a funnel plot, and its impact can be reduced, though not eliminated, by including unpublished studies, conference abstracts and other grey literature; the meta-analysis's own funding source may also introduce bias.
A funnel plot is one way to tell whether publication bias has occurred. The treatment effect of each study is plotted on the x axis against the size of the study on the y axis; if all studies are published, the result looks roughly like an inverted funnel, with a greater spread towards the bottom, where small samples make considerable variation in results likely, and a smaller spread towards the top, where large samples keep results close to the true one. Where there is publication bias, small negative studies are likely to go unpublished while small positive ones are published, and large studies are more likely to be published whatever they show, since a team that has gone to the trouble of a large study is more motivated to write it up; this produces asymmetry in the plot.
To construct a funnel plot, the effect size, such as the odds ratio or percent mean difference, is plotted on the x axis against a measure of imprecision, the standard error or the sample size, on the y axis. Some authorities prefer the standard error, partly because the funnel is then symmetrical when bias is absent and partly because smaller studies, which are more prone to bias, may be better represented; plotting the inverse of the variance instead lets differences in conclusions between small and larger trials be evaluated. A symmetrical inverted funnel makes publication bias unlikely, and an asymmetrical one suggests it. Cochrane's Review Manager (RevMan 5) software takes the data for each study of a meta-analysis and generates odds ratios with 95% confidence intervals, a forest plot, a funnel plot and I² values.
Publication bias is only one possible explanation of an asymmetrical funnel plot. Other potential contributors are artifacts of the choice of treatment-effect measure, citation bias, data irregularities including poor study design, insufficient analyses and fraud, differing intensities of intervention across trials, differences in the underlying risk of the sample populations, location bias, multiple-publication bias, and chance. Small-study effects suggest but do not prove that publication bias is present, and a contour-enhanced funnel plot may help to identify disproportionate influences of certain trials on the conclusions about it. With fewer than 10 studies, methods other than the funnel plot should be considered, because it then typically lacks the power to tell true asymmetry from chance, and caution, or another test, is needed when the studies have similar standard errors and similar sample sizes. Methods to test or adjust for publication bias include Begg and Egger's test, trim and fill, selection models, meta-regression and the funnel plot.
A funnel plot built from the data of a Cochrane review of the effect of pharmaceutical industry sponsorship on publications, whose reviewers claimed that industry-sponsored trials were more likely than independent studies to favour the sponsor's product, shows strong evidence of publication bias, which puts in question how far that meta-analysis's result can be trusted. Meta-analysis is a useful technique that can provide important insights when summarising the literature, but it is not a magic bullet and must be interpreted with the same caution as any other result. A meta-analysis based on poor-quality studies gives a questionable result, and beyond that, it matters whether the studies are similar enough for the analysis to make sense and, crucially, whether all the relevant studies were included.
Language bias may interact with publication bias: in a study of 62 paired English- and German-language randomised trials, researchers were nearly four times more likely to publish statistically significant findings in an English-language journal than in a German-language one, so the conclusions of meta-analyses involving only English-language trial reports may on average be biased away from the null.
A meta-epidemiologic study compared treatment effects between published and unpublished randomised trials, and between trials published in English and in other languages. It analysed 5,659 trials included in 698 meta-analyses, drawn from Cochrane reviews published between March 2011 and January 2017 and the trial references those reviews cited, among them 356 unpublished trials and a further 393 published in a language other than English. Treatment effects were larger in published than in unpublished trials, with a combined ratio of odds ratios across 174 meta-analyses of 0.90 (95% CI 0.82 to 0.98), and larger in trials published in a language other than English than in those published in English, with a combined ratio across 147 meta-analyses of 0.86 (95% CI 0.78 to 0.95). The first result confirms that restricting a search to published trials may overestimate treatment effects, possibly affecting a meta-analysis and its conclusions. The second is evidence of language bias, and the study questions the recommendation to consider all languages in a systematic review.
Trials suggesting a beneficial effect tend to be published more easily than similar trials showing negative results, so a review restricted to published studies can identify a false beneficial effect or miss an important adverse effect. Systematic reviews have traditionally been built on reports of randomised controlled trials, but clinical study reports were by 2018 becoming more widely available for use in them.
Although some clinical study reports were already available through the European Medicines Agency's website, most Cochrane reviews were not using them, and Cochrane was developing guidelines on the criteria for including them in a review, with methods and tools for extracting and analysing their content to follow. Whether the regulatory medical writers whose reports a review uses should be acknowledged was raised as a question.
Reporting the review
Considerations in reporting a meta-analysis include whether enough evidence exists to give meaningful answers to the research questions, and what amount of heterogeneity is unacceptable for reporting. Systematic reviews and meta-analyses may be regarded as the panacea of reporting data from multiple studies, but the author must still be cautious about whether enough quality evidence exists to make meaningful comparisons.
PRISMA asks the methods to state the process for selecting studies; the method of extracting data, such as piloted forms, independent extraction or extraction in duplicate, and any processes for obtaining and confirming data from investigators; the variables for which data were sought, with any assumptions and simplifications; and whether risk of bias in individual studies was assessed at the study or the outcome level, and how that information is used in the synthesis. It asks the results to give the numbers of studies screened, assessed for eligibility and included, with reasons for exclusion at each stage, ideally with a flow diagram; each study's characteristics and citation; for every outcome, each study's summary data for each intervention group and its effect estimate with confidence interval, ideally with a forest plot; the results of each meta-analysis done, with confidence intervals and measures of consistency; and the results of any additional analyses, such as sensitivity or subgroup analyses, which the methods identify as prespecified or not.
PRISMA asks the discussion to cover limitations at the study and outcome level and at the review level, and to interpret the results in the context of other evidence, with implications for future research. For a meta-analysis of observational studies, MOOSE asks the results to include a graphic summarising the individual study estimates and the overall estimate, a table giving descriptive information for each included study, the results of sensitivity testing such as subgroup analysis, and an indication of the statistical uncertainty of the findings; the discussion to give a quantitative assessment of bias such as publication bias, a justification for exclusions such as that of non-English-language citations, and an assessment of the quality of the included studies; and the conclusions to consider alternative explanations for the observed results, generalise only as far as the data and the domain of the review allow, and give guidelines for future research.
Reporting against PRISMA
As systematic reviews and meta-analyses began to appear, reports frequently omitted whether a protocol was used, how studies were identified and selected, how their quality was assessed, and who funded the work. The Quality of Reporting of Meta-analyses (QUOROM) statement of 1999 gave an 18-item checklist and a flow diagram for meta-analyses of clinical trials; ten years later the PRISMA statement replaced it, extending the scope to systematic reviews and expanding the checklist to 27 items to reflect progress in review science. The items are grouped under the sections of a typical research paper — title, abstract, introduction, methods, results and discussion — plus a section on funding, and the statement includes an updated flow diagram of study selection. The checklist and flow diagram let a reader see how far each limitation of a meta-analysis is present, and authors who expect to report against them are likely to consider those limitations when designing the meta-analysis. The PRISMA statement, created in 2009, spurred journals to use it as the criterion standard for reporting.
One study of how meta-analyses report against PRISMA used a data extraction form that rated each checklist item as adequate, inadequate, not reported or not applicable, and that defined adequate reporting item by item. An adequate title identifies the report as a systematic review, a meta-analysis or both; an adequate structured abstract carries all the applicable elements, with the review's registration number required; adequate objectives state all the PICOS elements, with the S referring to the design of the included studies. In the methods, an adequate protocol item says whether a protocol exists, where to access it and the registration details; eligibility needs both study-related and report-related criteria with a rationale for at least one; information sources need every source and the date of the last search; the search item needs at least one full electronic strategy with its limits, such that it could be repeated; selection and data collection need to say who did them and how — blinded or in duplicate, with forms, with raw data sought from authors — and, where done in duplicate, how disagreements were resolved; and risk of bias, summary measures, synthesis methods, risk of bias across studies and additional analyses each need their method stated, including the method for assessing heterogeneity and the choice of fixed or random effects, and for additional analyses whether they were prespecified.
In the results, adequate reporting of study selection needs the number of studies at every stage with reasons for exclusion at each stage, though a flow diagram is not required for an adequate rating; study characteristics need the variables collected for each study and its citation; the results of individual studies need summary data for each intervention group as actual numbers and effect estimates with confidence intervals, in a table or forest plot; synthesis needs the main results and every meta-analysis with its interval and measure of consistency; risk of bias within studies needs results for each study; and risk of bias across studies and additional analyses need results for every assessment planned in the methods. In the discussion, adequate reporting needs the main findings with the strength of evidence and their relevance, limitations at both the study or outcome level and the review level, and an interpretation in the context of other evidence with implications for research; funding needs the sources, whether the funder had any further role, and what that role was.
Against those criteria, none of 44 meta-analyses published online between 3 and 18 November 2014 reported every applicable item adequately; they were selected from the first 60 of 881 PubMed records retrieved with the search term meta-analysis, after excluding 13 at title and abstract screening and 3 at full text, and appeared in 38 journals. The median number of adequately reported items was 16 (range 4 to 25), a median of 57.7% of applicable items (range 16% to 92.6%); a further median 29.6% were rated inadequate and 14.8% not reported, and adequate and inadequate ratings combined reached 87.3%. Earlier studies had reported mean percentages of adequately reported items of 64% to 81%, and the lower percentage in this study may reflect its stricter criteria. Forty-one of the 44 (93.2%) included a PRISMA-like flow diagram. An earlier study found the same: none of its 312 systematic reviews and meta-analyses reported all items adequately.
The title, the rationale, the synthesis methods, study selection and the synthesis and additional-analysis results were most often adequate, in 90.9% to 100% of the meta-analyses, likely reflecting variously the ease of reporting them, standard practice for any scientific paper, and generally appropriate reporting of meta-analytic statistics. Three items were usually inadequate, in 81.8% to 84.1%: the structured summary, which was usually present but missing elements, to which journal word limits and content requirements may have contributed; the objectives, which gave some but not all PICOS elements, most often omitting the study design; and the information sources, which usually omitted the date of the last search. The protocol and registration item was not reported at all in 88.6%, which neither said whether a protocol existed nor gave a registration number. The methods for assessing risk of bias in individual studies were adequate in only 18.2% and inadequate in 63.6%, the full search strategy was not reported in 47.7%, the results of individual studies were adequate in 40.9% and inadequate in 47.7%, and funding was adequate in 36.4% and not reported in 31.8%.
The rating was somewhat subjective for several items, used unvalidated criteria devised for an earlier study, and was done by one person rather than by consensus; the sample came from one database over sixteen days, so it may not represent recent meta-analyses generally. The study concluded that the merits and limitations of meta-analyses have to be recognised and assessed, that the PRISMA statement facilitates this by promoting accurate, clear and transparent reporting, and that further effort is needed to promote more widespread use of the guidelines and to encourage adequate reporting of every checklist item.
Communicating the findings
Generating a review is not enough: its results also have to be communicated to a wider population. Methods of turning knowledge into action include linear push–pull models, relationship models and system models. A participative approach extends review methodology to where evidence-informed decision making meets stakeholder involvement, stakeholders being people with some self-interest in a piece of work, because they might use the findings or because decisions others make in light of the findings may affect them. Findings communicated to the media should include fact boxes and avoid jargon by using plain English.
In 2011 the US Institute of Medicine's report Finding What Works in Health Care, requested by Congress, set out twenty-one standards for systematic reviews of the comparative effectiveness of medical or surgical interventions. As well as how to select studies, synthesise the findings and assess the quality of the evidence, the standards specify the elements of the final manuscript, and they stipulate that it should be published in a manner that ensures free public access.
The volume of published reviews
Because they condense and synthesise data from multiple studies into a single document, systematic reviews and meta-analyses are important tools in medical decision-making, and the ever-increasing number of clinical treatments makes them very desirable to understand and use; but while the quantity of published meta-analyses had risen in the years before 2014, whether high-quality techniques were being used was in question.
In 2015 a surgeon warned of a trend in research away from conducting randomised clinical trials and towards systematic reviews and meta-analyses: if it is not inverted, there will be a growing number of synthetic studies using someone else's original data, and fewer raw data on which to base knowledge.
In 2016 the director of the Meta-Research Innovation Center at Stanford argued that the production of systematic reviews and meta-analyses had reached epidemic proportions, and that a massive production of unnecessary, misleading and conflicted reviews served mostly as easily produced publishable units or marketing tools rather than promoting evidence-based medicine and health care. In 2014, 9,135 meta-analyses were published and 28,959 systematic reviews were indexed in PubMed, more than the number of articles on new randomised trials, and it is debatable whether systematic methods for searching and integrating evidence were followed in generating all of them.
China had by then become the most prolific producer of English-language, PubMed-indexed meta-analyses, with its largest presence in meta-analyses of genetic associations, 63% of global production in 2014, where, on the same account, almost all results are misleading because they combine fragmented information from a mostly abandoned era of candidate-gene studies. Many contracting companies working on evidence synthesis receive industry contracts to produce meta-analyses, many of which probably remain unpublished. He concluded that suboptimal reviews can be harmful because of the major prestige and influence the format has acquired, and that their publication should be realigned to remove biases and vested interests and to integrate them better with the primary production of evidence.
Around 11,000 new meta-analyses were published in 2017, a third of them by Chinese authors, against fewer than 1,000 in 2000. Meta-analyses are popular because they can be done with little or no money, are publishable in high-impact journals and are in turn often cited. A good one starts with clear criteria for including and excluding studies, and scientists make several decisions and judgement calls that influence the outcome, so anyone who wants to manipulate the data has endless opportunities. Meta-analyses reaching conflicting conclusions have become frequent in fields such as antidepressants, antiviral therapy for hepatitis C, influenza treatments, the association between violence and games, and placebo effects. Funding is a potential source of bias but not the only one, and even where Cochrane reviews are more rigorous than others, that does not always eliminate conflict, as the public dispute among Cochrane directors that followed a systematic review on HPV vaccine showed. Resolving such conflicts is nearly impossible, and the hope is for more transparency in the future, opening up the data so that colleagues can redo the meta-analysis.