Systematic Reviews in Health Care

Meta-Analysis in Context

Matthias Egger, George Davey Smith, Douglas Altman, Matthias Egger, George Davey-Smith, Douglas Altman

• Publisher: BMJ Books
• Year: 2013
• ISBN:
  9781118700044
• Edition: 2
• Topic: Medicine
• Subtopic: Medical Theory, Practice & Reference

The second edition of this best-selling book has been thoroughly revised and expanded to reflect the significant changes and advances made in systematic reviewing. New features include discussion on the rationale, meta-analyses of prognostic and diagnostic studies and software, and the use of systematic reviews in practice.

Part I

Systematic reviews of controlled trials

2

Principles of and procedures for systematic reviews

MATTHIAS EGGER, GEORGE DAVEY SMITH

Summary points

• Reviews and meta-analyses should be as carefully planned as any other research project, with a detailed written protocol prepared in advance.
• The formulation of the review question, the a priori definition of eligibility criteria for trials to be included, a comprehensive search for such trials and an assessment of their methodological quality, are central to high quality reviews.
• The graphical display of results from individual studies on a common scale (“Forest plot”) is an important step, which allows a visual examination of the degree of heterogeneity between studies.
• There are different statistical methods for combining the data in meta-analysis but there is no single “correct” method. A thorough sensitivity analysis should always be performed to assess the robustness of combined estimates to different assumptions, methods and inclusion criteria and to investigate the possible influence of bias.
• When interpreting results, reviewers should consider the importance of beneficial and harmful effects of interventions in absolute and relative terms and address economic implications and implications for future research.

Systematic reviews allow a more objective appraisal of the evidence than traditional narrative reviews and may thus contribute to resolve uncertainty when original research, reviews and editorials disagree. Systematic reviews are also important to identify questions to be addressed in future studies. As will be discussed in the subsequent chapter, ill conducted reviews and meta-analyses may, however, be biased due to exclusion of relevant studies, the inclusion of inadequate studies or the inappropriate statistical combination of studies. Such bias can be minimised if a few basic principles are observed. Here we will introduce these principles and give an overview of the practical steps involved in performing systematic reviews. We will focus on systematic reviews of controlled trials but the basic principles are applicable to reviews of any type of study (see Chapters 12–14 for a discussion of systematic reviews of observational studies). Also, we assume here that the review is based on summary information obtained from published papers, or from the authors. Systematic reviews and meta-analyses based on individual patient data are discussed in Chapter 6. We stress that the present chapter can only serve as an elementary introduction. Readers who want to perform systematic reviews should consult the ensuing chapters and consider joining forces with the Cochrane Collaboration (see Chapters 25 and 26).

Developing a review protocol

Systematic reviews should be viewed as observational studies of the evidence. The steps involved, summarised in Box 2.1, are similar to any other research undertaking: formulation of the problem to be addressed, collection and analysis of the data, and interpretation of the results. Likewise, a detailed study protocol which clearly states the question to be addressed, the subgroups of interest, and the methods and criteria to be employed for identifying and selecting relevant studies and extracting and analysing information should be written in advance. This is important to avoid bias being introduced by decisions that are influenced by the data. For example, studies which produced unexpected or undesired results may be excluded by post hoc changes to the inclusion criteria. Similarly, unplanned data-driven subgroup analyses are likely to produce spurious results.^1,2 The review protocol should ideally be conceived by a group of reviewers with expertise both in the content area and the science of research synthesis.

Objectives and eligibility criteria

The formulation of detailed objectives is at the heart of any research project. This should include the definition of study participants, interventions, outcomes and settings. As with patient inclusion and exclusion criteria in clinical studies, eligibility criteria can then be defined for the type of studies to be included. They relate to the quality of trials and tothe combinability of patients, treatments, outcomes and lengths of follow-up. As discussed in detail in Chapter 5, quality and design features of clinical trials can influence the results.^3–5 Ideally, only controlled trials with proper patient randomisation which report on all initially included patients according to the intention-to-treat principle and with an objective, preferably blinded, outcome assessment would be considered for inclusion.⁶ Formulating assessments regarding study quality can be a subjective process, however, especially since the information reported is often inadequate for this purpose.^7–10 It is therefore generally preferable to define only basic inclusion criteria, to assess the methodological quality of component studies, and to perform a thorough sensitivity analysis, as illustrated below.

Box 2.1 Steps in conducting a systematic review^*

1 Formulate review question

2 Define inclusion and exclusion criteria

• participants
• interventions and comparisons
• outcomes
• study designs and methodological quality

3 Locate studies (see also Chapter 4)

Develop search strategy considering the following sources:

• The Cochrane Controlled Trials Register (CCTR)
• electronic databases and trials registers not covered by CCTR
• checking of reference lists
• handsearching of key journals
• personal communication with experts in the field

4 Select studies

• have eligibility checked by more than one observer
• develop strategy to resolve disagreements
• keep log of excluded studies, with reasons for exclusions

5 Assess study quality (see also Chapter 5)

• consider assessment by more than one observer
• use simple checklists rather than quality scales
• always assess concealment of treatment allocation, blinding and handling of patient attrition
• consider blinding of observers to authors, institutions and journals

6 Extract data

• design and pilot data extraction form
• consider data extraction by more than one observer
• consider blinding of observers to authors, institutions and journals

7 Analyse and present results (see also Chapters 8–11, 15, 16)

• tabulate results from individual studies
• examine forest plot
• explore possible sources of heterogeneity
• consider meta-analysis of all trials or subgroups of trials
• perform sensitivity analyses, examine funnel plots
• make list of excluded studies available to interested readers

8 Interpret results (see also Chapters 19–24)

• consider limitations, including publication and related biases
• consider strength of evidence
• consider applicability
• consider numbers-needed-to-treat to benefit / harm
• consider economic implications
• consider implications for future research

^* Points 1–7 should be addressed in the review protocol.

Literature search

The search strategy for the identification of the relevant studies should be clearly delineated. As discussed in Chapter 4, identifying controlled trials for systematic reviews has become more straightforward in recent years. Appropriate terms to index randomised trials and controlled trials were introduced in the widely used bibliographic databases MEDLINE and EMBASE by the mid 1990s. However, tens of thousands of trial reports had been included prior to the introduction of these terms. In a painstaking effort the Cochrane Collaboration checked the titles and abstracts of almost 300 000 MEDLINE and EMBASE records which were then re-tagged as clinical trials if appropriate. It was important to examine both MEDLINE and EMBASE because the overlap in journals covered by the two databases is only about 34%.¹¹ The majority of journals indexed in MEDLINE are published in the US whereas EMBASE has better coverage of European journals (see Box 4.1 in Chapter 4 for a detailed comparison of MEDLINE and EMBASE). Re-tagging continues in MEDLINE and EMBASE and projects to cover other databases are ongoing or planned. Finally, thousands of reports of controlled trials have been identified by manual searches (“handsearching”) of journals, conference proceedings and other sources.

All trials identified in the re-tagging and handsearching projects have been included in the The Cochrane Controlled Trials Register which is available in the Cochrane Library on CD ROM or online (see Chapter 25). This register currently includes over 250 000 records and is clearly the best single source of published trials for inclusion in systematic reviews. Searches of MEDLINE and EMBASE are, however, still required to identify trials that were published recently (see the search strategy described in Chapter 4). Specialised databases, conference proceedings and the bibliographies of review articles, monographs and the located studies should be scrutinised as well. Finally, the searching by hand of key journals should be considered, keeping in mind that many journals are already being searched by the Cochrane Collaboration.

The search should be extended to include unpublished studies, as their results may systematically differ from published trials. As discussed in Chapter 3, a systematic review which is restricted to published evidence may produce distorted results due to publication bias. Registration of trials at the time they are established (and before their results become known) would eliminate the risk of publication bias.¹² A number of such registers have been set up in recent years and access to these has improved, for example through the Cochrane Collaboration’s Register of Registers or the internet-based meta Register of Controlled Trials which has been established by the publisher Current Science (see Chapters 4 and 24). Colleagues, experts in the field, contacts in the pharmaceutical industry and other informal channels can also be important sources of information on unpublished and ongoing trials.

Selection of studies, assessment of methodological quality and data extraction

Decisions regarding the inclusion or exclusion of individual studies often involve some degree of subjectivity. It is therefore useful to have two observers checking eligibility of candidate studies, with disagreements being resolved by discussion or a third reviewer.

Randomised controlled trials provide the best evidence of the efficacy of medical interventions but they are not immune to bias. Studies relating methodological features of trials to their results have shown that trial quality influences effect sizes.^4,5,13 Inadequate concealment of treatment allocation, resulting, for example, from the use of open random number tables, is on average associated with larger treatment effects.^4,5,13 Larger effects were also found if trials were not double-blind.⁴ In some instances effects may also be overestimated if some participants, for example, those not adhering to study medications, were excluded from the analysis.^14–16 Although widely recommended, the assessment of the methodological quality of clinical trials is a matter of ongoing debate.⁷ This is reflected by the large number of different quality scales and checklists that are available.^10,17 Empirical evidence¹⁰ and theoretical considerations¹⁸ suggests that although summary quality scores may in some circumstances provide a useful overall assessment, scales should not generally be used to assess the quality of trials in systematic reviews. Rather, as discussed in Chapter 5, the relevant methodological aspects should be identified in the study protocol, and assessed individually. Again, independent assessment by more than one observer is desirable. Blinding of observers to the names of the authors and their institutions, the names of the journals, sources of funding and acknowledgments should also be considered as this may lead to more consistent assessments.¹⁹ Blinding involves photocopying of papers removing the title page and concealing journal identifications and other characteristics with a black marker, or scanning the text of papers into a computer and preparing standardised formats.^20,21 This is time consuming and potential benefits may not always justify the additional costs.²²

It is important that two independent observers extract the data, so errors can be avoided. A standardised record form is needed for this purpose. Data extraction forms should be carefully designed, piloted and revised if necessary. Electronic data collection forms have a number of advantages, including the combination of data abstraction and data entry in one step, and the automatic detection of inconsistencies between data recorded by different observers. However, the complexities involved in programming and revising electronic forms should not be underestimated.²³

Presenting, combining and interpreting results

Once studies have been selected, critically appraised and data extracted the characteristics of included studies should be presented in tabular form. Table 2.1 shows the characteristics of the long term trials that were included in a systematic review²⁴ of the effect of beta blockade in secondary prevention after myocardial infarction (we mentioned this example in Chapter 1 and will return to it later in this chapter). Freemantle et al.²⁴ included all parallel group randomised trials that examined the effectiveness of beta blockers versus placebo or alternative treatment in patients who had had a myocardial infarction. The authors searched 11 bibliographic databases, including dissertation abstracts and grey literature databases, examined existing reviews and checked the reference lists of each identified study. Freemantle et al. identified 31 trials of at least six months’ duration which contributed 33 comparisons of beta blocker with control groups (Table 2.1).

Standardised outcome measure

Individual results have to be expressed in a standardised format to allow for comparison between studies. If the endpoint is binary (e.g. disease versus no disease, or dead versus alive) then relative risks or odds ratios are often calculated. The odds ratio has convenient mathematical properties, which allow for ease in the combination of data and the testing of the overall effect for statistical significance, but, as discussed in Box 2.2, the odds ratio will differ from the r...

Table of contents

• cover
• Contents
• Dedication
• Title Page
• Copyright
• Contributors
• Foreword
• Introduction
• Part I: Systematic reviews of controlled trials
• Part II: Investigating variability within and between studies
• Part III: Systematic reviews of observational studies
• Part IV: Statistical methods and computer software
• Part V: Using systematic reviews in practice
• Part VI: The Cochrane Collaboration
• Index