Learning Objectives
After reading this chapter, you should be able to:
- Define terms such as systematic review and meta-analysis.
- Describe differences between “black box” and systematic reviews.
- Outline benefits of undertaking systematic reviews.
- Situate systematic reviews within an historical perspective.
- Appreciate the diversity of systematic reviews in sport, exercise, and physical activity.
Introduction
Combine 15 needles in a large concrete mixer along with 50 million straws of hay, and then pile them in a paddock. Now ask friends to find the needles and observe their reactions. The description illustrates the challenges facing people synthesizing research. Estimates indicate more than 50 million scientific publications exist with another 2.5 million being published per year (Ware & Mabe, 2015). Further, 15 studies, on average, are included in a systematic review (Page et al., 2016). Given these figures, the thought experiment above is not farfetched. Fortunately, however, research is not randomly scattered; for example, journals typically focus on specific disciplines which tends to gather together studies of a similar ilk. Nevertheless, given the various ways to disseminate data and the number of studies available, it can be daunting to undertake a systematic review, particularly in the multidisciplinary fields that the sport, exercise, and physical activity umbrellas embrace. To illustrate, I keep a database of research that has measured the drive for muscularity, which first starting appearing in the year 2000 (Edwards & Launder, 2000; McCreary & Sasse, 2000). The database currently holds 330 empirical studies, averaging 19.4 per year. Indicating the accelerating growth, 50% have been produced since 2014, meaning that currently 43 studies are being published each year. These studies have appeared across 71 journals and 87 postgraduate theses, although 39% are located in just 6 periodicals. It takes me considerable time to stay current, yet drive for muscularity is a niche area. The amount of energy and time needed to stay on top of the larger research areas in our fields requires substantially greater investment. My purpose in this book is to dismantle that challenge into its components so that you can make sense of, appreciate, and conduct a systematic review. In this chapter, I set the scene by describing what systematic reviews are and the contributions they make to our fields.
Defining Systematic Reviews
A systematic review :
Attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made. (Chandler, Higgins, Deeks, Davenport, & Clarke, 2017, p. 5)
Key features include (Chandler et al.): (a) clearly stated objectives; (b) pre-defined eligibility criteria; (c) explicit and reproducible methodology; (d) systematic searches to identify all studies meeting the eligibility criteria; (e) assessment of included studies’ design quality; (f) evaluation of the validity of the included studies’ findings; and (g) systematic presentation and synthesis of the included studies.
Sometimes people confuse systematic reviews with meta-analyses, but they are different beasts (Tod & Eubank, 2017). Meta-analysis involves the use of statistical procedures to synthesize the results from a set of studies. Meta-analytic techniques can be applied to any set of investigations, including those collected unsystematically, in which case the results will be biased. Systematic reviews need not use statistical procedures to integrate findings. The absence (or presence) of meta-analyses does not indicate the quality or usefulness of a systematic review. Both qualitative and quantitative investigations may be included in a research synthesis. Furthermore, high quality meta-analyses will be conducted on studies located via methodical rigorous means.
To help advertise the strengths of systematic reviews, authors pit them against narrative reviews, which they paint as the poor cousins in the synthesis family. The implication is that narrative reviews are unsystematic and of less value than those that synthesize research numerically. Such binary descriptions are misleading, because narrative reviews (where research is synthesized qualitatively) can adhere to Chandler et al.’s (2017) key features. All literature reviews should be underpinned by a systematic approach; otherwise they risk being unduly biased or even little more than opinion pieces (Booth, Sutton, & Papaioannou, 2016). The evaluation of whether a document can, or cannot, be labelled a systematic review does not depend on either the type of evidence included or the use of quantitative over qualitative analysis. Instead, literature reviews can be placed along a continuum from those where authors have adhered to the above features to those where reviewers have stuck loosely, if at all, to them (and have produced “black box” reviews).
Black Box to Systematic Review Continuum
To understand differences between the two review types, I find it helpful to consider systematic reviews as being somewhat similar to primary research (Gough, Oliver, & Thomas, 2017), where the sample consists of research reports instead of people. Some characteristics of good primary research include (a) identifying a relevant, justifiable, and answerable question; (b) employing a reproducible method; (c) being transparent, to allow an external evaluation of the work; and (d) adherence to ethical standards. These characteristics echo those by which Chandler et al. (2017) described systematic reviews. Primary research varies in the degree to which it satisfies these characteristics. For example, it is difficult sometimes to achieve transparency, given journal space limitations, variation in audience understanding, and the difficulty in reducing some procedures to a list of technical steps (Hammersley, 2006). Informed readers realize that all studies have limitations and weigh these against the findings to decide on the investigations’ knowledge contributions. Similarly, systematic reviews vary in their adherence to the above characteristics, but that does not make them unsystematic or invalidates their contribution. Again, readers need to evaluate the document in question and decide on the review’s usefulness for their needs.
Systematic Review Benefits
Systematic reviews in sport, exercise, and physical activity are similar to other scientific publications in the field: they are written for an audience, including fellow academics, practitioners, and policymakers. Considering the targeted audiences’ needs and preferences will help researchers pinpoint their reviews’ purposes and justifications. The benefits of undertaking systematic reviews can be classed as knowledge or decision-making support (Pope, Mays, & Popay, 2007).
Knowledge support
A valuable feature of reviews is the synthesis of primary documents. The Cambridge dictionary defines synthesis as “the mixing of things to make a whole that is different or new,” and this description signals what systematic reviews can achieve. A meta-analysis, for example, “mixes” or combines individual effect sizes to produce a new estimate, along with confidence intervals indicating the range of plausible alternatives to help evaluate the new result. Systematic reviews advance knowledge by informing us about what we know, what we do not know, what we need to know, and why we need to know it. Also, these projects describe the quality of the evidence, allowing us to assess how much confidence we have in what we think we know. More specifically, they allow us to critique evidence and question primary research findings.
Decision-making support
Systematic syntheses can inform audiences about a topic area so they can make evidence-based decisions and policies. When contemplating how to intervene in people’s lives, such as implementing exercise adoption programmes, systematic reviews can outline what actions are possible, what actions are not possible, the associated costs and risks, and the potential benefits. By providing support for decision-making processes, systematic reviews may have a real-world impact. For example, systematic reviews often influence the development of public health policy and health-related recommendations proposed by the United Kingdom’s National Institute of Clinical Excellence (NICE), as illustrated in their recent evaluation of exercise referral schemes (NICE, 2014).
Typically, primary research is unable to deliver on the above benefits as well as systematic rev...