Meta-Regression Analysis in Economics and Business

T.D. Stanley, Hristos Doucouliagos

• Editorial: Routledge
• Año: 2012
• ISBN:
  9781136279386
• Edición: 1
• Categoría: Negocios y empresa
• Categoría: Negocios en general

The purpose of this book is to introduce novice researchers to the tools of meta-analysis and meta-regression analysis and to summarize the state of the art for existing practitioners. Meta-regression analysis addresses the rising "Tower of Babel" that current economics and business research has become. Meta-analysis is the statistical analysis of previously published, or reported, research findings on a given hypothesis, empirical effect, phenomenon, or policy intervention. It is a systematic review of all the relevant scientific knowledge on a specific subject and is an essential part of the evidence-based practice movement in medicine, education and the social sciences. However, research in economics and business is often fundamentally different from what is found in the sciences and thereby requires different methods for its synthesis—meta-regression analysis. This book develops, summarizes, and applies these meta-analytic methods.

1 Introduction

This is but the start of their undertakings! There will be nothing too hard for them to do. Come, let us go down and confuse their language on the spot so that they can no longer understand one another.

(Genesis 11: 6–7)

1.1 The Tower of Research

We live in a wondrous age. Information technology has given billions access to the world’s accumulated scientific knowledge as well as this week’s viral video of some kid dancing. Inexpensive hand-held devices bring us the contents of a hundred libraries in seconds and the processing power of the best computers from only a generation ago. But has society’s knowledge become thousands of miles wide and mere nanometers deep? To some, these gigabytes, terabytes, and petabytes usher in a Renaissance of human knowledge and creativity. To others, like the Nobel econometrician, James Heckman, they represent a tsunami of noise and misinformation that threatens to drown out genuine scientific knowledge and informed policy action (Heckman, 2001). How will we be able to distinguish useful information from mere exaggeration, ideology and even lies?

Our extraordinary era has seen the rapid expansion of research publications, the meteoric rise in empirical economics and business research, and the proliferation of increasingly narrow areas of academic research. Is this not another “Tower of Babel,” one where these terabytes ensure “that [we] can no longer understand one another”?

Worse than the sheer mass of information are the large differences in what researchers report about a given phenomenon, treatment or effect. In social science, economics, and business research, one always finds a large variation in the reported estimates of a given parameter. The rising pressure to publish, with its concomitant demand to uncover something novel, is sufficient to generate ample conflict among empirical findings. Because economics and business ultimately depend on human behavior, the empirical phenomena that we study will always contain a great deal of natural variation; that is, a genuine heterogeneity that depends on prevailing socio-political institutions and history. Often, it seems as if researchers speak different languages.

Incentives in the media, science, and the academy all seem to accentuate the dissidence of reported research. Should science become too clear and uncontroversial (e.g. the health effects of smoking, global warming or evolution), concerned groups will fund researchers and spokesmen to manufacture uncertainty and controversy. Yet, wide variation in research findings will likely occur without any outside intervention. Even the best scientific practice will produce very disparate research findings without resorting to anything ethically questionable. Science progresses through critical discourse and by challenging what is believed. When virtually all researchers agree about a given theory, empirical phenomenon, or policy effect, scientific progress is likely to stagnate, and, ironically, we find larger, more distorting biases in what researchers report (Doucouliagos and Stanley, 2012).¹

Although we need not fear disparate scientific findings, practical policy demands clarity. Without some intelligent summary of business and economic research, understanding and informed policy actions are impossible. Yet, conventional narrative reviews are fatally flawed. Because there are no objective standards, conventional reviewers often dismiss studies or findings that do not fit into their preconceived notions or theories (Stanley, 2001). “Believing is seeing” (Demsetz, 1974: 164). Beliefs are often self-fulfilling. One can almost always find research papers or a literature review that interprets past research through the reader’s own priors or ideological lens. Yet, without the reliable coherence that a good narrative review is meant to provide, conflicting research results overwhelm any clear understanding of economic phenomena. The only informed and correct conventional summary of the research record on nearly any important economic phenomenon or policy question is: “it depends.”

What we need is some objective and critical methodology to integrate conflicting research findings and to reveal the nuggets of “truth” that have settled to the bottom. Meta-regression analysis (MRA), when replicable and conducted properly, offers such methodology. We believe that it is economics’ best hope for genuine empirical progress.

Meta-analysis is the statistical analysis of previously published, or reported, research findings on a given hypothesis, empirical effect, phenomenon, or policy intervention. It is a systematic review of all the relevant scientific knowledge on a specific subject and is an essential part of the “evidence-based practice” movement in medicine, education and the social sciences.

Medical researchers have long embraced meta-analysis to provide an objective and comprehensive summary of the often conflicting results from randomized clinical trials (RCTs) of some drug or medical procedure. Evidence-based medical practice has changed how sick people are treated and saved 100,000 lives within the first 18 months of its adoption (Berwick et al., 2006; Ayers, 2007). Because RCTs tend to be very expensive and time-consuming, medical practice is often based on only a few trials, trials which often report conflicting success and risks. To economize on this limited and expensive scientific evidence, medical researchers have been employing meta-analysis for over 30 years (Chalmers et al., 1977). Often, when several RCTs are statistically combined, a clearer, more accurate picture of a given treatment’s efficacy emerges.

Meta-analysis is the most objective and statistically rigorous approach to systematic reviews, which, in turn, provides the evidence for the evidence-based practice movement. A systematic review differs from more conventional narrative reviews by conducting exhaustive searches in a serious attempt to include all studies meeting explicitly stated criteria. When conducted properly, a systematic review is replicable by independent reviewers.

In economics, meta-analysis is almost entirely meta-regression analysis, and it has a somewhat different focus than how it is applied in other fields. MRA was initially proposed to correct known misspecification biases, endemic among econometrics estimates (Stanley and Jarrell, 1989). Meta-regression analysis is a multivariate empirical investigation, using multiple regression analysis, of what causes the large variation among reported regression estimates or transformations of regression estimates (e.g. elasticities, environmental values, or partial correlations). Because econometrics is typically observational (i.e. non-experimental), even the most rigorous econometric applications cannot eliminate all the potentially confounding influences.² By now, hundreds of MRAs have confirmed that such misspecification biases are routinely found in all areas of empirical economics research, and many of these are large enough to have a significant practical effect on how we view the phenomenon in question or on how a given policy intervention is evaluated.

Then there is the question of selection. Only a few of potentially millions of econometric models are reported – “I just ran two million regressions” (Sala-i-Martin, 1997).

Empirical results reported in economics journals are selected from a large set of estimated models. Journals, through their editorial policies, engage in some selection, which in turn stimulates extensive model searching and prescreening by prospective authors. Since this process is well known to professional readers, the reported results are widely regarded to overstate the precision of the estimates, and probably to distort them as well. As a consequence, statistical analyses are either greatly discounted or completely ignored.

(Leamer and Leonard, 1983: 306)

Each of these model specification choices affects the reported results, often by a lot, and there is no reliable way to know which model specification is correct.³ Enter meta-regression analysis.

Meta-regression analysis can explicitly model the effects of observed model specification variation and thereby directly estimate the associated misspecification biases. Accommodating and correcting the biases associated with applied econometrics is the central objective of MRA. Meta-regression analysis is a systematic and comprehensive review of all existing, yet comparable, empirical evidence. It allows the systematic reviewer to model and estimate any explanatory or biasing factor for which information or a proxy is available and thereby filters out their influence on our scientific knowledge. This applies to selection as well. Although MRA can accommodate the conventional sample selection biases that are often seen in empirical econometrics (Heckman, 1979; Stanley and Jarrell, 1998), it can do much more.

Publication selection, as opposed to sample selection, arises if researchers, editors, or reviewers use statistical significance as one model selection criterion. Publication biases have been identified in the majority of economics areas of research and often have important practical effects (Doucouliagos and Stanley, 2012).⁴ Because publication selection is caused by the process of conducting empirical economic research itself, conventional econometrics is incapable of correcting or estimating this effect. Hence, some “macro” perspective is required that looks across an entire research field, and this is precisely what MRA provides. Chapter 4 discusses how MRA can identify, estimate, and correct publication selection bias, and subsequent chapters illustrate and explain how MRA can filter out many other types of bias as well.

The purpose of this book is to introduce the tools of meta-analysis and meta-regression analysis to business and economic researchers unfamiliar with their use. Meta-regression analysis addresses the rising “Tower of Babel” that current economics and business research has become. Evidence-based policy requires a clear and objective assessment of the research record. Without a systematic and objective way to summarize and understand current research, policy discussions will be at the mercy of the subjective interpretation of our empirical knowledge. Moreover, there is a real danger that vested interest or ideology will dominate the discussion and thereby distort policy.

For example, it is clear that both of these forces dominated the anti-regulation atmosphere in the USA that preceded the global 2008 financial meltdown. Alan Greenspan, the former US Federal Reserve chairman, was a disciple of Ayn Rand and a libertarian (Greenspan, 2007; Leonhardt, 2007). Greenspan has been forthcoming about his free-market ideology. A case has been made that it was the opposition to the regulation of derivatives by both Greenspan and the financial industry that led to the worst recession in the USA since the Great Depression (Public Broadcasting Service, 2009), and Greenspan admitted the error of his ideology to the US Congress (Andrews, 2008).⁵

A more positive trend is that governmental agencies are funding dozens of systematic reviews and meta-analyses of their programs and policies.⁶ In 2011, the United Kingdom’s coalition government has renewed its pledge to protect its international development aid from the spending cuts and to double its international aid commitment. Needless to say, this puts the Cameron government under considerable political pressure, not the least of which comes from their party loyalists (Hennessy, 2011). In a climate of large cuts to domestic programs, it is especially important to ensure that government policies and programs are getting “value for money.” Here, too, MRA has an important role to play, because it can offer an objective, comprehensive and rigorous summary and evaluation of what is known, empirically, about a given intervention or policy.

In our view, we are at the dawning of a new era of empiricism in economics and business. Even though the capacity of future empirical methods cannot be fully known, there will remain conflict in what these methods reveal about specific business and economic effects. These phenomena are irreducibly contingent on prevailing cultural and political institutions, and we live in dynamic societies. Thus, an important role for meta-analysis is virtually assured.

If the past is any guide, future systematic reviews and meta-analyses will, on occasion, find that strongly held economic theories are not supported by the weight of empirical evidence. For example, minimum wage raises do not cause lower employment in the US (Doucouliagos and Stanley, 2009) – see Chapters 4 and 5. In other cases, intentionally weak governmental policy (i.e. non-mandatory regulation) will be found to have their intended effects – for example, chief executive pay and corporate performance (Doucouliagos et al., 2012a).⁷

1.2 A historical sketch of meta-regression analysis

One can begin the history of meta-analysis at several points. One choice is the early twentieth-century contributions of the legendary statisticians, Karl Pearson (1904) and R.A. Fisher (1932). Both sought a means to combine separate experiments statistically and rigorously. Because experiments tend to be expensive, the sample sizes employed are often too small to obtain statistically significant results in individual studies. Thus, an obvious statistical solution to economize scarce experimental knowledge is to combine several small-sample experiments to increase their overall statistical power and thereby obtain that all-pervading research goal, “statistical significance.”

Pearson’s solution was to average the correlation coefficients, while Fisher developed a new statistic that combined p-values. Pearson’s approach is very simple and obvious when we look back a hundred years. Nonetheless, weighted averages of correlation coefficients are still used by meta-analysts (see Chapter 3). The elegance of Pearson’s solution is that the correlation coefficient is a pure number with no units of measurement, allowing different, but related, outcome measures to be meaningfully compared and combined. This issue of which statistics and measures can be meaningfully combined remains a central issue confronting every meta-analysis. The second advantage of using correlation coefficients is that they reflect the underlying magnitude of the empirical phenomenon in question, not merely its statistical significance (Cohen, 1988).

Fisher’s approach is more complex, yet much less useful. It assumes, as the null hypothesis, that all studies have no genuine underlying experimental effect. By doing so, p-values become uniformly distributed and give the Fisher combined probability test:

where L is the number of statistical outcomes or studies in the literature, and P_i is the p-value of the ith the study. This Fisher test is distributed as a chi-squared wit...

Índice

• Cover
• Title
• Copyright
• Contents
• List of Figures
• List of Tables
• Acknowledgments
• 1. Introduction
• 2. Identifying and coding meta-analysis data
• 3. Summarizing meta-analysis data
• 4. Publication bias and its discontents
• 5. Explaining economics research
• 6. Econometric theory and meta-regression analysis
• 7. Further topics in meta-regression analysis
• 8. Summary and conclusions
• Notes
• References
• Index