Risk and Uncertainty in a Post-Truth Society

Sander van der Linden, Ragnar E. Löfstedt, Sander van der Linden, Ragnar E. Löfstedt

• Publisher: Routledge
• Year: 2019
• ISBN:
  9781000022926
• Edition: 1
• Topic: Business
• Subtopic: Insurance
• Index: Business

This edited volume looks at whether it is possible to be more transparent about uncertainty in scientific evidence without undermining public understanding and trust.

With contributions from leading experts in the field, this book explores the communication of risk and decision-making in an increasingly post-truth world. Drawing on case studies from climate change to genetic testing, the authors argue for better quality evidence synthesis to cut through the noise and highlight the need for more structured public dialogue. For uncertainty in scientific evidence to be communicated effectively, they conclude that trustworthiness is vital: the data and methods underlying statistics must be transparent, valid, and sound, and the numbers need to demonstrate practical utility and add social value to people's lives.

Presenting a conceptual framework to help navigate the reader through the key social and scientific challenges of a post-truth era, this book will be of great relevance to students, scholars, and policy makers with an interest in risk analysis and communication.

1 Trust in numbers¹

David J. Spiegelhalter

Introduction

This chapter gives me a fine opportunity to bring together two topical issues. First, the claims of a reproducibility crisis in science, which have led to concerns about the quality and reliability of at least parts of the scientific literature; second, the suggestion that we live in a ‘post-truth’ society abounding in fake news and alternative facts, in which emotional responses dominate evidence-informed judgement. These two topics have a close connection: both are associated with claims of a decrease in trust in expertise, and both concern the use of numbers and scientific evidence. They are therefore of vital importance to professional statisticians or any who analyze and interpret data.

A simple Internet search will reveal the daunting amount that has been written about these contested issues, and here I can only give a brief personal review of the evidence and the possible causes, focussing on the ‘filters’ that distort statistical evidence as it is passed through the information pipeline from the originators to its final consumption by the public. No single group can deal with these complex matters, but I shall argue that statisticians, and in particular the Royal Statistical Society (RSS), have an essential role both in improving the trustworthiness of statistical evidence as it flows through the pipeline and in improving the ability of audiences to assess that trustworthiness. On statistical shoulders rests a great responsibility.

Reproducibility and replication

The idea of a ‘reproducibility/replication crisis’ might reasonably be said to date from John Ioannidis’s 2005 article, which notoriously proclaimed ‘Why most published research findings are false’ (Ioannidis 2005). Although initially concerned with the biomedical literature, the idea has since been applied particularly to psychology and other social sciences. (Note that although attempts have been made to define ‘reproducibility’ and ‘replication’ precisely (Leek & Jager 2017), I feel we should try to avoid giving yet more technical definitions to words in routine use². So, I will treat the terms interchangeably and distinguish when an entire study is repeated or when data is re-analyzed).

The extent of this ‘crisis’ is contested. Ioannidis’s initial article was based on modelling rather than empirical evidence: he argued that reasonable assumptions about the design of studies, biases in conduct, selection in reporting, and the proportion of hypotheses investigated that were truly non-null meant a high rate of ‘false discoveries’, i.e., the proportion of published positive results that were actually null hypotheses that had been falsely rejected. In contrast, an analysis of published p-values (Jager & Leek 2014) came up with an estimated false-discovery rate of 14 per cent in the mainstream medical literature, and a recent review (Leek & Jager 2017) concluded, ‘We do not believe science is in the midst of a crisis of reproducibility, replicability, and false discoveries’.

So, was the claim about false claims itself a false claim? This is strongly disputed by Ioannidis (2014) and, in a recent exercise Szucs and Ioannidis (2017), scraped nearly 30,000 t statistics and degrees of freedom from recent psychology and neuroscience journals; and on the basis of the observed effect sizes and low power, concluded, ‘Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature’. Some of this apparent disagreement will be due to different literatures: Jager and Leek (2014) examined abstracts from top medical journals with many randomised controlled trials and meta-analyses, which would be expected to be much more reliable than first claims of ‘discoveries’. And even a 14 per cent false-discovery rate might be considered too high.

An alternative approach is purely empirical, in which the experiments behind past published claims are replicated by other teams of researchers: for example the effect of ‘power posing’, popularised in a TED talk that has been viewed over 40 million times (Cuddy 2012), has been subject to repeated failed replications (Ranehill et al. 2015). The Reproducibility Project was a major exercise in which 100 psychology studies were replicated with higher power (Open Science Collaboration 2015): whereas 97 per cent of the original studies had statistically significant results, only 36 per cent of the replications did. This was widely reported as meaning that the majority of the original studies were false discoveries, but Patil (Patil, Peng & Leek 2016) pointed out that 77 per cent of the new results lay within a 95 per cent predictive interval from the original study, which corresponds to there not being a significant difference between the original and replication studies. This illustrates that the difference between significant and not significant is often not significant (Gelman & Stern 2006). But it also means that 23 per cent of original and replication studies had significantly different results.

Perhaps the main lesson is that we should stop thinking in terms of significant or not significant as determining a ‘discovery’, and instead focus on effect sizes. The Reproducibility Project found that replication effects were on average in the same direction as the originals but were around half their magnitude (Open Science Collaboration 2015). This clearly displays the biased nature of published estimates in their literature and strong evidence for what might be termed regression-to-the-null.

What’s the cause of this ‘crisis’?

It’s important to note that deliberate fabrications of data do occur but appear relatively rare. A review estimated that 2 per cent of scientists admitted falsification of data (Fanelli 2009), and the US National Science Foundation and Office of Research Integrity deal with a fairly small number of deliberately dishonest acts (Mervis 2017), although substantial numbers of cases must go undetected as it is generally difficult to check raw material. Computational errors are more common but can be detected by repeating analyses if the original data is available.

Rather than deliberate dishonesty or computational incompetence, the main blame has been firmly placed on a ‘failure to adhere to good scientific practice and the desperation to publish or perish’ (Begley & Ioannidis 2015). The crucial issue is the quality of what is submitted to journals, and the quality of what is accepted, and deficits are a product of what have become known as ‘questionable research practices’ (QRPs).

Figure 1.1 shows the results of a survey of academic psychologists in the US, which had a 36 per cent response rate (John, Loewenstein & Prelec 2012). A very low proportion admitted falsification, but other practices that can severely bias outcomes were not only frequently acknowledged but generally seen as defensible: for example the 50 per cent who admitted selectively reporting studies gave an average score of 1.66 when asked whether this practice was defensible, where 0 = no, 1 = possibly, and 2 = yes. An Italian survey found similar rates, although the respondents were more inclined to agree that the practices were not defensible (Agnoli et al. 2017).

Figure 1.1 ‘Questionable research practices’ (QRPs) admitted by 2,155 US academic psychologists (John, Loewenstein & Prelec 2012).

These QRPs just involve experimentation. If we consider general observational biomedical studies and surveys, then there are a vast range of additional potential sources of bias: these might include

• Sampling things that are convenient rather than appropriate.
• Leading questions or misleading wording.
• Inability to properly adjust for confounders and make fair comparisons.
• Too small a sample.
• Inappropriate assumptions in a model.
• Inappropriate statistical analysis.

And to these we might add many additional questionable practices concerned with interpretation and communication, which we shall return to later.

These are not just technical issues of, say, lack of adjustment of p-values for multiple testing. Many of the problems arise through more informal choices made throughout the research process in response to the data: say in selecting the measures to emphasize, choice of adjusting variables, cut-points to categorize continuous quantities, and so on; this has been described as the ‘garden of forking paths’ (Gelman & Loken 2014) or ‘researcher degrees of freedom’ (Simmons, Nelson & Simonsohn 2011) and will often take place with no awareness that these are QRPs.

There have been strong arguments that the cult of p-values is fundamental to problems of reproducibility, and recent guidance from the American Statistical Association clearly revealed their misuse (Wasserstein & Lazar 2016). Discussants called for their replacement or at least downplaying their pivotal role in delineating ‘discoveries’ through the use of arbitrary thresholds. We’ve already seen that p-values are fragile things that need handling carefully in replication studies – for example a study with p = 0.05 would only be predicted a 50 per cent chance of getting p < 0.05 in a precise replication.

This is a complex issue, and in a recent article (Matthews, Wasserstein & Spiegelhalter 2017) I confessed that I liked p-values, that they are good and useful measures of compatibility between data and hypotheses, but there is insufficient distinction made between their informal use in exploratory analysis and their more formal use in confirmatory analyses that summarise the totality of evidence – perhaps they should be distinguished as p_exp and p_con.

Essentially there is too strong a tendency to use p-values to jump from selected data to a claim about the strength of evidence to conclusions about the practical importance of the research. P-values do what they say on the tin, but people don’t read the tin.

What gets into the scientific literature?

Questionable practices influence what is submitted to the scientific literature, and what finally appears depends on the publisher’s ability to critique and select from what is presented to them. Ideally, peer review would weed out inadequate research and reporting, and recommend publication of good science, regardless of the actual results. But we know that peer review is often inadequate, and there is an urge for the leading journals, to a varying amount across different disciplines, to publish newsworthy, positive ‘discoveries’ and hence produce a skewed resource.

We should not be surprised at this since traditionally journals were set up to report new findings rather than the totality of evidence. Now there is an explosion in the amount of research and publishing opportunities; I would agree that ‘most scientific papers have a lot more noise than is usually believed, that statistically significant results go in the wrong direction far more than 5% of the time, and that most published claims are overestimated, sometimes by a lot’ (Gelman 2013). Although Gelman adds, more positively, that even though there are identifiable problems with individual papers, areas of science could still be mov...

Table of contents

• Cover
• Half Title
• Series Page
• Title Page
• Copyright Page
• Table of Contents
• List of figures
• List of tables
• List of contributors
• Foreword
• Introduction: risk and uncertainty in a post-truth society
• 1 Trust in numbers
• 2 Science policy in a post-truth world
• 3 Trustworthiness, quality, and value: the regulation of official statistics in a post-truth age
• 4 Risk and uncertainty in the context of government decision-making
• 5 The handling of uncertainty: a risk manager’s perspective
• Index