Measurement Uncertainty in Forensic Science

A Practical Guide

Suzanne Bell

• Publisher: CRC Press
• Year: 2016
• ISBN:
  9781315400884
• Edition: 1
• Topic: Ciencias físicas
• Subtopic: Química física y teórica

In the courtroom, critical and life-changing decisions are made based on quantitative forensic science data. There is often a range in which a measured value is expected to fall and, in this, an inherent uncertainty associated with such measurement. Uncertainty in this context is not error. In fact, estimations of uncertainty can add to the utility and reliability of quantitative results, be it the length of a firearm barrel, the weight of a drug sample, or the concentration of ethanol in blood.

Measurement Uncertainty in Forensic Science: A Practical Guide describes and defines the concepts related to such uncertainty in the forensic context. The book provides the necessary conceptual background and framework—a baseline—for developing and deploying reasonable and defensible uncertainty estimations across forensic disciplines. Information is presented conceptually, using easily understood examples, to provide a readable, handy reference for scientists in the laboratory, as well as investigators and legal professionals who require a basic understanding of the science underpinning measurement results.

Forensic Measurements, Metrology, and Uncertainty 1

Every measurement made by human beings has an inherent uncertainty associated with it. It is unavoidable and inevitable. However, measurement uncertainty is not doubt, nor does it imply an error has occurred or a mistake had been made. In fact, estimating the uncertainty associated with a measurement increases the utility and reliability of that measurement because it can provide information vital to interpretation and application. A measurement without an estimate of the uncertainty is, at best, an incomplete picture and, at worst, misleading and a potential source of false conclusions. Making good measurements requires an understanding of uncertainty, how to estimate it, and how to communicate it to those needing this information.

1.1 Consequence

Consider what might seem to be a simple forensic example. A seized-drug analyst receives a single plastic bag containing a white powder. The analyst is tasked with determining if the powder contains a controlled substance, and if so, the weight of the powder. The measurement of interest here, the weight and weighing powders, is ostensibly a straightforward process. The powder is emptied into a preweighed or tared weighing dish, placed on an analytical balance, and the net weight of the powder is displayed. Suppose that the analyst follows standard laboratory procedure, works with care and good technique, and determines that the powder is methamphetamine that is greater than 99% pure. The analyst also obtains a net weight of 50.004 g using a reliable and properly functioning balance. If that were the end of the story, this would be a very short book.

The jurisdiction in which this hypothetical laboratory operates follows Drug Enforcement Administration (DEA)/Federal Trafficking Penalty guidelines. Currently, the sentencing guidelines for methamphetamine are categorized by severity based on the weight (pure or mixtures). “Pure” here denotes the weight of the controlled substance alone; “mixtures” refer to the combined weight of the controlled substance and all the other components of the sample. In this case, methamphetamine weighing “5–49 grams pure or 50–499 grams mixture” is associated with the lesser penalties and “50 grams of more pure or 500 grams or more mixture” (www.dea.gov/druginfor) with the more severe penalties, including the possibility of a life sentence.

The importance of the uncertainty of the measurement is obvious. The measured value of 50.004 g cannot be properly interpreted or applied without an associated range. That is what an uncertainty is—a range around a measured value. If the uncertainty is estimated to be ± 0.010 g, then the range is 49.994–50.014 g, which means there is chance that the weight is less than 50 g, the critical or threshold weight in this example. On the other hand, suppose the uncertainty is ± 0.001 g and the range is 50.003–50.005 g. The weight is above 50 g and now the sentencing guidelines change. This is not just paperwork or rounding. This may change a jail sentence, which impacts many lives.

This example illustrates why measurement uncertainty matters. Forensic data have consequences—consequences to individuals and to society. Critical and momentous decisions are based on such data, and as such, these measurements must be comprehensive and complete. When we make a measurement such as a weight, the goal is to determine, as best as we possibly can, the true weight of the substance. However, our measurement, no matter how good, thorough, or complete, generates an estimate of the true weight. We can never know the true value of anything we measure (think about it), but we can make reasonable and defensible estimates of it. In the same way, we can never know the exact and complete range of values (the uncertainty) but we can generate reasonable and defensible estimates of that uncertainty.

1.2 The “Best” Way

The goal of a forensic measurement should be to make the best measurement with the best uncertainty estimates for a given task. Judging what we mean by “best” is a bit trickier. For one thing, there is rarely one clearly best method to obtain a measurement, so the task becomes one of selecting which method(s) meets our performance criteria. Forensic scientists are tasked with and expected to produce data that are judged by their reliability and their utility, which are reasonable criteria to use in evaluating a measurement process. Questions to ask in this context include

• Do the measurement data answer the relevant forensic question?
• Is sufficient information about this measurement provided to utilize these data properly?
• Are the data trustworthy?
• How trustworthy? This is where confidence and probability will come in.
• Are the data sufficiently reliable and complete to be the basis of critical and consequential decisions?

The first criteria relate to utility—if sentencing for a drug infraction depends on the weight of a seized drug, then the measurement of weight must, at a minimum, meet the standards reflected in the legislation dictating sentencing guidelines. The remaining questions address reliability, a topic that underlies most of what we will talk about in this book. It is important to understand from the outset that reliability arises from the entire measurement process and not just from the instrument used to obtain a number. In our drug weight example, a good analytical balance is essential to generating reliable data, but it is far from the only consideration. Often the goodness of measurements is judged exclusively by the tool, which leads to misunderstandings and, in the worst cases, incomplete data.

To estimate the uncertainty of a measurement, we must consider the underlying process used to obtain it—not just the measuring tool or instrument, but the entirety of the measurement process from start to finish (sometimes called cradle to grave). The utility and reliability of data depend on more than the instrument or tool. When you approach uncertainty this way, method development, method validation, and estimating uncertainty are revealed to be intimately linked.^{1, 2, 3, 4, 5 and 6} Uncertainty arises from the measurement process, not just the instrument or tool.

Another common misconception is to assume that the more expensive the tool or instrument, the better the measurement. If the laboratory in the preceding example invested in a more expensive balance capable of reading six decimal places, the accuracy (estimate of true value) may indeed improve, but this says nothing about the uncertainty. Though the uncertainty estimate may be improved, it is conceivable that the range could become larger, not smaller. Accuracy and variability are two different descriptors, related for sure, but not the same. Changing one may or may not change the other. We will delve into terminology, including accuracy, in Section 1.4.

We return to the original question: How do we define what is best (what meets our performance criteria) in making forensic measurements? For the two tasks involved (making the measurement and estimating uncertainty), the data must be generated by methods that are reasonable, defensible, and fit-for-purpose. Not surprisingly, this implies that there usually is more than one way to approach measuring and more than one way to approach estimating uncertainty. That’s why all three criteria should be considered: Are my methods reasonable, defensible, and fit-for-purpose? In many forensic applications, fitness for purpose is the driving force. For example, the consequential threshold for blood alcohol measurements is 0.08%, which from the analytical chemistry point of view is not a difficult measurement to make. There are many analytical methods that could be used to obtain the data needed. However, forensic laboratories often run hundreds or thousands of such cases, so whatever method is selected cannot ignore these realities. The method must produce results that have utility, are reliable, and can be implemented on a production scale. There are undoubtedly analytical methods that can be developed that are capable of quantitating alcohol in blood to five decimal places down at the parts-per-million level or lower, but these methods are no more fit-for-purpose in a forensic setting than separating out the alcohol by distillation and measuring the volume in a graduated cylinder. Sometimes defining fitness for purpose for both the measurement process and the uncertainty estimate can be challenging. We discuss these points throughout the book as we go through examples. There is rarely one and only one best or right way to approach an uncertainty estimate. What is right for a given laboratory measurement is defined by reasonableness, defensibility, and fitness for purpose, and those criteria will vary among laboratories. Our goal here is to develop the understanding needed to create reasonable, defensible, and fit-for-purpose uncertainty estimations suitable to a variety of measurement processes.

1.3 Measurement Science and Traceability

Metrology is the broadly defined as the science of measurement. According to the Bureau International des Poids et Mesures (BIPM⁷), specifically metrology is “the science of measurement, embracing both experimental and theoretical determinations at any level of uncertainty in any field of science and technology.” The BIPM is the international umbrella organization that coordinates definitions, standards, and practices related to measurement. The need for coordination is clear—what we in the United States considered to be 1.0 kg should be the same as what someone in Brazil defines as a kilogram. These standards are fundamental for commerce and for science. If you are paying for a gallon of gasoline, you assume that the pump delivers a gallon. You further assume that a gallon is the same in New Mexico as it is in New York. There is nothing magical about a gallon any more than there is anything special about a kilogram. What is important is that everyone agrees on the volume that is defined as a gallon and the weight that is defined as a kilogram. The importance of international agreements has long been recognized, and by international treaty (signed in 1875), the BIPM oversees international definitions and agreements regarding the units of measurement. Many excellent free references regarding metrology and uncertainty are available from the BIPM website and these are referenced throughout the book. Many readers are familiar with SI units for example; these are coordinated by the BIPM. In the United States, the National Institute of Science and Technology (NIST⁸) is the agency that coordinates with international partnerships regarding global measurement standards.

These standards provide a critical foundation in measurement science, and by extension, in uncertainty estimates. The older name for NIST was the National Bureau of Standards and a key role of the NIST is to provide standards that will ensure consistency and accuracy in many types of measurements. The term familiar to many forensic scientists this regard is traceability. Informally, traceability is the ability to trace a measurement, equipment, or a thing back to an original source. The formal definition, taken from the International Vocabulary of Metrology (VIM) and adopted by the NIST, states that traceability is the “…property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the measurement uncertainty.”

Traceability can be defined as an unbroken record of documentation (“documentation traceability”) or an unbroken chain of measurements and associated uncertainties (“metrological traceability”). For example, if I purchase a 1.00 traceable kilogram weight, the weight of that kilogram is the same as the national and international standard kilograms ± whatever the certified tolerance. Measurements such as length, weight, time, and temperature can be traceable. An analytical balance can be calibrated and checked with weights that are certified as traceable to those maintained by the NIST and beyond to the global standard. Examples of traceable weights are shown in Figure 1.1. Thermometers can also be traceable, as can calipers, data loggers, and clocks.

Figure 1.1 Standard kilogram (kg) maintained by the National Institutes of Science and Technology (NIST).

It is important to realize that traceability does not have a direct connection to uncertainty. Uncertainty is a range in which we expect a value to lie. Traceability ensures that the measurements we take of weight, length, diameter, and so forth are as close to the true value as feasible given that we accept the standards kept by the NIST to be the true value. The phrase often used is “generally acce...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Dedication
• Table of Contents
• List of Figures
• List of Tables
• Preface
• Author
• 1 Forensic Measurements, Metrology, and Uncertainty
• 2 Sources of Uncertainty
• 3 Foundational Concepts
• 4 Process and Procedures
• 5 Measurement Assurance: Distances, Crime Scenes, and Firearms
• 6 Uncertainty and Weighing
• 7 Breath Alcohol
• 8 Miscellaneous Topics
• Index