Probability and Random Processes

With Applications to Signal Processing and Communications

Scott Miller, Donald Childers

• Publisher: Academic Press
• Year: 2012
• ISBN:
  9780123870131
• Edition: 2
• Topic: Mathematics
• Subtopic: Probability & Statistics
• Index: Mathematics

Probability and Random Processes, Second Edition presents pertinent applications to signal processing and communications, two areas of key interest to students and professionals in today's booming communications industry. The book includes unique chapters on narrowband random processes and simulation techniques. It also describes applications in digital communications, information theory, coding theory, image processing, speech analysis, synthesis and recognition, and others.

Exceptional exposition and numerous worked out problems make this book extremely readable and accessible. The authors connect the applications discussed in class to the textbook. The new edition contains more real world signal processing and communications applications. It introduces the reader to the basics of probability theory and explores topics ranging from random variables, distributions and density functions to operations on a single random variable. There are also discussions on pairs of random variables; multiple random variables; random sequences and series; random processes in linear systems; Markov processes; and power spectral density.

This book is intended for practicing engineers and students in graduate-level courses in the topic.

• Exceptional exposition and numerous worked out problems make the book extremely readable and accessible
• The authors connect the applications discussed in class to the textbook
• The new edition contains more real world signal processing and communications applications
• Includes an entire chapter devoted to simulation techniques

CHAPTER 1

Introduction

The study of probability, random variables, and random processes is fundamental to a wide range of disciplines. For example, many concepts of basic probability can be motivated through the study of games of chance. Indeed, the foundations of probability theory were originally built by a mathematical study of games of chance. Today, a huge gambling industry is built on a foundation of probability. Casinos have carefully designed games that allow the players to win just enough to keep them hooked, while keeping the odds balanced slightly in favor of the “house.” By nature, the outcomes of these games are random, but the casino owners fully understand that as long as the players keep playing, the theory of probability guarantees—with very high probability—that the casino will always come out ahead. Likewise, those playing the games may be able to increase their chances of winning by understanding and using probability.

In another application of probability theory, stock investors spend a great deal of time and effort trying to predict the random fluctuations in the market. Day traders try to take advantage of the random fluctuations that occur on a daily basis, while long-term investors try to benefit from the gradual trends that unfold over a much longer time period. These trends and fluctuations are random in nature and can only be described in a probabilistic fashion. Another business built on managing random occurrences is the insurance industry. Insurance premiums are calculated based on a careful study of the probabilities of various events happening. For example, the car insurance salesmen have carefully evaluated the inherent risk of various classes of drivers and will adjust the premiums of each class according to the probabilities that those drivers will have an accident. In yet another application of probability theory, a meteorologist tries to predict future weather events based on current and past meteorological conditions. Since these events are quite random, the weather forecast will often be presented in terms of probabilities (e.g., there is a 40% chance, or probability, of rain on Tuesday).

Since the theory of probability and random processes finds such a wide range of applications, students require various levels of understanding depending on the particular field they are preparing to enter. For those who wish to improve their proficiency at card games, a firm understanding of discrete probability may be sufficient. Those going into operations management need to understand queueing theory and therefore Markov and related random processes. A telecommunications engineer needs to have a firm understanding of models of noise and the design of systems to minimize the effects of noise.

This book is not intended to serve the needs of all disciplines, but rather is focused on preparing the students entering the fields of electrical and computer engineering. One of the main goals of the text is to prepare the student to study random signals and systems. This material is fundamental to the study of digital signal processing (voice, image, video, etc.), communications systems and networks, radar systems, power systems, and many other applications within the engineering community. With this readership in mind, a background which is consistent with most electrical and computer engineering curricula is assumed. That is, in addition to fundamental mathematics including calculus, differential equations, linear algebra, and complex variables, the student is assumed to be familiar with the study of deterministic signals and systems. We understand that some readers may be very strong in these areas, while others may need to “brush up.” Accordingly, we have included a few appendices which may help those that need a refresher and also provide a quick reference for significant results.

Throughout the text, the reader will find many examples and exercises which utilize MATLAB. MATLAB is a registered trademark of the MathWorks, Inc.; it is a technical software computing environment. Our purpose for introducing computer-based examples and problems is to expand our capabilities so that we may solve problems that might be too tedious or complex to do via hand calculations. Furthermore, MATLAB has nice plotting capabilities that can greatly assist the visualization of data. MATLAB is used extensively in practice throughout the engineering community; therefore, we feel it is useful for engineering students to gain exposure to this important software package. Examples in the text which use MATLAB are clearly marked with a small computer logo.

Before diving into the theory of discrete probability in the next chapter, we first provide a few illustrations of how the theory of probability and random processes is used in a few engineering applications. At the end of each subsequent chapter, the reader will find engineering application sections which illustrate how the material presented in that chapter is used in the real world. These sections can be skipped without losing any continuity, but we recommend that the reader at least skim through the material.

1.1 A Speech Recognition System

Many researchers are working on methods for computer recognition of speech. One application is to recognize commands spoken to a computer. Such systems are presently available from several vendors. A simple speech recognition system might use a procedure called template matching, which may be described as follows. We define a vocabulary, or a set of possible words for a computerized dictionary. This restricts the number of possible alternatives that must be recognized. Then a template for each word is obtained by digitizing the word as it is spoken. A simple dictionary of such templates is shown in Figure 1.1. The template may be the time waveform, the spectrum of the word, or a vector of selected features of the word. Common features might include the envelope of the time waveform, the energy, the number of zero crossings within a specified interval, and the like.

Figure 1.1 A simple dictionary of speech templates for speech recognition.

Speech recognition is a complicated task. Factors that make this task so difficult include interference from the surroundings, variability in the amplitude and duration of the spoken word, changes in other characteristics of the spoken word such as the speaker’s pitch, and the size of the dictionary to name a few. In Figure 1.2, we have illustrated some of the variability that may occur when various talkers speak the same word. Here, we see that the waveform templates may vary considerably from speaker to speaker. This variability may be described by the theory of probability and random processes, which in turn may be used to develop models for speech production and recognition. Such models may then be used to design systems for speech recognition.

Figure 1.2 Variations in speech templates for different speakers.

1.2 A Radar System

A classical problem drawing heavily on the theory of probability and random processes is that of signal detection and estimation. One example of such a problem is a simple radar system, such as might be used at an airport to track local air traffic. A known signal is converted to an electromagnetic wave and propagated via an antenna. This wave will reflect off an aircraft and return back to the antenna (as illustrated in Figure 1.3), where the signal is processed to gather information about the aircraft. In addition to being corrupted by a random noise and interference process, the returning signal itself may exhibit randomness as well. First, we must determine if there is a reflected signal present. Usually, we attempt to maximize the probability of correctly detecting an aircraft subject to a certain level of false alarms. Once we decide that the aircraft is there, we attempt to estimate various random parameters of the reflected signal to obtain information about the aircraft. From the time of arrival of the reflected signal, we can estimate the distance of the aircraft from the radar site. The frequency of the returned signal will indicate the speed of the aircraft. Since the desired signal is corrupted by noise and interference, we can never estimate these various parameters exactly. Given sufficiently accurate models for these random disturbances, however, we can devise procedures for providing the most accurate estimates possible. We can also use the theory of probability and random processes to analyze the performance of our system.

Figure 1.3 A radar system.

1.3 A Communication Network

Consider a node in a computer communication network, such as depicted in Figure 1.4, that receives packets of information from various sources and must forward them along toward their ultimate destinations. Typically, the node has a fixed, or at least a maximum, rate at which it can transmit data. Since the arrival of packets to a node will be quite random, the node will usually have some buffering capability, allowing the node to temporarily store packets which it cannot forward immediately. Given a random model of the arrival process of packets at a node, the theory of probability and random processes developed in this text will allow the network designer to determine how large a buffer is needed to insure a minimal probability of buffer overflow (and a resulting loss of information). Or, conversely, given a set buffer size, a limit on the amount of traffic (i.e., throughput) that the node can handle can be determined. Other random quantities such as the delay a packet encounters at the node can also be statistically characterized.

Figure 1.4 Nodes and links in a communications network.

On a less local basis, when information is generated at one of the nodes with a specified destination, a route must be determined to get the packet from the source to the destination. Some nodes in the network may be more congested than others. Congestion throughout the network tends to be very dynamic and so the routing decision must be made using probability. Which route should the packet follow so that it is least likely to be dropped along the way? Or, maybe we want to find the path that will lead to the smallest average delay. Protocols for routing, flow control, and the likes are all based in the foundations of probability theory.

These few examples illustrate the diversity of problems that probability and random processes may model and thereby assist in the development of effective design solutions. By firmly understanding the concepts in this text, the reader will open up a vast world of engineering applications.

CHAPTER 2

Introduction to Probability Theory

Many electrical engineering students have studied, analyzed, and designed systems from the point of view of steady-state and transient signals using time domain or frequency domain techniques. However, these techniques do not provide a method for accounting for variability in the signal nor for unwanted disturbances such as interference and noise. We will see that the theory of probability and random processes is useful for modeling the uncertainty of various events (e.g., the arrival of telephone calls and the failure of electronic components). We also know that the performance of many systems is adversely affected by noise, which may often be present in the form of an undesired signal that degrades the performance of the system. Thus, it becomes necessary to design systems that can discriminate against noise and enhance a desired signal.

How do we distinguish between a deterministic signal or function and a stochastic or r...

Table of contents

• Cover Image
• Table of Contents
• Title
• Copyright
• Preface
• Chapter 1. Introduction
• Chapter 2. Introduction to Probability Theory
• Chapter 3. Random Variables, Distributions, and Density Functions
• Chapter 4. Operations on a Single Random Variable
• Chapter 5. Pairs of Random Variables
• Chapter 6. Multiple Random Variables
• Chapter 7. Random Sums and Sequences
• Chapter 8. Random Processes
• Chapter 9. Markov Processes
• Chapter 10. Power Spectral Density
• Chapter 11. Random Processes in Linear Systems
• Chapter 12. Simulation Techniques
• APPENDIX A. Review of Set Theory
• APPENDIX B. Review of Linear Algebra
• APPENDIX C. Review of Signals and Systems
• APPENDIX D. Summary of Common Random Variables
• APPENDIX E. Mathematical Tables
• APPENDIX F. Numerical Methods for Evaluating the Q-Function
• Index