Least Squares Fitting

9 Key excerpts on "Least Squares Fitting"

• 

  eBook - ePub

  Multivariable System Identification For Process Control

  • Y. Zhu(Author)
  • 2001(Publication Date)
  • Elsevier Science

    (Publisher)

  Chapter 4

  Identification by the Least-Squares Method

  The least-squares principle was invented by Karl Gauss at the end of the eighteenth century for determining the orbits of planets. Since then this method has become a major tool for parameter estimation using experimental data. Most existing parametric identification methods can be related to the least-squares method. The method is easy to comprehend and, due to the existence of a closed solution, it is also easy to implement. The least-squares method is also called linear regression (in statistical literature) and equation error method (in identification literature).

  Section 4.1 will introduce the principle of least-squares; in Section 4.2 the method will be applied to the estimation of finite impulse response (FIR) models and the estimation of parametric models. Before making assumptions and pursuing a theoretical analysis of the method, we first test the method on two industrial processes, a single stand rolling mill and a glass tube production process (Section 4.3 ). The method is successful for the first process; but it fails for the second one. Why? In Section 4.4 some theoretical analysis is carried out, and reasons are given why the least-squares method can fail. Finally in Section 4.5 we will draw conclusions about the least-squares method.

  4.1 The Principle of Least-Squares

  The least-squares technique is a mathematical procedure by which the unknown parameters of a mathematical model are chosen (estimated) such that the sum of the squares of some chosen error is minimized. Suppose a mathematical model is given in the form

  (4.1.1)

  where y (t ) is the observed variable, {θ1 , θ2 , · · ·, θn } is a set of constant parameters, x 1 (t ), x 2 (t ), · · ·,

  xn

  (t ) are known functions that may depend on other known variables. The variable t often denotes time.

  Assume that N samples of measurements of y (t ) and x 1 (t ), x 2 (t ), · · ·,

  xn

  (t ) are made at time 1, 2, · · ·, ·, N . Filling the data samples into equation (4.1.1

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Engineering Analysis

  Interactive Methods and Programs with FORTRAN, QuickBASIC, MATLAB, and Mathematica

  • Yen-Ching Pao(Author)
  • 2019(Publication Date)
  • CRC Press

    (Publisher)

  2 Exact, Least-Squares, and Cubic Spline Curve-Fits 2.1    INTRODUCTION

  Engineers conduct experiments and collect data in the laboratories. To make use of the collected data, these data often need to be fitted with some particularly selected curves. For example, one may want to find a parabolic equation

  y   =  

  c 1

  +

  c 2

  x+c 3

  x 2

  which passes three given points (xi ,yi ) for i = 1,2,3. This is a problem of exact curvefit. Or, since knowing in advance that these three points should all fall on a straight line, but the reason that they are not is because of bad calibration of the measuring equipment or because of presence of noises in the testing environment.

  In case that we may want express this straight line by the equation y = c1 + c2 x for the stress and strain data collected for a stretching test of a metal bar in the elastic range, then the question of how to determine the two coefficients c1 and c2 is a matter of deciding on which criterion to adopt. The Least-Squares method is one of the criteria which is most popularly used. The two cases cited are the consideration of adopting the two and three lowest polynomial terms, x0 , x1 , and x2 , and linearly combining them.

  If the collected data are supposed to represent a sinusoidal function of time, the curve to be determined may have to be assumed as x(t) = c1 sint + c2 sin3t + c3 sin5t + c4 sin7t by linearly combining 4 odd sine terms. This is the case of selecting four particular functions, namely, fi (t) = sin(2i−l)t for i = 1,2,3,4., and to determine the coefficients c1−4 by application of the least-squares method.

  Often some special form of curve needs to be selected to fit a given set of data, the least-squares criterion can still be applied if mathematical transformations can be found to convert the equation describing the curve into linear equations. This is discussed in a section devoted to transformed least-squares curve-fit.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Beginning Statistics with Data Analysis

  • Frederick Mosteller, Stephen E. Fienberg, Robert E.K. Rourke, Stephen E. Fienberg, Robert E.K. Rourke(Authors)
  • 2013(Publication Date)
  • Dover Publications

    (Publisher)

  Fitting StraightLines UsingLeast Squares

       11

                 

  Learning Objectives

  1. Predicting the values of new observations using linear relations

  2. Reviewing the aims of fitting straight lines to data

  3. Choosing a criterion for fitting straight lines

  4. Fitting straight lines to data using the method of least squares to minimize the sum of the squares of the residuals

  11-1 PLOTTING DATA AND PREDICTING

  Were predicting the outcomes or values of future events not such a difficult job, we would all be rich. To make prediction a reasonable statistical task, we need information on approximate relationships between the variable whose value we would like to predict and other variables whose values we can observe or control, such as the relationship between the breaking strength of pottery and its temperature at firing.

           Today, high-speed computers can do most of the computational work for plotting data and fitting linear relations. In this chapter we give, without proof, formulas for fitting straight lines to data. By using such formulas, many people have written computer programs to help us compute. We need to learn the logic behind the formulas and then let the computer do the work of calculating and printing out the information we request.

           In Chapter 3 we explored in an informal manner how to find and summarize approximate linear relationships. We now offer formal methods that are widely used in practice for fitting and examining linear relations. We turn to an example that uses these methods for prediction purposes.

  EXAMPLE 1 Winning speeds at the Indianapolis 500. The most famous American automobile race, the Indianapolis 500, is held each Memorial Day at the Indianapolis Speedway. Owing to advancing technology, each year the racers tend to go faster than in previous years. We can assess this phenomenon by examining the speed of the winning drivers over a period of years. Table 11-1

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Advanced Kalman Filtering, Least-Squares and Modeling

  A Practical Handbook

  • Bruce P. Gibbs(Author)
  • 2011(Publication Date)
  • Wiley

    (Publisher)

  The assumptions and properties of each method are discussed in later sections. Since maximum likelihood and maximum a posteriori solutions are based on assumed probability distributions, they are not really “least-squares” methods. However, maximum likelihood and maximum a posteriori are included in this chapter because the estimates are identical to other least-squares solutions when the system is linear and distributions are Gaussian. 4.1 LEAST-SQUARES DATA FITTING Curve fitting and linear regression are the simplest forms of the least-squares method. Often the goal of curve fitting is to fit a low-order polynomial to a time series of noisy data points, y (t i), i = 1 to m. If only two points are available, the solution is obvious: fit a straight line through the two points using one point as a constraint, and compute the slope from the difference between the points. Then the solution is essentially linear interpolation/extrapolation of the form If more data points are available than the number of coefficients in the polynomial model, then it is generally not possible to fit a curve exactly through all the data points. Thus the question becomes: “What criterion should be used to determine the ‘best’ fit to the data for the given model?” Two obvious possibilities are (1) the sum of absolute values of the differences (residuals) between the data and the polynomial curve; or (2) the sum-of-squared residuals. Gauss and Legendre selected the sum-of-squares as the cost function to be minimized, and that has been the preferred approach ever since. There are two arguments for preferring the residual sum-of-squares

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Signals and Systems Primer with MATLAB

  • Alexander D. Poularikas(Author)
  • 2018(Publication Date)
  • CRC Press

    (Publisher)

  chapter 14

  *Least square system design, Wiener filter, and the LMS filter

  In this chapter we shall develop three basic and important principles used extensively in signal processing, communications, system identification, denoising signals, etc. The least squares principle, proposed first by Gauss when he was 18 years old, is widely applicable to the design of digital signal processing systems. We will first describe the use of the least square approach to modeling, interference canceling, as well as the cases involving prediction. Next, we will study the celebrated Wiener filter, which was developed during the Second World War, and finally, we will study another widely used filter known as the least mean squares (LMS) filter.

  14.1 The least-squares technique

  The principle of least squares will be used in this text for signals of one variable, although the concept applies equally well for signals with more than one variable. Furthermore, we shall study discrete signals, which can always be derived from continuous signals if the sampling frequency is sufficiently high enough so that aliasing does not take place.

  Let us, for example, have a discrete signal that is represented in its vector form as follows:

  x _

  =

  [

  x

  ( 1 )

  x

  ( 2 )

    ⋯   x

  ( N )

  ]

  . We would next like to approximate the function with a polynomial of the form

  x ^

  (

  n T

  )

  =

  c 1

  +

  c 2

  (

  n T

  )

  +

  c 3

  (

  n T

  )

  2

  + ⋯ +

  c M

  (

  n T

  )

  M − 1

  T = sampling   time

  (14.1)

  Since the approximating polynomial is a linear function of the unknown

  ci

  ’s, we are dealing with linear least squares approximation. The difference between the exact and approximate function is the error. We can define a cost function J , which is equal to the sum of the difference of errors squared, the total square error. Hence, we write

  Figure 14.1.1 Least squares approximation of sampled data T = 1.

  J =

  ∑

  n = 1

  N

  [ x ( n ) −

  x ^

  ( n ) ]

  2

  (14.2)

  Our next step is to minimize this error. To see how the above equation works, let us have the discrete function plotted in Figure 14.1.1 as black dots. It is required to find a straight line that minimizes the mean square error, the cost function. To do so, we assume an approximate function of the form x = a + b (nT ) T = 1, where a and b

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Applied Regression Analysis

  • Norman R. Draper, Harry Smith(Authors)
  • 2014(Publication Date)
  • Wiley-Interscience

    (Publisher)

  CHAPTER 1 Fitting a Straight Line by Least Squares

  1.0. INTRODUCTION: THE NEED FOR STATISTICAL ANALYSIS

  In today’s industrial processes, there is no shortage of “information.” No matter how small or how straightforward a process may be, measuring instruments abound. They tell us such things as input temperature, concentration of reactant, percent catalyst, steam temperature, consumption rate, pressure, and so on, depending on the characteristics of the process being studied. Some of these readings are available at regular intervals, every five minutes perhaps or every half hour; others are observed continuously. Still other readings are available with a little extra time and effort. Samples of the end product may be taken at intervals and, after analysis, may provide measurements of such things as purity, percent yield, glossiness, breaking strength, color, or whatever other properties of the end product are important to the manufacturer or user.

  In research laboratories, experiments are being performed daily. These are usually small, carefully planned studies and result in sets of data of modest size. The objective is often a quick yet accurate analysis, enabling the experimenter to move on to “better” experimental conditions, which will produce a product with desirable characteristics. Additional data can easily be obtained if needed, however, if the decision is initially unclear.

  A Ph.D. researcher may travel into an African jungle for a one-year period of intensive data-gathering on plants or animals. She will return with the raw material for her thesis and will put much effort into analyzing the data she has, searching for the messages that they contain. It will not be easy to obtain more data once her trip is completed, so she must carefully analyze every aspect of what data she has.

  Regression analysis is a technique that can be used in any of these situations. Our purpose in this book is to explain in some detail something of the technique of extracting, from data of the types just mentioned, the main features of the relationships hidden or implied in the tabulated figures. (Nevertheless, the study of regression analysis techniques will also provide certain insights into how to plan the collection of data, when the opportunity arises. See, for example, Section 3.3.)

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Numerical Methods and Methods of Approximation in Science and Engineering

  • Karan S. Surana(Author)
  • 2018(Publication Date)
  • CRC Press

    (Publisher)

  7 Curve Fitting 7.1    Introduction

  In the interpolation theory, we construct an analytical expression, say f(x), for the data points (x

  i

  , f

  i

  ); i = 1, 2, …, n. The function f(x) is such that it passes through the data points, i.e., f(x

  i

  ) = f

  i

  ; i = 1, 2, …, n. This polynomial representation f(x) of the data points may some times be a poor representation of the functional relationship described by the data (see Figure 7.1 ).

  Figure 7.1:  Polynomial representation of data and comparison with true functional relationship

  Thus, a polynomial representation of data points may not be the best analytical form describing the behavior represented by the data points. In such cases we need to find an analytical expression that represents the best fit to the data points. In doing so, we assume that we know the analytical form of the function g(x) that best describes the data. Generally g(x) is represented by a linear or non-linear combination of the suitably chosen functions using unknown constants or coefficients. The unknown constants or coefficients in g(x) are determined to ensure that g(x) is the best fit to the data. The method of least squares fit is one such method.

  When g(x) is a linear function of the unknown constants or coefficients, we have a linear least squares fit to the data. When g(x) is a non-linear function of the unknown constants or coefficients, we obviously have a nonlinear least squares fit. In this chapter we consider linear as well as non-linear least squares fits. In case of linear least squares fit we also consider weighted least squares fit in which the more accurate data points can be assigned larger weight factors to ensure that the resulting fit is biased towards these data points. The non-linear least squares fit is first presented for a special class of g(x) in which taking log or ln of both sides of g(x) yields a form that is suitable for linear least squares fit with appropriate correction so that true residual is minimized. This is followed by a general non-linear least squares fit process that is applicable to any form of g(x) in which g(x) is a desired non-linear function of the unknown constants or coefficients. A weighted non-linear least squares formulation of this non-linear least squares fit is also presented. It is shown that this non-linear least squares fit formulation naturally degenerates to linear and weighted linear least squares fit when g(x

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Mathematical Programming with Data Perturbations II, Second Edition

  • Fiacco(Author)
  • 2020(Publication Date)
  • CRC Press

    (Publisher)

  CHAPTER 8  LEAST SQUARES OPTIMIZATION WITH IMPLICIT MODEL EQUATIONS

  AIVARS CELMIņš / Ballistic Research Laboratory, Aberdeen Proving Ground, Maryland

  ABSTRACT

  This article is concerned with algorithmic aspects of nonlinear least squares model fitting problems. Such problems differ from general optimization problems due to the special structure of the least squares normal equations. In simple least squares problems, that structure allows one to partition the normal equation system, whereby significant algorithmic simplification can be achieved. The more general least squares problems, considered here, often can be made partitionable by proper parameter manipulations. In this article, sufficient partitionability conditions are derived and parameter manipulations are discussed for the achievement of partitionability.

  I. INTRODUCTION

  We consider least squares model fitting tasks that can be formulated as constrained minimization problems. The constraints representing the model equations are, in general, implicit nonlinear relations between observables and parameters. The solution of such problems can be found by the Lagrange multiplier technique which provides a system of coupled nonlinear normal equations. The equations determine the optimal values of the model parameters and residuals, and can be used to analyze the sensitivity of the solution to data perturbations. However, the numerical solution of the system is not necessarily trivial, because the size of the system is proportional to the often very large number of data. Fortunately, many typical least squares model fitting problems have such a normal equation structure that the equation system can be partitioned by manipulations of the model equations. Manipulations of parameters, e.g., an introduction of new parameters and/or a formal elimination of some parameters, can be particularly effective. We will consider the effects of such manipulations and derive partitionability conditions that can be used as a guide for a rational model equation formulation, and for a rational planning of experiments.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Symbolic Mathematics for Chemists

  A Guide for Maxima Users

  • Fred Senese(Author)
  • 2018(Publication Date)
  • Wiley

    (Publisher)

  Even if the model is correct, then, the residuals will not be zero. However, the form of the model equation and the model parameters (like α) can be optimized so that all of the residuals are as small as possible. We require a collective function of the residuals to minimize. The most common choice is the sum of the squared residuals (SSE): (9.5) The SSE is called a figure of merit. When it is small, the residuals will be small (and the fit will be “good”). To minimize the SSE, find the derivative dSSE/d α, set it to zero, and solve for α (see Section 6.4): (9.6) (9.7) Optimization of model parameters to minimize the SSE is the basis for the ordinary least-squares method. Worksheet 9.0: Fitting data to a proportionality In this worksheet, we’ll use least-squares fitting to find proportionality constants, along with their uncertainties. 9.1 The Ordinary Least-Squares Method Suppose we have a set of N data points (x i, y i), with. In a linear model with only one independent variable x, we have two parameters to fit: the slope (m) and the intercept (b). The y values can be predicted from the x values by (9.8) where the hat marks the predicted value. Each of the data points will satisfy (9.9) Figure 9.1 compares y i, ŷ i, and the residual e i with the mean. It shows how the variation of y i around its mean has two parts: variation explained by the model () and unexplained variation found in the residuals (). Figure 9.1 Residuals e i reflect the variation in the y i data around that the model does not explain. As before, the SSE is the sum of squared residuals. The ordinary least-squares method becomes (9.10) The SSE will be at its minimum value when its derivatives with respect to m and b are zero

  Sign up to read

  Learn more about book