A. Brief Background

In trying to solve a problem, researchers sometimes get data points and seek a function with a graph passing through these points. Usually, the nature of the problem dictates the kind of function they need. For instance, if a car moves at a constant speed and we measure the distance s(t) covered every minute, we expect the graph of s(t) to be a straight line. A high-degree polynomial or an exponential function would be unsuitable for this situation.

Suppose our problem suggests a straight line and we have the points (1,2), (2,4), (3,3). Let y = b + mx be the equation of this line. We want to find the slope m and the y-intercept b. The line should pass through the three point, thus we have:

2 = b + m1 ; 4 = b + m2 ; 3 = b + m3 .

Unfortunately, the resulting linear system in unknowns m and b

is easily seen to be inconsistent. So our problem cannot be solved exactly. The next best thing then is to try to find the straight line that best "fits" these points.

Best fitting may have different meanings, depending on what aspects of the solution we need emphasized. In this case, suppose we want our best line to be such that if d1, d2, and d3 are the errors in the y-direction, and

d1 = 2 - b - m1 ; d2 = 4 - b - m2 ; d3 = 3 - b - m3 ,

then the number

d1^2 + d2^2 + d3^2

is minimum. A solution for m and b that minimizes this sum of the squares of the errors is called a least squares solution.

How could we find the least square solution for the mentioned points and in general? Let we have an inconsistent linear system

Ax = b (*)

where A is an m x n matrix. Because for any n-vector x, the product Ax is never b, the resulting error d

d = b - Ax

is a nonzero m-vector for all n-vectors x. Solving the least squares problem for (*) amounts to finding an n-vectors x~ such that the length of d = b - Ax~ is minimum. Then x~ would be our least square solution.

It is possible to show the following. If A is an m x n matrix, then there are always least squares solutions x~ of Ax = b. Furthermore:

• x~ is a least squares solution of Ax = b if and only if x~ is a solution of the so-called normal equation

  

• A has linearly independent columns if and only if ATA is invertible. In this case, the least squares solution is unique, and it can be computed by

  

These statements provide a practical procedure for finding x~.

B. Problems

1. A college instructor has been teaching Linear Algebra course for a while. He has collected the following records regarding a B grade in this course:

• Find the least squares line for the grade data, and
• Find the expected percentage of Bs after the 10th term (presumably if this trend of grading performance continues).

2. A steel producer gather the following data:

• Find the least squares line relating the years and the annual sales.
• Use the equation obtained in the previous point to estimate the annual sales for the year 2006.

Hint: Represent the years 1997, ..., 2002 as 0, ..., 5 respectively, and let x denote the year. Let y denote the annual sales (in millions of dollars).

C. Helpful Sources

The Course Text, Chapter 8, Section 8.1.
References from Further Readings sub-section, Section 8.1.