Lecture
No.
|
Date
|
Topics
Covered
|
Lecture
1
|
Thursday,
1/14
|
Gave
an
overview
of
the
course.
Noted
the
two
main
problems:
determining
solutions
of
a
linear
system
Ax
=
b
and
determining
solutions of an eigenvalued
problem Ax = lambda x, where lambda is a (scalar)
eigenvalue and x is a
non-zero vector. We will focus first on the linear system
problem.
|
Lecture 2
|
Friday,
1/15
|
Began
looking
at
the
linear
system
problem,
for
now
considering
the
case
of
an
equal
number
of
equations
and
unknowns. With simple examples,
illustrated Gaussian elimination and developed some
general principles:
Whether Ax = b has a unique solution depends only on the
coefficient
matrix A; if the system doesn't have a unique solution,
then it has
either no solutions or infinitely many soluions, depending
on b.
|
Lecture 3
|
Tuesday,
1/19
|
Noted
that
our
Gaussian
elimination
algorithm
can
break
down
even
on
systems
that
have
a
unique
solution,
even
though
a simple "fix" (interchanging
the order of the equations in the system) would have
allowed the
algorithm to proceed. We will refer to this as "naive"
Gaussian
elimination. Formulated general algorithms for
naive Gaussian
elimination and back substitution. Addressed the breakdown
issue: The
problem is that if, at some step of the elimination, the
algorithm
encounters a zero pivot element,
then
it
cannot proceed. Outlined a particular pivoting strategy, partial
pivoting,
that
allows
the
algorithm
to
continue
in
such
cases
if
at
all
possible.
In
this,
at the
kth step, the algorithm finds an entry on or below the kth
row that is
largest in absolute value; it then interchanges rows (if
necessary) to
make that entry the new pivot element.
|
Lecture 4
|
Thursday,
1/21
|
Showed
that
Gaussian
elimination
with
partial
pivoting
applied
to
a
system
Ax
=
b
goes
to
completion
(i.e.,
succeeds
in reducing the system to an
upper-triangular system) if and only if the system has a
unique
solution. Since the success of Gaussian elimination with
partial
pivoting depends only A, this confirms in general what we
observed
earlier in some simple cases: Whether Ax = b has a unique
solution
depends only on A and not on b. If Ax = b has a unique
solution (for
any and every b), we say A is nonsingular;
otherwise,
we
say A is singular.
Began
looking
at
the
cost
of
Gaussian
elimination.
For
comparative
purposes,
a
useful
measure
of
the
cost is the number of multiplications
required by the algorithm. Showed that this is
approximately (n^3)/3,
where n is the number of equations and unknowns.
|
Lecture 5
|
Friday,
1/22
|
Continued
with
the
cost
analysis
of
solving
Ax
=
b
using
Gaussian
elimination.
Showed
that
the
number
of
multiplications
required to perform the row
operations on b followed by back substitution is about
n^2. Thus the
cost of Gaussian elimination (approximately (n^3)/3
multiplications) is
far greater for all but the smallest values of n. Briefly
reviewed
vector and matrix notation and vector and matrix
arithmetic (addition,
multiplication, and multiplication by a scalar).
|
Lecture 6
|
Monday,
1/25
|
Looked at some
special
matrices:
upper
and
lower
triangular,
diagonal,
the
identity
matrix
I,
and
the transpose
A^T of a matrix A. (We say that a matrix is symmetric
if
A^T = A.) Saw that if
A
is nonsingular, then A is invertible,
i.e.,
there is a unique matrix A^{-1} (called the inverse
of
A) such that A^{-1}A = A
A^{-1} = I. Concluded that the
following are equivalent:
- A is invertible.
- A is nonsingular.
- Ax = b has a unique
solution
for every b.
- Ax = 0 if and only
if x =
0.
|
Lecture 7
|
Tuesday,
1/26
|
Gave
the
definition
of
a vector
space and a subspace.
Explored some
examples involving vectors, matrices, polynomials, and
continuous
functions.
|
Lecture
8
|
Thursday,
1/28
|
Defined
two
very
important
vector
subspaces
associated
with
an
mxn
matrix
A:
the column
space C(A), which is the set of all vectors in
R^m that are
linear combinations of the columns of A, and the null
space N(A),
which
is
the
set
of
all
vectors
v
in
R^n
such
that
Av=0.
These
give
great insight into the existence and uniqueness of
solutions of Ax=b:
- A solution exists <=> b is in C(A).
- If a solution exists, then it is unique <=>
N(A) =
{0}, i.e., x=0 is the only solution of Ax=0.
Began deriving the echelon form
and reduced
row echelon form of a
matrix, illustrating the echelon form in a simple example.
|
Lecture
9
|
Friday,
1/29
|
Further
developed
the echelon form
and reduced
row echelon form of a
matrix. Saw that the algorithm for producing the echelon
form always
goes to completion. The resulting matrix is
characterized as follows:
- The pivots are the first nonzero entries in their
rows.
- Each pivot has only zeros below it.
- Each pivot lies to the right of the pivot in the row
above.
The pivot
variables are the variables corresponding to the
pivot columns,
i.e., the columns containing the pivots. The remaining
variables are
the free
variables. Defined the rank of a matrix to be the
number of (nonzero) pivot
rows in the reduced matrix,
which
is
the
number
of pivot variables.
Note that if the column dimension of the matrix is n and
the rank is r,
then the number of free variables is n - r.
Saw the following from the echelon form and the reduced
row echelon
form:
- If the rank is r, then the system has solutions if
and only
if the reduced right-hand side vector has only zeros
after the rth
component.
- If the system has solutions, then the free variables
can be
arbitrarily specified; once the free variables are
assigned, the pivot
variables are uniquely determined.
|
Lecture
10
|
Monday,
2/1
|
Began
looking
into
the
fundamentally
important
notions
of linear
independence,
basis, and dimension. Defined linear
independence of vectors and
looked into several examples. Saw that a matrix A has
linearly
independent columns if and only if N(A) = 0. Saw that the
pivot columns
of a matrix are linearly independent and that any larger
set of columns
is linearly dependent.
|
Lecture 11
|
Tuesday,
2/2
|
Continued
looking
into
linear
independence,
basis,
and
dimension.
Defined
a basis
of a
vector space to be a set of vectors that are linearly
independent and
span the space, i.e., are such that any vector in the
space can be
written as a linear combination of the basis vectors.
(Since the basis
vectors are linearly independent, this linear combination
is unique.)
Saw that a vector space (other than {0}) has infinitely
many bases, but any two
bases have the same
number of vectors. Thus we can define the dimension
of
a space to be the number of vectors in any (and every)
basis. Began
discussing the
four
fundamental
subspaces of a matrix A.
|
Lecture
12
|
Thursday,
2/4
|
Discussed
further
the
four
fundamental
subspaces.
These
are the
column space
C(A), the nullspace N(A), the row space C(A^T), and the
left nullspace
N(A^T). (Note: "^" indicates a superscript.) Saw
that if A is
mxn and has rank r, then the dimensions of the subpaces
are as follows:
dim C(A) = r, dim N(A) = n-r, dim C(A^T) = r, and dim
N(A^T) = m-r.
Went through an example of finding C(U) and N(U) for U in
echelon form.
Noted that, in general, if U is the echelon form of A,
then N(A) =
N(U). It is not true that C(A) = C(U) in general. However,
it is true
that dim C(A) = dim C(U) and that the pivot columns of A
(the columns
of A corresponding to the pivot columns in U) constitute a
basis of
C(A).
|
Lecture
13
|
Friday,
2/5
|
Continued
discussing
the
four
fundamental
subspaces.
Saw
that, in
general, if U is the echelon form of A, then
their row spaces are the same, i.e., C(A^T) = C(U^T).
Their left null
spaces are not the same in general; however, their
dimensions are the
same. Went through several examples and reviewed for the
mid-term. |
Lecture
14
|
Tuesday, 2/9
|
Discussed
the
mid-term
exam.
Gave
an
informal
introduction
to linear
transformations through several examples.
|
Lecture
15
|
Thursday,
2/11
|
Gave
a
formal
definition
of
a
linear
transformation
from
one vector space to
another. Defined what it means for linear
tranformation to be one-to-one
and to be onto.
Defined inverse
transformations. Discussed several examples.
|
| Lecture
16 |
Friday,
2/12
|
Continued
discussing
linear
transformations,
focussing
on
the
relationships
of
linear
transformations, linear independence, spanning sets, and
bases.
Discussed more examples.
|
| Lecture
17 |
Monday,
2/15
|
Discussed
some
special
2x2
matrices
associated
with
special
transformations in
R^2, specifically stretchings/shrinkings,
rotations,
reflections,
and
projections.
|
Lecture
18
|
Tuesday,
2/16
|
Discussed matrix
representations
of linear transformations.
|
Lecture
19
|
Friday,
2/19
|
Began
looking
at
the inner
product of two vectors, the norm
(length) of a vector, and the very important notion of orthogonality
of vectors. For now, considered only the inner product
determined by
the "dot" product of vectors u and v in R^n: <u,v> =
(u^T)v =
u_1*v_1 + ... + u_n*v_n. With this, the norm (length) of v
is ||v|| is
the square root of <v,v>. Noted that the
characteristic norm and
inner-product properties hold for these. Derived the
famous Schwarz's inequality
|<u,v>| \le ||u||*||v|| (aka the
Cauchy-Schwarz-Buniakowsky
inequality). Said that vectors u and v are orthogonal
if <u,v>=0. For a subset S of a vector space V,
defined the orthogonal
complement S^(perp) to be the set of v in V that
are orthogonal
to every vector in S. Noted that S^(perp) is always
a subspace
and that 0 is always in S^(perp).
|
Lecture
20
|
Monday,
2/22
|
Defined
a
basis
{v_1,...,v_k}
to
be
an orthogonal
basis
if <v_i,v_j>=0 whenever i \ne j. Defined an orthonormal
basis
to be an orthogonal basis {q_1,...,q_k} such that
<q_i,q_i>=1 for
each i. (So an orthogonal
basis is also an orthonormal
basis if and only if each of its vectors has length one.)
If a basis is
orthogonal or (even better) orthonormal, then it is very
easy to
determine the linear combination of basis vectors that
represents a
given vector. As an aside, noted that an nxn matrix Q is
said to be an orthogonal matrix
if its columns are orthonormal. In this case, we have
(Q^T)Q=I, and so
Q is invertible (nonsingular) and Q^(-1)=Q^T. Saw that,
for vectors u
and v, <Qu,Qv>=[(Qu)^T](Qv)=(u^T)(Q^T)Qv=(u^T)v,
from which it
follows that ||Qv|| = ||v||. Thus, an orthogonal matrix
preserves
(i.e., does not change) inner products and norms of
vectors. Concluded
by outlining the very important Gram-Schmidt
orthogonalization process, which allows one to
produce an
orthonormal basis {q_1,...,q_k} from a given basis
{v_1,...,v_k}. This
orthonormal basis has the property that
span{q_1,...,q_j}=span{v_1,...,v_j} for j=1,...,k.
|
Lecture
21
|
Tuesday,
2/23
|
Went
through
an
example
of
Gram-Schmidt
orthogonalization.
Returned to the
matter of a subset S of a vector space V and its
orthogonal complement
S^(perp), and considered the particular case in which S is
a subspace.
Developed a number of useful results.
|
Lecture
22
|
Thursday,
2/25
|
Stated
and
proved
the Projection
Theorem, which states the if S is a subspace of a
vector space
V, then every v in V can be uniquely written as v = q + u,
where q is
in S and u is in S^(perp). Noted the Pythagorean
Theorem:
||v||^2
=
||q||^2
+
||u||^2.
|
Lecture
23
|
Friday,
2/26
|
Explored
consequences
of
the
Projection
Theorem.
Defined the projection
of a vector space V onto a subspace S as follows: If v = q
+ u, where q
is in S and u is in S^(perp), then define P(v) = q. The
following are
easy to verify:
- P: V -> V is a linear transformation.
- The range of P is S and the nullspace of P is
S^(perp).
- P^2 = P.
- P(v) is the closest
thing in S
to v, i.e., ||v-P(v)|| < ||v-w|| for any w in S
distinct from P(v).
Everything so far holds in a general inner-product space,
i.e., a
vector space with an inner product defined on it. Looked
at the particular
case when S is a
subspace of R^n and saw that P can
be
represented as P(v) = QQ^Tv, where the columns of
Q form an
orthonormal basis of S. (Saw that Q^TQ=I, the kxk identity
matrix, and
extended our previous definition to say that this Q is
orthogonal.)
Looked at consequences for the solution of Ax = b, where A
is mxn. Saw
that C(A) =
N(A^T)^(perp), so Ax = b has at least one
solution <=>
<b,v> = 0 whevever A^Tv = 0. Also saw that N(A) = C(A^T)^(perp),
so Ax = b has at most one solution <=> {0}
= N(A) =
C(A^T)^(perp).
Finally, considered least-squares problems. The problem is
as follows:
Given b in R^m and mxn A with n < m, find x in R^n such
that ||b -
Ax|| is minimal. Note that we cannot expect a solution to
exist for all
b in R^m, so finding an x for which Ax is as close as
possible to b is
the best we can hope for. With the Projection Theorem, we
can write b =
q + u, where q is in C(A) and u is in C(A)^(perp).
Moreover, q is the
closest thing in C(A) to b and q = Ax for some x in R^n,
and we
conclude that ||b-Ax|| is minimal. To further characterize
x, we note
that b - Ax = u is in C(A)^(perp). Therefore A^Tb - A^TAx
= 0, that is,
we have the normal
equation
of
least-squares
A^TAx
=
A^Tb. The name is due to the
fact that the solution x makes the residual
b - Ax normal (orthogonal) to C(A). Noted that A^TA is nxn
and A^Tb is
in R^n. To explore existence and uniqueness of solutions
of the normal
equation, noted that A^TAv=0 <=> Av = 0, i.e.,
N(A^TA) = N(A).
Since (A^TA)^T = A^TA, we then have that N((A^TA)^T)
=
N(A),
and
so A^TAx = A^Tb has at least one solution
<=>
<A^Tb,v> = 0 whenever Av = 0. Since Av = 0 => <A^Tb,v> = b^TAv = 0,
we conclude that the
normal equation
always has at least one solution. Also, A is of full rank n if and
only if N(A^TA) =
N(A) =
{0}, and the normal equation has at most one
solution. Since we already
know a solution exists, we conclude that the
normal equation
has a unique solution if and only if A is of full
rank n.
|
Lecture
24
|
Monday,
3/1
|
Went
through
examples
showing
how
to find the projection of a vector onto a
subspace and how to solve a least-squares problem using
the normal
equations. Talked briefly about determinants in
preparation to
discussing eigenvalue problems. (Read Secs. 4.1-4.2 of the
text.)
|
Lecture
25
|
Tuesday,
3/2
|
Introduced eigenvalues
and
eigenvectors, first for general linear
transformations and
then for square matrices. Discussed the characteristic
polynomial: If A
is an nxn matrix, then the characteristic
polynomial of A is det(A - lambda*I), a
polynomial of degree n.
The roots (zeros) of this polynomial are the eigenvalues
of A. Since a
polynomial of degree n can have at most n distinct roots,
it follows
that an nxn
matrix has at most n distinct eigenvalues. Worked
through
two simple examples for 2x2 matrices. In the first, the
matrix was
symmetric and had two distinct real eigenvalues. In the
second, the
matrix was a rotation matrix and had two distinct
eigenvalues, but they
were complex.
Noted (without
proof) that a
symmetric matrix has only real eigenvalues; then
proved
that the
eigenvectors of a symmetric matrix corresponding to
distinct
eigenvalues are orthogonal.
|
| Lecture
26 |
Thursday,
3/4
|
Reviewed for the final
exam.
|