EE263 Autumn 2015 S. Boyd and S.

Lall

Symmetric matrices and quadratic forms

I eigenvectors of symmetric matrices

I quadratic forms

I inequalities for quadratic forms

I positive semidefinite matrices

1
Eigenvalues of symmetric matrices

if A ∈ Rn×n is symmetric, i.e., A = AT , then the eigenvalues of A are real

to see this, suppose Av = λv, v 6= 0, v ∈ Cn , then

n
X
v T Av = v T (Av) = λv T v = λ |vi |2
i=1

but also
n
T T X
v T Av = (Av) v = (λv) v = λ |vi |2
i=1

so we have λ = λ, i.e., λ ∈ R (hence, can assume v ∈ Rn )

2
Eigenvectors of symmetric matrices

there is a set of n orthonormal eigenvectors of A

I i.e., q1 , . . . , qn s.t. Aqi = λi qi , qiT qj = δij

I in matrix form: there is an orthogonal Q s.t.

Q−1 AQ = QT AQ = Λ

I hence we can express A as

n
X
A = QΛQT = λi qi qiT
i=1

I in particular, qi are both left and right eigenvectors

3
Interpretations

A = QΛQT corresponds to

x QT x ΛQT x Ax
QT Λ Q

linear mapping y = Ax can be decomposed as

I resolve into qi coordinates

I scale coordinates by λi

I reconstitute with basis qi

4
Geometrical interpretation

multiplication by A is the same as

I rotate by QT

I diagonal real scale (‘dilation’) by Λ

I rotate back by Q

decomposition
n
X
A= λi qi qiT
i=1

expresses A as linear combination of 1-dimensional projections

5
Example:

 
−1/2 3/2
A=
3/2 −1/2
      T
1 1 1 1 0 1 1 1
= √ √
2 1 −1 0 −2 2 1 −1

q1
q2 q2T x
x

q1 q1T x λ1 q1 q1T x
q2
λ2 q2 q2T x
Ax

6
Proof

eigenvectors corresponding to distinct eigenvalues are orthogonal

I since λi distinct, have v1 , . . . , vn , a set of linearly independent eigenvectors

of A
Avi = λi vi , kvi k = 1

I then viT (Avj ) = λj viT vj = (Avi )T vj = λi viT vj

I and (λi − λj )viT vj = 0

I for i 6= j, λi 6= λj , hence viT vj = 0

I in this case we can say: eigenvectors are orthogonal

I in general case (λi not distinct) we must say: eigenvectors can be chosen to
be orthogonal

7
Quadratic forms

a quadratic form is a function f : Rn → R of the form

n
X
f (x) = xT Ax = Aij xi xj
i,j=1

I in a quadratic form we may as well assume A = AT since

xT Ax = xT ((A + AT )/2)x

((A + AT )/2 is called the symmetric part of A)

I uniqueness: if xT Ax = xT Bx for all x ∈ Rn and A = AT , B = B T , then

A=B

8
Examples

quadratic forms

I kBxk2 = xT B T Bx
Pn−1
I i=1 (xi+1 − xi )2

I kF xk2 − kGxk2

sets defined by quadratic forms:

I { x | f (x) = a } is called a quadratic surface

I { x | f (x) ≤ a } is called a quadratic region

9
Inequalities for quadratic forms

suppose A = AT , A = QΛQT with eigenvalues sorted so λ1 ≥ · · · ≥ λn then

xT Ax ≤ λ1 xT x

because

xT Ax = xT QΛQT x
= (QT x)T Λ(QT x)
n
X
= λi (qiT x)2
i=1
n
X
≤ λ1 (qiT x)2
i=1

= λ1 kxk2

10
Inequalities

I similar argument shows xT Ax ≥ λn kxk2 , so we have

λn xT x ≤ xT Ax ≤ λ1 xT x

I sometimes λ1 is called λmax , λn is called λmin

I note also that

q1T Aq1 = λ1 kq1 k2 , qnT Aqn = λn kqn k2 ,

so the inequalities are tight

11
Positive semidefinite and positive definite matrices

suppose A = AT ∈ Rn×n

we say A is positive semidefinite if xT Ax ≥ 0 for all x

I this is written A ≥ 0 (and sometimes A  0)

I A ≥ 0 if and only if λmin (A) ≥ 0, i.e., all eigenvalues are nonnegative

I not the same as Aij ≥ 0 for all i, j

we say A is positive definite if xT Ax > 0 for all x 6= 0

I denoted A > 0

I A > 0 if and only if λmin (A) > 0, i.e., all eigenvalues are positive

12
Matrix inequalities

I we say A is negative semidefinite if −A ≥ 0

I we say A is negative definite if −A > 0

I otherwise, we say A is indefinite

matrix inequality: if A and B are both symmetric, we use A < B to mean

B − A > 0.

I many variations, for example A ≥ B means A − B ≥ 0,

I A > B means xT Ax > xT Bx for all x 6= 0

13
Matrix inequalities

many properties that you’d guess hold actually do, e.g.,

I if A ≥ B and C ≥ D, then A + C ≥ B + D

I if B ≤ 0 then A + B ≤ A

I if A ≥ 0 and α ≥ 0, then αA ≥ 0

I A2 ≥ 0

I if A > 0, then A−1 > 0

matrix inequality is only a partial order: we can have

A 6≥ B, B 6≥ A

(such matrices are called incomparable)

14