INTERMEDIATE METHODS:

OPTIMISATION
QUADRATIC FORM + DEFINITENESS
 You can express a quadratic function in a matrix form by using a
symmetric matrix:
 Q(x) = xT . A . X
 An example is show below.
a11 x12  a12 x1 x2  a22 x22
 a11 1
a12   x1 
  x1 x2    1 2
   
 2 a12 a22   x2 
 Consider the quadratic form y=ax2.
 If a > 0, y ≥ 0, and the quadratic form is POSITIVE DEFINITE.
 If a < 0, y ≤ 0, and the quadratic form is NEGATIVE DEFINITE.
 If the quadratic form can be either negative or positive (except for when at the
origin), it is INDEFINITE.
 If y is always ≥ 0, but is zero when x ≠ 0, it is POSITIVE SEMIDEFINITE.
 E.g., (x + y)2
 If y is always ≤ 0, but is zero when x ≠ 0, it is NEGATIVE SEMIDEFINITE.
 E.g., -(x + y)2
PRINCIPAL MINORS
 To determine the Definiteness of a Matrix, we must use principal
minors:
 Given an n x n Matrix A, we want a ‘k’ x ‘k’ submatrix.
 A principal minor is calculated by deleting a certain number (‘n - k’) of
columns, say column 1 and column 2 from a 3 x 3 matrix, from a square
matrix, and deleting the corresponding number of rows, so row 1 and row 2.
 The resulting Matrix is the ‘kth’ Principal Submatrix of A. In the above case,
we’d have a 1st order Principal Submatrix.
 A Principal Minor is calculated by taking the determinant of the Principal
Submatrix.
 We are only interested in the LEADING principal minors, however.
 A leading principal submatrix is formed by deleting only the LAST ‘n- k’
columns and rows from a square matrix.
 For example, a 3 x 3 matrix where we deleted rows 1 + 3 and columns 1 + 3
would be a principal submatrix, but not a LEADING principal submatrix.
 The leading principal Minor is the determinant of the respective leading principal
submatrix.
LEADING PRINCIPAL MINORS +
DEFINITENESS
 If we call the leading Principal Minors
‘Ak’, where ‘k’ is the order of the leading
principal minor, then the following is
true:
 The symmetric Matrix A is positive
definite if, and only if, ALL it’s leading
principal minors are > 0.
 It is negative definite if, and only if, it’s a11  1st order leading principal matrix
‘n’ leading principal minors
ALTERNATE in sign as follows: a11 a12
 A1 <0 , A2 > 0, A3 < 0 ..., where the ‘kth’  2 nd order leading principal matrix
order leading principal minor has the a21 a22
same sign as (-1)k.
 If a leading principal minor is nonzero, but a11 a12 a13
does not fit the above two criteria, the
matrix is indefinite. a 21 a22 a23  3rd order leading principal matrix
 However, if a leading principal minor is a31 a32 a33
zero, yet the other leading principal
minors appear to fit one of the first two
criteria, then we must look at ALL the
principal minors of the matrix.
 If ALL the principal minors of A are ≥
0, A is POSITIVE SEMIDEFINITE.
 If ALL even ordered principal minors
of A are ≥ 0, and ALL ODD ordered
principal minors of A are ≤ 0, A is
NEGATIVE SEMIDEFINITE.
  a11  a1n  x1 
  
(1)   x
THE BORDERED MATRIX 1  xn        
 a  a  x 
 n1 nn  n 
 When you want to discover whether the
answer to a quadratic equation with ‘n’
variables subject to ‘m’ linear constraints is a
global maxima or a global minima, you must
 B11  B1n  x1 
ascertain the definiteness of the BORDERED
matrix.   
 Note: This won’t tell you the ANSWER to the
(2)        
constrained optimization problem. B  
 To border a matrix, consider the following  m1  Bmn  xn 
problem (on the right hand side):
 (1) is a Quadratic function in the Form
Q(x)=xT Ax
 (2) is a set of ‘m’ constraints, each of the form  0  0 B11  B1n 
Bi1.x1+Bi2.x2+...+Bin.xn  
 The resulting matrix is an (n + m) x (n + m)       
matrix.  0  0 Bm1  Bmn 
 DEFITENESS CONDITIONS:
(3)   
Check the last ‘n – m’ leading principal
 B11  Bm1 a11  a1n 

minors.
 NEGATIVE DEFINITE: If det H has the        
same sign as (-1)n and the last n-m leading

principal minors ALTERNATE in sign.
B  Bmn an1  ann 
 POSITIVE DEFINITE: If det H and the last
n-m L.P.M all have the SAME sign as (-1)m.
 1n
 0 B
  T H
B A 
AND IN ENGLISH...
 You have to check the LARGEST leading principal minors.
 So if you’ve got a 6 x 6 Bordered Matrix with 4 variables and
2 constraints (n + m = 4 + 2 = 6), you have to check the
largest n – m = 4 – 2 = 2 leading principal minors.
 Hence, you evaluate the determinant of H6 and H5.
 For only one constraint, there is a slightly easier to
remember approach:
 Form the usual (n + 1) X (n + 1) bordered matrix
 If the last ‘n’ matrices are the same sign, the function is positive
definite, if they alternate, the function is negative definite.
 If a function is +ve definite, then the solution x = 0 will be a
constrained MIN, and vice versa for –ve definite.
THE HESSIAN
The application of Definiteness to
 2F 2F 

optimisation problems allows us to
ascertain whether the solution is a local   
minimum or a local maximum.
  x1
2
xn x1 
 Let us start with UNCONSTRAINED
OPTIMISATION.     H
 To determine if the critical point to an
unconstrained optimisation problem,
 2F 2F 
simply set up a matrix of second-order   2 
derivatives  This is called the Hessian  x1xn xn 
Matrix, and is shown to the side.  
 To determine whether or not you have a
local max or min, compute the
 F11  Fn1 
Definiteness of the matrix:
 
 H
 If Positive Definite  strict local min.
 If Negative Definite  strict local max.   
If Indefinite  Neither!
F  Fnn 

 In the case of semidefinite conclusions,
the ultimate conclusion remains the same:  1n
+ve semidefinite is a local min and vice
versa.
THE BORDERED HESSIAN
 CONSTRAINED OPTIMISATION
 h1 h1 
 To test for minima and maxima for a solution  0  0  
derived from the constrained optimisation  x1 xn 
problem, one needs to set up a Bordered
Hessian matrix        
 This is very similar to a normal bordered  hk hk 
matrix, as we’ve already encountered, and is
shown to the side.
 0  0
x1

xn 
 Assume optimizing a function F, under ‘k’  H
constraints  h1, h2, ... ,hk  h1  hk 2L 2L 

 Therefore, it differs slightly from the  x1 x1 x12 xn x1 
unconstrained optimisation principle as you are
not taking a derivative of the objective        
function, but rather the LAGRANGE set up. 
 h1  hk  L 2L 
2
 However, due to the envelope theorem, they are 
equal anyway.  x xn x1 xn xn2 
 REMEMBER TO EVALUATE THE  n
Dh x*  
DERIVATIVES AT THE CRITICAL
VALUES.  0
 H
 Standard Definiteness analysis follows: check   Dh x*  T Dx L  x ,   
2 * * 
the determinants of the leading ‘n – k’ 
submatrices.
 IF ONLY ONE CONSTRAINT AND 2
EQUATIONS, then if det H > 0, LOCAL
MAX.