Introduction QP Methods Final Remarks
Lecture 4: Quadratic Programming
Prof. Marcelo Escobar
PPGEQ – UFRGS
Prof. Jorge Otávio Trierweiler
September 11, 2011
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Outline
1 Introduction
2 QP Methods
Equality Constrained QP
General QP
3 Final Remarks
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Basic Concepts
Quadratic Programming
1 T
minimize x Hx + c T x
f (x) :
x 2
subject to h(x) : Ax = b
g(x) : Cx ≤ d
where:
f (x): quadratic objective function
h(x): linear equality constraints
g(x): linear inequality constraints
Sufficient Conditions for Unique Optimum:
Convexity - H must be positive (semi)-definite logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Basic Concepts
Quadratic Programming: Geometry
x*
x*
The solution can be either inside the feasible region or at the border, It
is not a vertex point!
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
Quadratic Programming (QP)
Consider the equality constrained mathematical program:
minimize x T Hx + c T x
x
subject to Ax = b
H: symetric matrix n × n
A: a matrix m × n
x: a vector n × 1
Note: Any square matrix M can be converted in symetric matrix
H = (M + M T )/2 and x T Hx = x T Mx! logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
Quadratic Programming (QP)
Consider the equality constrained mathematical program:
1 T
minimize x Hx + x T c
x 2
subject to Ax = b
Lagrangean Function:
1 T
L(x, λ) = x Hx + x T c + (Ax − b)T λ
2
First order necessary conditions (KKT):
∇x L(x, λ) = 0 ∇λ L(x, λ) = 0
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
KKT System
First order necessary conditions:
H AT x
−c
=
A 0 λ b
The system can be rewritten in a form that is useful for computation:
H AT d
−c − Hxk
=
A 0 λ b − Axk
where xk+1 = xk + d
Note: The system is called KKT system and the matrix of coefficients
is the KKT Matrix.! logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
Numerical Example (qp0.m )
minimize 3x12 + 2x1 x2 + x1 x3 + 2.5x22 + 2x2 x3 + 2x32 − 8x1 − 3x2 − 3x3
x
subject to x1 + x3 = 3
x2 + x3 = 0
In a matricial form:
6 2 1 −8
1 0 1 3
H = 2 5 2 c = −3 A= b=
0 1 1 0
1 2 4 −3
The solution x ∗ and optimal Lagrange multiplier vector λ∗ are given by:
2
∗ ∗ 3
x = −1 λ =
−2 logo
1
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
Methods for solving the KKT System
Full Space Method, i.e. direct solution of KKT system
Range-Space Method
Null-Space Method
Note: Exploit the block structure in the KKT system to decouple
it into two smaller systems!
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
Range-Space Method
Perform block elimination on the system. Multiply the first equation by
AH −1 and then subtract the second equation:
(AH −1 AT )λ = (−AH −1 (Hxk + c) + Axk − b)
Solve this symetric system for λ∗ and recover d from the first equation:
Hd = −AT λ∗ − (Hxk + c)
This method is most useful when:
H is well conditioned and easy to invert;
H −1 is known explicitly through a Quasi-Newton Method;
the number of equality constraints m is small.
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
Null-Space Method
Suppose that we partition the vector d into two components:
d = YdY + ZdZ
where Z is the n ×
(n − m) null-space matrix (AZ = 0), and Y is any n × m
matrix such that Y | Z is nonsingular. By substituting d into the second
equation of the KKT system:
AYdY = −(Axk − b)
From the first equation:
HYdY + HZdZ + AT λ = −(Hxk + c)
multiplty this equation by Z T to obtain:
Z T HZdZ = −Z T HYdY − Z T (Hxk + c)
To obtain the Lagrange multiplier, we multiply the first equation by Y T :
logo
(AY )T λ = Y T (Hxk + c + Hd)
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
Null-Space Method: Example (qp0a.m )
Consider the previous numerical example:
−1 2 −1
1
Z = 1 Y = −1 2
3
1 1 1
For this particular choice of Y , we have AY = I . Suppose that we have
x T = 0, 0, 0 . Then:
T
dY = 3, 0 dZ = 0
so that T
d = YdY + ZdZ = 2, −1, 1
After recovering λ and x:
T T
x ∗ = 2, −1, 1 λ∗ = 3, −2
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Equality Constrained QP
Null-Space: Numerical Computation
Find the n × (n − m) matrix Z such that:
AZ = 0
QR Factorization:
AT = Q1
Q2 R
where Q1 is n × m and Q2 is n × (n − m)
The columns of Q2 span the null space of A:
Z = Q2 logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Inequality constrained Problem
Quadratic Programming:
1
minimize cT x + xT H x
x 2
subject to Cx = d
Ax ≤ b
Methods for general QP:
Classical active-set method (most widely used);
Gradient Projection Method;
Interior Point Method;
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Inequality constrained Problem
Quadratic Programming:
1
minimize cT x + xT H x
x 2
subject to Cx = d
Aj x = bj , j ∈ A(x ∗ )
Lagrangean Function:
1
L(x, λ, µ) = c T x + x T H x + λ(Cx − d) + µ(AA x − bA )
2
Note: A(x ∗ ) is the active set at the optimal point x ∗ which is unknown
in advance! logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
QP Subproblem
Quadratic Programming Subproblem at Iteration k:
1
minimize (Hxk + c)T dx + dxT H dx
dx 2
subject to Cdx = d − Cxk
Aj dx = bj − Aj xk , j ∈ Wk
where dx = x − xk and Wk is the working set at iteration k.
For a given point xk and the working set Wk solve the KKT system:
H C T AT
j dx −Hxk − c
C 0 0 λ = d − Cxk
Aj 0 0 µ b j − A j xk logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Line Search
Suppose that dx is nonzero:
xk+1 = xk + αk dx αk ∈ (0, 1]
decide how far to move along this direction retaining feasibility:
ajT (xk + αk dx ) ≤ bj , ∀j ∈
/ Wk
This constraint will be satisfied for all nonengative choices of αk only if:
bj − ajT xk
αk ≤
ajT dx
Since we want αk to be as large as possible in (0, 1]:
bj −aT xk
αk = min 1, min aT dj x
j ∈W
/ k j
Note: Blocking constraint(s) for which αk is defined must be added to the logo
working set for the next iteration!
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Active-Set Method for Convex QP: Algorithm
0. Compute a feasible starting point x0 , Set W0 to be a subset of the active
constraints at x0 ;
1. Solve QP subproblem to find dx
1 T
minimize (Hxk + c)T dx + d H dx
dx 2 x
subject to Cdx = d − Cxk
aj dx = bj − aj xk , j ∈ Wk
2. If dx = 0, compute Lagrange Multipliers
If µ ≥ 0, STOP with solution x ∗ = xk ;
else drop one constraint with wrong sign (µ < 0)
xk+1 = xk ; Wk+1 ← Wk − j;
3. If dx 6= 0, Compute αk and xk+1 ← xk + αk dx ;
If there are blocking constraints, add one blocking constraint to Wk
else Wk+1 ← Wk ; logo
4. Set k = k + 1 and Go to 2
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Active-Set Method: Example 01
minimize (x1 − 1)2 + (x2 − 2.5)2
x1 ,x2
subject to g1 (x) : x1 − 2x2 + 2 ≥ 0
g2 (x) : −x1 − 2x2 + 6 ≥ 0
g3 (x) : −x1 + 2x2 + 2 ≥ 0
g4 (x) : x1 ≥ 0
g5 (x) : x2 ≥ 0
Suppose that we choose x0 = (2, 0). Constraints 3 and 5 are active at this
point, and we set W0 = 3, 5 .
Note: We could have chosen W0 = 3 , or W0 = 5 or even W0 = ∅, each logo
would lead the algorithm to perform quite different.
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Active-Set Method: Example 01 (qp01.m )
Iteration 0: dx = (0, 0)T and (µ∗3 , µ∗5 ) = (−2, −1)
Remove the constraint 3 from the active set. Set W1 = 5
Iteration 1: dx = (−1, 0)T , α1 = 1, then x2 = (1, 0)T
There is no blocking constraint, thenW2 = W1 = 5
Iteration 2: dx = (0, 0)T , and µ∗5 = −5
Remove the constraint 5 from the active set. Set W3 = ∅
Iteration 3: dx = (0, 2.5)T , and α3 = 0.6, then x4 = (1, 1.5)T
Add the blocking constraint 1 to the active set. Set W4 = 1
Iteration 4: dx = (0.4, 0.2)T , and α3 = 1, then x5 = (1.4, 1.7)T
There is no blocking constraint, thenW5 = W4 = 1 .
Iteration 5: dx = (0, 0)T , and µ∗1 = 1.25
logo
This is the SOLUTION!!!
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Active-Set Method: Example 02
minimize x12 + x22 − 4x1 − 4x2
x1 ,x2
subject to g1 (x) : −2x1 + x2 + 2 ≥ 0
g2 (x) : −x1 + x2 + 1 ≥ 0
g3 (x) : x1 + x2 + 1 ≥ 0
g4 (x) : 2x1 − x2 + 2 ≥ 0
Suppose that we choose x0 = (−1, 0). Constraints 3 and 4 are active at this
point, and we set W0 = 3, 4 .
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Active-Set Method: Example 02 (qp02.m )
Iteration 0: dx = (0, 0)T and (µ∗3 , µ∗4 ) = 31 (−14, −2)
Remove the constraint 3 from the active set. Set W1 = 4
Iteration 1: dx = (1.4, 2.8)T , α1 = 0.71, then x2 = (0, 2)T
Add the blocking constraint 1, then W2 = 1, 4
Iteration 3: dx = (0, 0)T , and (µ∗1 , µ∗4 ) = (1, −1)
Remove the constraint 4 from the active set. Set W4 = 1
Iteration 4: dx = (0.4, −0.8)T , and α4 = 1, then x5 = (0.4, 1.2)T
There is no blocking constraint, thenW5 = W4 = 1 .
Iteration 5: dx = (0, 0)T , and µ∗1 = 1.6
This is the SOLUTION!!!
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
General QP
Final Remarks: Active Set Method
Different initial working set W0 leads to a different iteration
sequence;
Different choices of the blocking constraint to remove leads to a
different iteration sequence;
Selecting the constraints with the most negative µ always works
well in practice;
The working set changes slowly along iterations;
This algorithm has a finite termination!
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Final Remarks
Final Remarks
For indefinite QP the direction must be corrected as a descent
direction (direction of negative curvature) in order to avoid
convergence to a saddle point
The Gradient Projection Method is based on variable elimination.
It turns out to be Reduced Gradient Method and will be
presented for general NLP;
The Interior Point Method will be presented for general NLP;
Main Application in Chemical Engineering: Model Predictive logo
Control.
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Final Remarks
Further Readings
Numerical Optimization. Nocedal
and Wright (1999): Chapter 16.
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
Introduction QP Methods Final Remarks
Final Remarks
Help, Comments, Suggestions
logo
Prof. Marcelo Escobar Optimization of Chemical Processes
This presentation is over
Thank You for your attention!!!!
What you want to do now:
restart presentation
quit
