Nonlinear Optimization: Introduction
Linear vs. nonlinear objective functions
Linear Nonlinear
When objective function is linear Optimum always attained at constraint boundaries A local optimum is also a global optimum When objective function is nonlinear Optima may be in the interior as well as at boundaries of constraints A local optimum is not necessarily a global optimum
September 12, 2007 (1 : 6)
Nonlinear Optimization: Introduction
Convexity and optimality
f(x)
A convex function f on a convex set S : f is below or on the linear interpolant between any two points in the set x1 , x2 S
)+ (x 1 f
x1
( )f 1
x 2)
x2
f x1 + (1 )x2 f (x1 ) + (1 )f (x2 ) [0, 1] Examples of convex functions on R n (a) f (x ) = c T x (also concave!) T (b) f (x ) = + c T x + 1 2 x Qx (Q positive semidenite matrix)
September 12, 2007 (2 : 6)
Nonlinear Optimization: Introduction
S Global minimizer: x S such that f (x ) f (x ) x S Local minimizer: x S such that, for some > 0, f (x ) f (x ) x S B (x ; ) where B (x ; ): ball of radius
Rn
f(x)
Local Local
Global
centered at x
Theorem. For convex functions on convex sets holds that each local minimizer is a global minimizer
Example: min f (x ) such that
x
Ax b , where f (x ) = c T x (linear program) or T f (x ) = + c T x + 1 2 x Qx , Q positive semidenite (quadratic program)
September 12, 2007 (3 : 6)
Nonlinear Optimization: Introduction
Proof. Let x be a local minimizer. Then > 0 s. t. f (x ) f (x ) x S B (x ; ). Let x S arbitrarily. Will show f (x ) f (x ). S convex x + (1 )x = x S x local min f (x ) f (x ) = f x + (1 )x [f convex] f (x ) + (1 )f (x ), that is, f (x ) f (x ) f (x ) f (x )
[0, 1]
Choose > 0 such that x S B (x ; )
>0
September 12, 2007
(4 : 6)
Nonlinear Optimization: Introduction
Unconstrained optimization
Will start to consider unconstrained optimization min f (x ) n or, equivalently, Find x R n such that f (x ) f (x ) x R n
x R
Function f is nonlinear in x . Unconstrained optimization meaningless for linear f , since linear f on R n are unbounded or constant Most common application for unconstrained optimization: inverse problems, parameter estimation
September 12, 2007
(5 : 6)
Nonlinear Optimization: Introduction
Unconstrained optimization: example application
At time intervals t1 , t2 , . . . , tm a physical process generates a time sequence of m observations (measurements) b1 , b2 , . . . , bm A model of the process says that bk b (tk ) where b (t ) = x1 + x2 e x3 t + x4 e x5 t Model not exact: measuring errors (noise), modeling errors We want to nd the coecients x1 , . . . , x5 that best matches the observations Dene the residual rk = bk (x1 + x2 e x3 tk + x4 e x5 tk ) and solve the unconstrained optimization problem
m x1 ,...,x5
min
2 rk k =1
September 12, 2007
(6 : 6)
