M. Balda LMFnlsq2– Nonlinear least squares 2013-04-15
Let us have a measured values (column vector) y dependent on operational variables
x. We try to make a regression of y(x) by some function f (x, c), where c is a vector of
unknown parameters of the model. The difference
f(x, c) − y(x) = r(x, c) (1)
is a column vector of so called residuals r(x,c), which should be zero vector, provided our
model f (x,c) is physically perfect and y(x) is also without any measurement deviations.
The method of least squares of residuals searches such vector c, for which sum of
squares S = r
T
r becomes minimum. It follows from the condition of minimum that
∂S(c)/∂c = 2 J
T
r = 2v, (2)
where ∂ is a symbol for partial differential, J is a matrix of partial derivatives of S due
to c, so called Jacobi’s matrix (Jacobian matrix), and r is a vector of residuals.
It is clear that v, r and J are functions of unknown parameters c. In general case,
c may not be obtained in closed form, it is necessary to solve it in iterations. Let c in
(k+1)st step of iteration has the simplest form
c
(k+1)
= c
(k)
+ ∆c
(k)
. (3)
It is possible to assume that the vector of residuals in (k+1)st iteration, provided r is
continuous in c, will have the form given by Taylor’s expansion
r
(k+1)
= r
(k)
+ ∂r
(k)
/∂c
(k)
∆c
(k)
+ ... (4)
After multiplication of the equation by a matrix J
(k)
T
from left, we get the following
equation:
J
(k)
T
r
(k+1)
= J
(k)
T
r
(k)
+ J
(k)
T
J
(k)
∆c
(k)
, (5)
which after introducing
A
(k)
= J
(k)
T
J
(k)
(6)
takes the form
A
(k)
∆c
(k)
− J
(k)
T
r
(k+1)
= −v
(k)
. (7)
This equation (7) is the starting point for a series of methods:
Newton-Raphson:
It assumes that norms or residual vectors are dropping so fast, that the second term
in eqn. (7) may be neglected. Hence,
A
(k)
∆c
(k)
= −v
(k)
(8)
may be used to obtain ∆c
(k)
and then
c
(k+1)
= c
(k)
+ ∆c
(k)
. (9)
There are some modifications of the method, say that only a ∆c
(k)
is added, where a <= 1.
