Lecture 8
Math Foundations Team
Introduction
▶ Till now we have discussed about Taylor/Maclaurian series,
Partial Derivatives and Gradients.
▶ Now we are interested in Higher order Derivatives.
▶ Multivariate Taylor Series and its uses in the expansion of a
function with multivariables.
Higher-Order Derivatives
Consider a function f : R2 → R
Notations for Higher-Order Partial Derivatives:
∂2f
∂x 2
: Second Partial Derivative of x w.r.t. x
∂nf
∂x n : nth Partial Derivative of x w.r.t. x
∂2f ∂ ∂f
∂y ∂x = ∂y ( ∂x ) : is the partial derivative obtained by first partial
differentiating with respect to x and then with respect to y.
∂2f ∂ ∂f
∂x∂y = ∂x ( ∂y ) : is the partial derivative obtained by first partial
differentiating by y and then x.
Hessian Matrix
The Hessian is the collection of all second-order partial derivatives.
If f(x, y) is a twice (continuously) differentiable function, then
∂2f ∂2f
∂x∂y = ∂y ∂x i.e., the order of differentiation does not matter, and
the corresponding Hessian matrix
∂2f ∂2f
" #
∂x 2 ∂x∂y
H= ∂2f ∂2f
∂x∂y ∂y 2
is symmetric. The Hessian is denoted as ∇2x,y f (x, y )
Linearization and Multivariate Taylor Series
The gradient ∇f of a function f is often used for a locally linear
approximation of f around x0 :
f (x) ≈ f (x0 ) + (∇x f )(x0 )(x − x0 ) (1)
Here (∇x f )(x0 ) is the gradient of f with respect to x, evaluated
atx0 . Figure illustrates the linear approximation of a function f at
an input x0 . The original function is approximated by a straight
line.
Linearization and Multivariate Taylor Series...
This approximation is locally accurate, but the farther we move
away from x0 the worse the approximation gets. Equation (1) is a
special case of a multivariate Taylor series expansion of f at x0 ,
where we consider only the first two terms. We discuss the more
general case in the following, which will allow for better
approximations.
Multivariate Taylor Series
Consider a function f : RD → R, x → f (x),
x ∈ RD
that is smooth at x0 . When we define the difference vector
δ := x − x0 the multivariate Taylor series of f at (x0 ) is defined as
multivariate Taylor series
∞
X D k f (x0 )
x
f (x) = δk (2)
k!
k=0
where Dxk f (x0 )
is the k th (total) derivative of f with respect to x,
evaluated at x0 .
Taylor Polynomial
The Taylor polynomial of degree n of Taylor polynomial f at x0
contains the first n + 1 components of the series in (2) and is
defined as
n
X Dxk f (x0 ) k
Tn (x) = δ (3)
k!
k=0
In (2) and (3), we used the slightly sloppy notation of δ k , which is
not defined for vectors
x ∈ RD ,
D > 1, and k > 1. Note that both Dxk f and δ k are k th order
tensors, i.e., k-dimensional arrays.
Taylor Polynomial...
Taylor Polynomial...
.
Taylor Polynomial...
In general, we obtain the terms in the Taylor series, where
Dxk f (x0 )δ k contains k th order polynomials.Now that we defined the
Taylor series for vector fields, let us explicitly write down the first
terms Dxk f (x0 )δ k of the Taylor series expansion for
Taylor Series Expansion of a Function with Two Variables
Consider the function f (x, y ) = x 2 + 2xy + y 3 .
We want to compute the Taylor series expansion of f at
(x0 , y0 ) = (1, 2).
Before we start, let us discuss what to expect: The function in
f (x, y ) is a polynomial of degree 3. We are looking for a Taylor
series expansion,which itself is a linear combination of polynomials.
Therefore, we do not expect the Taylor series expansion to contain
terms of fourth or higher order to express a third-order polynomial.
This means that P it shouldk be sufficient to determine the first four
terms of f (x) = ∞ k=0
Dx f (x0 ) k
k! δ for an exact alternative
representation of f (x, y ). To determine the Taylor series
expansion, we start with the constant term and the first-order
derivatives, which are given by f (1, 2) = 13
Taylor Series Expansion of a Function with Two Variables...
Taylor Series Expansion of a Function with Two Variables...
When we collect the second-order partial derivatives, we obtain the
Hessian
Taylor Series Expansion of a Function with Two Variables...
The third-order derivatives are obtained as
Taylor Series Expansion of a Function with Two Variables...
Since most second-order partial derivatives in the Hessian, are
constant, the only nonzero third-order partial derivative is
∂3f ∂3f
∂y 3
= 6 =⇒ ∂y 3 (1, 2) = 6 Higher-order derivatives and the mixed
∂3f
derivatives of degree 3 (e.g., ∂x 2 ∂y
) vanish, such that
Taylor Series Expansion of a Function with Two Variables...
which collects all cubic terms of the Taylor series. Overall, the
(exact) Taylor series expansion of f at (x0 , y0 ) = (1, 2) is
