Introduction to Time Series Analysis. Lecture 3.

1. Sample autocorrelation function

2. ACF and prediction

3. Properties of the ACF

35
 Mean, Autocovariance, Stationarity

A time series {Xt } has mean function µt = E[Xt ]

and autocovariance function

γX (t + h, t) = Cov(Xt+h , Xt )
= E[(Xt+h − µt+h )(Xt − µt )].

It is stationary if both are independent of t.

Then we write γX (h) = γX (h, 0).
The autocorrelation function (ACF) is
γX (h)
ρX (h) = = Corr(Xt+h , Xt ).
γX (0)

2
 Linear Processes

An important class of stationary time series:

∞
X
Xt = µ + ψj Wt−j
j=−∞
2
where {Wt } ∼ W N (0, σw )
and µ, ψj are parameters satisfying
∞
X
|ψj | < ∞.
j=−∞

3
 Linear Processes

∞
X
Xt = µ + ψj Wt−j
j=−∞

Examples:
• White noise: ψ0 = 1.
• MA(1): ψ0 = 1, ψ1 = θ.
• AR(1): ψ0 = 1, ψ1 = φ, ψ2 = φ2 , ...

4
 Estimating the ACF: Sample ACF

Recall: Suppose that {Xt } is a stationary time series.

Its mean is
µ = E[Xt ].
Its autocovariance function is

γ(h) = Cov(Xt+h , Xt )
= E[(Xt+h − µ)(Xt − µ)].

Its autocorrelation function is

γ(h)
ρ(h) = .
γ(0)

5
 Estimating the ACF: Sample ACF

For observations x1 , . . . , xn of a time series,

n
1X
the sample mean is x̄ = xt .
n t=1
The sample autocovariance function is
n−|h|
1 X
γ̂(h) = (xt+|h| − x̄)(xt − x̄), for −n < h < n.
n t=1

The sample autocorrelation function is

γ̂(h)
ρ̂(h) = .
γ̂(0)

6
 Estimating the ACF: Sample ACF

Sample autocovariance function:

n−|h|
1 X
γ̂(h) = (xt+|h| − x̄)(xt − x̄).
n t=1

≈ the sample covariance of (x1 , xh+1 ), . . . , (xn−h , xn ), except that

• we normalize by n instead of n − h, and
• we subtract the full sample mean.

7
Sample ACF for white Gaussian (hence i.i.d.) noise

1.2

0.8

0.6

0.4

0.2

−0.2
−20 −15 −10 −5 0 5 10 15 20 Red lines=c.i.

8
 Sample ACF

We can recognize the sample autocorrelation functions of many non-white

(even non-stationary) time series.

Time series: Sample ACF:

White zero
Trend Slow decay
Periodic Periodic
MA(q) Zero for |h| > q
AR(p) Decays to zero exponentially

9
 Sample ACF: Trend

−1

−2

−3

−4
0 10 20 30 40 50 60 70 80 90 100

10
 Sample ACF: Trend

1.2

0.8

0.6

0.4

0.2

−0.2
−60 −40 −20 0 20 40 60 (why?)

11
 Sample ACF

Time series: Sample ACF:

White zero
Trend Slow decay
Periodic Periodic
MA(q) Zero for |h| > q
AR(p) Decays to zero exponentially

12
 Sample ACF: Periodic

−1

−2

−3

−4
0 10 20 30 40 50 60 70 80 90 100

13
 Sample ACF: Periodic

6
signal
signal plus noise
5

−1

−2

−3

−4
0 10 20 30 40 50 60 70 80 90 100

14
 Sample ACF: Periodic

0.8

0.6

0.4

0.2

−0.2

−0.4

−0.6

−0.8
−100 −80 −60 −40 −20 0 20 40 60 80 100 (why?)

15
 Sample ACF

Time series: Sample ACF:

White zero
Trend Slow decay
Periodic Periodic
MA(q) Zero for |h| > q
AR(p) Decays to zero exponentially

16
 ACF: MA(1)

MA(1): X = Z + θ Z
t t t−1

0.8

0.6

0.4 θ/(1+θ2)

0.2

0
−10 −8 −6 −4 −2 0 2 4 6 8 10

17
 Sample ACF: MA(1)

1.2
ACF
Sample ACF

0.8

0.6

0.4

0.2

−0.2
−10 −8 −6 −4 −2 0 2 4 6 8 10

18
 Sample ACF

Time series: Sample ACF:

White zero
Trend Slow decay
Periodic Periodic
MA(q) Zero for |h| > q
AR(p) Decays to zero exponentially

19
 ACF: AR(1)

AR(1): X = φ X +Z
t t−1 t
1

0.9

0.8

0.7

0.6

0.5

0.4

0.3
φ|h|
0.2

0.1

0
−10 −8 −6 −4 −2 0 2 4 6 8 10

20
 Sample ACF: AR(1)

1.2
ACF
Sample ACF

0.8

0.6

0.4

0.2

−0.2
−10 −8 −6 −4 −2 0 2 4 6 8 10

21
 Introduction to Time Series Analysis. Lecture 3.
1. Sample autocorrelation function
2. ACF and prediction

3. Properties of the ACF

22
 ACF and prediction

−1
white noise
MA(1)
−2

−3
0 2 4 6 8 10 12 14 16 18 20

1.2
ACF
1 Sample ACF

0.8

0.6

0.4

0.2

−0.2
−10 −8 −6 −4 −2 0 2 4 6 8 10

23
 ACF of a MA(1) process

5 5 5 5

0 0 0 0

−5 −5 −5 −5
−5 0 5 −5 0 5 −5 0 5 −5 0 5
lag 0 lag 1 lag 2 lag 3

24
 ACF and least squares prediction

Best least squares estimate of Y is EY :

min E(Y − c)2 = E(Y − EY )2 .

c

Best least squares estimate of Y given X is E[Y |X]:

2 2
 
min E(Y − f (X)) = min E E[(Y − f (X)) |X]
f f
2
 
= E E[(Y − E[Y |X]) |X]
= var[Y |X].

Similarly, the best least squares estimate of Xn+h given Xn is

f (Xn ) = E[Xn+h |Xn ].

25
 ACF and least squares prediction

Suppose that X = (X1 , . . . , Xn+h ) is jointly Gaussian:

 
1 1 ′ −1
fX (x) = exp − (x − µ) Σ (x − µ) .
(2π)n/2 |Σ|1/2 2
Then the joint distribution of (Xn , Xn+h ) is
   
µn σn2 ρσn σn+h
N  ,  ,
2
µn+h ρσn σn+h σn+h

and the conditional distribution of Xn+h given Xn is

 
σn+h 2
N µn+h + ρ (xn − µn ), σn+h (1 − ρ2 ) .
σn

26
 ACF and least squares prediction

So for Gaussian and stationary {Xt }, the best estimate of Xn+h given
Xn = xn is
f (xn ) = µ + ρ(h)(xn − µ),
and the mean squared error is

E(Xn+h − f (Xn ))2 = σ 2 (1 − ρ(h)2 ).

Notice:
• Prediction accuracy improves as |ρ(h)| → 1.
• Predictor is linear: f (x) = µ(1 − ρ(h)) + ρ(h)x.

27
 ACF and least squares linear prediction

Consider a linear predictor of Xn+h given Xn = xn . Assume first that

{Xt } is stationary with EXn = 0, and predict Xn+h with f (xn ) = axn .
The best linear predictor minimizes
2 2

2 2


E (Xn+h − aXn ) = E Xn+h − E (2aXn+h Xn ) + E a Xn
= σ 2 − 2aγ(h) + a2 σ 2 ,

and this is minimized when a = ρ(h), that is,

f (xn ) = ρ(h)Xn .

For this optimal linear predictor, the mean squared error is

E(Xn+h − f (Xn ))2 = σ 2 − 2ρ(h)γ(h) + ρ(h)2 σ 2

= σ 2 (1 − ρ(h)2 ).

28
 ACF and least squares linear prediction

Consider the following linear predictor of Xn+h given Xn = xn , when

{Xn } is stationary and EXn = µ:

f (xn ) = a(xn − µ) + b.

The linear predictor that minimizes

2
E (Xn+h − (a(Xn − µ) + b))

has a = ρ(h), b = µ, that is,

f (xn ) = ρ(h)(Xn − µ) + µ.

For this optimal linear predictor, the mean squared error is again

E(Xn+h − f (Xn ))2 = σ 2 (1 − ρ(h)2 ).

29
 Least squares prediction of Xn+h given Xn

f (Xn ) = µ + ρ(h)(Xn − µ).

E(f (Xn ) − Xn+h )2 = σ 2 (1 − ρ(h)2 ).

• If {Xt } is stationary, f is the optimal linear predictor.

• If {Xt } is also Gaussian, f is the optimal predictor.
• Linear prediction is optimal for Gaussian time series.
• Over all stationary processes with that value of ρ(h) and σ 2 , the optimal
mean squared error is maximized by the Gaussian process.
• Linear prediction needs only second order statistics.
• Extends to longer histories, (Xn , Xn − 1, . . .).

30
 Introduction to Time Series Analysis. Lecture 3.
1. Sample autocorrelation function
2. ACF and prediction

3. Properties of the ACF

31
 Properties of the autocovariance function

For the autocovariance function γ of a stationary time series {Xt },

1. γ(0) ≥ 0, (variance is non-negative)

2. |γ(h)| ≤ γ(0), (from Cauchy-Schwarz)

3. γ(h) = γ(−h), (from stationarity)

4. γ is positive semidefinite.

Furthermore, any function γ : Z → R that satisfies (3) and (4) is the

autocovariance of some stationary time series.

32
 Properties of the autocovariance function

A function f : Z → R is positive semidefinite if for all n, the matrix Fn ,

with entries (Fn )i,j = f (i − j), is positive semidefinite.
A matrix Fn ∈ Rn×n is positive semidefinite if, for all vectors a ∈ Rn ,

a′ F a ≥ 0.

To see that γ is psd, consider the variance of (X1 , . . . , Xn )a.

33
 Properties of the autocovariance function

For the autocovariance function γ of a stationary time series {Xt },

1. γ(0) ≥ 0,
2. |γ(h)| ≤ γ(0),

3. γ(h) = γ(−h),

4. γ is positive semidefinite.
Furthermore, any function γ : Z → R that satisfies (3) and (4) is the
autocovariance of some stationary time series (in particular, a Gaussian
process).
e.g.: (1) and (2) follow from (4).

34