SCHOOL OF MATHEMATICAL AND COMPUTER

SCIENCES

DEPARTMENT OF STATISTICS AND OPERATIONS

REREARCH

TIME SERIES ANALYSIS

SSTA031

STUDY GUIDE

(2022)

1
SSTA031 STUDY GUIDE 2022

INTRODUCTION
A time series is a collection of observations made sequentially over time. Examples occur
in a variety of fields, ranging from economics to engineering. The module will introduce a
variety of examples of time series from diverse areas of application. Studying models that
incorporate dependence is the key concept in time series analysis

LECTURER INFORMATION:

Name Mr KN Maswanganyi ; Ms M Phaho

Office Mathematical Sciences Building, Room (2004D, 2019)
Telephone 015 268 3680/015 268 2881
E-mail address [email protected]/[email protected]

STUDY COMPONENTS

A. Purpose of the Module

The purpose of the module is:

 To provide students with sufficient background in time series analysis by

choosing techniques suited both to the data-source and the time series model.

 To understand or model the stochastic mechanism that gives rise to an observed

series and,

 To predict or forecast the future values based on the history of that series.

B. CHAPTERS

1. INTRODUCTION TO TIME SERIES

1.1 Learning Outcomes

1.2 Introduction

1.3 Time series plots

1.4 General approach to time series modelling

1.5 Components of time series

1.6 Decomposition of time series

1.7 Smoothing methods

2
SSTA031 STUDY GUIDE 2022

1.8 Trend analysis

1.9 Seasonal analysis

2. THE MODEL BUILDING STRATEGY

2.1 Learning Outcomes

2.2 Introduction

2.3 The Box-Jenkins technique

2.4 Stationary time series

2.5 Filtering

2.6 Stochastic processes

2.7 Means, Variances and Covariances

3. MODELS FOR STATIONARY TIME SERIES

3.1 Learning Outcomes

3.2 Introduction

3.3 Autocorrelation function (ACF)

3.4 Purely random process

3.5 Random walk

3.6 Moving average (MA) processes

3.7 Partial autocorrelation function (PACF)

3.8 Autoregressive (AR) processes

3.9 The dual relation between AR and MA processes

3.10 Yule-Walker equation

3.11 Mixed ARMA processes

3.12 SARMA processes

4. MODEL FOR NON STATIONARY SERIES

4.1 Learning Outcomes

3
SSTA031 STUDY GUIDE 2022

4.2 ARIMA models

4.3 SARIMA models

5. PARAMETER ESTIMATION

5.1 Learning Outcomes

5.2 Introduction

5.3 Method of moments estimation (MME)

5.4 Least square estimation (LSE)

5.5 Confidence interval for mean

6. MODEL DIAGNOSTICS

6.1 Learning Outcomes

6.2 Introduction

6.3 Autocorrelation of residuals

6.4 Overfitting

7. FORECASTING

7.1 Learning Outcomes

7.2 Minimum mean square error (MMSE) Forecast

7.3 Computation of forecasts

7.4 Prediction interval

7.5 Forecasting AR  p  and MAq  models

7.6 Forecasting ARMA p, q  models

7.7 Forecasting ARIMA p, d , q  models

4
SSTA031 STUDY GUIDE 2022

C. ACTIVITIES

LECTURES AND LECTURE TIMES:

You have to attend 3 lectures per week. The times are as follows:

DAY TIME VENUE

WEDNESDAY 11H10-12H50 N40/online

FRIDAY 07H30-08H15 B0006/online

ASSIGNMENTS AND PRACTICALS

You have to attend 2 practical /tutorial sessions per week and shall also be given 2
assignments and 2 Quizzes.

TESTS AND EXAMINATION

2 tests shall be written and the final mark will be obtained from a 2hr comprehensive
written examination.

A student is admitted to the final exam based on a module mark of at least 40%.

CALCULATION OF MARKS

MODULE MARK = 10% ASSIGNMENT + 20% QUIZ AVERAGE+ 70% TEST

AVERAGE
FINAL MARK = 60% MODULE MARK + 40% FINAL EXAM MARK

D. COURSE PLAN:
Your performance on this module is assessed through two written Quizzes, tests
and assignments. The Quizzes are written a week before each test.

5
SSTA031 STUDY GUIDE 2022

This module has a duration of 15 weeks and a tentative course plan is given
hereunder.

SCHEDULE OF LECTURES

Week Topics Chapters

1 Introduction to Time Series 1
2 Introduction to Time Series 1
3 The Model Building Strategy 2
4 Models for Stationary Time Series 3
5 Models for Stationary Time Series 3
6 Models for Stationary Time Series 3
7 Models for Non-Stationary Time Series 4
8 Models for Non-Stationary Time Series 4
9 Models for Non-Stationary Time Series 4
10 Parameter Estimation 5
11 Parameter Estimation 5
12 Model Specification and Diagnostics 6
13 Forecasting 7
14 Forecasting 7
15 REVISION WEEK

IMPORTANT DATES

WEEK DATES CHAPTERS ASSESSMENT

?? ?? 1,2,3 QUIZ 1
?? ?? 1,2,3 TEST 1
?? ?? 4,5,6 QUIZ 2
?? ?? 4,5,6,7 TEST 2
?? ?? 4,5,6,7 ASSIGNMENT 1

6
SSTA031 STUDY GUIDE 2022

E. NOTES AND REFERENCES

Take note of the main aspects (like test and quiz dates) indicated in this study guide
and, also read the headings and sub-headings of the relative material in the textbook
or other study materials mentioned in this study guide.

You are expected to purchase atleast one of the following recommended textbooks:
RECOMMENDED TEXTS

 Chris Chatfield (2004). The Analysis of Time Series, An Introduction, Six Edition,
Chapman & Hall/CRC.
 Cryer, D. C. and Chan, K (2008). Time Series Analysis with Application in R, 2nd
Edition, Springer.

 Wei W.W.S (2006) Time Series Analysis, Univariate and Multivariate Methods, Second
Edition, Pearson Addison Wesley.

SUPPLEMENTARY TEXTS

 Cryer, D. J. (1986) Time Series Analysis, Duxbury Press

 Abraham B. and Ledolter J. (1983) Statistical Methods for Forecasting

 Wiley Series, New York

 Peter J. Brockwell and Richard A.Davis (2002), Introduction to Time Series and
Forecasting ,Second Edition,Springer.

F. LECTURE NOTES, EXAMPLES and ASSIGNMENTS

LECTURE NOTES

A copy of Lecture notes will be available on blackboard. The notes do not include
proofs of stationarity/invertibility , so students are advised to take additional notes.

EXAMPLES
As already noted, there are examples in every chapter to work on. It is very
important that you attempt them. Your understanding of the material you have
studied in the chapter will be greatly improved if you do the examples yourself.
ASSIGNMENTS

You will be given two assignments and make sure that you hand in each assignment
on/before the due date. All work submitted by each student must be an authentic
product of his/her efforts and should not be copied from another student.

7
SSTA031 STUDY GUIDE 2022

CHAPTER 1
INTRODUCTION TO TIME SERIES

1.1 Learning Outcomes:

Upon successful completion of this chapter, student should be able to have a better
understanding of what time series is all about and be able to:

 Define time series.

 Describe the general approaches of time series modelling.
 Identify the components of time series.
 Know the methods for isolating the influence of time series components.
 Calculate the seasonal index using ratio-to moving average method.

1.2 Introduction

What is time series? A time series may be defined as a set of observations of a random
variable arranged in chronological (time) order. We also say it’s a series of observations
recorded sequentially at equally spaced intervals of time.

Why do we need time series?

The aim of time series is “to identify any recurring patterns which could be useful in
estimating future values of the time series”. Time series analysis assumes that the actual
values of a random variable in a time series are influenced by a variety of environmentally
forces operating over time.

Time series analysis attempts to isolate and quantify the influence of these different
environmental forces operating on the time series into a number of different components.
This is achieved through a process known as decomposition of the time series.

Once identified and quantified, these components are used to estimate future values of the
time series. An important assumption in time series analysis is the continuation of past
patterns into the future (i.e. the environment in which the time series occurs is stable.)

Notation:

 The time series is denoted by xt , t  T where T is the index.

8
SSTA031 STUDY GUIDE 2022

1.3 Time Series plots

The most important step in time series analysis is to plot the observations against time. This
graph should show up important features of the series such as a trend, seasonality, outliers
and discontinuities. The plot is vital, both to describe the data and to help in formulating a
sensible model. This is basically a plot of the response or variable of interest  x  against
time t  , denoted xt .

1.4 General Approach to Time Series Modeling

 Plot the series and examine the main features: This is usually done with the aid of
some computer package e.g. SPSS, SAS, etc.

 Reform a transformation of the data if necessary.

 Remove the components to get stationary residuals by differencing the data.(i.e

Replacing the original series xt by yt  xt  xt 1 .

 Choose a model to fit the residuals and do the forecasting

1.5 Component of a time series

One way to examine a time series is to break it into components. A standard approach is to
find components corresponding to a long –term trend, any cyclic behavior, seasonal
behavior and a residual, irregular part.

1.5.1 Trend (T)

A smooth or regular underlying movement of a series over a fairly long period of time. A
gradual and consistent pattern of changes.

1.5.2 Seasonal variation (S)

Movement in a time series which recur year after in some months or quarters with more less
the same intensity.

1.5.3 Cyclical variation (C)

Period variations extending over a long period of time, caused by different factors such as
cycles, recession, depression, recovery, etc.
9
SSTA031 STUDY GUIDE 2022

1.5.4 Irregular variation (I)

Variations caused by readily identifiable special events such as elections, wars, floods,
earthquakes, strikes, etc.

1.6 Decomposition of a time series

The main time series analysis is to isolate the influence of each of the four components of
the actual series. The multiplicative time series model is used to analyse the influence of
each of these four components. The multiplicative model is based on the idea that the actual
values of a time series xt can be found by multiplying the trend component T  by cyclical
component C  , by seasonal index S  and by irregular component I  . Thus, the
multiplicative time series is defined as: x  T  C  S  I .

Another model we can use is the additive model given by: x  T  C  S  I .

1.7 Smoothing methods

Smoothing methods are used in attempting to get rid of the irregular, random component of
the series.

1.7.1 Moving averages:

A moving average (ma) of order M is produced by calculating the average value of a variable
over a set of M values of the series.

1.7.2 Running median:

A running median of order M is produced by calculating the median value of a variable over
a set of M values of the series.

1.7.3 Exponential smoothing:

xˆ t 1   xt  1    xˆ t

Where xˆ t 1  the exponential smoothing forecast at time t.

x̂t  the old forecast.

xt  the actual value (observation) at time t.

10
SSTA031 STUDY GUIDE 2022

1.8 Trend Analysis

The trend in a time series can be identified by averaging out the short term fluctuations in
the series. This will result in either a smooth curve or a straight line.

1.8.1 Methods for trend isolation

 The moving average method: Produces a smooth curve.

 Regression analysis method: Involves fitting a straight line.

1.8.2 Calculating moving average

 The three –year moving total for an observation x would be the sum of the
observation immediately before x , x itself and the observation immediately after x .

 The three- year moving average would be each of these moving totals divided by 3.

 The five-year moving total for an observation x would be the sum of the two
observations immediately before x , x itself and the two observations immediately
after x .The five-year moving average would be each of these moving totals divided
by 5.

Example: 1.1
Australia’s official development assistance (ODA) from 1984-85 until 1992-93 is shown (at
current prices, $ million) in Table 1.

11
SSTA031 STUDY GUIDE 2022

Table 1

Year ODA($
million)

1984-85 1011

1985-86 1031

1986-87 976

1987-88 1020

1988-89 1195

1989-90 1174

1990-91 1261

1991-92 1330

1992-93 1384

(a) Find the three- year moving averages to obtain the trend of the data.

(b) Find the five-year moving averages for the data.

1.8.3 Regression Analysis

A trend line isolates the trend (T) component only. It shows the general direction in which
the series is moving and is therefore best represented by a straight line. The method of least
squares from regression analysis is used to determine the trend line of best fit.

Regression line is defined by: y   0  1 x . If the variable x , are not given, must be coded.

Methods for coding the time variable, x

 The sequential numbering method.

 The zero-sum method.

12
SSTA031 STUDY GUIDE 2022

Example 1.2
Table 2

Year 1977 1978 1979 1980 1981 1982 1983

Y 2 6 1 5 3 7 2

a) Calculate the regression line using sequential numbering method.

b) Calculate the regression line using Zero-sum method.

1.9 Seasonal Analysis

Seasonal analysis isolates the influence of seasonal forces on a time series. The ratio-to-
moving average method is used to measure these influences. The seasonal influence is
expressed as an index number.

1.9.1 The ratio-to-moving average method

Step-1

Identify the trend/cyclical movement.

Step-2

Find seasonal ratio

Step-3

Average the seasonal ratios across corresponding periods within each year.

Step-4

Compute adjusted seasonal indices.

13
SSTA031 STUDY GUIDE 2022

Example: 1. 3
The average daily sales (in litres) of milk at a country store are shown in Table 3 for each of
the years 1983 to 1985.

Table 3

Year Quarter Average daily

sales ( Yt )

1 47.6

2 48.9

1983 3 51.5

4 55.3

1 57.9

1984 2 61.7

3 65.3

4 70.2

1 76.1

1985 2 84.7

3 93.2

4 97.2

(a) Find the four- year moving averages.

(b) Calculate the seasonal index by making use of the ratio-to-moving average method.

14
SSTA031 STUDY GUIDE 2022

CHAPTER 2
THE MODEL BUILDING STRATEGY

2.1 Learning Outcomes:

Upon successful completion of this chapter, students should have a better understanding
on how to develop a model building strategy and be able to:

 Identify, estimate and model validation (model- checking).

 Define the concepts of stochastic processes.

2.2 Introduction

Perhaps the most important question we ask now is “how do we decide on the model to
use?” Finding appropriate models for time series is not an easy task. We will develop a
model building strategy which was developed by Box and Jenkins in 1976.

2.3 The Box-Jenkins Technique

There are three main steps in the Box-Jenkins procedure, each of which may be used
several times:

 Model specification

 Model fitting.

 Model diagnostics.

2.3.1 Model specification

In model specification (or identification) we select classes of time series that may be
appropriate for a given observed series. In this step we look at the time plot of the series,
compute many different statistics from the data, and also apply knowledge from the subject
area in which the data arise, such as economics, physics, chemistry, or biology. The model
chosen at this point is tentative and may be revised later in the analysis. In the process of
model selection we shall try to adhere to the principle of parsimony.

Definition 2.3.1 (The principle of parsimony): The model used should require the smallest
possible number of parameters that will adequately represent the data.

15
SSTA031 STUDY GUIDE 2022

a) Test for white noise

For the data set to be purely random sequence/white noise the sample autocorrelation

2
̂ k  (i.e the sample ACF must be within the boundaries).
n

Example 2.1
200 observations on a stationary series were analyzed and gave the following sample
autocorrelation :

Table 4

k 1 2 3 4 5

̂ k 0.59 0.56 0.46 0.38 0.31

Is the data set a white noise?

2.3.2 Model fitting

Model fitting consists of finding the best possible estimates of those unknown parameters
within a given model. After we have identified the model and estimated the unknown
parameters we need to check if the model is a good model, this is done through diagnostic
checking.

2.3.3 Model Diagnostics

Here we are concerned with analyzing the quality of the model that we have specified and
estimated. We ask the following questions to guide us:

 How well does the model fit the data?

 Are the assumptions of the model reasonably satisfied?

16
SSTA031 STUDY GUIDE 2022

2.4 Stationary time series

Definition 2.4.1: A time series is said to be stationary if there is no systematic change in

mean (no trend), if there is no systematic change in variance, and if strictly periodic
variations have been removed.

2.4.1 Transformations

If the process is not stationary to analyze its series, we must transform the series to be
stationary. There are various transformations that we can use to make a time series
stationary. Some of them are:

 Differencing.

 Log transformation.

 Square root transformation.

 Arcsine transformation.

 Power transformation.

The three main reasons for making a transformation are as

follows:

 To stabilize the variance.

 To make the seasonal effect additive.
 To make the data normally distributed.

2.4.2 Stationarity through differencing

Models that are not stationary when subjected to differencing often yield processes that are
stationary. Thus if we difference a time series we denote it by  xt  xt  xt 1 . We will use the

operator  to denote the difference operation. In some instances, differencing ones may not
yield a stationary process.

17
SSTA031 STUDY GUIDE 2022

2.5 Analyzing Series Which Contain a Trend

2.5.1 Filtering

Definition 2.5.1: A linear filter is a linear transformation or any operator which converts one
time series, xt  into another series yt  through the linear operation.

Example: 2.2

1 1  1 1 
(a) Consider the following two filters: A   ,  , B   ,  Compute A * B where
2 2 2 2
*denotes the convolution operator.

(b) Consider the following two filters: A   1,1, B   1,1Compute A * B where *denotes
the convolution operator.

2.6 Stochastic processes

Definition 2. 6.1: A time series x0 , x1 ,... is a sequence of observation. More technically a

time series is a sample path or realization of a stochastic (random) process xt , t  T where

T is an ordered set.

Definition 2.6.2: Let xt , t  T be a stochastic process and let  be a set of all possible

realization or sample of path then  is called the ensemble for the process xt .

Remarks: In time series literature, the terms “time series” and process are used (often)
interchangeably.

2.7 Means, Variances and Covariances

The mean and autocovariance function (ac.vf) are given by:

 t  xt  and  t  k  xt   t xt  k   t k  respectively.

The variance function  t is defined by var xt    t

2 2

18
SSTA031 STUDY GUIDE 2022

CHAPTER 3
MODELS FOR STATIONARY TIME SERIES

3.1 Learning Outcomes:

Upon successful completion of this chapter, students should be able to:

 Define a strictly stationary, weakly stationary, purely random, IID noise and random
walk processes.
 Identify the properties of AR and MA processes.
 Express and manipulate time series model using Backshift operator (B-notation).
 Calculate ACF and PACF for both AR and MA processes.
 Classify the model as ARMA/ARIMA process.
 Determine whether the process is stationary/invertible or both.

3.2 Introduction

In order to be able to analyze or make meaningful inference about the data generating
process it is necessary to make some simplifying and yet reasonable assumption about the
process. A characteristic feature of time series data which distinguishes it from other types
of data is that the observations are, in general, correlated or dependent and one principal
aim of time series is to study , investigate, explore and model this unknown correlation
structure.

3.2.1 Strictly Stationary Processes

Definition 3.2.1: A time series xt is said to be strictly stationary if the joint density
functions depend only on the relation location of the observations, so that
f xt1 h , xt 2 h ,..., xtk  h   f xt1 , xt 2 ,..., xtk . meaning that xt1h , xt 2h ,..., xtk h  and xt1 , xt 2 ,..., xtk 
have the same joint distributions for all h and for all choices of the time points ti .

3.2.2 Weakly Stationary Processes

Definition 3.2.2: A stochastic process z t is weakly stationary (or of second order

stationary), if both the mean function and the autocovariance function do not depend on time
t .thus,
19
SSTA031 STUDY GUIDE 2022

 t  xt  = (a constant) and  t  k   k = (a constant).Note that  t k  cov( xt , xt k )

Example: 3.1
Prove whether the following process is covariance stationary:

(a) z t   1 A ,where A is a random variable with zero mean and unit variance?
t

3.3 Autocorrelation function of stationary processes

Definition 3.3.1: Suppose a stationary stochastic process xt  has mean  , variance  2 and

 k   k 
acv.f.  k  , then the autocorrelation function (ac.f) is given by:  k   
 0   2

Properties

1.  0   1

2.  k    k  .

3.  k   1

3.4 Purely random process

Definition 3.4.1: A discrete time process is called a purely random process (white noise)
if it consists of a sequence of random variable z t which are mutually independent and
identically distributed. It follows that both mean and variance are constants and the acv.f. is:
 k   covz t , z t  k 
 0 for k  1,2...

1 , for k  0
The ac.f.is given by  k   
0 , for k  1,2,...

20
SSTA031 STUDY GUIDE 2022

Definition 3.4.2 (IID Noise Process)

A process xt  is said to be an IID noise with mean 0 and variance  x ,written xt ~ IID 0,  x
2 2
 
if the random variable xt are independent and identically distributed with xt   0 and

var xt    x2 .

3.5 Random walk

Let z1 , z 2 ,...be independent identically distributed random variables, each with mean 0 and
variance  z2 . The time series that can be observed xt  is called a random walk if it can be

expressed as xt  z1  z 2  z3    zt .................

Backshift operator

The backshift operator is used to express and manipulate time series models. The backshift
operator denoted B on the time index of a series and shifts time back 1 time unit to form a
new series i.e: Bxt   xt 1 .

3.6 Moving average processes

3.6.1 First- order moving average [MA (1)] process

The MA (1) for the actual data, as opposed to deviations to deviation from the mean will be
written as xt    z t  1 z t 1 or xt    z t  1 z t 1 where  is the mean of the series

corresponding to the intercept in the moving average case.

3.6.2 Second –order moving average [MA (2)] process

Invertible conditions

The second order moving average process is defined by xt  zt  1 zt 1   2 zt 2 and is

stationary for all values of 1 and  2 .However , it is invertible only if the roots of the
characteristic equation 1  1 B   2 B 2  0 lie outside the unit circle, that is ,

(i)  2   1  1
(ii)  2   1  1
(iii)  1   2  1
21
SSTA031 STUDY GUIDE 2022

Example: 3.2
Find the ACF of the following process: xt  zt  1 zt 1   2 zt 2

3.6.3 qth-order moving average [ MA (q) ] process.

Definition 3.6.1: Suppose that zt is a purely random process, such that
E z t   0, var z t    z .then a process xt is said to be a moving average process of order
2

q (abbreviated as MA (q)) if xt  z t  1 z t 1  ...   q z t q .Where  i are constants.

Example: 3.3

Consider a MA (q) given by xt  z t  1 z t 1  ...   q z t q .

Find ACVF and ACF of xt .

3.7 Partial autocorrelation function

Definition 3.7.1: The partial correlation (PACF) is defined as the correlation between xt and

xt  k with their linear dependency on the intervening variables xt  k ,..., xt  k 1 removed,

 kk  corr xt , xt  k xt 1 ,..., xt  k 1 .

 0 1 1
 0 1 1  0 2
   2 1 3
Where  11  1 ,  22  1 2 and  33 
 0 1  0 1 2
1  0 1  0 1
 2 1 0

ˆ kk  1
Definition 3.7.2: Standard error for 
n

22
SSTA031 STUDY GUIDE 2022

3.8 Autoregressive processes

Definition 3.8.1: Let z t be a purely random process with mean zero and variance  z .
2

Then a process xt is said to be an autoregressive process of order p if

xt  1 xt 1  ...   p xt  p  z t .

3.8.1 First -order Autoregressive (Markov) process

Example: 3.4
Consider a model xt  1 xt 1  zt where xt is a white noise.

Find the ACVF and ACF.

3.8.2 Second – order autoregressive process.

Stationary condition

The second-order autoregressive process may be written as: xt  1 xt 1  2 xt 2  z t .

For stationarity, the roots of  B   1  1 B  2 B 2  0 must lie outside the unit circle, which
implies that the parameters 1 and  2 must lie in the triangular region:

(i )  2  1  1
(ii)  2  1  1
(iii)  1   2  1

3.8.3 pth-order autoregressive process

xt  1 xt 1  2 xt 2  ...   p xt  p  zt or 1  1 B  2 B 2  ...   p B p xt  z t

Stationary conditions:

The roots Bi , i  1,2..., p of the characteristic equation 1  1 B  2 B 2  ...   P B p  0 must lie

outside the unit circle.

23
SSTA031 STUDY GUIDE 2022

3.9 The dual relation between AR & MA processes

Process ACF PACF Stationary Invertible

condition condition

AR( p ) Damps out Cuts off after lag- Roots of Always

p characteristic invertible
equation
outside unit
circle.

MA( q ) Cuts off after Damps out Always Roots of

lag- q stationary characteristic
equation
outside unit
circle.

ARMA( p, q ) Damps out Damps out Roots of Roots of

characteristic characteristic
equation equation
outside unit outside unit
circle. circle.

3.10 Yule-Walker equation

Yule-Walker equation using the fact that  k     k  for all k is given by

 k   1  k  1  ...   p  k  p  , for all k  0

The general solution is  k   A1 1 k  ...  A p  p k where  i ,are the roots of the auxiliary

equation: y p  1 y p 1  ...   p ; Ai  are constant,  0   1 ,and A i 1

24
SSTA031 STUDY GUIDE 2022

Example: 3.5

1
Consider the AR (2) process given by xt   xt 1  xt  2  z t
4

Show that xt is stationary and then calculate its ACF.

3.11 Mixed ARMA models

Definition 3.11: A mixed autoregressive/moving –average process containing p AR terms

and q MA terms is said to be an ARMA process of order (p, q) and is given by:
xt  1 xt 1  ...   p xt  p  zt  1 zt 1  ...   q zt q .

The backshift operator B, is given by  B xt   B zt where  B ,  B  are polynomials of

order p, q respectively, such that  B   1  1 B  ...   P B p and  B   1  1 B  ...   q B q .

Stationary process

The roots of  B   0 must lie outside the unit circle.

Invertible process

The roots of  B   0 must lie outside the unit circle.

3.11.1 ARMA (1, 1)

xt  1 xt 1  z t  1 z t 1 , using a backshift operator we have 1  1 B xt  1  1 z t .

Stationary and invertible conditions

The process is stationary if  1  1  1 , and invertible if  1  1  1 .

Example: 3.6

1
Consider the following process: xt  xt 1  z t  2 z t 1 .
2

a) Is the process stationary/invertible? b) Find the ACF

25
SSTA031 STUDY GUIDE 2022

3.11.2 Weights ( ,  )

From the relations  1  1 0  1  1  1 and j  1 j 1 for j  1 , we find that the  j

weights are given by  j  1  1 1 j 1 , for j  1, and similarity it is easily seen that

 j  1  1 1 j 1 , for j  1 ,for the stationary and ARMA (1, 1) process.

The  weights or  weights may be obtained directly by division or by equating powers of B

in an equation such as B  B    B  .

Example: 3.7
Find the  weights and  weights for the ARMA (1, 1) process given by

xt  0.5xt 1  z t  0.3zt 1 .

3.12 Seasonal ARMA models

Definition 3.12.1: A process xt  is called a seasonal ARMA process of non-seasonal order

p, q and seasonal component P, Q and a seasonality order S if xt  satisfies

 B B  xt   B B  z t .

26
SSTA031 STUDY GUIDE 2022

CHAPTER 4
MODEL FOR NON STATIONARY SERIES

4.1 Learning Outcomes:

Upon successful completion of this chapter, students should be able to:

 Identify the time series model as Seasonal ARIMA models.

 Fit SARIMA model.
 Calculate ACF for SARIMA model.

4.2 ARIMA models

A series xt is said to follow Integrated Autoregressive- Moving Average (ARIMA) model if

the d th difference wt   xt is stationary ARIMA process. If wt is ARMA p, q  , we say that

d

xt is ARIMA p, d , q  .

Example: 4.1
Identify the following models

a) 1  0.8B 1  B  xt  zt .

b) 1  B  xt  1  0.75zt .

4.3 Non stationary seasonal processes

4.3.1. SARIMA model

Definition 4.3.1: A process xt  is called a seasonal ARIMA process of non-seasonal

component p, d , q and seasonal components P, D, Q if xt

satisfies  B  B   s xt   B  B  z t .

d D

27
SSTA031 STUDY GUIDE 2022

Example: 4.2
Let us consider the ARIMA0,1,1  0,1,112 model.

 
12
 12

where , Wt  1  B  1  B xt  1  B  1  B z t .

Find the autocovariance and the autocorrelations of Wt .

CHAPTER 5
PARAMETER ESTIMATION

5.1 Learning Outcomes:

Upon successful completion of this chapter, students should be able to:

 Estimate the model parameters.

 Derive the least squares estimates for autoregressive process.
 Estimate the sample partial ACF.
 Practically apply your knowledge to identify a model for the series.
 Calculate the confidence interval for mean.

5.2 Introduction

Having tentatively specified ARIMA p, d , q  the next step is to estimate the parameters of
this model. This chapter focuses on estimation of parameters of an AR and MA models. We
shall deal with the most commonly used method of estimating parameters, these are:

 Method of moments.

 Least square method

 Maximum-likelihood method.

28
SSTA031 STUDY GUIDE 2022

5.3 Method of moments

This method consist of equating sample moments such as the sample mean x ,sample
variance  0 and sample autocorrelation function to the theoretical counterparts and solving
the resultant equation(s).

5.3.1 Mean

With only a single realization (of length n ) of the process, a natural estimator of the mean,
1 n
 is the sample mean x   xt .where
n t 1
x is the average time average of n observation.

1 n
Since E x    E xt    , x is an unbiased estimator of 
n t 1

If xt  is a stationary process with autocorrelation function  k  then

 0  n 1
 k 
var x   1  2  1    k 
n  k 1  n 

ESTIMATION OF k and k

Suppose that we have n observations x1, x2 ,, xn then the corresponding sample

autocovariance and autocorrelation functions (or estimates) at lag k are:

nk

1 nk  xt  x xt  k  x 

ˆk   xt  x xt  k  x  and ̂ k  t 1 n

 x  x
n t 1 2
t
t 1

n 1

1 n 1  xt  x xt 1  x 
ˆ
As an example, 1    xt  x  xt 1  x  ̂ 
and 1 t 1 n are used to estimate  1
 xt  x 2
n t 1
t 1

and 1 .

29
SSTA031 STUDY GUIDE 2022

Example: 5.1

Suppose that xt  is a MA1 process defined by xt  z t   z t 1 where  is a fixed parameter

 
and z t ~ IID 0,  Z . Where IID stand for Independent Identically Distributed noise.
2

Calculate var  x  in terms of  .

5.3.2 Autoregressive model

Example: 5.2

Consider an AR 1 model: xt  1 xt 1  zt Find the estimate of 1 using the method of

moments.

5.3.3 Moving average models

Method of moments is not convenient when applied to moving average models. However,
for purposes of illustration, we shall consider MA1 process given in the following example:

Example: 5.3

Consider an MA1 model: xt  zt  1 zt 1 Find the estimate of 1 using the method of

moments.

5.4 The Least Square Estimation (LSE)

The method of Least Square Estimation is an estimation procedure developed for standard
regression models. In this section we discuss LSE procedure and its associated problems
in time series analysis.

Recall: For sample linear regression model given by: yt   xt  zt , t  1,2,..., n

n  xt y t   xt  y t
The Least Square Estimate is given by: ˆ 
n xt2   xt 
2

30
SSTA031 STUDY GUIDE 2022

In the next subsection we shall apply the LSE method to time series model.

5.4.1 Autoregressive models

Considering an AR 1 model: xt  1 xt 1  zt ….*

Model *can be viewed as a regression model with predictor variable xt 1 and responded

variable xt . On LSE method we minimize the sum of squares of the difference xt  xt 1 that

is S     z t2   xt   xt 1 
2

Consider the minimization of S   with respect to  we have

d S  
 2 xt   xt 1  xt 1 ...............* *
d  


Setting this equal to zero and solving for  yields  2 xt  ˆ xt 1 xt 1  0  ˆ   x t 1 xt
.
2
x t 1

Example: 5.4
Consider an AR 1 model: xt     ( xt 1   )  zt .

Find the estimates  and ˆ using the LSE method.

5.5 Confidence interval for mean

Suppose that xt    z t where  is constant, zt ~ N 0,  0  and zt is a stationary process.

Under these assumptions, xt ~ N  ,  0  and xt is stationary , therefore

   n 1
 k  
X ~ N  , 0 

1  2 1    k  
k 1  n  
 n

If  0 and  k ' s are known then a 1001    percent confidence interval for  is

0   k  
X  z 1  2 1    k  . Where z  is upper quantile from the standard normal
2
n   n  2
2

distribution. Note that if  k  0 for all k ,then this confidence interval formula reduces to

0
X  z .
2
n
31
SSTA031 STUDY GUIDE 2022

Example 5.5

Suppose that in a sample of size 100 from AR 1 : xt  1 xt 1  z t process with mean  ;   0.6

and  2  2 we obtain X 100  0.271

(a) Construct an approximate 95% confidence interval for  .

(b) Are the data computable with the hypothesis that   0 ?

CHAPTER 6
MODEL DIAGNOSTICS

6.1 Learning Outcomes:

Upon successful completion of this chapter, students should be able to:

 Identify whether the process is a white noise or not.

 Check the adequacy of the model.
 Test for independence, normality and constant variance.
 Determine if the residuals ACF are significantly different from zero using the
Portmanteau test.

6.2 Introduction

Model diagnostics is primarily concerned with testing the goodness of fit of a tentative model.
Two complementary approaches are: Analysis of residuals from fitted models and analysis
of over parameterized model will be considered in this chapter.

6.2.1 Residual Analysis

Before a model can be used for inference the assumptions of the model should be assessed
using residuals. Recall from regression analysis, residuals are given by :

Residual=Actual Value-Predicted Value.

Residual can be used to assess if the ARMA is adequate and if the parameter estimates
are close to the true values. Model adequacy is checked by assessing whether the model
assumptions are satisfied.

32
SSTA031 STUDY GUIDE 2022

The basic assumption is that , z t are white noise .That is they possess the properties of

independence, identically and normally distributed random variables with zero mean and
constant variance  z2 .

A good model is one with residuals that satisfy these properties, that is, it should have
residuals which are:

 Independent (uncorrelated errors),

 Normally distributed and

 Constant variance.

6.2.2 Test of independence

A test of independence can be performed by:

-Examining ACF: Compute the sample ACF of the residual. Residuals are independent if
they do not form any pattern and are statistically insignificant, that is, they are within Z 
2

standard deviation.

6.2.3 Test for normality

Test of normality can be performed by:

-Constructing Histogram: Gross normality can be assessed by plotting histogram of

residuals.

Histogram of normally distributed residuals should approximately be symmetric and bell

shaped.

6.2.4 Test of constant variance

Test of constant variance can be inspected by plotting the residuals over time. If the model
is adequate, we expect the plot to suggest a rectangular scatter around zero horizontal level
with no trends whatsoever.

33
SSTA031 STUDY GUIDE 2022

6.3 Autocorrelation of residuals

The basic idea behind the ARIMA modeling is to account for any autocorrelation pattern in
the series xt  with a parsimonious combination of AR and MA terms, leaving random terms

zt as a white noise. If the residuals are white noise this implies that they are correlated,
that is, they are serial independent. To determine if the residuals ACF are significantly
different from zero we use the following Portmanteau test.

6.3 1 Test for combined residual ACF: Portmanteau test

This test uses the magnitude of residual autocorrelations as a group to check for model
adequacy.

The test is as follows:

Hypothesis

H 0 : 1   2  ...   k  0 mod el is correct 

H 1 :  k  0 for atleast one k  1,2,..., K mod el is incorrect

Here we use the modified Box-Pierce Statistic (Q).

K
Test statistic: Q  N  rz ,k
2

k 1

where N is the number of terms in the differenced series.

K is the number of lags we wish to use in the test.

z, k  denote the autocorrelation coefficient at lag k of the residual ẑt .

Decision rule: Reject H 0 if Q   k2,1 and conclude that the random term z t  from the

estimated model are correlated and that the estimated model may be inadequate.

Remark: If a correct ARMA p, q  model is fitted, then Q should be approximately distributed

as  2 with k  p  q degree of freedom, where p, q are numbers of AR and MA terms,
respectively, in the model.

Note: The maximum lag K is taken large enough so that the weights are negligible for j  K

34
SSTA031 STUDY GUIDE 2022

 
K
Alternative test: Ljung-Box test, Q  N N  2 rz ,k / N  k  .
* 2

k 1

Example: 6.1
The AR2  model xt  C  1 xt 1   2 xt 2  zt was fitted to a data set of length 121.

24
a) The Box-Pierce statistic value was Q  121 rz k   31.5
2

k 1

At 95% level of significance, test the hypothesis that AR2  model fit the data.

b) The parameter estimates are ˆ1  1.5630 , ˆ 2  0.6583 and Cˆ  0.4843

The last four values in the series are 707, 6.90, 6.63, 6.20 .

Use AR2  model to predict the next observation value.

6.4 Over fitting

Another basic diagnostic tool is that of over fitting. In this diagnostic check we add another
coefficient to see if the model is better. Recall that any ARMA p, q  model can be considered
as a special case of a more general ARMA model with the additional parameters equal to
zero. Thus, after specifying and fitting our tentative model, we fit a more general model that
contains the original model as a special case.

There is no unique way to over fit a model, but one should be careful not to add coefficients
to both sides of the model. Over fitting both AR and MA terms at the same time leads to
estimation problems because of the parameter redundancy as well as violating the principle
of parsimony.

If our tentative model is AR2  we might over fit with AR3 . The original AR2  will be
confirmed if :

 The estimates of the additional parameter 3 is not significantly different from zero.

 The estimates for the parameters 1 and  2 do not change significantly from the
original states.

35
SSTA031 STUDY GUIDE 2022

CHAPTER 7
FORECASTING

7.1 Learning Outcomes:

Upon successful completion of this chapter, students should be able to:

 Identify whether the process is a white noise or not.

 Check the adequacy of the model.
 Test for independence, normality and constant variance.
 Determine if the residuals ACF are significantly different from zero using the
Portmanteau test.

7.2 Minimum Mean Square Error

The minimum mean square error (MMSE) forecasts xˆ n l  of z n l at the forecast origin n is

given by the conditional expectation: xˆ n l   E xn l z n , z n1 ,... .

Example: 7.1

Let  B    B Wt   B 1  B d  1   1 B  ...   p  q B p  d 

7.2.1 ARMA model forecast equation in infinite MA form

The ARMA model for xt is xt   B zt  xt   1 zt 1   2 zt 2  ...  zt

Assume that we have x1 , x2 ,..., xn and we want to forecast xnl (i.e l step-ahead forecast from
origin, the actual value is: xnl   1 z nl 1   2 z nl 2  ...  z nl .

The “minimum mean square error” forecast for xnl is xˆ n l    1 z nl 1    2 z nl 2   ...

This form is not very useful for computing forecasts, but is useful in finding the forecast error.

7.3 Computation of Forecasts

7.3.1 Forecast error and forecast error variances

(a) One step –ahead ( l  1 )

36
SSTA031 STUDY GUIDE 2022

en 1  xn1  xˆ n 1  z n1

var en 1  var z n1    z

2

(b) Two steps –ahead ( l  2 )

en 2  xn 2  xˆ n 2  z n 2   1 z n1

var en 2  var z n  2    1 var z n1    z   1 z z   z 1   1

2 2 2 2 2
 2

(c) Three steps-ahead ( l  3 )

en 3  xn3  xˆ n 3  z n3   1 z n 2   2 z n1

var en 3  var z n3    1 var z n 2    2 var z n1    z   1  z   2  z   z 1   1   2

2 2 2 2 2 2 2 2
 2 2

(d) In general, l steps-ahead

en l   xnl  xˆ n l   z nl   1 z nl 1  ...   l 1 z n1

 
l 1
var en l    z 1   1   2  ...   l 1   z 
2 2 2 2 2 2
i where  0  1 .
i 0

To forecast l steps-ahead from origin n :

The actual value is xnl   1 z nl 1   2 z nl 2  ...  z nl

The minimum mean square error forecast for xnl is xˆ n l    1 z nl 1    2 z nl 2   ...

7.4 Prediction Interval

A 95% prediction interval for l step ahead forecast is xˆ n l   Z 0.025  var en l  .

7.5 Forecasting AR  p  and MAq  models.

Example: 7.2
For each of the following models,

AR 1 process: xt     xt 1     z t .

37
SSTA031 STUDY GUIDE 2022

MA1 process: xt  z t   z t 1 .

Find: a) xˆ n 1 b) xˆ n 2 c) xˆ n 1

7.6 Forecasting ARMA p, q  models

Example: 7.3
For 1   B xt     1   B  zt

a) Find the first, 2nd and l -step ahead forecast.

b) Calculate: i)   weight ii) forecast error variance

iii) 95% forecast limits for xˆ n l .

7.7 Forecasting ARIMA p, d , q  models

Consider the ARIMA p, d , q  model at time t  n  l ,  p B 1  B d xt   q B  z t

38