CPSC 531:

System Modeling and Simulation

Carey Williamson
Department of Computer Science
University of Calgary
Fall 2017
 Overview

 The world a model-builder sees is probabilistic rather

than deterministic:
— Some probability model might well describe the variations

Goals:
 Review the fundamental concepts of probability
 Understand the difference between discrete and
continuous random variable
 Review the most common probability models

2
 Outline

 Probability and random variables

— Random experiment and random variable
— Probability mass/density functions
— Expectation, variance, covariance, correlation
 Probability distributions
— Discrete probability distributions
— Continuous probability distributions
— Empirical probability distributions

3
 Outline

 Probability and random variables

— Random experiment and random variable
— Probability mass/density functions
— Expectation, variance, covariance, correlation
 Probability distributions
— Discrete probability distributions
— Continuous probability distributions
— Empirical probability distribution

4
 Probability

Is widely used in mathematics, science, engineering,

finance and philosophy to draw conclusions about the
likelihood of potential events and the underlying
mechanics of complex systems

 Probability is a measure of how likely it is for an event to

happen
 We measure probability with a number between and
 If an event is certain to happen, then the probability of
the event is
 If an event is certain not to happen, then the probability
of the event is 5
 Random Experiment

 An experiment is called random if the outcome of the

experiment is uncertain
 For a random experiment:
— The set of all possible outcomes is known before the
experiment
— The outcome of the experiment is not known in advance
 Sample space of an experiment is the set of all
possible outcomes of the experiment
 Example: Consider random experiment of tossing a
coin twice. Sample space is:

6
 Probability of Events

 An event is a subset of sample space

Example 1: in tossing a coin twice, E={(H,H)} is the event of
having two heads
Example 2: in tossing a coin twice, E={(H,H), (H,T)} is the
event of having a head in the first toss

 Probability of an event E is a numerical measure of the

likelihood that event will occur, expressed as a number
between 0 and 1,

— If all possible outcomes are equally likely:

— Probability of the sample space is 1:

7
 Joint Probability

 Probability that two events and occur in a single

experiment:

 Example: drawing a single card at random from a

regular deck of cards, probability of getting a red king
— : getting a red card
— : getting a king

8
 Independent Events

 Two events and are independent if the occurrence

of one does not affect the occurrence of the other:

 Example: drawing a single card at random from a

regular deck of cards, probability of getting a red king
— : getting a red card
— : getting a king
— and are independent

9
 Mutually Exclusive Events

 Events and are mutually exclusive if the occurrence of one

implies the non-occurrence of the other, i.e.,:

 Example: drawing a single card at random from a regular deck

of cards, probability of getting a red club
— : getting a red card
— : getting a club

 Complementary event of event is event , i.e., the event that

does not occur, denoted by
— Events and are mutually exclusive

10
 Union Probability

 Union of events and :

 If and are mutually exclusive:

 Example: drawing a single card at random from a regular deck

of cards, probability of getting a red card or a king
— : getting a red card
— : getting a king

11
 Conditional Probability

 Probability of event given the occurrence of some event :

 If events and are independent:

 Example: drawing a single card at random from a regular deck

of cards, probability of getting a king given that the card is red
— : getting a red card
— : getting a king

12
 Random Variable

 A numerical value can be associated with each

outcome of an experiment
 A random variable X is a function from the sample
space  to the real line that assigns a real number
X(s) to each element s of 
X:  → R
 Random variable takes on its values with some
probability

13
 Random Variable

 Example: Consider random experiment of tossing a

coin twice. Sample space is:
 = {(H,H), (H,T), (T,H), (T,T)}
Define random variable X as the number of heads in
the experiment:
X((T,T)) = 0, X((H,T))=1,
X((T,H)) = 1, X((H,H))=2
 Example: Rolling a die.
Sample space  = {1,2,3,4,5,6).
Define random variable X as the number rolled:
X(j) = j, 1≤j≤6
14
 Random Variable

 Example: roll two fair dice and observe the outcome

Sample space = {(i,j) | 1 ≤ i ≤ 6, 1 ≤ j ≤ 6}
i: integer from the first die
j: integer from the second die
(1,1) (1,2) (1,3) (1,4) (1,5) (1,6)
(2,1) (2,2) (2,3) (2,4) (2,5) (2,6)
(3,1) (3,2) (3,3) (3,4) (3,5) (3,6)
(4,1) (4,2) (4,3) (4,4) (4,5) (4,6)
(5,1) (5,2) (5,3) (5,4) (5,5) (5,6)
(6,1) (6,2) (6,3) (6,4) (6,5) (6,6)

Possible outcomes

15
 Random Variable

 Random variable : sum of the two faces of the dice

X(i,j) = i+j
— (X = 12) = ( (6,6) ) = 1/36
— (X = 10) = ( (5,5), (4,6), (6,4) ) = 3/36

 Random variable Y: value of the first die

— (Y = 1) = 1/6
— ( Y = i) = 1/6, 1≤i≤6

16
 Types of Random Variables

 Discrete
— Random variables whose set of possible values can be
written as a finite or infinite sequence
— Example: number of requests sent to a web server
 Continuous
— Random variables that take a continuum of possible values
— Example: time between requests sent to a web server

17
 Probability Mass Function (PMF)

 X: discrete random variable

 : probability mass function of , where

 Properties:

18
 PMF Examples

 Number of heads in tossing  Number rolled in rolling

three coins a fair die

0 1/8
1 3/8
2 3/8
3 1/8

19
 Probability Density Function (PDF)

 X: continuous random variable

 f(x): probability density function of X
d
f ( x)  F ( x)
dx CDF of
 Note:
— !!

 Properties:

—
20
 Probability Density Function

Example: Life of an inspection device is given by , a

continuous random variable with PDF:

, for

— has an exponential distribution with mean 2 years

— Probability that the device’s life is between 2 and 3 years:

21
 Cumulative Distribution Function (CDF)

 X: discrete or continuous random variable

 : cumulative probability distribution function of X, or simply,
probability distribution function of X

— If is discrete, then

— If is continuous, then

 Properties
— is a non-decreasing function, i.e., if , then
— , and

 All probability questions about can be answered in terms of the CDF, e.g.:
for all

22
 Cumulative Distribution Function

Discrete random variable example.

 Rolling a die, X is the number rolled

—p(i) = (X = i) = 1/6, 1≤i≤6
—F(i) = (X ≤ i)
= p(1) + … + p(i)
= i/6

23
 Cumulative Distribution Function

Continuous random variable example.

 The inspection device has CDF:

1 x t / 2
F ( x)   e dt  1  e  x / 2
2 0

— The probability that the device lasts for less than 2 years:

— The probability that it lasts between 2 and 3 years:

24
 Joint Probability Distribution

 Joint probability distribution of random variables

and is defined as

 and are independent random variables if

— Discrete:
— Continuous:

25
 Expectation of a Random Variable

 Mean or Expected Value:

n
 xi p ( xi ) discrete X
 i 1
  E[ X ]   
 xf ( x)dx continuous X
 
 Example: number of heads in tossing three coins
E[X] = 0 p(0) + 1 p(1) + 2 p(2) + 3 p(3)
= 1 3/8 + 2 3/8 + 3 1/8
= 12/8
= 1.5 26
 Expectation of a Function

 : a real-valued function of random variable

 How to compute ?
— If is discrete with PMF :

— If is continuous with PDF

 Example: is the number rolled when rolling a die

— PMF: , for

27
 Properties of Expectation

 : two random variables

 : two constants

28
 Misuses of Expectations

 Multiplying means to get the mean of a product

E[ XY ]  E[ X ]E[Y ]
 Example: tossing three coins
— X: number of heads
— Y: number of tails
— E[X] = E[Y] = 3/2  E[X]E[Y] = 9/4
— E[XY] = 3/2
 E[XY] ≠ E[X]E[Y]
 Dividing means to get the mean of a ratio

 X  E[ X ]
E  
 Y  E[Y ] 29
 Variance of a Random Variable

 The variance is a measure of the spread of a

distribution around its mean value
 Variance is symbolized by or or :
— Mean is a way to describe the location of a distribution
— Variance is a way to capture its scale or degree of being
spread out
— The unit of variance is the square of the unit of the original
variable
 : standard deviation
 Defined as the square root of variance
 Expressed in the same units as the mean

30
 Variance of a Random Variable

 Variance: The expected value of the square of

distance between a random variable and its mean
 2  V[X ]
 n
 i   2
( x ) p ( xi ) discrete X
 i 1
 E[( X   ) ]   
2

 ( x   ) 2 f ( x)dx continuous X

where, μ= E[X]

 Equivalently:
σ2 = E[X2] – (E[X])2 31
 Variance of a Random Variable

 Example: number of heads in tossing three coins

E[X] = 1.5

σ2 = (0 – 1.5)2p(0) + (1 – 1.5)2p(1)
+ (2 – 1.5)2p(2) + (3 – 1.5)2p(3)
= 9/4 1/8 + 1/4 3/8 + 1/4 3/8 + 9/4 1/8
= 24/32
= 3/4

32
 Variance of a Random Variable

 Example: The mean of life of the previous inspection

device is:

1  x / 2 x / 2
E[ X ]   xe dx   xe

  e  x / 2 dx  2
2 0 0
0

 To compute variance of , we first compute :


1 x / 2
E[ X ]   x e dx   x e
 
2 2 x / 2 2  2  xe  x / 2 dx  8
2 0 0
0

 Hence, the variance and standard deviation of the

device’s life are:
V [ X ]  8  22  4
  V[X ]  2
33
 Properties of Variance

 : two random variables

 : two constants

 If and are independent:

34
 Coefficient of Variation

 Coefficient of Variation:

 Example: number of heads in tossing three coins

3/ 4 1
CV  
 Example: inspection3 /device
2 3

E[ X ]  2
  CV  1
 2 
35
 Covariance

 Covariance between random variables and denoted by or is

a measure of how much and change together

 For independent variables, the covariance is zero:

 Note: Although independence always implies zero covariance,

the reverse is not true
36
 Covariance

 Example: tossing three coins

— X: number of heads x y xy p(x)
0 3 0 1/8
— Y: number of tails
1 2 2 3/8
— E[X] = E[Y] = 3/2 2 1 2 3/8
 E[XY]? 3 0 0 1/8
— X and Y depend on each other
—Y = 3 – X xy p(xy)
— E[XY] = 0×P(0) + 2×P(2) 0 2/8
2 6/8
= 3/2
 = E[XY] – E[X]E[Y]
= 3/2 – 3/2 × 3/2
= – 3/4 37
 Correlation

 Correlation Coefficient between random variables and ,

denoted by , is the normalized value of their covariance:

 Indicates the strength and direction of a linear relationship

between two random variables
 The correlation always lies between -1 and +1

Negative linear Positive linear

correlation No correlation correlation
-1
 Example: tossing three coins0 +1

38
 Outline

 Probability and random variables

— Random experiment and random variable
— Probability mass/density functions
— Expectation, variance, covariance, correlation
 Probability distributions
— Discrete probability distributions
— Continuous probability distributions
— Empirical probability distribution

39
 Outline

 Probability and random variables

— Random experiment and random variable
— Probability mass/density functions
— Expectation, variance, covariance, correlation
 Probability distributions
— Discrete probability distributions
— Continuous probability distributions
— Empirical probability distribution

40
 Discrete Uniform Distribution

 A random variable has discrete uniform distributed if

each of the values in its range, say , has equal
probability.

 PMF:
𝑝(𝑥𝑖)

1
4

1 2 3 4
41
 Discrete Uniform Distribution

 Consider a discrete uniform random variable on the

consecutive integers ,
for . Then:

42
 Bernoulli Trial

 Consider an experiment whose outcome can be a success

with probability or a failure with probability :
— if the outcome is a success
— if the outcome is a failure
 is a Bernoulli random variable with parameter
— where is the success probability

 PMF:

 Properties:
— and

43
 Binomial Distribution

 : number of successes in independent Bernoulli trials with

success probability

 is a binomial random variable with parameters

 PMF: Probability of having () successes in trials

where,
 Properties:
— and

44
 Example: Binomial Distribution

Binomial distribution PMF Binomial distribution CDF

 Geometric Distribution

 : number of Bernoulli trials until achieving the first success

 is a geometric random variable with success probability

 PMF: probability of trials until the first success

 CDF:

 Properties:
, and

46
 Example: Geometric Distribution

Geometric distribution PMF

Geometric distribution CDF
 Poisson Distribution

 Number of events occurring in a fixed time interval

— Events occur with a known rate and are independent

 Poisson distribution is characterized by the rate 

— Rate: the average number of event occurrences in a fixed time interval

 Examples
— The number of calls received by a switchboard per minute
— The number of packets coming to a router per second
— The number of travelers arriving to the airport for flight registration
per hour

48
 Poisson Distribution

Random variable is Poisson distributed with rate parameter

 PMF: the probability that there are exactly events in a time interval
,
 CDF: the probability of at least events in a time interval

 Properties:

49
 Example: Poisson Distribution

Poisson distribution PMF Poisson distribution CDF

 Example: Poisson Distribution

The number of cars that enter a parking lot follows a

Poisson distribution with a rate equal to cars/hour

— The probability of having exactly 15 cars entering the

parking lot in one hour:

— The probability of having more than 3 cars entering the

parking lot in one hour:

51
 Outline

 Probability and random variables

— Random experiment and random variable
— Probability mass/density functions
— Expectation, variance, covariance, correlation
 Probability distributions
— Discrete probability distributions
— Continuous probability distributions
— Empirical probability distribution

52
 Uniform Distribution

 A random variable X has continuous uniform

distribution on the interval [a,b], if its PDF and CDF
are: PDF
— PDF: , for

— CDF:

 Properties: CDF
, and

53
 Uniform Distribution Properties

 is proportional to the length of the interval

 Special case: standard uniform distribution denoted

by

𝑓 (𝑥 )= {
1,0≤ 𝑥≤1
0 , otherwise
— Very useful for random number generation in simulations

54
 Exponential Distribution

 A random variable is exponentially distributed with

parameter if its PDF and CDF are:
— PDF: , for
— CDF: , for

 Properties:
, and

 The exponential distribution describes the time

between consecutive events in a Poisson process of
rate
55
 Example: Exponential Distribution

Exponential distribution PDF Exponential distribution CDF

 Memoryless Property

 Memoryless is a property of certain probability distributions

such as exponential distribution and geometric distribution
—future events do not depend on the past events,
but only on the present event

 Formally: random variable has a memoryless distribution if

for

 Example: The probability that you will wait more minutes

given that you have already been waiting minutes is the same
as the probability that you wait for more than minutes from
the beginning! 57
 Example: Exponential Distribution

 The time needed to repair the engine of a car is exponentially

distributed with a mean time equal to 3 hours.
— The probability that the car spends more than the average wait time in
repair:

— The probability that the car repair time lasts between 2 to 3 hours is:

— The probability that the repair time lasts for another hour given that it has
already lasted for 2.5 hours:
Using the memoryless property of the exponential distribution,

58
 Normal Distribution

 The Normal distribution, also called the Gaussian

distribution, is an important continuous probability
distribution applicable in many fields

 It is specified by two parameters: mean (μ) and variance ()

 The importance of the normal distribution as a statistical

model in natural and behavioral sciences is due in part to the
Central Limit Theorem

 It is usually used to model system error (e.g. channel error),

the distribution of natural phenomena, height, weight, etc.

59
 Why Normal?

 There are two main reasons for the popularity of the normal
distribution:

1. The sum of n independent normal variables is a normal

variable. If,X i ~ N ( i ,  i )
X  i 1 ai X i
n

then has a normal distribution with

  i 1 ai i
n
mean 2 n and variance
  i 1 ai2 i2

2. The mean of a large number of independent observations from

any distribution tends to have a normal distribution. This result,
which is called central limit theorem, is true for observations from
all distributions
=> Experimental errors caused by many factors are normal
60
 Central Limit Theorem

Histogram plot of average proportion of heads in a fair coin

toss, over a large number of sequences of coin tosses.
61
 Normal Distribution

 Random variable X is normally distribution with

parameters , i.e., :
— PDF: , for
— CDF: does not have any closed form!
— , and

 Properties:

— Normal PDF is a symmetrical, bell-shaped curve

centered at its expected value
— Maximum value of PDF occurs at

62
 Standard Normal Distribution

 Random variable has Standard Normal Distribution if it is

normally distributed with parameters , i.e.,
— PDF: , for
— CDF: commonly denoted by :

63
 Normal Distribution

 Evaluating the distribution :

Two techniques:
1. Use numerical methods (no closed form)
2. Use the standard normal distribution
— is widely tabulated
— Use the transformation
— If then , i.e., standard normal distribution:

64
 Normal Distribution

 Example: The time required to load an oceangoing vessel, , is

distributed as
— The probability that the vessel is loaded in less than 10 hours:

— Using the symmetry property, F(1) is the complement of F (-1)

65
 Stochastic Process

Stochastic Process:
Collection of random variables indexed over time

 Example:
— N(t): number of jobs at the CPU of a computer system over
time
— Take several identical systems and observe N(t)
— The number N(t) at any time is a random variable
— Can find the probability distribution functions for N(t) at
each possible value of t

 Notation:
66
 Poisson Process

 Counting Process:
A stochastic process that represents the total number of
events occurred in the time interval

 Poisson Process:
The counting process is a Poisson process with rate , if:

— The process has independent increments

— The number of events in any interval of length is Poisson distributed
with mean . That is, for all

Property: equal mean and variance:

67
 Interarrival Times

 Consider the interarrival times of a Poisson process with

rate , denoted by , where is the elapsed time between arrival and
arrival

 Interarrival times, are independent identically distributed

exponential random variables having the mean

Arrival counts Interarrival times

~ Poisson() ~ Exponential()

 Proof?

68
 Splitting and Pooling

 Pooling:
— : Poisson process with rate
— : Poisson process with rate
— : Poisson process with rate

N1(t) ~ Poisson(l1) l1
l 1 + l2
N(t) ~ Poisson(l1 + l2)

N2(t) ~ Poisson(l2) l2
 Splitting:
— : Poisson process with rate
— Each event is classified as Type I, with probability and Type II, with probability
— : The number of type I events is a Poisson process with rate
— : The number of type II events is a Poisson process with rate
— Note:

lp N1(t) ~ Poisson(lp)

N(t) ~ Poisson(l) l

N2(t) ~ Poisson(l(1-p))
l(1-p) 69
 More on Poisson Distribution

 : a Poisson process with arrival rate l

 Probability of no arrivals in a small time interval :

 Probability of one arrivals in a small time interval :

 Probability of two or more arrivals in a small time

interval :
70
 Outline

 Probability and random variables

— Random experiment and random variable
— Probability mass/density functions
— Expectation, variance, covariance, correlation
 Probability distributions
— Discrete probability distributions
— Continuous probability distributions
— Empirical probability distribution

71
 Empirical Distribution

 A distribution whose parameters are the observed

values in a sample of data:
— Could be used if no theoretical distributions fit the data
adequately
— Advantage: no assumption beyond the observed values in
the sample
— Disadvantage: sample might not cover the entire range of
possible values

72
 Empirical Distribution

 “Piecewise Linear” empirical distribution

— Used for continuous data 1

0.9
— Appropriate when a large 0.8

sample data is available 0.7

0.6
— Empirical CDF is approximated 0.5

by a piecewise linear function: 0.4

0.3
 the ‘jump points’ connected 0.2

by linear functions 0.1

Piecewise Linear
Empirical CDF

73
 Empirical Distribution

 Piecewise Linear empirical distribution

— Organize -axis into intervals
— Interval is from to for
— : relative frequency of interval
— : relative cumulative frequency of interval , i.e.,
interval

•••
𝑎0 𝑎𝑖−1 𝑎 𝑖 𝑎𝐾
intervals
— Empirical CDF:
 If is in interval , i.e., , then:

where, slope is given by

74
 Example Empirical Distribution

 Suppose the data collected for 100 broken machine repair times are:
Interval Relative Cumulative
i (Hours) Frequency Frequency Frequency Slope
1 0.0 < x ≤ 0.5 31 0.31 0.31 0.62
2 0.5 < x ≤ 1.0 10 0.10 0.41 0.2
3 1.0 < x ≤ 1.5 25 0.25 0.66 0.5
4 1.5 < x ≤ 2.0 34 0.34 1.00 0.68

0.9 Piecewise Linear

0.8 Empirical CDF
0.7

0.6

0.5 Slope
0.4

0.3

0.2

0.1

0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
𝑎0 𝑎1 𝑎2 𝑎3 𝑎4
75