Random Number Generation

Outline
1. Properties of Random Numbers
2. Generation of Pseudo-Random Numbers (PRN)
3. Techniques for Generating Random Numbers
4. Tests for Random Numbers
 1. Properties of Random Numbers
A sequence of random numbers R1, R2, …, must have two important statistical properties:
• Uniformity
• Independence.
Random Number, Ri, must be independently drawn from a uniform distribution with pdf:

pdf for random

1, 0  x  1 numbers
f ( x)  
0, otherwise
1
2
1 x 1

E ( R )  xdx 
0 2
0

2
3 1 2
x 1 1 1 1
V ( R )   x dx   E  R  
1 2
2
     
0 3 0 2 3 4 12 3
1. Properties of Random Numbers

Uniformity: If the interval [0,1] is divided into n classes, or

subintervals of equal length, the expected number of
observations in each interval is N/n, where N is the total number
of observations
Independence: The probability of observing a value in a
particular interval is independent of the previous value drawn

4
2. Generation of Pseudo-Random Numbers (PRN)

“Pseudo”, because generating numbers using a known method

removes the potential for true randomness.
• If the method is known, the set of random numbers can be
replicated!!
Goal: To produce a sequence of numbers in [0,1] that simulates,
or imitates, the ideal properties of random numbers (RN) -
uniform distribution and independence.

5
2. Generation of Pseudo-Random Numbers (PRN)
Problems that occur in generation of pseudo-random numbers
(PRN)
• Generated numbers might not be uniformly distributed
• Generated numbers might be discrete-valued instead of continuous-
valued
• Mean of the generated numbers might be too low or too high
• Variance of the generated numbers might be too low or too high
• There might be dependence (i.e., correlation)

6
2. Generation of Pseudo-Random Numbers (PRN)

Departure from uniformity and independence for a particular

generation scheme can be tested.
If such departures are detected, the generation scheme should be
dropped in favor of an acceptable one.

7
2. Generation of Pseudo-Random Numbers (PRN)

Important considerations in RN routines:

• The routine should be fast. Individual computations are inexpensive,
but a simulation may require many millions of random numbers
• Portable to different computers – ideally to different programming
languages. This ensures the program produces same results
• Have sufficiently long cycle. The cycle length, or period represents the
length of random number sequence before previous numbers begin to
repeat in an earlier order.
• Replicable. Given the starting point, it should be possible to generate
the same set of random numbers, completely independent of the
system that is being simulated
• Closely approximate the ideal statistical properties of uniformity and
independence.

8
3. Techniques for Generating Random Numbers
3.1 Linear Congruential Method (LCM).
• Most widely used technique for generating random numbers
3.2 Combined Linear Congruential Generators (CLCG).
• Extension to yield longer period (or cycle)
3.3 Random-Number Streams.

9
3. Techniques for Generating Random Numbers: Linear
Congruential Method
To produce a sequence of integers, X1, X2, … between 0 and m-1 by
following a recursive relationship:

X i 1  (aX i  c) mod m, i  0,1,2,...

The The The

multiplier increment modulus

X0 is called the seed

The selection of the values for a, c, m, and X0 drastically affects the
statistical properties and the cycle length.
If c 0 then it is called mixed congruential method
When c=0 it is called multiplicative congruential method

10
3. Techniques for Generating Random Numbers: Linear
Congruential Method

The random integers are being generated in the range

[0,m-1], and to convert the integers to random numbers:

Xi
Ri  , i  1,2,...
m

11
3. Techniques for Generating Random Numbers: Linear
Congruential Method

EXAMPLE: Use X0 = 27, a = 17, c = 43, and m = 100.

The Xi and Ri values are:
X1 = (17*27+43) mod 100 = 502 mod 100 = 2, R1 = 0.02;
X2 = (17*2+43) mod 100 = 77 mod 100 =77, R2 = 0.77;
X3 = (17*77+43) mod 100 = 1352 mod 100 = 52 R3 = 0.52;
…

Notice that the numbers generated assume values only from the set I =
{0,1/m,2/m,….., (m-1)/m} because each Xi is an integer in the set {0,1,2,
….,m-1}
Thus each Ri is discrete on I, instead of continuous on interval [0,1]

12
3. Techniques for Generating Random Numbers: Linear
Congruential Method
Maximum Density
• Such that the values assumed by Ri, i = 1,2,…, leave no large gaps on
[0,1]
• Problem: Instead of continuous, each Ri is discrete
• Solution: a very large integer for modulus m (e.g., 231-1, 248)
Maximum Period
• To achieve maximum density and avoid cycling.
• Achieved by: proper choice of a, c, m, and X0.
Most digital computers use a binary representation of numbers
• Speed and efficiency are aided by a modulus, m, to be (or close to) a
power of 2.

13
3. Techniques for Generating Random Numbers: Linear
Congruential Method

Maximum Period or Cycle Length:

For m a power of 2, say m=2b, and c0, the longest possible period is
P=m=2b, which is achieved when c is relatively prime to m (greatest
common divisor of c and m is 1) and a=1+4k, where k is an integer
For m a power of 2, say m=2b, and c=0, the longest possible period is
P=m/4=2b-2, which is achieved if the seed X0 is odd and if the multiplier a
is given by a=3+8k or a=5+8k for some k=0,1,….
For m a prime number and c=0, the longest possible period is P=m-1, which
is achieved whenever the multiplier a has the property that the smallest
integer k such that ak-1 is divisible by m is k=m-1

14
3. Techniques for Generating Random Numbers : Linear
Congruential Method
Example: Using the multiplicative congruential method, find the period of the generator for a=13,
m=26=64 and X0=1,2,3 and 4

i 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Xi 1 13 41 21 17 29 57 37 33 45 9 53 49 61 25 5 1

Xi 2 26 18 42 34 58 50 10 2

Xi 3 39 59 63 51 23 43 47 35 7 27 31 19 55 11 15 3

Xi 4 52 36 20 4

m=64, c=0; Maximal period P=m/4 = 16 is achieved by using odd seeds X0=1 and X0=3 (a=13 is of
the form 5+8k with k=1)
With X0=1, the generated sequence {1,5,9,13,…,53,57,61} has large gaps
Not a viable generator !! Density insufficient, period too short
15
3. Techniques for Generating Random Numbers (1): Linear
Congruential Method
Example: Speed and efficiency in using the generator on a digital
computer is also a factor
Speed and efficiency are aided by using a modulus m either a
power of 2 (=2b)or close to it
After the ordinary arithmetic yields a value of aXi+c, Xi+1 can be
obtained by dropping the leftmost binary digits and then using
only the b rightmost digits

16
3. Techniques for Generating Random Numbers : Linear
Congruential Method
Example: c=0; a=75=16807; m=231-1=2,147,483,647 (prime #)
Period P=m-1 (well over 2 billion)
Assume X0=123,457

X1=75(123457)mod(231-1)=2,074,941,799
R1=X1/231=0.9662
X2=75(2,074,941,799) mod(231-1)=559,872,160
R2=X2/231=0.2607
X3=75(559,872,160) mod(231-1)=1,645,535,613
R3=X3/231=0.7662
……….
Note that the routine divides by m+1 instead of m. Effect is negligible for such large
values of m.

17
3. Techniques for Generating Random Numbers: Combined Linear
Congruential Generators.

With increased computing power, the complexity of simulated

systems is increasing, requiring longer period generator.
• Examples: 1) highly reliable system simulation requiring
hundreds of thousands of elementary events to observe a
single failure event;
• 2) A computer network with large number of nodes,
producing many packets
Approach: Combine two or more multiplicative congruential
generators in such a way to produce a generator with good
statistical properties

18
3. Techniques for Generating Random Numbers: Combined Linear
Congruential Generators
L’Ecuyer suggests how this can be done:
• If Wi,1, Wi,2,….,Wi,k are any independent, discrete valued
random variables (not necessarily identically distributed)
• If one of them, say Wi,1 is uniformly distributed on the
integers from 0 to m1-2, then

 k 
Wi    Wi , j  mod m1  1
 j 1 
is uniformly distributed on the integers from 0 to m1-2

19
3. Techniques for Generating Random Numbers: Combined Linear
Congruential Generators

Let Xi,1, Xi,2, …, Xi,k, be the ith output from k different multiplicative
congruential generators.
• The jth generator:
– Has prime modulus mj and multiplier aj and period
is mj-1
– Produced integers Xi,j is approx ~ Uniform on
integers in [1, mj-1]
– Wi,j = Xi,j -1 is approx ~ Uniform on integers in [0,
mj-2]

20
3. Techniques for Generating Random Numbers: Combined Linear
Congruential Generators

Suggested form:
 Xi
 m , Xi  0
 k 
X i    (1) X i , j  mod m1  1
j 1
Hence, Ri   1
 j 1  m 1
 1 , Xi  0
 m1

• The maximum possible period for such a generator is:

(m1  1)(m2  1)...(mk  1)
P
2 k 1
21
3. Techniques for Generating Random Numbers: Combined Linear
Congruential Generators

Example: For 32-bit computers, L’Ecuyer [1988] suggests combining k = 2 generators

with m1 = 2,147,483,563, a1 = 40,014, m2 = 2,147,483,399 and a2 = 40,692. The
algorithm becomes:
Step 1: Select seeds
– X1,0 in the range [1, 2,147,483,562] for the 1st generator
– X2,0 in the range [1, 2,147,483,398] for the 2nd generator.
Step 2: For each individual generator,
X1,j+1 = 40,014 X1,j mod 2,147,483,563
X2,j+1 = 40,692 X1,j mod 2,147,483,399.
Step 3: Xj+1 = (X1,j+1 - X2,j+1 ) mod 2,147,483,562.
Step 4: Return  X j 1
 , X j 1  0
R j 1   2,147,483, 563
 2,147,483,562 , X j 1  0
 2,147,483,563
Step 5: Set j = j+1, go back to step 2.
• Combined generator has period: (m1 – 1)(m2 – 1)/2 ~ 2 x 1018
22
3. Techniques for Generating Random Numbers: Random-
Numbers Streams

The seed for a linear congruential random-number generator:

• Is the integer value X0 that initializes the random-number sequence.
• Any value in the sequence can be used to “seed” the generator.
A random-number stream:
• Refers to a starting seed taken from the sequence X0, X1, …, XP.
• If the streams are b values apart, then stream i could defined by starting
seed:

S X
i b (bi =
• Older generators:
for i  1,2, , P b b = 1037.
1) 105; Newer generators:

23
3. Techniques for Generating Random Numbers: Random-
Numbers Streams
A single random-number generator with k streams can act like k
distinct virtual random-number generators
To compare two or more alternative systems.
• Advantageous to dedicate portions of the pseudo-random number
sequence to the same purpose in each of the simulated systems.

24
4. Tests for Random Numbers: Principles

Desirable properties of random numbers: Uniformity and Independence

Number of tests can be performed to check these properties been achieved or
not
Two type of tests:
• Frequency Test: Uses the Kolmogorov-Smirnov or the Chi-square
test to compare the distribution of the set of numbers generated to a
uniform distribution
• Autocorrelation test: Tests the correlation between numbers and
compares the sample correlation to the expected correlation.

25
4. Tests for Random Numbers: Principles

Two categories:
• Testing for uniformity. The hypotheses are:
H0: Ri ~ U[0,1]
H1: Ri ~ U[0,1] /

– Failure to reject the null hypothesis, H0, means that

evidence of non-uniformity has not been detected.
• Testing for independence. The hypotheses are:
H0: Ri ~ independently distributed
/
H1: Ri ~ independently distributed
– Failure to reject the null hypothesis, H0, means that
evidence of dependence has not been detected.
26
4. Tests for Random Numbers: Principles

For each test, a Level of significance must be stated.

The level is the probability of rejecting the null hypothesis H0 when the
null hypothesis is true:
 = P(reject H0|H0 is true)
The decision maker sets the value of for any test
Frequently  is set to 0.01 or 0.05

27
4. Tests for Random Numbers: Principles

When to use these tests:

• If a well-known simulation languages or random-number generators is
used, it is probably unnecessary to test
• If the generator is not explicitly known or documented, e.g., spreadsheet
programs, symbolic/numerical calculators, tests should be applied to many
sample numbers.
Types of tests:
• Theoretical tests: evaluate the choices of m, a, and c without actually
generating any numbers
• Empirical tests: applied to actual sequences of numbers produced. Our
emphasis.

28
4. Tests for Random Numbers: Frequency Tests

Test of uniformity
Two different methods:
• Kolmogorov-Smirnov test
• Chi-square test
Both these tests measure the degree of agreement between the distribution of a
sample of generated random numbers and the theoretical uniform distribution
Both tests are based on null hypothesis of no significant difference between
the sample distribution and the theoretical distribution

29
4. Tests for Random Numbers: Frequency Tests Kolmogorov-
Smirnov Test
Compares the continuous cdf, F(x), of the uniform distribution with the
empirical cdf, SN(x), of the N sample observations.
• We know: F ( x )  x, 0  x  1
• If the sample from the RN generator is R1, R2, …, RN, then the empirical
cdf, SN(x) is:
number of R1 , R2 ,..., Rn which are  x
S N ( x) 
N
The cdf of an empirical distribution is a step function with jumps at each
observed value.

30
4. Tests for Random Numbers: Frequency Tests Kolmogorov-
Smirnov Test

Test is based on the largest absolute deviation statistic between F(x) and SN(x) over the
range of the random variable:
D = max| F(x) - SN(x)|
The distribution of D is known and tabulated (A.8) as function of N
Steps:
1. Rank the data from smallest to largest. Let R(i) denote ith smallest observation,
so that R(1)R(2)…R(N)
2. Compute i   i  1
D   max   R i  ; D   max  R i   
1i  N N N
  1 i  N
 
3. Compute D= max(D+, D-)
4. Locate in Table A.8 the critical value D, for the specified significance level 
and the sample size N
5. If the sample statistic D is greater than the critical value D, the null
hypothesis is rejected. If D D, conclude there is no difference
31
4. Tests for Random Numbers: Frequency Tests Kolmogorov-
Smirnov Test

Example: Suppose 5 generated numbers are 0.44, 0.81, 0.14, 0.05, 0.93.

Arrange R(i) from

R(i) 0.05 0.14 0.44 0.81 0.93
Step 1: smallest to largest
i/N 0.20 0.40 0.60 0.80 1.00

i/N – R(i) 0.15 0.26 0.16 - 0.07 D+ = max {i/N – R(i)}

Step 2: R(i) – (i-1)/N 0.05 - 0.04 0.21 0.13
D- = max {R(i) - (i-1)/N}

Step 3: D = max(D+, D-) = 0.26

Step 4: For  = 0.05,
D = 0.565 > D

Hence, H0 is not rejected.

32
4. Tests for Random Numbers: Frequency Tests - Chi-Square
Test
Chi-square test uses the sample statistic:

n is the # of classes Ei is the expected

n
(Oi  Ei ) 2
 02   # in the ith class

i 1 Ei Oi is the observed
# in the ith class

• Approximately the chi-square distribution with n-1 degrees of freedom

(where the critical values are tabulated in Table A.6)
• For the uniform distribution, Ei, the expected number in the each class
is:
N
Ei  , where N is the total # of observation
n
Valid only for large samples, e.g. N >= 50
Reject H0 if 02 > ,N-12

33
4. Tests for Random Numbers : Frequency Tests - Chi-Square
Test
Example 7.7: Use Chi-square test for the data shown below with =0.05. The
test uses n=10 intervals of equal length, namely [0,0.1),[0.1,0.2), ….,
[0.9,1.0)

34
4. Tests for Random Numbers: Frequency Tests Chi-Square Test

The value of 02=3.4; The critical value from table A.6 is 0.05,92=16.9.
Therefore the null hypothesis is not rejected

35
What does autocorrelation look like?
Here is a list of random integers generated. See if you can identify
a pattern by looking at the values in the sequence.

1 2 3 4 5 6 7 8 9 10
0.63 0.28 0.30 0.42 0.97 0.05 0.71 0.63 0.17 1.0
0.61 0.19 0.94 0.64 0.84 0.54 0.56 0.57 0.09 0.99
0.01 0.10 0.69 0.38 0.93 0.85 0.68 0.14 0.18 0.84
0.19 0.71 0.44 0.72 0.95 0.28 0.96 0.51 0.50 0.89
0.66 0.31 0.50 0.33 0.89 0.54 0.73 0.76 0.62 0.92
If you look carefully at the following table of random integers
generated, you'll notice that every number in the 5th, 10th, 15th, and
20th position is a larger value.

1 2 3 4 5 6 7 8 9 10
0.63 0.28 0.30 0.42 0.97 0.05 0.71 0.63 0.17 1.0
0.61 0.19 0.94 0.64 0.84 0.54 0.56 0.57 0.09 0.99
0.01 0.10 0.69 0.38 0.93 0.85 0.68 0.14 0.18 0.84
0.19 0.71 0.44 0.72 0.95 0.28 0.96 0.51 0.50 0.89
0.66 0.31 0.50 0.33 0.89 0.54 0.73 0.76 0.62 0.92
4. Tests for Random Numbers: Tests for Autocorrelation

The test for autocorrelation are concerned with the dependence between
numbers in a sequence.
Consider:

Though numbers seem to be random, every fifth number is a large number in

that position.
This may be a small sample size, but the notion is that numbers in the
sequence might be related

38
4. Tests for Random Numbers: Tests for Autocorrelation
Testing the autocorrelation between every m numbers (m is a.k.a. the lag),
starting with the ith number
• The autocorrelation im between numbers: Ri, Ri+m, Ri+2m, Ri+(M+1)m
• M is the largest integer such that
i  (M  1 )m  N
Hypothesis:
H 0 :  im  0, if numbers are independent
H1 :  im  0, if numbers are dependent

If the values are uncorrelated:

• For large values of M, the distribution of the estimator of im, denoted
is approximately normal.
̂ im

39
4. Tests for Random Numbers : Tests for Autocorrelation

ˆ im
Test statistics is: Z0 
ˆ ˆim

• Z0 is distributed normally with mean = 0 and variance = 1, and:

1 M 
ˆρim   
M  1  k 0
Ri  km Ri (k 1 )m   0.25

13M  7
σˆ ρim 
12(M  1 )

If im > 0, the subsequence has positive autocorrelation

• High random numbers tend to be followed by high ones, and vice versa.
If im < 0, the subsequence has negative autocorrelation
• Low random numbers tend to be followed by high ones, and vice versa.
40
4. Tests for Random Numbers: Tests for Autocorrelation

After computing Z0, do not reject the hypothesis of independence if –z/2Z0 

z/2
 is the level of significance and z/2 is obtained from table A.3

41
4. Tests for Random Numbers: Tests for Autocorrelation

Example: Test whether the 3rd, 8th, 13th, and so on, for the output on Slide 35 are
auto-correlated or not.
• Hence,  = 0.05, i = 3, m = 5, N = 30, and M = 4. M is the largest
integer such that 3+(M+1)530.

1 (0.23)(0.28)  (0.25)(0.33)  (0.33)(0.27)

ρˆ 35     0.25
4  1  (0.28)(0.05)  (0.05)(0.36) 
 0.1945
13(4)  7
σˆ ρ35   0.128
12( 4  1 )
0.1945
Z0    1.516
0.1280

• From Table A.3, z0.025 = 1.96. Hence, the hypothesis is not rejected.
42
4. Tests for Random Numbers: Tests for Autocorrelation

Shortcoming:
The test is not very sensitive for small values of M, particularly when the
numbers being tested are on the low side.
Problem when “fishing” for autocorrelation by performing numerous tests:
• If  = 0.05, there is a probability of 0.05 of rejecting a true hypothesis.
• If 10 independent sequences are examined,
– The probability of finding no significant autocorrelation, by
chance alone, is 0.9510 = 0.60.
– Hence, the probability of detecting significant
autocorrelation when it does not exist = 40%

43