Probability Theory Essentials
Probability Theory Essentials
n n
P ∩ Aik = ∏ P( A ik ). (2-3)
k =1 k =1 3
PILLAI
• Let
A = A1 ∪ A2 ∪ A3 ∪ ∪ An , (2-4)
a union of n independent events. Then by De-Morgan’s
law
A = A1 A 2 An
and using their independence
n n
P ( A ) = P ( A1 A 2 An ) = ∏ P ( A ) = ∏ (1 − P ( A )). (2-5)
i =1
i
i =1
i
1 1 6
= ∏ (1 − 1)
p2
= ∞ = 2 = 2 = 0.6079,
π /6 π
p prime
∑ 1/ k 2
k =1
1 1 6
= ∞ = 2 = 2.
π /6 π
∑ 1/ k 2
7
k =1 PILLAI
Note: To add an interesting twist to the ‘square free’ number
problem, Ramanujan has shown through elementary but
clever arguments that the inverses of the nth powers of all
‘square free’ numbers add to Sn / S2 n , where (see (2-E))
∞
Sn = ∑1/ k n .
k =1
Fig.2.1
Solution: a. Let Ai = “Switch Si is closed”. Then P ( Ai ) = p ,
i = 1 → 3 . Since switches operate independently, we have
P ( Ai A j ) = P ( Ai ) P ( A j ); P ( A1 A2 A3 ) = P ( A1 ) P ( A2 ) P ( A3 ).
9
PILLAI
Let R = “input signal is received at the output”. For the
event R to occur either switch 1 or switch 2 or switch 3
must remain closed, i.e.,
R = A1 ∪ A2 ∪ A3 . (2-7)
Using (2-3) - (2-6),
P( R) = P( A1 ∪ A2 ∪ A3 ) = 1 − (1 − p)3 = 3 p − 3 p 2 + p 3. (2-8)
We can also derive (2-8) in a different manner. Since any
event and its compliment form a trivial partition, we can
always write
P ( R ) = P ( R | A1 ) P ( A1 ) + P ( R | A1 ) P ( A1 ). (2-9)
But P ( R | A1 ) = 1, and P ( R | A1 ) = P ( A2 ∪ A3 ) = 2 p − p 2
and using these in (2-9) we obtain
P ( R ) = p + ( 2 p − p 2 )(1 − p ) = 3 p − 3 p 2 + p 3 , (2-10)
which agrees with (2-8). 10
PILLAI
Note that the events A1, A2, A3 do not form a partition, since
they are not mutually exclusive. Obviously any two or all
three switches can be closed (or open) simultaneously.
Moreover, P ( A1 ) + P ( A2 ) + P ( A3 ) ≠ 1 .
b. We need P ( A1 | R ). From Bayes’ theorem
P ( R | A1 ) P ( A1 ) ( 2 p − p 2 )(1 − p ) 2 − 2 p + p 2 (2-11)
P ( A1 | R ) = = = .
P( R) 3p − 3p + p
2 3
3p − 3p + p
2 3
11
PILLAI
Repeated Trials
Consider two independent experiments with associated
probability models (Ω1, F1, P1) and (Ω2, F2, P2). Let
ξ∈Ω1, η∈Ω2 represent elementary events. A joint
performance of the two experiments produces an
elementary events ω = (ξ, η). How to characterize an
appropriate probability to this “combined event” ?
Towards this, consider the Cartesian product space
Ω = Ω1× Ω2 generated from Ω1 and Ω2 such that if
ξ ∈ Ω1 and η ∈ Ω2 , then every ω in Ω is an ordered pair
of the form ω = (ξ, η). To arrive at a probability model
we need to define the combined trio (Ω, F, P).
12
PILLAI
Suppose A∈F1 and B ∈ F2. Then A × B is the set of all pairs
(ξ, η), where ξ ∈ A and η ∈ B. Any such subset of Ω
appears to be a legitimate event for the combined
experiment. Let F denote the field composed of all such
subsets A × B together with their unions and compliments.
In this combined experiment, the probabilities of the events
A × Ω2 and Ω1 × B are such that
P ( A × Ω 2 ) = P1 ( A), P (Ω1 × B ) = P2 ( B ). (2-12)
k
Fig. 2.2 21
PILLAI
Pn ( k − 1) n! p k − 1 q n − k + 1 ( n − k )! k ! k q
= k n−k
= . (2-28)
Pn ( k ) ( n − k + 1)! ( k − 1)! n! p q n − k +1 p
Thus Pn ( k ) ≥ Pn ( k − 1 ), if k (1 − p ) ≤ ( n − k + 1 ) p or k ≤ ( n + 1) p .
Thus Pn ( k ) as a function of k increases until
k = ( n + 1) p (2-29)
5000
400
= ∑ ( 0 .1) k ( 0 .9 ) 5000 − k . (2-31)
k =0 k
Equation (2-31) has too many terms to compute. Clearly,
we need a technique to compute the above term in a more
efficient manner.
From (2-29), kmax the most likely number of successes in n
trials, satisfy
( n + 1) p − 1 ≤ k max ≤ ( n + 1) p (2-32)
or q k max p
p− ≤ ≤ p+ , (2-33)
n n n 24
PILLAI
so that
km
lim = p. (2-34)
n→ ∞ n
From (2-34), as n → ∞, the ratio of the most probable
number of successes (A) to the total number of trials in a
Bernoulli experiment tends to p, the probability of
occurrence of A in a single trial. Notice that (2-34) connects
the results of an actual experiment (km / n) to the axiomatic
definition of p. In this context, it is possible to obtain a more
general result as follows:
Bernoulli’s theorem: Let A denote an event whose
probability of occurrence in a single trial is p. If k denotes
the number of occurrences of A in n independent trials, then
k pq
P − p > ε < . (2-35)
n nε 2
25
PILLAI
Equation (2-35) states that the frequency definition of
k
probability of an event n and its axiomatic definition ( p)
can be made compatible to any degree of accuracy.
Proof: To prove Bernoulli’s theorem, we need two identities.
Note that with Pn ( k ) as in (2-23), direct computation gives
n n −1 n
n! n!
∑
k =0
k Pn ( k ) = ∑
k =1
k
( n − k )! k !
p k n −k
q = ∑
k =1 ( n − k )! ( k − 1)!
p k n −k
q
n −1
n! n −1
( n − 1)!
=∑ i +1 n − i −1
p q = np ∑ p i q n −1−i
i = 0 ( n − i − 1)! i! i = 0 ( n − 1 − i )! i!
= np ( p + q ) n −1 = np . (2-36)
Proceeding in a similar manner, it can be shown that
n n n
n! n!
∑
k =0
k2
Pn ( k ) = ∑
k =1
k
( n − k )! ( k − 1)!
p k n −k
q = ∑
k = 2 ( n − k )! ( k − 2 )!
p k q n −k
n
n!
+∑ p k q n − k = n 2 p 2 + npq . (2-37)
k =1 ( n − k )! ( k − 1)!
26
PILLAI
Returning to (2-35), note that
k
− p >ε is equivalent to ( k − np ) 2 > n 2ε 2 , (2-38)
n
which in turn is equivalent to
n n
∑
k =0
( k − np ) Pn ( k ) >
2
∑k =0
n 2 ε 2 Pn ( k ) = n 2 ε 2 . (2-39)
∑ ( k − np )
k =0
2
Pn ( k ) = ∑k
k =0
2
Pn ( k ) − 2 np ∑ k Pn ( k ) + n 2 p 2
k =0
≥ ∑ ( k − np )
k − np > n ε
2
Pn ( k ) > n 2ε 2 ∑
k − np > n ε
Pn ( k )
(2-41) 27
= n ε P { k − np > n ε }.
2 2
PILLAI
Using (2-40) in (2-41), we get the desired result
k pq
P − p > ε < . (2-42)
n nε 2
Let us assume that the two dice are slightly loaded in such
a manner so that the faces 1, 2 and 3 appear with probability
1 − ε and faces 4, 5 and 6 appear with probability
6
1 + ε , ε > 0 for each dice. If T represents the combined
6
total for the two dice (following Text notation), we get 33
PILLAI
p4 = P{T = 4} = P{(1,3),(2, 2),(1,3)} = 3( 16 − ε ) 2
1 − ε 2 ) + 2( 1 − ε ) 2
p5 = P{T = 5} = P{(1, 4),(2,3),(3, 2),(4,1)} = 2( 36 6
1 − ε 2 ) + ( 1 − ε )2
p6 = P{T = 6} = P{(1,5),(2, 4),(3,3),(4, 2),(5,1)} = 4( 36 6
1 −ε2)
p7 = P{T = 7} = P{(1,6),(2,5),(3, 4),(4,3),(5, 2),(6,1)} = 6( 36
1 − ε 2 ) + ( 1 + ε )2
p8 = P{T = 8} = P{(2,6),(3,5),(4, 4),(5,3),(6, 2)} = 4( 36 6
1 − ε 2 ) + 2( 1 + ε ) 2
p9 = P{T = 9} = P{(3,6),(4,5),(5, 4),(6,3)} = 2( 36 6
p10 = P{T = 10} = P{(4,6),(5,5),(6, 4)} = 3( 16 + ε )2
p11 = P{T = 11} = P{(5,6),(6,5)} = 2( 16 + ε ) 2 .
(Note that “(1,3)” above represents the event “the first dice
shows face 1, and the second dice shows face 3” etc.)
For ε = 0.01, we get the following Table:
34
PILLAI
T=k 4 5 6 7 8 9 10 11
pk = P{T = k} 0.0706 0.1044 0.1353 0.1661 0.1419 0.1178 0.0936 0.0624
Thus
P{winning the game} = P1 + P2 = 0.5002 (2-49)
Although perfect dice gives rise to an unfavorable game,
35
PILLAI
a slight loading of the dice turns the fortunes around in
favor of the player! (Not an exciting conclusion as far as
the casinos are concerned).
Even if we let the two dice to have different loading
factors ε1 and ε 2 (for the situation described above), similar
conclusions do follow. For example, ε1 = 0.01 and ε 2 = 0.005
gives (show this)
P{winning the game} = 0.5015. (2-50)
Once again the game is in favor of the player!
Although the advantage is very modest in each play, from
Bernoulli’s theorem the cumulative effect can be quite
significant when a large number of game are played.
All the more reason for the casinos to keep the dice in
perfect shape. 36
PILLAI
In summary, small chance variations in each game
of craps can lead to significant counter-intuitive changes
when a large number of games are played. What appears
to be a favorable game for the house may indeed become
an unfavorable game, and when played repeatedly can lead
to unpleasant outcomes.
37
PILLAI
Appendix: Euler’s Identity
S. Ramanujan in one of his early papers (J. of Indian
Math Soc; V, 1913) starts with the clever observation that
if a2 , a3 , a5 , a7 , a11 , are numbers less than unity where
the subscripts 2,3,5,7,11, are the series of prime
numbers, then1
1 1 1 1
⋅ ⋅ ⋅ = 1 + a2 + a3 + a2 ⋅ a2 + a5
1 − a2 1 − a3 1 − a5 1 − a7
+ a 2 ⋅ a3 + a7 + a 2 ⋅ a 2 ⋅ a 2 + a3 ⋅ a3 + . (2-A)
Notice that the terms in (2-A) are arranged in such a way
that the product obtained by multiplying the subscripts are
the series of all natural numbers 2,3, 4,5,6,7,8,9, . Clearly,
(2-A) follows by observing that the natural numbers
38
1The relation (2-A) is ancient.
PILLAI
are formed by multiplying primes and their powers.
Ramanujan uses (2-A) to derive a variety of
interesting identities including the Euler’s identity that
follows by letting a2 = 1/ 2 s , a3 = 1/ 3s , a5 = 1/ 5s , in
(2-A). This gives the Euler identity
∞
∏ (1 − p1s ) = ∑1/ n s .
−1
(2-B)
p prime n =1
2(2n )! 1 1 1 1
= 2n 2n
+ 2n + 2n + 2n +
(2π ) 1 2 3 4
which gives
∞
(2π ) Bn
2n
S2 n = ∑1/ k =
∆ 2n (2-E)
k =1 2(2n )!
Thus1
∞
π 2 ∞
π 4
∑ 1/ k 2
=
6
; ∑ 1/ k 4
=
90
etc.
k =1 k =1
∞
1The ∑ 1/ k 2
series k =1 can be summed using the Fourier series expansion of a periodic ramp 40
signal as well. PILLAI