Scribd
Upload
46 views15 pages

Approximations To Probability Distributions: Limit Theorems

- Limit theorems can be used to approximate probability distributions when exact distributions are difficult to obtain for large sample sizes. They describe the behavior of estimators as sample sizes increase. - The weak law of large numbers states that the sample mean converges in probability to the population mean as the sample size increases. - The central limit theorem establishes that the sampling distribution of the sample mean will be approximately normally distributed for large sample sizes, regardless of the underlying population distribution.

Uploaded by

theboyisbobby
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
46 views15 pages

Approximations To Probability Distributions: Limit Theorems

- Limit theorems can be used to approximate probability distributions when exact distributions are difficult to obtain for large sample sizes. They describe the behavior of estimators as sample sizes increase. - The weak law of large numbers states that the sample mean converges in probability to the population mean as the sample size increases. - The central limit theorem establishes that the sampling distribution of the sample mean will be approximately normally distributed for large sample sizes, regardless of the underlying population distribution.

Uploaded by

theboyisbobby
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 15

Approximations to Probability

Distributions: Limit Theorems


Sequences of Random Variables
Interested in behavior of functions of random
variables such as means, variances, proportions
For large samples, exact distributions can be
difficult/impossible to obtain
Limit Theorems can be used to obtain properties
of estimators as the sample sizes tend to infinity
Convergence in Probability Limit of an estimator
Convergence in Distribution Limit of a CDF
Central Limit Theorem Large Sample Distribution of
the Sample Mean of a Random Sample
Convergence in Probability
The sequence of random variables, X
1
,,X
n
, is said to
converge in probability to the constant c, if for every c>0,


Weak Law of Large Numbers (WLLN): Let X
1
,,X
n
be iid
random variables with E(X
i
)= and V(X
i
)=o
2
< . Then the
sample mean converges in probability to :
1 ) | (| lim = s

c c X P
n
n
( ) ( )
n
X
X
X P X P
n
i
i
n
n
n
n
n

=

=
= s = >
1
where
1 lim or 0 lim c c
Proof of WLLN
( ) ( )
( )

c
c
o
c
c
o
c
o
o
c
o
o
c
o
c o
c
o
o
o o
o o
o
o
o

Prob
2
2
2
2
2
2
2
2 2
2
2
2
2 2
2
2
0 0 lim | | lim
1
| |
1
: Let
1
| |
1
) | (|
1
1 ) | (|
) 1 (
1
1 ) ( : Inequality s Chebyshev'

> = s >
= s |
.
|

\
|
= = >
= = = =
s |
.
|

\
|
= >
s > > s
> > + s s
= = = =

n
n
X
n
n
X X
n
X X
n
X X X X
X X X X
X
n
X
n
X
n
X P
n k
n
k
k X P
n k
n
k
n
k
n
k
k
n
k
k X P
k
k X P
k
k X P
k
k
k X k P
n
n
X V X E
Other Case/Rules
Binomial Sample Proportions






Useful Generalizations:
p p
n
p p
p V p p E
n
X
n
X
p
p np X V np X E X X
p p X V p X E
i
i
X p n X
n
i
i
n
i
i
i i i
Prob ^
^ ^
1
^
1
) 1 (
, Let
) 1 ( ) ( , ) (
) 1 ( ) ( ) (
Failure a is Trial if 0
Success a is Trial if 1
) , ( Binomial ~

=
|
.
|

\
|
=
|
.
|

\
|
= =
= = =
= =

=
=
) 1 ) 0 ( provided ( ) 4
) 0 provided ( / / ) 3
) 2
1)
: Then and : Suppose
Prob
Prob
Prob
Prob
Prob Prob
= >
=

+ +

n X n
Y Y X n n
Y X n n
Y X n n
Y n X n
X P X
Y X
Y X
Y X
Y X





Convergence in Distribution
Let Y
n
be a random variable with CDF F
n
(y).
Let Y be a random variable with CDF F(y).
If the limit as n of F
n
(y) equals F(y) for every point
y where F(y) is continuous, then we say that Y
n

converges in distribution to Y
F(y) is called the limiting distribution function of
Y
n

If M
n
(t)=E(e
tYn
) converges to M(t)=E(e
tY
), then Y
n

converges in distribution to Y
Example Binomial Poisson
X
n
~Binomial(n,p) Let =np p=/n

M
n
(t) = (pe
t
+ (1-p))
n
= (1+p(e
t
-1))
n
= (1+(e
t
-1)/n)
n

Aside: lim
n
(1+a/n)
n
= e
a

lim
n
M
n
(t) = lim
n
(1+(e
t
-1)/n)
n
= exp((e
t
-1))

exp((e
t
-1)) MGF of Poisson()

X
n
converges in distribution to Poisson(=np)
Example Scaled Poisson N(0,1)
( ) { }
( )
( ) ( )
( )
( ) ( )
( )
( ) ( ) ( ) ( )
( ) ( )
)) 1 , 0 ( (
! 3
/
! 2
exp lim ) ( lim
: as limit taking Now
! 3
/
! 2
exp
! 3
/
! 2
exp
! 3
/
! 2
/
/ 1 1 exp ) (
! 3
/
! 2
/
/ 1 1 1
!
: Aside
1 exp ) (
) ( ) (
,
1
) (
) (
) ( ), ( ) ( ) ( ~
2 /
2 / 1 3 2
2 / 1 3 2 2 / 1 3 2
2 / 3 3 2
2 / 3 3 2
/
0
/ ) 1 (
) 1 (
2
/
N MGF e
t t
t M
t t t t
t t
t t
t t t M
t t
t e
i
x
e
e t e e t M
at M e t M
b a b aX
X
X V
X E X
Y
e t M X V X E Poisson X
t
Y
Y
t
i
i
x
t e t
Y
X
bt
b aX
e
X
t
t
=
)
`

|
|
.
|

\
|
+ + =

)
`

|
|
.
|

\
|
+ + =
)
`

|
|
.
|

\
|
+ + + + =
)
`

|
|
.
|

\
|
+ + + + + + =
+ + + + + = =
+ = =
=
= = + =

=
= = =

=

+

Poisson/Normal CDF Y=(X-L)/sqrt(L) L=25


0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-6 -4 -2 0 2 4 6 8
y
F
(
y
)
Poisson CDF
Z CDF
Central Limit Theorem
Let X
1
,X
2
,,X
n
be a sequence of independently and
identically distributed random variables with finite
mean , and finite variance o
2
. Then:





Thus the limiting distribution of the sample mean is a
normal distribution, regardless of the distribution of the
individual measurements

( )
n
X
X N
X n
n
i
i
=
=

1
Dist
where ) 1 , 0 (
o

Proof of Central Limit Theorem (I)
Additional Assumptions for this Proof:
The moment-generating function of X, M
X
(t), exists in
a neighborhood of 0 (for all |t|<h, h>0).
The third derivative of the MGF is bounded in a
neighborhood of 0 (M
(3)
(t) B< for all |t|<h, h>0).
Elements of Proof
Work with Y
i
=(X
i
-)/o
Use Taylors Theorem (Lagrange Form)
Calculus Result: lim
n
[1+(a
n
/n)]
n
= e
a
if lim
n
a
n
=a

Proof of CLT (II)
( )
( )
( )
( )
target" " our is This ) ( ) ( ) (
1
1 1
) (
1 ) ( 0 ) ( nt) (independe : Define
1
1
1
1
1
/ ) / ( /
t M t M t M
Y
n
n
Y
n Y n
X n
X
X
n
Y
t
M e e E e e E t M
Y V Y E
X
Y
n
Y
n
X n
n
i
i
n
i
i
n
i
i
X
t t X t
X
t
Y
i i
i
i
n
i i
i
i
i
i
= =
= = =

=
|
.
|

\
|
= =
|
|
.
|

\
|
=
= =

=
=
=

|
.
|

\
|

=

o

o o o o

o

o o

o
o

Proof of CLT (III)
( ) ( )
x a t a x
k
t f
x r
a x
k
t f
a x
k
a f
a x a f a f x f
n
t
M
n
t
M
n
t
M
e e E e E t M
x
k
x
k
k
k
x
k
k
k
n
Y Y Y
Y n t Y n t
Y n t
n
n
n
i
and between strictly with ) (
)! 1 (
) (
) ( : where
) (
)! 1 (
) (
) (
!
) (
) )( ( ' ) ( ) (
: form) (Lagrange Theorem s Taylor' : Aside
) (
1
) (
1
) 1 ( ) (
) / ( ) / (
) / (
1
1
+
+
+

+
=

+
+ + + + =
(

|
.
|

\
|
= |
.
|

\
|
|
.
|

\
|
= =

Proof of CLT (IV)


( )
( )
( ) | |
2 / 3
3 2
3 2
) 3 ( ) 3 (
2
2 ) 2 ( ) 2 (
0
1
) 1 ( ) (
6 2
1 0
! 3
0
! 2
1
0 0 1
) assumption (Previous ) ( ) (
1 0 1 ) ( ) ( ) 0 ( ) (
0 ) ( ) 0 ( ' ) ( '
1 ) 1 ( ) 0 ( ) (
2 , 0 0 ) ( ) (
: n Applicatio Current
) max( ) min(
) (
)! 1 (
) (
) (
!
) (
) )( ( ' ) ( ) (
n
t B
n
t
n
t B
n
t
n
t
n
t
M
B B t M t f
Y E Y V Y E M a f
Y E M a f
E e E M a f
k
n
t
t a
n
t
x M f
a,x , a,x t
a x
k
t f
a x
k
a f
a x a f a f x f
n n
Y
n x Y x
Y
Y
Y
Y
x Y
x
k
x
k
k
k
+ + = |
.
|

\
|
+ |
.
|

\
|
+ |
.
|

\
|
+ = |
.
|

\
|

< s = =
= + = + = = =
= = =
= = = =
= |
.
|

\
|
e = = - = -
e

+
+ + + + =
+
+

Proof of CLT (V)


( )
) 1 , 0 (
)) 1 , 0 ( ( ) ( lim
) (
2 6 2
lim
6 2
lim lim
6 2
ere wh 1 lim
6 2
1
1 lim
6 2
1 lim lim ) ( lim
Dist
2 /
2
2 / 1
3 2
2 / 1
3 2
2 / 1
3 2
2 / 1
3 2
2 / 3
3 2
2
N
X n
N MGF e e t M
B a
t
n
Bt t
n
t B t
a
n
t B t
a
n
a
n
t B t
n
n
t B
n
t
n
t
M t M
t a
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
Y
n
n
n

= =
< = =
|
|
.
|

\
|
+ s
|
|
.
|

\
|
+ =
+ =
(

+ =
(

|
|
.
|

\
|
+ + =
(

+ + =
(

|
.
|

\
|
=




o

You might also like