Estimation of Parameters
Unbiased estimator
Let 𝜃̂ be an estimate for the unknown parameter 𝜃 associated with the distribution of random
variable 𝑋. Then 𝜃̂ is an unbiased estimator for 𝜃 if 𝐸(𝜃̂) = 𝜃 ∀ 𝜃.
Consistent estimate
Let 𝜃̂ be an estimate of the parameter 𝜃. We say that 𝜃̂ is a consistent estimate of 𝜃 if
lim 𝑃{|𝜃̂ − θ| > 𝜖} = 0 ∀ 𝜖 > 0 𝑜𝑟 lim 𝑃{|𝜃̂ − θ| ≤ 𝜖} = 1 .
𝑛→∞ 𝑛→∞
Note: The above definition indicates that as the sample size increases, the estimate becomes
better.
Unbiasedness and consistency of estimate can be found using the following theorem:
Theorem
Let 𝜃̂ be an estimate of the parameter 𝜃 based on a sample size 𝑛. If 𝐸(𝜃̂) = 𝜃 and
lim 𝑉(𝜃̂) = 0 then 𝜃̂ is a consistent estimate of 𝜃.
𝑛→∞
Proof:
We shall prove by using Chebyshev’s inequality.
1 2 1 2
∴ lim 𝑃{|𝜃̂ − θ| > 𝜖} ≤ 2 𝐸(𝜃̂ − θ) = 2 𝐸{(𝜃̂ − E(𝜃̂)) + (E(𝜃̂) − θ)}
𝑛→∞ 𝜖 𝜖
(Add and subtract E(𝜃̂))
1
= {𝐸[𝜃̂ − E(𝜃̂)]2 + 2𝐸{[𝜃̂ − E(𝜃̂) ](E(𝜃̂) − θ)} + E(E(𝜃̂) − θ)2 }
𝜖2
1 2
= 2 {𝑉(𝜃̂) + 2[𝐸(𝜃̂) − E(𝜃̂)(E(𝜃̂) − θ)] + (E(𝜃̂) − θ) }
𝜖
1 2
= 2 {𝑉(𝜃̂) + (E(𝜃̂) − θ) } → 0 𝑎𝑠 𝑛 → ∞ using given condition.
𝜖
∴ 𝜃̂ is a consistent estimate of 𝜃.
Exercise
1. Show that sample mean is an unbiased and consistent estimate of population mean.
Solution:
Let 𝑋1 , 𝑋2 , … 𝑋𝑛 be samples taken from the distribution of 𝑋 having mean 𝜇
∑𝑛
𝑖=1 𝑋𝑖
∴ 𝑋̅ = is the sample mean
𝑛
To prove: 𝐸(𝑋̅) = 𝜇 and 𝑉(𝑋̅) = 0
𝑛 𝑛
1 1 ∑𝑛𝑖=1 𝐸(𝑋𝑖 ) 𝑛𝜇
̅
𝐸(𝑋) = 𝐸 ( ∑ 𝑋𝑖 ) = ∑ 𝐸(𝑋𝑖 ) = = =𝜇
𝑛 𝑛 𝑛 𝑛
𝑖=1 𝑖=1
That is, the sample mean is an unbiased estimate of the population mean
𝑛 𝑛
∑𝑛𝑖=1 𝑋𝑖 1 1 𝑛𝑉(𝑋) 𝜎 2
̅
𝑉(𝑋) = 𝑉 ( ) = 2 𝑉 (∑ 𝑋𝑖 ) = 2 ∑ 𝑉(𝑋𝑖 ) = =
𝑛 𝑛 𝑛 𝑛2 𝑛
𝑖=1 𝑖=1
2
𝜎
∴ lim 𝑉(𝑋̅) = lim
=0
𝑛→∞ 𝑛→∞ 𝑛
That is, the sample mean is a consistent estimate of the population mean.
2. Show that sample variance 𝑆 2 is not an unbiased estimate of population variance.
Solution:
Let 𝜎 2 be the variance of the distribution of 𝑋, i.e., 𝜎 2 is the population variance
Let 𝑆 2 be the sample variance
To prove: 𝐸(𝑆 2 ) ≠ 𝜎 2
∑𝑛 ̅ 2
𝑖=1(𝑋𝑖 −𝑋) ∑𝑛 2 ̅ 2 ̅
𝑖=1(𝑋𝑖 +(𝑋 ) −2𝑋 𝑋𝑖 ) [ ∑𝑛 2 ̅ 2 ̅2
𝑖=1(𝑋𝑖 ) ] +𝑛(𝑋) −2𝑛(𝑋 )
By definition, 𝑆 2 = = =
𝑛 𝑛 𝑛
∑𝑛 2
𝑖=1 𝑖 )
(𝑋
= − (𝑋̅)2
𝑛
(since ∑𝑛𝑖=1(𝑋̅)2 = 𝑛(𝑋̅)2 and 𝑛𝑋̅ = ∑𝑛𝑖=1 𝑋𝑖 )
∑𝑛𝑖=1(𝑋𝑖 2 ) ∑𝑛𝑖=1(𝑋𝑖 2 )
2)
𝐸(𝑆 = 𝐸 { ̅ 2
− (𝑋) } = 𝐸 { } − 𝐸{(𝑋̅)2 }
𝑛 𝑛
1
= 𝐸(∑𝑛𝑖=1(𝑋𝑖 2 )) − {𝐸(𝑋̅̅̅)2 }
𝑛
(since 𝐸(𝑋 2 ) = 𝑉(𝑋) + 𝜇2 , 𝐸(𝑋̅ 2 ) = 𝑉(𝑋̅) + 𝜇2 )
1
= 𝐸(𝑛𝑋 2 ) − {𝐸(𝑋̅̅̅)2 }
𝑛
= 𝐸(𝑋 2 ) − {𝐸(𝑋̅)2 } = 𝑉(𝑋) + 𝜇2 - {𝑉(𝑋̅) + 𝜇2 }
𝜎2 𝜎2
= [(𝜎 2 + 𝜇2 )] − { + 𝜇2 } (since sample variance is )
𝑛 𝑛
𝜎2
= (𝜎 2 + 𝜇2 ) − { + 𝜇2 }
𝑛
𝜎2
2
𝑛−1
=𝜎 − = 𝜎2 ( )
𝑛 𝑛
∴ 𝐸(𝑆 2 ) ≠ 𝜎 2 i.e., sample variance 𝑆 2 is not an unbiased estimate of population
variance
3. Show that if 𝑋 is a random sample of size 𝑛 with pdf
1 −𝑥
𝑓(𝑥, 𝜃) = {𝜃 𝑒 0 < 𝑥 < ∞, 0<𝜃<∞
𝜃
0 𝑒𝑙𝑠𝑒𝑤ℎ𝑒𝑟𝑒
𝜃2
then 𝑋̅ is an unbiased estimate of 𝜃 and has variance .
𝑛
Solution:
∞ ∞
1 −𝑥
𝐸(𝑋) = 𝜇 = ∫ 𝑥 𝑓(𝑥, 𝜃) 𝑑𝑥 = ∫ 𝑥 𝑒 𝜃 𝑑𝑥
−∞ 0 𝜃
𝑥 𝑥 ∞
1 𝑒− 𝜃 𝑒− 𝜃
= {𝑥 − } =𝜃
𝜃 −1 1
𝜃 𝜃2 0
∞ ∞
2)
1 −𝑥
𝐸(𝑋 = ∫ 𝑥 𝑓(𝑥, 𝜃) 𝑑𝑥 = ∫ 𝑥 2
2
𝑒 𝜃 𝑑𝑥
−∞ 0 𝜃
𝑥 𝑥 𝑥 ∞
1 𝑒 𝑒− 𝜃 𝑒 −
𝜃
−
𝜃
{ 𝑥2
= − 2𝑥 + 2 } = 2𝜃 2
𝜃 −1 1 −1
𝜃 𝜃 2 𝜃3 0
∴ 𝑉(𝑋) = 𝜎 2 = 2𝜃 2 − 𝜃 2 = 𝜃 2
The random variable 𝑋 has mean 𝜃 and variance 𝜃 2
𝜃2 𝜎2
Hence, 𝐸(𝑋̅) = 𝜃 , 𝑉(𝑋̅) = and lim 𝑉(𝑋̅) = lim =0
𝑛 𝑛→∞ 𝑛→∞ 𝑛
4. Let 𝑋1 , 𝑋2 , … 𝑋𝑛 be samples taken from a normal distribution with 𝜇 = 0 and variance 𝜎 2
∑ 𝑥𝑖 2
= 𝜃, 0 < 𝜃 < ∞. Show that 𝑌 = is an unbiased and consistent estimate of 𝜃.
𝑛
Solution:
𝜃 𝜃
𝑋~𝑁(0, 𝜃) ⟹ 𝑋̅~𝑁 (0, ) i.e., 𝐸(𝑋̅) = 0 and 𝑉(𝑋̅) =
𝑛 𝑛
∑ 𝑥𝑖 2
Let 𝑌 =
𝑛
To prove: 𝐸(𝑌) = 𝜃 and 𝑉(𝑌) = 0 𝑎𝑠 𝑛 → ∞
∑ 𝑥𝑖 2 1 2
𝑛 𝐸(𝑋 2 )
𝐸(𝑌) = 𝐸 ( ) = 𝐸 (∑ 𝑥𝑖 ) = = 𝐸(𝑋 2 )
𝑛 𝑛 𝑛
𝐸(𝑌) = 𝐸(𝑋 2 ) = 𝑉(𝑋) + [ 𝐸(𝑋)]2 = 𝜃 which implies 𝑌 is an unbiased estimate of 𝜃.
2 𝑋− 𝜇 2 𝑋− 𝜇 2 𝑋− 0 2 𝑋2
We know that 𝑋 ∼ 𝑁( 𝜇, 𝜎 ) , 𝑍 = ∼ 𝑁( 0, 1) and 𝑍 = ( ) =( ) = ∼
𝜎 𝜎 √𝜃 𝜃
𝜒 2 (1)
𝑋2 𝑋2
∴ 𝐸 ( ) = 1 𝑎𝑛𝑑 𝑉 ( ) = 2
𝜃 𝜃
This implies 𝐸(𝑋 2 ) = 𝜃 , 𝑉(𝑋 2 ) = 2𝜃 2
∑ 𝑥𝑖 2 1 𝑛 𝑉(𝑋 2 ) 𝑉(𝑋 2 ) 2𝜃2
Consider 𝑉(𝑌) = 𝑉 (
𝑛
) = 𝑛2 𝑉(∑ 𝑥𝑖 2 ) = 𝑛2
=
𝑛
=
𝑛
2𝜃2
∴ lim 𝑉(𝑌) = lim = 0 which implies 𝑌 is a consistent estimate of 𝜃.
𝑛→∞ 𝑛→∞ 𝑛
5. Let 𝑌1 and 𝑌2 be two independent unbiased statistics for 𝜃. The variance of 𝑌1 is twice the
variance of 𝑌2 . Find the constants 𝑘1 and 𝑘2 such that 𝑌 = 𝑘1 𝑌1 + 𝑘2 𝑌2 is an unbiased
statistic for 𝜃 with smallest possible variance for such a linear combination.
Solution:
Given that 𝐸(𝑌1 ) = 𝐸(𝑌2 ) = 𝜃, 𝑉(𝑌1 ) = 2𝑉( 𝑌2 ) = 2𝜎 2
To find: 𝑘1 𝑎𝑛𝑑 𝑘2 such that 𝐸(𝑌) = 𝐸(𝑘1 𝑌1 + 𝑘2 𝑌2 ) = 𝜃
𝑘1 𝐸(𝑌1 ) + 𝑘2 𝐸(𝑌2 ) = 𝜃
𝑘1 𝜃 + 𝑘2 𝜃 = 𝜃
𝑘1 + 𝑘2 = 1 ⇒ 𝑘2 = 1 − 𝑘1
2 2
𝑉(𝑌) = 𝑉(𝑘1 𝑌1 + 𝑘2 𝑌2 ) = 𝑘1 2 𝑉(𝑌1 ) + 𝑘2 𝑉(𝑌2 ) = 𝜎 2 (2𝑘1 2 + 𝑘2 )
2
𝑉(𝑌) = 𝜎 2 (2𝑘1 2 + 𝑘2 ) = 𝜎 2 (2𝑘1 2 + (1 − 𝑘1 )2 )
𝑑𝑉(𝑌)
𝑉(𝑌) has minima if =0
𝑑𝑘1
𝑑𝑉(𝑌) 𝑑[𝜎 2 (2𝑘1 2 +(1−𝑘1 )2 )]
= =0
𝑑𝑘1 𝑑𝑘1
1 2
⇒ 4𝑘1 − 2(1 − 𝑘1 ) = 0 ⇒ 𝑘1 = and 𝑘2 =
3 3
6. Let 𝑋1 , 𝑋2 , … , 𝑋25 ; 𝑌1 , 𝑌2 , … , 𝑌25 be two independent random samples from the
𝑋̅
distribution 𝑁(3, 16) , 𝑁(4, 9) respectively. Evaluate P ( ̅ > 1)
𝑌
Solution:
Given that 𝑋 ~𝑁(3, 16), 𝑌~𝑁(4, 9).
16 9
Then, 𝑋̅ ~𝑁 (3, ) , 𝑌̅ ~𝑁(4, )
25 25
𝑋̅
Now > 1 ⇒ 𝑋̅ > 𝑌̅ ⇒ 𝑋̅ − 𝑌̅ > 0.
𝑌̅
16 9
Since 𝑋̅ − 𝑌̅ ~ 𝑁[ 3 × 1 + 4 × (−1), 12 × + (−1)2 × ]~𝑁(−1, 1)
25 25
We have, 𝑍 = 𝑋̅ − 𝑌̅ + 1 ~𝑁(0, 1)
𝑋̅
Consider 𝑃 ( ̅ > 1) = 𝑃(𝑋̅ − 𝑌̅ > 0)
𝑌
= 𝑃(𝑋̅ − 𝑌̅ + 1 > 1)
= 𝑃(𝑍 > 1)
= 1 − 𝜙(1) = 1 − 0.841 = 0.159
Interval estimation
Let 𝑋 be a random variable with some probability distribution, depending on an unknown
parameter 𝜃. An estimate of 𝜃 given by two magnitudes within which 𝜃 can lie is called an
interval estimate of the parameter 𝜃. The process of obtaining an interval estimate for 𝜃 is called
interval estimation.
Confidence interval
Let 𝜃 be an unknown parameter to be determined by a random sample 𝑋1 , 𝑋2 , 𝑋3 , … 𝑋𝑛 of size
𝑛. The confidence interval for the parameter θ is a random interval containing the parameter
with high probability say 1 − 𝛼 where 1 − 𝛼 is called the confidence coefficient.
NOTE:
Let 𝑋1 , 𝑋2 . . 𝑋𝑛 be a random sample of size 𝑛 from a normal distribution 𝑁(𝜇, 𝜎 2 )
∑𝑛
𝑖=1 𝑋𝑖 𝜎2
1. 𝑋̅ = ~ 𝑁(𝜇, ) for 𝜇, 𝜎 2 is known
𝑛 𝑛
𝑋̅−𝜇
2. 𝑇 = 𝑆 ~ 𝑇(𝑛 − 1) for 𝜇, 𝜎 2 is unknown
( )
√𝑛−1
∑𝑛 ̅ 2
𝑖=1(𝑋𝑖 −𝑋)
3. 𝑌 = ~𝜒 2 (𝑛) for 𝜎, 𝜇 is known
𝜎2
𝑛𝑆 2
4. 𝑌 = ~𝜒 2 (𝑛 − 1) for 𝜎, 𝜇 is unknown
𝜎2
Confidence Interval for mean
Confidence interval for mean
𝝈𝟐 is known: 𝝈𝟐 is unknown:
∑𝑛 𝑋 𝜎2 𝑋̅−𝜇
Consider 𝑋̅ = 𝑖=1 𝑖 ~ 𝑁(𝜇, ) Consider 𝑇 = 𝑆 ~𝑇(𝑛 − 1) i.e., t-
𝑛 𝑛
√𝑛−1
𝑋̅ − 𝜇
∴ 𝑍 = 𝜎 ~𝑁(0,1) distribution with 𝑛 − 1 degrees of freedom
√𝑛
To find 𝑎 such that 𝑃(−𝑎 < 𝑍 < 𝑎) = 1 −
𝛼 To find 𝑎 such that 𝑃(−𝑎 < 𝑇 < 𝑎) = 1 −
𝛼
𝑋̅ − 𝜇
𝑃 (−𝑎 < 𝜎 < 𝑎) = 1 − 𝛼 𝑋̅ − 𝜇
√𝑛 𝑃 (−𝑎 < < 𝑎) = 1 − 𝛼
𝑆
𝑎𝜎 𝑎𝜎
𝑃 (𝑋̅ − < 𝜇 < 𝑋̅ + ) = 1 − 𝛼 √𝑛 − 1
√𝑛 √𝑛 𝑎𝑆 𝑎𝑆
𝑎𝜎 𝑎𝜎 𝑃 (𝑋̅ − < 𝜇 < 𝑋̅ + )
𝜇 ∈ (𝑋̅ − , 𝑋̅ + ) √𝑛 − 1 √𝑛 − 1
√ 𝑛 √𝑛 =1−𝛼
𝑎𝑆 𝑎𝑆
𝜇 ∈ (𝑋̅ − , 𝑋̅ + )
√𝑛 − 1 √𝑛 − 1
Exercise
1. Let the observed value of 𝑋̅ of size 20 from a normal distribution with 𝜇 and 𝜎 2 = 80 be
81.2. Obtain 95% confidence interval for the mean 𝜇.
Solution:
80
Let 𝑋~ 𝑁(𝜇, 80) ⟹ 𝑋̅ ~ 𝑁 (𝜇, ) = 𝑁(𝜇, 4)
20
𝑋̅−𝜇 𝑋̅ −𝜇
∴𝑍= 𝜎 = ~𝑁(0,1)
2
√𝑛
𝑃(−𝑎 < 𝑍 < 𝑎) = 0.95
1.95
2 𝜙(𝑎) − 1 = 0.95 ⇒ 𝜙(𝑎) = = 0.975 ⇒ 𝑎 = 1.96
2
𝑎𝜎 𝑎𝜎
⇒ 𝜇 ∈ (𝑋̅ − , 𝑋̅ + ) = (81.2 − 1.96 × 2, 81.2 + 1.96 × 2)
√𝑛 √𝑛
⇒ 𝜇 ∈ (77.28, 85.12)
2. Let a random sample of size 17 from 𝑁(𝜇, 𝜎 2 ) yield 𝑋̅ = 4.7 and 𝑆 2 = 5.76. Determine
90% confidence interval for 𝜇.
Solution:
Given: 𝑛 = 17, 𝑋̅ = 4.7 and 𝑆 2 = 5.76
𝑋̅−𝜇 4(𝑋̅ −𝜇)
Let 𝑇 = 𝑆 = ~𝑇(17 − 1) ~𝑇(16)
√5.76
√𝑛−1
To find 𝑎 such that 𝑃(−𝑎 < 𝑇 < 𝑎) = 0.90
2𝜙(𝑎) − 1 = 0.90 ⇒ 𝑎 = 1.75
1.75 × √5.76 1.75 × √5.76
⟹ 𝜇 ∈ (4.7 − , 4.7 + ) ⇒ 𝜇 ∈ (3.65, 5.75)
√16 √16
Confidence interval for variance
Confidence interval for variance
𝝁 is known: 𝝁 is unknown:
∑𝑛 ̅ 2
𝑖=1(𝑋𝑖 −𝑋) 𝑛𝑆 2
Let 𝑌 = ~𝜒 2 (𝑛) Let 𝑌 = ~𝜒 2 (𝑛 − 1)
𝜎2 𝜎2
To find a and b such that 𝑃(𝑎 < 𝑌 < 𝑏) = To find 𝑎 and 𝑏 such that 𝑃(𝑎 < 𝑌 < 𝑏) =
1−𝛼 1−𝛼
𝛼 𝛼 𝛼 𝛼
i.e., 𝑃(𝑌 < 𝑎) = , 𝑃(𝑌 > 𝑏) = i.e., 𝑃(𝑌 < 𝑎) = , 𝑃(𝑌 > 𝑏) =
2 2 2 2
∑𝑛𝑖=1(𝑋𝑖 − 𝑋̅)2 ∴ 𝑃 (𝑎 <
𝑛𝑆 2
< 𝑏) = 1 − 𝛼
∴ 𝑃 (𝑎 < < 𝑏) = 1 − 𝛼 𝜎2
𝜎2
2
1 𝜎2 1
1 𝜎 1 𝑃( < < )=1−𝛼
𝑃( < < )=1−𝛼 𝑏 𝑛𝑆 2 𝑎
𝑏 ∑𝑛𝑖=1(𝑋𝑖 − 𝑋̅)2 𝑎
𝑛𝑆 2 2
𝑛𝑆 2
∑𝑛𝑖=1(𝑋𝑖 −𝑋 ̅ )2 ∑𝑛𝑖=1(𝑋𝑖 ̅ )2
−𝑋 𝑃( < 𝜎 < )=1−𝛼
𝑃( < 𝜎2 < ) 𝑏 𝑎
𝑏 𝑎
2
𝑛𝑆 2 𝑛𝑆 2
=1−𝛼 ⇒ 𝜎 ∈( , )
𝑏 𝑎
∑𝑛𝑖=1(𝑋𝑖 − 𝑋̅)2 ∑𝑛𝑖=1(𝑋𝑖 − 𝑋̅)2
2
⇒ 𝜎 ∈( , )
𝑏 𝑎
∑ 𝑋𝑖
where 𝜇 = = 𝑋̅
𝑛
Exercise
1. If 8.6, 7.9, 8.3, 6.4, 8.4, 9.8, 7.2, 7.8, 7.5 are the observed values of a random sample of
size 9 from a distribution 𝑁(8, 𝜎 2 ), construct 90% confidence interval for 𝜎 2 .
Solution:
Given: 𝜇 = 8, 𝑛 = 9
∑𝑛 ̅ 2
𝑖=1(𝑋𝑖 −𝑋)
𝑌=
𝜎2
1
= {0.62 + 0.12 + 0.32 + 1.62 + 0.42 + 1.82 + 0.82 + 0.22 + 0.52 }
𝜎2
7.35
= ~𝜒 2 (9)
𝜎2
To find a and b such that 𝑃(𝑎 < 𝑌 < 𝑏) = 1 − 𝛼 = 0.90
⇒ 𝛼 = 0.10
𝛼 0.10
𝑃(𝑌 < 𝑎) = = = 0.05 ⇒ 𝑎 = 3.33 ,
2 2
𝛼 0.10
𝑃(𝑌 > 𝑏) = = = 0.05
2 2
⇒ (𝑌 < 𝑏) = 1 − 0.05 = 0.95 ⇒ 𝑏 = 16.9
using chi square table for 9 degrees of freedom.
∑𝑛 ̅ 2 ∑𝑛
𝑖=1(𝑋𝑖 −𝑋 ) (𝑋 −𝑋̅)2 7.35 7.35
∴ 𝜎2 ∈ ( , 𝑖=1 𝑖 ) =( , ) = (0.43, 2.21)
𝑏 𝑎 16.9 3.33
2. A random sample of size 15 from a normal distribution 𝑁(𝜇, 𝜎 2 ) yields 𝑋̅ = 3.2 , 𝑆 2 =
4.24 . Determine a 90% confidence interval for 𝜎 2 .
Solution:
Given: 1 − 𝛼 = 0.9 ⇒ 𝛼 = 0.1
𝑛𝑆 2
𝑌 = 2 ~𝜒 2 (15 − 1) = 𝜒 2 (14)
𝜎
𝛼 0.10
∴ 𝑃(𝑌 < 𝑎) = = = 0.05 ⇒ 𝑎 = 6.57
2 2
𝛼 0.10
𝑃(𝑌 > 𝑏) = = = 0.05
2 2
⇒ (𝑌 < 𝑏) = 1 − 0.05 = 0.95 ⇒ 𝑏 = 23.7
using chi square table for 14 degrees of freedom.
15 × 4.24 15 × 4.24
∴ 𝜎2 ∈ ( , ) = (2.68, 9.68)
23.7 6.57
3. A random sample of size 9 from a normal distribution 𝑁(𝜇, 𝜎 2 ) yields 𝑆 2 = 7.63.
Determine a 95% confidence interval for 𝜎 2 .
(Ans: 𝜎 2 ∈ (3.924, 31.5))
4. A random sample of size 15 from a normal distribution 𝑁(𝜇, 𝜎 2 ) yields 𝑋̅ = 3.2 , 𝑆 2 =
4.24 . Determine a 95% confidence interval for 𝜇.
(Ans: 𝜇 ∈ (2.02, 4.38))
5. A random sample of size 25 from a normal distribution 𝑁(𝜇, 4) yields 𝑋̅ = 78.3 , 𝑆 2 =
4.24 . Determine a 99% confidence interval for 𝜇.
(Ans: 𝜇 ∈ (77.268, 79.332))
Maximum Likelihood Estimate for 𝜽 (MLE)
Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be a random sample from the random variable 𝑋 and let 𝑥1 , 𝑥2 , … , 𝑥𝑛 be
sample values. We denote likelihood function 𝐿 as the following function
𝐿(𝑋1 , 𝑋2 , … , 𝑋𝑛 ; 𝜃) = 𝑓(𝑥1 , 𝜃)𝑓(𝑥2 , 𝜃) … 𝑓(𝑥𝑛 , 𝜃)
Maximum Likelihood Estimate (MLE) of 𝜃, say 𝜃̂ based on a random sample 𝑋1 , 𝑋2 , … 𝑋𝑛 is that
value of 𝜃 that maximizes 𝐿(𝑋1 , 𝑋2 , … , 𝑋𝑛 ; 𝜃).
Exercise
1. Let 𝑋1 , 𝑋2 , … 𝑋𝑛 denote a random sample of size 𝑛 from a distribution having pdf
𝜃 𝑥 (1 − 𝜃)1−𝑥 , 0 ≤ 𝜃 ≤ 1
𝑓(𝑥, 𝜃) = {
0, 𝑒𝑙𝑠𝑒𝑤ℎ𝑒𝑟𝑒
Find the MLE for 𝜃.
Solution:
𝐿(𝑋1 , 𝑋2 , … , 𝑋𝑛 ; 𝜃) = 𝑓(𝑥1 , 𝜃)𝑓(𝑥2 , θ) … 𝑓(𝑥𝑛 , 𝜃)
= 𝜃 𝑥1 (1 − 𝜃)1−𝑥1 . 𝜃 𝑥2 (1 − 𝜃)1−𝑥2 . . . 𝜃 𝑥𝑛 (1 − 𝜃)1−𝑥𝑛
𝑛 𝑛
= 𝜃 ∑𝑖=1 𝑥𝑖 (1 − 𝜃)𝑛−∑𝑖=1 𝑥𝑖
Taking log on both sides and then partially differentiating with respect to 𝜃,
𝜕 (log 𝐿) 1 1
= ∑𝑛𝑖=1 𝑥𝑖 × + (𝑛 − ∑𝑛𝑖=1 𝑥𝑖 ) (− )
𝜕𝜃 𝜃 1−𝜃
𝜕 (log 𝐿)
For maximum, =0
𝜕𝜃
On simplifying, we get MLE of 𝜃 = 𝜃̂ = 𝑋̅
2. Let 𝑋1 , 𝑋2 , … 𝑋𝑛 denote a random sample of size 𝑛 from a distribution having pdf
𝜃 𝑥 𝑒 −𝜃
𝑓(𝑥, 𝜃) = { 𝑥! , 0≤𝜃≤1
0, 𝑒𝑙𝑠𝑒𝑤ℎ𝑒𝑟𝑒
Find MLE for 𝜃.
Solution:
𝐿(𝑋1 , 𝑋2 , … , 𝑋𝑛 ; 𝜃) = 𝑓(𝑥1 , 𝜃)𝑓(𝑥2 , θ) … 𝑓(𝑥𝑛 , 𝜃)
𝜃 𝑥1 𝑒 −𝜃 𝜃 𝑥2 𝑒 −𝜃 𝜃 𝑥𝑛 𝑒 −𝜃
= × × …×
𝑥1 ! 𝑥2 ! 𝑥𝑛 !
𝑛
= 𝜃 ∑𝑖=1 𝑥𝑖 𝑒 −𝑛𝜃
𝜕 (log 𝐿)
Taking log on both sides and = 0 gives MLE, 𝜃̂ = 𝑋̅
𝜕𝜃
3. Find the MLE of normal distribution 𝑁(𝜃1 , 𝜃2 ) where −∞ < 𝜃1 < ∞, 0 < 𝜃2 < ∞.
Solution:
(𝑥−𝜇)2
2 2) 1 −
If 𝑋~𝑁(𝜇, 𝜎 ), then 𝑓(𝑥, 𝜇, 𝜎 = 𝑒 2𝜎2
√2𝜋𝜎
(𝑥−𝜃1 )2
1 −
Therefore, 𝑓(𝑥, 𝜃1 , 𝜃2 ) = 𝑒 2𝜃2
√2𝜋𝜃2
𝐿(𝑋1 , 𝑋2 , … , 𝑋𝑛 ; 𝜃) = 𝑓(𝑥1 , 𝜃)𝑓(𝑥2 , θ) … 𝑓(𝑥𝑛 , 𝜃)
1 (𝑥 −𝜃 )2 1 (𝑥 −𝜃 )2 1 (𝑥 −𝜃 )2
− 1 1 − 2 1 − 𝑛 1
= 𝑒 2𝜃2 × 𝑒 2𝜃2 × ⋯× 𝑒 2𝜃2
√2𝜋𝜃2 √2𝜋𝜃2 √2𝜋𝜃2
1 1 𝑛
− ∑ (𝑥 −𝜃 )2
= 𝑛 𝑒 2 𝑖=1 𝑖 1
2𝜃
(2𝜋𝜃2 )2
Taking logarithm on both sides
𝑛
1 2 ∑𝑛𝑖=1(𝑥𝑖 − 𝜃1 )2
log 𝐿 = log ( ) − log 𝑒
2𝜋𝜃2 2𝜃2
𝑛 1 ∑𝑛𝑖=1(𝑥𝑖 − 𝜃1 )2
= log ( )−
2 2𝜋𝜃2 2𝜃2
𝑛 1 ∑𝑛𝑖=1(𝑥𝑖2 + 𝜃12 − 2𝑥𝑖 𝜃)
= log ( )−
2 2𝜋𝜃2 2𝜃2
Differentiating partially with respect to 𝜃1 , we get
𝜕(log 𝐿) 1
=− ∑𝑛𝑖=1(2𝜃1 − 2𝑥𝑖 )
𝜕𝜃1 2𝜃2
𝜕 (log 𝐿)
For maximum, =0
𝜕𝜃1
On simplifying, we get MLE of 𝜃1 = 𝜃̂1 = 𝑋̅
Differentiating partially with respect to 𝜃1 , we get
𝑛
𝜕(log 𝐿) 𝑛 1
=− + 2 [∑(𝑥𝑖 − 𝜃1 )2 ]
𝜕𝜃2 2𝜃2 2𝜃2
𝑖=1
𝜕 (log 𝐿)
For maximum, =0
𝜕𝜃2
𝑛 2
∑ (𝑥 −𝑋̅)
On simplifying, we get MLE of 𝜃2 = 𝜃̂2 = 𝑖=1 𝑖 = 𝑠2
𝑛
