Estimation Theory Estimation Theory

Chapter 3
Likelihood Function Likelihood Function
Higher dependence of data PDF on unknown parameter
results in higher estimation accuracy results in higher estimation accuracy
2
) 0 (
] 0 [ ] 0 [ : Example
N
W A X
|
+ =
Only 1 data sample
2
2
) , 0 (
POOR A

l If
W[0] A X[0] A

Choose
N |
+ = =
o
2
2
GOOD A

small, If
POOR A large, If
o
o
A A E = ]

[
( )
( )
2
2
] 0 [
2
1
1
]; 0 [ P
: function) d (likelihoo paramter unknown of function a as PDF Data
A X
i
e A X

o
( )
i
2
]; 0 [ P
i
i
e A X =
o t
Likelihood Function
( ) ; 3 ] 0 [
1
A X P =
Small noise variance
More dependence on A
S a o se a a ce
3 A
( ) ; 3 ] 0 [
2
A X P =
Large noise variance
( )
2
3
A
Likelihood Function
P
i
(X[0]=3;A) vs A is the probability (likelihood)
of observing X[0]=3 when A is the true value o obse g [0] 3 e s t e t ue a ue
If
1
=1/3 => probability ~0 for A>4 since most
likely values of A are 3 3
1
=[2,4]

Concentration of P
i
(X[0]=3;A) restricts possible
values of A
PDF Concentration
Parameter Accuracy

|
|
Parameter Accuracy
|
PDF Curvature
To measure the sharpness of the PDF, use curvature
2
1 ) ln( c P
d fi d b (f G i df)
As
2
, curvature increases.
2 2
) (
o
=
c

A
+
defined by (for Gaussian pdf)
As , curvature increases.
Note:
+
]) 0 [ ( )

(
2
X V A V
P by ) P(X; denote we
notation, simplify to : Note
u
Curvature
1
) ln(
1

]) 0 [ ( ) (
2
2
P
X Var A Var
=
c
=
= = o
y ) ( ;
) ln(
Curvature ) ln(
-
2
2
P
A
P
c
c
c
variable random a is
) ln(
- general, In
2
A
P
c
c
PDF Curvature
t PDF t Th f
u
|
|
.
|

\
|
c
c

) ln(
E
use we curvature PDF measure to Therefore,
2
2
P
u
. \
c

Scalar CRLB Theorem
0 ]
)) ; ( ln(
[ =
c
c
u
u X P
E
u
u
satisfies

estimator unbiased any of variance the

>
of value true
)) ; ( ln(
1
)

Var(
y
2
2
X P
E
u
u
u
c
(

c
)) ( l (
if only and If Also,
of value true
2
X P
E
Note : expectation is w.r.t.
where is deterministic ) ; ( u X p
u
( ) u u
u
u
=
c
c
) ( ) (
)) ; ( ln(
X g I
X P
where is deterministic ) ; ( u X p
u
is satisfied for some functions g and I an estimator can is satisfied for some functions g and I, an estimator can
be found that attains CRLB. That estimator is
and the minimum variance is
) ( / 1 u I
) (

x g = u
1 - N 0,1,...., n W[n] A X[n] : Example = + =
) (
exists) it (if A of estimator MVU Find : Problem
1 N 0,1,...., n W[n] A X[n] : Example
2
Variance
with WGN |
+
o
h h b d i f i
found be can estimator If CRLB, Compute : Approach
exists) it (if A of estimator MVU Find : Problem
( )
1
) ; X p(
! MVUE have then we bound, CRLB satisfying
2
2
] [
2
1
1
A n X
N
e A

[
o

2
) ; X p(
2
0 n
e A
=
[
=
o
o t
Example (Contd)
( )
e
A n X
N
N
n
] [
2
1
N/2
) (2
1

2
1
0
2
o t
o

=


=
( ) A n X
A A
P
N
2
2
N/
] [
2
1

) ln(

) (2 o t
|
.
|

\
|

c
c
=
c
c

( )
( ) ( ) A X A n X
A A
n
2 2
2
N
] [
1
] [
2o
= =
. \
c c

( ) ( )
N
P
A X A n X
n
2
2
2 2
) ln(
] [
o o

=
c

N
A
2
2
2
1
) A

Var(

o
o
= >
=
c
N
N
2
) A Var(
o
>
Example (Contd)
n X N
N
=

1
Unbiased is ] [ 1 X but
n

=
2
0
MVUE! is X
N
variance has And
] [
o
iff h ld lit Th CRLB F
? properties these had X that recall not did we if What
N
A X g A I
A
=
c
) ) ( )( (
)) ; X ln(P(

iff holds equality Theorem, CRLB From
( ) A X
N A
g
A
=
c
c
2
)) ; X ln(P(
but
) ) ( )( (
( )
A c
2
but
o
CRLB
d
1
) A

(
and e on varianc bound satisfies
2
X
o

expected. as
I(A)
1
) A Var(
N
o
= =
Summary: CRLB Theorem not only gives us a
lower bound on variance of any unbiased estimator
but also finds the estimator (if it exisits) that
attains the lower bound
Efficient Estimator
Estimator that attains Cramer Rao lower bound on
variance termed as EFFICIENT (it makes efficient
use of data)
VAR VAR
u

u
u u
MVU & efficient

u
efficient not but MVU

u
MVU & efficient u
Example
( )
Estimation Phase
( )
estimate known, f A,
1 - 0,1,..N n : ] [ f 2 ACos X[n]
0
0
|
| t = + + = n W n
( )
( )
1
; X p
) 2 ( ] [
2
1
2 2
0
2
0
2
|
|
| t
o

=
+
e
n f ACos n X
N
n
( )
( ) ) 2 cos( ] [
1 ln
) 2 (
; p
2 2
| t
to
|
+
c

n f A n X
P
N
( )
( ) ) 2 sin(
) 2 cos( ] [
0
2
| t
| t
o |
+
+ =
c

n f A
n f A n X
n
( )

) 2 sin(
0
| t + n f A
Example (Contd)
)] 2 4 sin(
2
) 2 sin( ] [ [
0 0
2
| t | t + + =

n f
A
n f n X
A
) 2 cos( ] [ [
ln
2
0
2
2
| t
o
+

=
c

n f n X
A P
n
)] 2 4 cos(
) 2 cos( ] [ [
0
0
2 2
| t
| t
o |
+
+
c

n f A
n f n X
n
( ) 2 ( cos
ln
E -
)] (
0
2
2 2
2
0
| t
o |
|
+ =
(

c
c

n f A
A P
f
) ) 2 4 cos(
0
| t
o |
+

c

n f A
n

Example (Contd)
) 2 4 cos( ) 2 4 cos(
2
1
2
1
0 0
2
2
| t | t
o
|
.
|
+ + +

\
|
=

f f
A
) 2 4 cos(
2 2
2 2
| t
o
+ =
. \

n f
A NA
n
1/2 f f 0 ) 2 4 (
1
B
) 2 4 cos(
2 2

1
0
2 2
| |
| t
o o
+ =

f
n f
N
n (See problem 3.7 in SK)
2
1/2 or near not f for 0 ) 2 4 ( cos
N
But
2
0 0
0
| | t ~ +

=
n f
n
Exercise : Compute CRLB for estimating
2
)

var(
2
2
2
o
| >
NA
Exercise : Compute CRLB for estimating
A, fo , or noise variance for this example
increases
2
A
SNR as decreases Bound
2
2
o
=
Fisher Information
increases
2
A
SNR as decreases CRLB
2
2
=
o
MVUE?
exist? estimator efficient Does
expected. as increases N and/or
No, cant factorize
Dont Know
| c
c P ln
define We
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MVUE?
When CRLB is attained, estimator variance is
Don t Know
n Informatio Fisher the as
) ln(
-E ) I(
define We
2
2
|
|
.
|

\
|
c
c
=
u
u
P
reciprocal of Fisher information
below) remark (see 0 ) 1)I(
: Note
>
. \
c
u
u
when Information increases
variance (uncertainty) decreases
ns observatio t independen for 2)Additive
) ( ) ) (
Fisher Information Matrix
) ] [ ( l ) ( l
1
u u X P X P
N
[

) ]; [ ( ln ) , ( ln
1
0
u u n X P X P
N
n
=
[

=
) ]; [ ( ln
0
u n X P
n
( | |
=

=
) ]; [ ( ln ln
) (
1
0
2
2
2
2
u
u
u
u
n X P
E
P
E I
N
n
(

c
c
=
|
|
.
|

\
|
c
c
=

=
) ( ) ( then X[n]) each for pdf
(same d distribute y identicall also are ns observatio If
u u Ni I =
. \
) ( ) ( then X[n]), each for pdf u u Ni I =
Fisher Information Matrix
) ]; [ ( ln
-E ) i(
2
2
(

c
c
=
u
u
u
n X P
l d i ll d l h h
0 CRLB case IID for , N As

cu
SK in 3.9 Problem See ? samples data
correlated - partially and perfectly for happens What
) ; ( ln
: Remark
2
(
| |
c u X P
( proof soon)
0
) ; ( ln
) ( >
(
(

|
.
|

\
|
c
c
=
u
u
u
X P
E I
squaring increases powers while
differentiation decreases powers
form original the use easier to Usually
General CRLB for Signals in WGN General CRLB for Signals in WGN
u + 1 N 0 1 2 ] [ ] [ ] [ W S X u
u
= + =
| |
1 - .N 0,1,2,.... n ] [ ] ; [ ] [
WGN
on
Depends
n W n S n X
( )
u
u
o

=


=
1
) ; X P(
] ; [ ] [
2
1
1
0
2
2
e
n S n X
N
N
n
( )
u
u
to
c c

] ; [
] [ ] [
1 ) ln(
) 2 (
) ; (
1
2
2
n S
S X
P
N
N
( )
u
u
o u c
=
c

=
] ; [
] ; [ ] [
) (
0
2
n S n X
n
The only dependence of X(n) on the unknown parameter is through
] ; [ u n S
General CRLB for Signals in WGN
] ; [ ] ; [
]) ; [ ] [ (
1 ) ln(
1
2
2
2
2 2
2

(
(

|
.
|

\
|
c
c

c
c
=
c
c
N
n S n S
n S n X
P
u
u
u
u
u
o u
] ; [ 1 ) ln(
2
1
2 2
2
0

=
|
|

|
c
=
|
|
|

|
c
(

. \
c c c
N
n
n S P
E
u
u u o u
)

Var(

2
0
2 2

=
>
|
.

\
c
|
|
.

\
c
n
E
o
u
u o u
] ; [
) Var(
1
0
2

=
|
.
|

\
|
c
c
>
N
n
n S
u
u
u
unknown for their estimators accurate more in result change
parameters unknown their as rapidly more change that Signals
paramters
u u A) (S[n] ] S[n; If : Example = =
o
o
) A

Var(
) ( [ ] ] [ ; p
2
2
2
= >

N
| u ) ( l h f h l T
(1)
) A Var(
N
2
>

N
| u
is here signal a of aspect important that the Note
) ( example phase for the also Try =
u with change of rate its
g p p
Transformation of Parameters
u
u o =
some is g(.) and by zed parameteri is pdf
where ) g( estimate to CRLB
( ) u
o
c c
>
g
)

Var(
general) in linear (non tion transforma
2
u
u
o
|
|
.
|

\
|
c
c

>
)) ; ( ln(
) Var(
2
2
X p
E
u o
u

'
=
c
)) (

)(
)) ; ( ln(
iff achieved is Equality
g ( I
X p
Proof soon
u
u o
u
|
|
|

|
c

=
c
)) ; ( ln(
)) ( )(
2
2
X p
E
g ( I
u
u
u
c c
|
|
.

\
c
=
'
g
) ( where
2
E
I
E l DC L l i WGN Example: DC Level in WGN
] [ ] [ n W A n X + =
4 ) 2 ( ) (

) ( A for CRLB
2 2 2 2
2 2
o
o
A A A g
A A g
c c
= =
depends on A !
4 ) 2 (
)) ; ( ln(
) (
) A

Var(
2
2
2
o
o
N
A
N
A
A
A X p
E
A g
= =
(

c
c

c c
>
) ( A

Is A. for efficient was

2 2
X X A
A
= =

c
slide) previous from
theorem using (prove No ? A for efficient
2
) N , N(A as d distribute is X
2
o
) ( ) ( ) ) ((
2 2
+ X Var X E X E
A
) ( ) ( ) ) ((
2 2 2
= + =
+ =
A N
X Var X E X E
o
linear non by destroyed efficiency general In
unbiased! even Not

formation. trans
linear non by destroyed efficiency general In
mations transfor (affine) linear over Maintained
Affine Transformations Affine Transformations
b + = + b a for efficient

a efficient

u o u u
b a b aE b a + + + )

( )

E(
since
o u u u
Var a b a Var
b a b aE b a
= +
= + = + = +
2
)

( )

(
) ( ) E(
u u
o u u u
( )
CRLB
p
=
|
|

|
c
c c
=
2
2
) ln(
g

u
p
E
|
|
.
|

\
|
c
c

2
) ln(
u
Now, we are ready to prove the CRLB Theorem
Proof of Scalar Cramer-Rao Lower
B d (CRLB) Th Bound (CRLB) Theorem
) 1 ( ) ( d ) (

) ( )

E( : Estimator Unbiased
) ; x p( given ) ( :
g
g Estimate Objetive
= =
=
}
u u
u o o
u u o
dx ) ; x p(
) ; x lnp( ) ; x lnp(
E
: Condition Regularity
) 1 ( ) ( dx ) ; x p(

g
c
(
(

c
=
}
}
u
u u
u u o

) ; x p(
) ; x p(
1 ) ; x lnp(
dx
) ; x p(

dx ) ; x p(
) p( ) p(
E
(

c
c
=
c
c
c
c
=
c
=
(

c
}
}
u
u
u u
u
u
u
u
u u

| | 1 dx ) ; x p( dx ) ; x p(
) ; x p(
step key
= =
c
c
=

c c c
} }
u | u
u
u u u u

n. integratio and ation differenti of order e interchang
can we as long as hold always will condition regularity , Hence
cu
Proof (cont.)
u , w.r.t. (1) of sides both ating differenti by Now
u
u
u
u
o
) g(
dx
) ; (

c
c
=
c
c
}
x p
u u
u
u
u
u
u
o
) g( ) ; ( ln
) g(
dx ) ; (
) ; ( ln

c c
c
c
=
c
c

}
}
x p
x p
x p
u
u
o
u
u
u
u
u
o o
dx ) ; (
) ; ( ln
) 2 (
) g(
dx ) ; (
) ; ( ln
)

(
c
c
c
=
c
c

}
}
x p
x p
Since
x p
x p
|
u
u
o
u
u
o

) ; ( ln

dx ) ; ( ,
=
(

c
c
=
c
}
x p
E
x p Since
u
Re
(

c
_
Condition gularity
Proof (cont.) ( )
: form) (function inequality Schwarz - Cauchy consider us let Now
| |
} }
s
}
(x)dx h w(x) . (x)dx g w(x) g(x)h(x)dx w(x)
) ( q y y
2 2 2
| |
|
=
>
c'. ' constant for h(x) c g(x) iff equality with
x for w(x)
| | s y x x.y : form or with vect Compare
2 2 2
o o
| u
=
> =

g(x)
) ; x p( w(x) , Let
u
u
c
c
=
) ; x lnp(
h(x)
Proof (cont.) ( )
(2), to inequality Schwarz Cauchy Apply
( ) dx ) ; x p(
) ; x lnp(
. dx ) ; x p(

) g(
2
) ( var
2
2
|
.
|

\
|
c
c
s
(

c
c
} }
u
u
u
u o o
u
u
o
_
( )
( ) | |
(3)
) ; x lnp(
) g(
)

var(
2
2
(
c c
c c
>
u u
u u
o
E
) ; x lnp(
condition) y (regularit 0
) ; x lnp(
E Now,
c
=
(

c
c
}
u
u
u
w.r.t. ating differenti By
0 dx ) ; x p(
) ; x lnp(
=
c
c

}
u
u
u
u
0 dx ) ; x p(
) ; x lnp(
=
|
.
|

\
|
c
c
c
c
}
u
u
u
u
Proof (cont.)
=
(
(

c c
+
c

}
2
0 dx
) ; x ( p ) ; x ( p ln
) ; x p(
) ; x ( p ln u u
u
u
c
c
c
c
=
(

c
c

=
(

c c
+
c

}
}
2
2
2
dx ) ; x ( p
) ; x ( p ln ) ; x ( p ln ) ; x ( p ln
E -
0 dx ) ; x p(
u
u
u
u
u
u
u
u u
u
u
(
(

|
.
|

\
|
c
c
=
c c
(

c
}
2
2
) 4 (
) ; x ( p ln
E
u
u
u u u

(

. \
c
(4) and ) 3 ( from
u
Alternative form of
Fisher Information
(
|
.
|

\
|
c
c
>
2
2
) g(
) var(
( ) ) (
u
u
o
(

c
c
2
2
) ; x ( p ln
E -
) (
u
u
Proof (cont.)
) (
1 ) ; x lnp(
g(x)
1
h(x)
iff occurs Equality
o o
u
=
c
=
) 5 ( )

(
1 ) ; x lnp(

) g( if , particular In
) (
) c(
g( )
) c(
( )
u u
u
u u o
o o
u u u
c
= =
c

. w.r.t. (5) ate differenti ) c( determine To

(
) c(
1
) c(
1 ) ; x lnp(
2
2
2
u
u u
u u u u
u
(
c

|
|
.
|

\
|
c
c
+ =
c
c
1 1
) c(
) c(
1 ) ; x lnp(
-E
2
u
u u
u
= =
=
(

c
c

) ( I
) ; x lnp(
E
) c(
2
2
u
u
u
u =
(

c
c
=
Proof for General ) (u g
)) ( l (
iff Schwartz - Cauchy in Equality
u X c
(6) - - - )) (

)(
)) ; ( ln(
u o
u
u
g ( I
X p

'
=
c
c

)) ; ( ln(

ns expectatio take & )) (

( by (6) Multiply
2
u
u o
X p
g
'
c

(2) Using
] )) (

[( ) ]
)) ; ( ln(
)) (

[(
2
u o
u
u
u o g E ( I
X p
g E
'
=
c
c

(7) - - - - - - )

var( ) ) g (
(2) Using
o u ( I
'
= c c
Proof for General
) (u g
'
c c
2
) )) ; ( ln(
(6) From
u ( I X

c
'
c
+
'
c c =
c
c
2
2
)) (

(
)
) ) g (
)) ; ( ln(
u o
u
u
u
u
g
( I
( I
X p
'
c c =
c
c

2
2
(8) - - - ) ) g ( ]
)) ; ( ln(
[ u
u
u
( I
X p
E
c c
c
2
) (
(8) and (7) From
u
u
|
|
|

|
c
c c
=
2
2
)) ; ( ln(
) g (
)

var(
u
u
o
X p
E
|
|
.

\
c
2
)) ; ( (
u
p
E
of estimator MVU Find
1 - N , 0,1, n ) G(0, ~ w(n) x(n)
: ) with varies variance estimator MVU (where
2
2
o
o
u Example
= =
( )
] [
2
1
exp
2
1
) ; (
. of estimator MVU Find
1
0
2
2 2 / 2
2
o to
o
o
n x x p
N
n
N
|
|
.
|

\
|
|
.
|

\
|
=

=
What if we are trying to
estimate the standard
deviation of the noise ? o
1 ) ; ( ln
] [
2
1
ln
2
) 2 ln(
2
) ; ( ln
1
2
2
1
0
2
2
2 2
o
o
o t o
N x p
n x
N N
x p
N
N
n
c
=

=
deviation of the noise
Is there an efficient
estimator in this case ?
? o
1
] [
2
1
2
) ; ( ln
1
0
2
4 2 2
o o o
o
N
n x
N x p
N
n
(
(

+ =
c
c

2 1
Efficient & MVU ] [
N
1
2

4 ^ 1 ^
2
) (
0
2
) (
4
o
o
u
u
n x
N
N
x g
n
I
(
(
(
(

=
_
on depends which
2
) var( )

var( and ] [
N
1

2
4
2
1
0
2 2
o
o
o u o u
N
n x
N
n
= = = =

=
CRLB for Vector Parameter CRLB for Vector Parameter
| |
2 1
= u u u u
T
p

proof in SK appendix 3B
( ) ( )

BIASED UN )

E( Assume
1 1
=u u
proof in SK appendix 3B
( ) ( )
)) ( l (
general) (in ) ( ) ( )

Var(
2
1 1
| |
c
= >

u
u u u
X
I I
ii
ii i
)) ; ( ln(
)] ( I [ where
2
|
|
.
|

\
|
c c
c
=
u u
u
u
X p
E
j i
ij
definite - positive and symmetric p p is ) (
. \
u I
j i
) invertible (hence Matrix n Informatio Fisher
Fisher Information Matrix for Signal in WGN g
n W n S n X u = + =
| |
1 - .N 0,1,2,.... n ] [ ] ; [ ] [
WGN Depends
( ) n S n X
e
N
n
u
u
o
u

=
1
) ; P(X
] ; [ ] [
2
1
WGN
on
Depends
1
0
2
2
( )
N
N
n S P
e
n
u
to
u
c c
=

=
] ; [ 1 ) ln(
) 2 (
) ; P(X
1
2
2
0
( )
i
n
i
n S
n S n X
P
u
u
u
o u c
c
=
c
c

=
] ; [
] ; [ ] [
1 ) ln(
0
2
(
] ; [ ] ; [ 1 ) ln(
] ; [ ] ; [ ] ; [
]) ; [ ] [ (
1 ) ln(
1
2
1
0
2
2
2

=
c c
|
|

|
c
(
(

c
c
c
c

c c
c
=
c c
c
N
N
n
j i j i j i
n S n S P
E
n S n S n S
n S n X
P
u u
u
u
u
u
u u
u
u
o u u

] ; [ ] ; [ ) (

0
2

=
c c
=
|
|
.

\
c c
n
j i j i
E
u u o u u
Example
+ =
2
) , 0 ( ~ ] [ where ] [ ] [
: ) pg.40 3.6 (SK
G n w n w A n x
Example
o
( )
( )

|
.
|

\
|
= =
=
1
0
2
2 2 / 2
T 2
] [
2
1
exp
2
1
) ; x p( ; ] [A
1 - N , 0,1, n
N
N
A n x
o to
u o u

( )
( )

=
=
=
. \
2
1
1
0
2
2
2
0
] [
2
1
ln
2
) 2 ln(
2
) ; ( ln
2 2
N
N
n
n
A n x
N N
x p
o
o t u
o to
( )
( )

=
+ =
c
=
c
c
=
c
c
1
2
2 2
2
1
0
2
] [
1 ln

ln
; ] [
1 ln
N
N
n
A n x
N p
N
A
p
A n x
A
p
o o
( )
( )
( )

=
=
c
c
+ =
c
1
0
2
6 4 2 2
2
0
4 2 2
] [
1
2
ln
] [
2 2
N
n
A n x
N p
A n x
o o o
o o o
random !
( )
( )

=
=
=
c c
c
c
1
0
4 2
2
0
] [
1 ln
&
2
N
n
n
A n x
A
p
o o
o o o
Example (cont.)
N
N
N p
E
4
2
6 4 2 2
2
2
.
1
2 ) (
ln

o
o
o o o
= =
(

c
c
N
0
) (

(

N
N
I
4
2
general in not but
case in this diagonal

0
0
) (
o
u
|
|
.
|

\
|

(
(
(

=
2
4
2o

)

N
A
4 ^
2
2
)

var(
o
>
CRLB for joint estimation of the 2 parameters
is same as CRLB for separate estimation
assuming the other parameter is known
N
4
2
2
) var(
o
o >
Fitting Line : Example
=
(

= + + =
|
0) B had Ex. Level (DC ; W[n] Bn A X[n]
g p
B
A
u

| WGN
first ) ( I compute CRLB Find To
B
u
(
(
(

c c
c

c
c

2
2
2
) ln( ) ln(
) ( I
B A
p
E
A
p
E
u
(
(
(

c
c

c c
c

=
2
2 2
) ln( ) ln(
) ( I
B
p
E
A B
p
E
u


Example (Contd) Example (Cont d)
( )
1
] [
2
1
1
2
2
Bn A n X
N

( )
) 2 (
1
) ; (
2
2 2
0
2
to
u
o
e X p
N
n

=
=
( ) ] [
1 ) ln(
2
o
Bn A n X
A
p
n
=
c
c

( ) ] [
1 ) ln(
2
o
n Bn A n X
B
p
=
c
c

) ln(
2
2
o
o
N
p
B
n
=
c
c

2
o N
A c
Example (Contd)
=
c

2
1 ) ln(
n
p
Example (Cont d)
c c
=
c c

2 2
2
) ln( 1 ) ln(

p p
n
B A
n
o
c
c c
= =
c c

2
2
1 ) ln(
) ( ) (

p
B A
p
n
A B
p
n
o
always, why ?
(
=
c
c

2
2 2
1 ) ln(
n
B
p
n
o
(
(

=


2
2
1
) (
n n
n N
I
o
u
(

Example (Contd)

=

=

6
) 1 2 )( 1 (
n ;
2
) 1 (
n But
1 - N
2
1 - N
N N N N N
(
(

= =
2
) 1 (
1
6 2
0 n 0 n
N N
N
(
(
(

=
6
) 1 2 )( 1 (
2
) 1 (
2
1
) (
2
N N N N N
I
o
u
(

6 ) 1 2 ( 2
why?) - symmetric Always (
6 2
N
(
(
(
(

=
12 6
) 1 (
6
) 1 (
) 1 2 ( 2
) ( I
2 1 -
N N N N
N
o u
(
(

+ ) 1 (
12
) 1 (
6
2
N N N N
Example (Contd)
2
11
1
) 1 (
) 1 2 ( 2
)] ( [ )

= >
N N
N
I A Var
o
u
2
2
22
1
12
)] ( [ )

(
) 1 (

= >
+
I B Var
N N
o
u
Question : Compute
CRLB for data
samples x(k),
2
22
: ns Observatio
) 1 (
)] ( [ ) (

>
N N
I B Var u
x(k+1),.....
x(k+N) where
k is an integer
2
N ) A

Var(
W[n] A X[n] for Recall 1)
>
+ =
o
2
1
) 1 2 ( 2
N
) A

Var( Now,
N ) A Var(
|
.
|

\
|
+

>
>
N
N o
o
See Prob. 3.12
for general proof
_
2 N for 1
1 N
> >
. \
+ N
more estimate we as decrease not can CRLB

task) difficult (more parameters

CRLB(
) A

CRLB(
Also,
or S[n] in change large causes B in
change slight is Reason estimate. easier to is B
A A
A
n S
A A = A
c
c
= A
c
c
~ A
] [
A
X[n]
X[n]
or S[n] in change large causes B in
B n B
n S
B
A
A = A
c
= A
c
~ A
c c
] [ X[n]
X[n]
A
B c cB
[ ]
x(n) more sensitive to changes in B than to A
No Noise
X[n]
1
A=1,B=0
A=B=0
1 2 3 4
n
X[n]
0
A 0 B 1
4
3
X[n]
A=0,B=1
2
1
A=B=0
1 2 3 4
n
Example
57) p. 3.14 (SK : Example
2
1
0 & 0 A where,
1 - N , 0,1, n ) ( ) 2 cos( ) (
< < >
= + + =
o
o
f
n w n f A n x | t
| |
shown that be can It
2 ;
2
+ = =
o
T
o
o
n f f A
f
| t o | u
| |
2 2
2 cos 1 1
cos
1
) (
shown that be can It
2
1
2
1
2
2
1 , 1
~
+
= =


=

=
N
o n
N
o n
N
I
o
o
o
o
o
u
(do calculation and submit next class)
| | sin cos 2
1
) (
2 2
2
2 , 1
=

= = o n o n
n A I o o t
o
u
o o o
0 2 sin
2
~ =

n
n
A
o
o
t
Example (cont.)
f
N
i

1
0 ) 2 4 i (
1
) them! (show identities following the using By
|
f for
n f n
n f n
N
o
i
n
o
i
=

~ +
~ +

+
=
+
0
1
1 i
0
1 i
1/2 or 0 near not and 0,1,2 i
0 ) 2 4 cos(
N
1
0 ) 2 4 sin(
N
1
| t
| t
N
be can It
f
n
o
(

=
+
0
1 i
2
shown that
) (
N
| |
|
NA n A
n A n A
N
(
(
(

2 2
2 2 2 2
2
get we inversion, After .
2
2
2
1
) I(
t |
t t |
| |
o
u
( )
A
N N
N
A
+
>

> >
2
2
2
o
2
1) N(N
1) - 2(2N
)

var( ;
) 1 ( 2
12
) f

var( ;
2
)

var(
q
|
q t
o
Blows up for N=1; why ?
SNR
A
= =
2
2
2
where,
o
q
Question : how do these bounds compare to those
Of single-parameter and two parameter estimation ?
Blows up for N 1; why ?
CRLB Theorem- Vector Parameter
(
X

C as

of matrix covariance Denote u

=
Z
Y
X
3) (p If u
=
(

T
E E E
Z

] ))

))(

[( C u u u u
u
|
|
|

|
(
(
(

(
(

(
(

T
X E X )

|
|
|
|

(
(
(

(
(
(

(
(
(

=
Z E
Y E
Z
Y
_
)

(
)

(
) (

E
|
|
.

\


_
_
Vector Row
Vector Column
) (
) Z

Cov( ) Y

Cov( ) X

Var(
(
)

( )

C ( )

C (
) Z

, Y

Cov( ) Y

Var( ) Y

, X

Cov(
) Z , X Cov( ) Y , X Cov( ) X Var(

(
(
(
(

=
then , conditions regularity satisfies PDF If
) Z Var( ) Z , Y Cov( ) Z , X Cov(
(

TE. SEMIDEFINI POSITIVE is 0 ) ( C

u
u
>

I
iff CRLB attains estimator unbiased An
). of value at true ) ( I (Evaluate u u
) ) ( )( ( I
) ; ( ln
u u
u
u
=
c
c
X g
X p
(*)
Note : regularity condition implies , from Equation (*), that efficient estimator is unbiased
i i MVU) h ( d ffi i Th
. I and g some for
X g = ) (

is estimator MVU) hence (and efficient That

u
u
symmetric is it if matrix te semidefini positive a is A
N EXPLANATIO
( )
T
X A > X all for 0 X and
y p
1
T
( )
i
T
T
e X
X I X
= =
>

] 0 ... 010 ... 00 [ let , particular In
0 ) ( C
1

u
u
i
|
] [ , p
PLACE
th
i
] A [ e A e But
i i
=
ii
T
( ) 0 e ) ( I - C e
i
1

i
>
T
( )
( ) ( )
( )
) C (
1
ii

>

ii
I
( ) before as ) ( )

Var(
1
>

ii i
I u
Now, consider
GRADIENT
ln(p)


=
c
c

Now, consider
c
ln(p)
(
(

c
c
u
ln(p)

1
(
(
(
(

c
c
=
u
.
require we hold to CRLB in equality For
p
(
(

cu
require we hold, to CRLB in equality For
) ) ( )( (
ln(p)

g I

=
c
c
|
X u
efficient is

Remarks Remarks
CRLB theorem can be applied whenever
) ; ( u X p
CRLB theorem can be applied whenever
is available. In particular, it is directly
applicable to all additive-noise models where it
) ; ( u X p
applicable to all additive noise models where it
is easy to write
) ; ( u X p
Noise need not be i.i.d. or zero-mean Gaussian
Key step is to write down the expression for y p p
general N-dimensional noise pdf
Revisiting EXAMPLE on Vector CRLB

(
(
c
) ] [ (
) ln(
Fitting Line
B A X
p

(
(
(

=
(
(
(
(

c
c
c
=
c
c
2
) ] [ (
) ] [ (
1
) ln(
) ln(
ln(p)

n
n Bn A n X
Bn A n X
p
A
p
o u

(
(

(
(

c
2
2

2 ) 1 (
n
A A
N N
N
B
o
(

(
(

=
-

) ( I
2 2
2

6 ) 1 2 )( 1 ( 2 ) 1 (
) (

B B
N N N N N
_
_
o o
o


=

=
+

=
1
0
1
0
) ( I
] [
) 1 (
6
] [
) 1 (
) 1 2 ( 2

N
n
N
n
n nX
N N
n X
N N
N
A
Verify it is
bi d
= = 0 0
) ( ) (
n n
N N N N
unbiased
EXAMPLE (Contd)
1 1 N N
] [
) 1 (
12
] [
) 1 (
6

1
0
1
0
2


=

=

+
+

=
N
n
N
n
n nX
N N
n X
N N
B
Verify it is
unbiased
) obvious! Not (
Model Linear of case special a is example This
Chapter 4 Chapter 4
Announcements Announcements
HW2 : Chapter 3 : (from SK) HW 2 : Chapter 3 : (from SK)
3.2, 3.5,3.7,3.11,3.12
Review problems : 3.4, 3.9, 3.13
Sections not covered in Chapter 3 : Sections not covered in Chapter 3 :
3.8,3.9,3.10,3.11 (except Example 3.14)