Classical Detection and Estimation Theory

Reference: "Detection, Estimation and Modulation Theory" by H.L. Van Trees

Probabilistic Observation
Source
transition mechanism Space

Decision
Source o/p is one of two choices rule
Hypothesis H0, H1
1 H1 Decision
0 H0

* Observation Space is finite dimensional, i.e., observations consist of a

set of N numbers and can be represented as point in an N-dimensional
space.
n
H1
+1 r
Source
-1
H0
Transition Observation
Mechanism Space

pn(N)

1/2

1/4 1/4

-1 0 +1 N

H1 : r = 1+n
H0 : r = -1+n

p r|H1 (R|H1), p r|H 0 (R|H0) ??

DPAWCS 1 Jan12
DCE
Simple Binary Hypothesis Tests
r1
r * two known conditional
2 probability densities
r .
pr|H1 (R|H1), pr|H0 (R|H0) ??
.
rN

Use this info. to develop a suitable decision rule.

Decision Criteria : Either H0 or H1 is true.

One of the following can happen;
1. H0 true : Choose H0 correct
2. H0 true : Choose H1
3. H1 true : Choose H1 correct
4. H1 true : Choose H0

Bayes Criterion
Based on
1) The source o/p's are with P0, P1 a priori probabilities.
2) A cost is assigned to each possible course of action.C00, C10 , C11, C01
where Ci, j the jth hypothesis true and ith is hypothesis chosen.

Each time a certain cost is incurred.

Design s.t. "on the average" the cost will be as small as possible
decision rule

Thus, the expected value of the cost Risk

R = C00P0Pr(say H0|H0 true)

+ C10P0Pr(say H1|H0 true)
+ C11P1Pr(say H1|H1 true)
+ C01P1Pr(say H0|H1 true)

Because the decision rule must say either H1 or H0 rule for dividing
the total observation space Z into two parts Z0 and Z1.

p r|H1 (R|H1) Say H0

Z1 Z0

R
Source Z0
R
Z Observation
p r|H 0 (R|H0) Space

Say H1
DPAWCS 2 Jan12
DCE
The Risk in terms of transition probabilities and decision region.
R = C00P0 pr|H0 (R|H0) dR
Z0

+ C10P0 pr|H0 (R|H0) dR

Z1

+ C11P1 pr|H1 (R|H1)dR

Z1

+ C01P1 pr|H1 (R|H1)dR

Z0

N dimensional space Integrals are N fold.

Assumption : Cost of a wrong decision is higher than the cost of a correct
decision.
C10 > C00
C01 > C11

We must choose regions Z0 and Z1 s.t. risk minimum

Z = Z0 + Z1
Thus
R = P0C00 pr|H0 (R|H0) dR + P0C10 pr|H0 (R|H0) dR
Z0 Z Z0

+ P1C01 pr|H1 (R|H1)dR + P1C11 pr|H1 (R|H1)dR

Z0 Z Z0

pr|H0 (R|H0) dR = pr|H1 (R|H1)dR = 1

Z Z

R = P0C10 + P1C11 + [ {P1(C01-C11) pr|H1 (R|H1)}

Z0
- {P0(C10-C00) pr|H0 (R|H0)} ] dR
First two terms fixed cost.
C01-C11 > 0 ; C10-C00 > 0 (assumption)

Thus : all values of R where the second term is larger than the first should
be included in Z0 because they contribute a negative amount to the
integral.

DPAWCS 3 Jan12
DCE
Similarly, all values of R where the first term is larger than the second
should be excluded in Z0 (assigned to Z1) because they contribute a
positive amount to the integral.

Thus if:
P1(C01-C11) pr|H1 (R|H1) P0(C10-C00) pr|H0 (R|H0)

assign R to Z1 H1 true ; otherwise assign R to Z0 H0 true.

H1
pr|H1 (R|H1) P0(C10-C00)
>
< P (C -C )
pr|H0 (R|H0) 1 01 11
H0

"Likelihood ratio (R)"

pr|H1 (R|H1)
(R) p (R|H )
r|H 0 0

Because it is the ratio of two functions of a random variable, it is a

random variable. We see that regardless of the dimensionality of R, (R)
is a one-dimensional variable.
P0(C10-C00)
Threshold
P1(C01-C11)
Thus Bayes Criterion leads us to a "likelihood ratio test".

* All the data processing is involved in computing (R) and is not

affected by a priori probabilities or cost assignments.

Because the natural logarithm is a monotonic function, and equivalent

test is
H1
ln (R) >
< ln
H0

DPAWCS 4 Jan12
DCE
 Threshold
R Data (R) Device Decision
Processor (R) >
<
(a)

Threshold Decision
R Data ln (R) Device
Processor ln (R) >
< ln
(b)

Likelihood ratio processors

Example 1
N-observations
Under H0 : R1 = -1 + n1 Under H1 : R1 = 1 + n1
R2 = -1 + n2 R2 = 1 + n2
. .
. .
RN =-1 + nN RN = 1 + nN

ni - zero mean, statistically independent Gaussian noise samples of

variance 2.

p r|H1 (R | H 1 ) p r |H (R i | H1 )
i 1
i 1
(R ) N
p r |H 0 ( R | H 0 )
p r |H
i 0
(R i | H 0 )
i 1
N (R i 1) 2

1
exp H1
2 2 2

<
i 1 >

N (R i 1) 2

1
exp H0
2 2 2
i 1

DPAWCS 5 Jan12
DCE
 N H1
Ri
1 2
>
< 2N
ln
N i 1
H0
l(R) or l function of the received data

* Sufficient statistics
When making a decision, knowing the value of the sufficient statistic is
just as good as knowing R.

* Several kinds of Bayes cost

C00 = C11 = 0 ; C01 = C10 = 1

R = P0 pr|H0 (R|H0) dR + P1 pr|H1 (R|H1)dR

Z1 Z0
This is the total probability of making an error.

Thus, for this cost assignment Bayes cost is minimizing the total
probability of error. The test is then

H1
P1
(R) minimum error probability
< P0
H0
If in addition P0=P1 then
H1
pr|H1 (R | H1 )
(R ) 1 maximum likelihood test.
pr|H0 (R | H 0 ) <
H0

We denote the integrals in the Bayes test in the following manner.

PF = pr|H0 (R|H0) dR
Z1
-false alarm : (We say the target is present when it is not.)
From now onward, we'll use the notation p(X) instead of px(X).
, .

PD = p(R|H1) dR
Z1
-Detection : (We say the target is present when it is.)

DPAWCS 6 Jan12
DCE
PM = pr|H1 (R|H1)dR = 1 - PD
Z0
-Miss : (We say the target is absent when it is present.)

Consider the earlier example

N
H1
Ri
1 2
test
N < 2 N ln
i 1
H0
l
- Sum of N statistically independent Gaussian random variables.
l is Gaussian.

2
E[l|H0] = -1; E[l|H1] = 1, var[l|H0]= var[l|H1]= =12
N
1 (l 1) 2
p(l|H1)= exp
2 1 2
2 1
1 (l 1) 2
p(l|H0)= exp
2 1 2
2 1
PF =Pr[H1 chosen | H0 true] = Pr[l> | H0 true]

1 (l 1) 2
=
2 1
exp

2 1
2
dl

1
= Q

1

1 x2
[ because Q(x) =
2 exp dx
2
x
( y )2
exp dy
1
2 2 2


1 z2 y
=
2 y
exp dz (Substituting
2
= z)

= Q ]

DPAWCS 7 Jan12
DCE
 PM = Pr[H0 chosen | H1 true] = Pr[l< | H1 true]

1
= pl | H1 dl

2 1

1 (l 1) 2
=1-
2 1
exp
2 1
2
dl

1
= 1- Q

1

Example 2
H0 : Ri N(0, 02)
H1 : Ri N(0, 12), 1 > 0 , i=1, 2, ..., N

log LRT H1
N
1

N
R i 1
2
i T
<
H0
* For performance need to find statistics of the sufficient
statistic R 2i (Chi-squared random variable.)
i

Minimax Criterion - Useful when a priori probabilities are unknown.

- Use PF, PM and PD.
Bayes Risk

R = P0C10 + P1C11 + P1(C01-C11) PM- P0(C10-C00) (1-PF)

But P0 + P1 = 1 P0 = 1- P1

R = C00(1-PF)+ C10PF+ P1[(C11-C00) + (C01-C11) PM- (C10-C00) PF]

= RB(P1) (because PF and PM are functions of P1)

Now, if all costs and a priori probabilities are known, we can find a Bayes
test.

DPAWCS 8 Jan12
DCE
 R
* Observe that as P1 changes
the decision regions for Bayes
test change and therefore PF and
C00 C11 PM change.

0 P1 = 1 P1

PM=PF=0
If we assume a P1, say P1 = P1* and design Bayes test then,
(1-P1*)(C10-C00)
Threshold = is fixed.
P1* (C01-C11)
and PF = Pr[(R)> | H0]
PM = Pr[(R)< | H1] are also fixed since is fixed.

If now P1 is allowed to vary in the earlier equation RF(P1) is a straight line

which touch RB(P1) at P1 = P1*.

RF(P1)
RB(P1)

0 1 P1
P1=P1* Pmin-max

Thus RF(P1) RB(P1) because the Bayes test minimizes the Risk.
A Bayes test designed to minimize the maximum possible risk is called a
"minimax test". Hence we choose RF to be the horizontal minimax
equation.
(C11-C00) + (C01-C11)PM - (C10-C00)PF = 0

Use C00 = C11 =0 (This guarantees maximum is interior.)

Let C01 = CM
C10 = CF the risk is,
RB = CFPF + P1(CMPM-CFPF)
= P0CFPF + P1CMPM

DPAWCS 9 Jan12
DCE
The minimax equation is

CMPM = CFPF

Neyman - Pearson Criterion

(tries to get around cost-specifications)

Try to work with conditional probabilities PD, PF i.e. make PF as small

as possible, PD as large as possible.

Constrain one, and maximize (minimize) the other.

Thus, constrain PF = '

Design a test to maximize PD (or minimize PM) under this
constraint.
Solution to this by "Lagrange mutipliers".
Construct function F;
F = PM + ( PF - ')
or = p(R|H1)dR + [ p(R|H0) dR - ']
Z0 Z1

F = ( 1 - ') + [ p(R|H1) - p(R|H0) ] dR

Z0
F is minimized by assigning R Z0 whenever p(R|H1) - p(R|H0) < 0.
H1
If >0, then (R )
p ( R | H1 )
p(R | H 0 )
<
H0
To satisfy the constraint so that PF = '.
If the density of p(|H0) then under H0

PF = p(|H0) d = ' (**)

Solving (**) for gives the threshold. The value of given by (**) will
be non-negative because p(|H0) is zero for negative values of .

decreasing is equivalent to increasing Z1 H1.

DPAWCS 10 Jan12
DCE
Therefore we decrease until we obtain the largest possible '.

In most cases PF is a continuous function of and we have PF = .

Under this assumption the Neyman-Pearson criterion leads to a likelihood

ratio test (LRT).

Sufficient Statistic

l - Sufficient statistic
R Processor
(Observables) t - Indifferent statistic

t is indifferent if p(t|H1, l) = p(t|H0, l)

pl , t | H1 pl | H1 pt | H1 , l
Since (l, t) =
pl , t | H1 pl | H 0 pt | H 0 , l
pl | H1
=
pl | H1

knowledge of only l allows one to distinguish between hypothesis.

e.g
i) H1 : R1 = m +n1 H0 : R1 = n1
R2 = n2 R2 = n2
n1 , n2 are statistically independent. R2 irrelevant statistics

Note i) If l is passed through an invertible operation results in a

sufficient statistic.
ii) (l, t) together are sufficient statistics through not minimal
sufficient statistics.

Performance : Receiver Operating Characteristic

Evaluating the perfromance of the LRT.

Neyman Pearson test PF , PD specify the performance.
Bayes Risk RB follows from PF, PD.
Concentrate on calculating PF and PD.

DPAWCS 11 Jan12
DCE
e.g
H1 : ri = m + ni , i = 1, 2, , N
H0 : ri = ni , i = 1, 2, , N
H1
N

R i ><
1 Nm
l= ln
N i 1 H0 Nm 2

Nm
d

p(l|H0) p(l|H1)

PF

Nm ln d
Threshold : ln
Nm 2 d 2

PD

l ~ N(0, 1) under H0.

Nm
~ N( , 1) under H1.

Nm
d is the distance between the means of two densities.

Thus,

DPAWCS 12 Jan12
DCE

1 x2
PF = exp dx
2 2
ln d

d 2
ln d
= Q
d 2

(x - d) 2
1
PD = exp dx
2 2
ln d

d 2
y2
1
= exp dy
2
ln d 2

d 2
ln d
= Q
d 2

Plot PD versus PF for various values of d with as the varying parameter.

=0 ; ln = - PF = PD = 1 (H1)
; PF = PD = 0 (H0)

d=2.0

d=1.0 Receiver
Operating
d=0.5 Characteristic
(ROC)
PD
Increasing

PF

DPAWCS 13 Jan12
DCE
Performance increases monotically with d. (As we would expect)

Special Case : Minimizing the total probability of error.

Pr() P0 PF + P1 PM
P0 = P1 = ( = 1)
Pr() = (PF + PD)

1 x2
= exp dx

d 2 2
2
d
= Q
2

Bounds for Q(x) : For x > 0

1 1 x 2 1 x2
1 exp Q(x) exp
2 x
x 2
2 2 x
2
1 x2
Q(x) < exp ; x>0
2 2

Example 2
Unequal variances
H1
N
2 02 12
l (R) = R i2 >
< ln - N ln 0
12 02 1
i 1 H0

= (1>0)

Consider N=2

PF = Pr (l |H0) r1, r2 statistically independent var. 02

= Pr (r12 + r22 |H0) r1 = z cos and r2 = z sin
z= r12 r22 and = tan-1 (r2/r1)

DPAWCS 14 Jan12
DCE
 2 z2
1
Pr(z |H0) = d z
2
exp dz
2
2 0 2 0
2
0

1 z2
= 2 .z.exp dz
2
0 2 0

1 l d l =exp
= 2 2 2
exp
2 2
2 0 0 0

Similarly, PD = exp
2 2
1

PD = (PF ) 0 / 1
2 2
(ROC)
02
ln PD = 2 ln PF
1

PF < 1 PD 12/02

e.g
12/02 = 2 PF = 0.2 PD = (0.2)1/2 = 0.447
12/02 = 4 PD = (0.2)1/4 = 0.6687

Example 3
Poisson distribution of events : Our observation is just this number which
obeys a Poisson distribution on both hypothesis, i.e.,

Pr( n events ) =
m i n e m i
, n = 0, 1, 2,
n!
i = 0, 1
E[n] = mi

m1
n H1
(n) = exp m1 m 0 >
<
0
m
H0
H1
n ln 1 m1 m 0
m
m0 < ln
H0

DPAWCS 15 Jan12
DCE
 H1
n [ ln m1 ln m0]
ln + ( m1 m0)
<
H0

If m1 > m0, H1
n ln m1 m 0
< ln m1 ln m 0
H0

If m1 < m0, H0
n ln m1 m 0
< ln m1 ln m 0
H1

Now consider the case m1 > m0.

H1
n ln m1 m 0 = 1 consider the integer values
< ln m1 ln m 0
H0

integer only
H1
n 1 , 1 = 0, 1, 2, ...
<
H0
Now,
PD = Pr[n 1|H1] = 1- Pr[n < 1|H1]
1 1
m1 n e m1
= 1- , 1 = 0, 1, 2, ...
n 0 n!

Similarly,
PF = Pr[n 1|H0] = 1- Pr[n < 1|H0]
1 1
m 0 n e m0
= 1-
n 0 n!

DPAWCS 16 Jan12
DCE
 1

ROC
* consists of a
series of points
PD
1 0 - 1
PF 0 1- e m0

m0 = 2, m1 = 4
m0 = 4, m1 = 10

PF

LRT 1 PF PD
0 0 1 1

1 1 1- e m0 1- e m1

1 m0
If PF to have an intermediate value between 1 and 1- e m0 , say 1- e .
2
PF PF
LRT0 LRT1
1 .P
F + 1 PF
2 2
LRT0 LRT1
1 1
= .1 + .(1- e m0 )
2 2
1
= 1- . e m0
2
Therefore the test is
1
If n = 0, say H1 with probability .
2
1
say H0 with probability .
2

DPAWCS 17 Jan12
DCE
 n 1, say H1

Here we mix two LRTs in same probabilistic manner

randomized decision rule

1 1
PD = . PD + .PD
2 2
LRT0 LRT1
= 0.5 . 1 + 0.5 (1- e m1 )
1
= 1 . e m1
2

Reason : Observed r.vs are discrete (R) is a discrete r.v.

Only certain values of PF possible.

DPAWCS 18 Jan12
DCE