CSC446: Pattern Recognition (LN6)

ASU-CSC446 : Pattern Recognition. Prof. Dr. Mostafa Gadal-Haqq slide - 1
CSC446 : Pattern Recognition
Prof. Dr. Mostafa G. M. Mostafa
Faculty of Computer & Information Sciences
Computer Science Department
AIN SHAMS UNIVERSITY
Parameter Estimation :
Ch3 : Maximum-Likelihood
& Problem of Dimensionality
(Study DHS-Chapter 3 – Sec. 3.1, 3.2, 3.3, 3.7, 3.8)

• Introduction
• Maximum-Likelihood Estimation
–The General Principle
– The Gaussian Case: unknown  and 
– Bias
•Problem of Dimensionality
•Component Analysis and Discrimination
ML & Bayesian Estimation

• Introduction
– Data availability in a Bayesian framework
• We could design an optimal classifier if we knew:
– P(i) (priors)
– p(x | i) (class-conditional densities)
Unfortunately, we rarely have this complete
information!
– Design a classifier from a training sample
• No problem with prior estimation
• Samples are often too small for class-conditional
estimation (large dimension of feature space!)

– A priori information about the problem
– Normality of P(x | i)
• Characterized by 2 parameters
P(x | i) ~ N( i, i)
• Estimation techniques
– Maximum-Likelihood (ML) and the Bayesian
estimations
– Results are nearly identical, but the approaches
are different

• Maximum-Likelihood vs. Bayesian
– Parameters in ML estimation are fixed but
unknown!
– Best parameters are obtained by maximizing the
probability of obtaining the samples observed
– Bayesian methods view the parameters as
random variables having some known
distribution
– In either approach, we use P(i | x) for our
classification rule!

• Advantages:
– Has good convergence properties as the sample
size increases
– Simpler than alternative methods.
• General principle
– Assume we have c classes and
where: ),( jjj  
Maximum-Likelihood Estimation
P(x | j)  P (x | j ) ~ N( j, j)

• Our goal is to use the information provided by the training
samples D to estimate  = (1, 2, …, c), where i (i = 1, 2,
…, c) defines the parameters associated with each category
• Suppose that D contains n samples, x1, x2,…, xn that are iid
• ML estimate of  is, by definition the value that maximizes
P(D | )
“It is the value of  that best agrees with the actually
observed training sample”
)|()|(
1
 



nk
k
kxPDP
Maximum-Likelihood Estimation

• Optimal Estimation
– Let  = (1, 2, …, p)t and let  be the gradient
operator
– We define l() as the log-likelihood function
l() = ln P(D | )
– New problem statement:
“determine  that maximizes the log-likelihood”
t
p 















 ,...,,
21
)(argˆ
max 

l
The MLE Method

Set of necessary conditions for an optimum is:
or
0)|(ln)(
1






 


nk
k
kxPl  
The MLE Method
0))ˆ|((ln
1



nk
k
kxP 

• ML: Univariate Gaussian Case, unknown  and  :
– That is, p(xi | ) ~ N(,  2), where (1, 2) =(, 2)
– The log-likelihood for a single point xk is:
2
1
2
2 )(
2
1
2ln
2
1
)|(ln)( 

  kk xxpl
0
)(
)(
)(
2
1























l
l
l 0
2
)(
2
1
)(
1
2
2
2
1
2
1
2





















k
k
x
x
MLE: Example (1)

Summation over the n points gives:
from eq.(1) and eq.(2), we obtain:






nk
k
k
nk
k
k x
n
x
n 1
22
1
)(
1
;
1





nk
k
kx
1
1
2
(1)0)(
ˆ
1


MLE: Example (1)
 







nk
k
nk
k
kx
1 1
2
2
2
1
2
(2)0
ˆ
)ˆ(
ˆ
1




MLE: Example (2)
• The Multivariate Gaussian case:
– That is, p(xi |  ) ~ N(, ) , (unknown  and )
(Samples are drawn from a multivariate normal
population). Therefore:
  )()(
2
1
)2(ln
2
1
)|(ln 1
  
k
t
k
d
k xxxp


n
k
t
kk
n 1
)μˆx)(μˆx(
1ˆ
Similar analysis yields  and  as:




nk
k
kx
n 1
1
ˆ

Given the following samples of the feature values of
two different classes:
C1 = {10, 8, 7, 11, 9, 12, 6}
C2 = {15, 16, 11, 14, 18, 9, 22, 8}
Use Bayesian Decision Theory to find the class from
which the feature value of x = 12 is drawn from, if
the priors P(C1) = 0.6 and P(C2) = 0.4.
Assignment 3

Problem of Dimensionality
ASU-CS446 : Pattern Recognition. Prof. Mostafa Gadal-Haqq slide - 16

• Having problems involving 50 or 100
features. Two issues arises:
– How classification accuracy depends upon the
dimensionality (and the amount of training
data)?
– What is the computational complexity of
designing a classifier?
Problems of Dimensionality

1- Accuracy, Dimension, and Training Sample Size:
• If the features are statistically independent,
theoretically we could have excellent performance.
• For example: Two-class multivariate normal with
the same covariance
– if the priors are equal, we can show that:
0)(lim
)()(:
2
1
)(
21
1
21
2
2
2/
2








errorP
rwhere
dueerrorP
r
t
u
r



1- Accuracy, Dimension, and Training Sample Size:
• If features are independent then:
• Most useful features are the ones for which the
difference between the means is large relative to the
standard deviation
• It has frequently been observed in practice that, beyond
a certain point, the inclusion of additional features leads
to worse rather than better performance
2
1
212
22
2
2
1 ),...,,(








 


di
i i
ii
d
r
diag




77

2- Computational Complexity
• In describing the computational complexity of an
algorithm we are generally interested in the number
of basic mathematical operations, such as additions,
multiplications and divisions it requires, or in the
time and memory needed on a computer.

• Gaussian priors in d dimensions classifier
with n training samples for each of c classes
Total = O(d2..n)
• We have to compute the discriminant
function for each category c ,
• The total for c classes = O(cd2.n)  O(d2.n)
• Cost increase when d and n are large!
Computational Complexity: The MLE

• Goal: Combine features in order to reduce the
dimension of the feature space:
– Linear combinations are simple to compute & tractable.
– Projecting high dimensional data onto a lower
dimensional space.
• Two classical approaches for finding “optimal”
linear transformation:
– PCA (Principal Component Analysis) “Projection that
best represents the data in a least- square sense”
– MDA (Multiple Discriminant Analysis) “Projection that
best separates the data in a least-squares sense”
Component Analysis and Discriminants

• Fisher Linear Discriminant (Sec. 4.10)

• Multiple Discriminant Analysis (Sec. 4.11)

Assignment 4
• Generate an image that consists of Four classes
using the following distributions:
C1 = N( 50, 5) + N(0,10)
C2 = N(100,5) + N(0,20)
C3 = N(150,5) + N(0,30)
C4 = N(200,5) + N(0,40)
• Then, use the MLE to find the parameter of the
density functions (µ,) for each class in the image,
and use the Bayesian Decision Theory to classify
each pixel in the image to its corresponding class.
ASU-CS446 : Pattern Recognition. Prof. Mostafa Gadal-Haqq slide - 26

CSC446: Pattern Recognition (LN6)

More Related Content

What's hot (20)

Similar to CSC446: Pattern Recognition (LN6) (20)

More from Mostafa G. M. Mostafa (14)

Recently uploaded (20)

CSC446: Pattern Recognition (LN6)