1

What is a Pattern?
• A set of instances that
- share some regularities and similarities
- is repeatable
- is observable, sometimes partially, using sensors
- May have noise and distortion
 2

What is Pattern Recognition?

• Pattern recognition (PR) is the scientific discipline

that concerns the description and classification
(recognition) of patterns (objects)

• PR techniques are an important component of

intelligent systems and are used for many
application domains
- Decision making
- Object and pattern classification
 3

Human Perception

• Humans have developed highly sophisticated skills for

sensing their environment and taking actions
according to what they observe, e.g., I recognizing a
face, I understanding spoken words, I reading
handwriting, I distinguishing fresh food from its smell. I
We would like to give similar capabilities to machines
 4

• Every day, we recognize faces around us, but we do

it unconsciously and because we cannot explain our
expertise, we find it difficult to write a computer
program to do the same.

• Each person’s face is a pattern composed of a

particular combination of structures (eyes, nose,
mouth, . . .) located in certain positions on the face.
By analyzing sample images of faces, a program
should be able to capture the pattern specific to a face
and identify (or recognize) it as a face (as a member of
a category or class we already know); this would be
pattern recognition
 5
A class is a collection of objects that are similar, but
not necessarily identical, and which is distinguishable
from other classes. Fig.1 illustrates the difference
between classification where the classes are known
beforehand and classification where classes are
created after inspecting the objects

Fig 1 Classification when the classes are (A) known and (B) unknown beforehand
 6

Human and Machine Perception

• We are often influenced by the knowledge of how

patterns are modeled and recognized in nature when
we develop pattern recognition algorithms. Research
on machine perception also helps us gain deeper
understanding and appreciation for pattern recognition
systems in nature. Yet, we also apply many
techniques that are purely numerical and do not have
any correspondence in natural systems.
 7

Machine Perception

• Build a machine that can recognize patterns:

• Speech recognition
• Fingerprint identification
• OCR (Optical Character Recognition)
• DNA sequence identification

Pattern Classification, Chapter 1

 8

An Example

• “Sorting incoming Fish on a conveyor according to

species using optical sensing”

Sea bass
Species
Salmon
 9

• Problem Analysis
• Set up a camera and take some sample images to extract
features

• Length
• Lightness
• Width
• Number and shape of fins
• Position of the mouth, etc…
• This is the set of all suggested features to explore for use in our
classifier!

Pattern Classification, Chapter 1

 10

• Preprocessing

• Use a segmentation operation to isolate fishes from one

another and from the background

• Information from a single fish is sent to a feature extractor

whose purpose is to reduce the data by measuring certain
features

• The features are passed to a classifier

 11

Pattern Classification, Chapter 1

• Classification
12

• Select the length of the fish as a possible feature for

discrimination

Pattern Classification, Chapter 1

 13
The length is a poor feature alone!
Select the lightness as a possible feature.

Pattern Classification, Chapter 1

 14

• Threshold decision boundary and cost relationship

• Move our decision boundary toward smaller values of
lightness in order to minimize the cost (reduce the number
of sea bass that are classified salmon!)

Task of decision theory

Pattern Classification, Chapter 1

 15

• Feature extraction
Task: to extract features which are good for classification.
Good features:
• Objects from the same class have similar feature values.

• Objects from different classes have different values.

Pattern Classification, Chapter 1

 16

Therefore… Basic concepts

Feature vector : x Є X
- A vector of observations (measurements).
- x is a point in feature space X

Hidden state: y Є Y
- Cannot be directly measured.
-Patterns with equal hidden state belong to the same class.

Task
- To design a classifier (decision rule) q : X → Y
which decides about a hidden state based on an observation.
Pattern Classification, Chapter 1
 17

• Adopt the lightness and add the width of the fish

Fish xT = [x1, x2]

Lightness Width

Pattern Classification, Chapter 1

 18

An example of Industrial Inspection

In our case…

Pattern Classification, Chapter 1

 19

Linear (simple) decision boundary; Cost of misclassification ?

Pattern Classification, Chapter 1

 20

• We might add other features that are not correlated

with the ones we already have. A precaution should
be taken not to reduce the performance by adding
such “noisy features”

• Ideally, the best decision boundary should be the one

which provides an optimal performance such as in the
following figure:

Pattern Classification, Chapter 1

 21

Pattern Classification, Chapter 1

 22

• However, our satisfaction is premature

because the central aim of designing a classifier
is to correctly classify novel input

Issue of generalization!

Pattern Classification, Chapter 1

 23

An example of Industrial Inspection

Pattern Classification, Chapter 1

 24

An example of Industrial Inspection

Overfitting and underfitting

Pattern Classification, Chapter 1

 25
Emerging Applications
Interest in pattern recognition and classification has
grown due to emerging applications, which include :-
• Data mining (sifting through a large volume of data to
extract a small amount of relevant and useful information,
e.g., fraud detection).
• Biometrics (personal identification based on physical
attributes of the face, iris, fingerprints, etc.)
• Machine vision (e.g., automated visual inspection in an
assembly line)
• Character recognition [ (automated teller machines)]
• Document recognition (e.g., recognize whether an e-mail
is spam or not, based on the message header and
content)
•
26
Computer-aided diagnosis [e.g., helping doctors make
diagnostic decisions based on interpreting medical data
such as mammographic images.
• Medical imaging [e.g., classifying cells as malignant or
benign based on the results of magnetic resonance
imaging (MRI) scans .
• Speech recognition (e.g., helping handicapped patients
to control machines).
• Bioinformatics (e.g., DNA sequence analysis to detect
genes related to particular diseases).
• Remote sensing (e.g., land use and crop yield).
• Astronomy (classifying galaxies based on their shapes
 27
The methods used have been developed in
various fields, often independently.
• In statistics, going from particular observations to general
descriptions is called inference, learning [i.e., using example
(training) data] is called estimation, and classification is known
as discriminant analysis.

• In engineering, classification is called pattern recognition

and the approach is nonparametric and much more empirical .
Other approaches have their origins in machine learning ,
artificial intelligence , artificial neural networks , and data
mining. Fig. 2 illustrated the Pattern recognition and related
fields
 28

Fig. 2 Pattern recognition and related

fields
 29
Pattern Recognition Systems
• Data acquisition and sensing:
- Use of a transducer (camera or microphone)
- Important issues: bandwidth, resolution,
sensitivity, distortion, SNR, latency, etc.
• Pre-processing:
-Removal of noise in data.
- Isolation of patterns of interest from the background

• Segmentation and grouping

Patterns should be well separated and should not overlap

Pattern Classification, Chapter 1

 30

Pattern Recognition Systems

Pattern Classification, Chapter 1

 31

Pattern Recognition Systems

• Feature extraction
Finding a new representation in terms of features
- Discriminative features
- Invariant features with respect to translation, rotation and
scale.

• Post Processing
• Exploit context input dependent information other than from
the target pattern itself to improve performance

• Classification
• Use a feature vector provided by a feature extractor to
assign the object to a category
 32

The Design Cycle

• Data collection
• Feature Choice
• Model Choice
• Training
• Evaluation
• Computational Complexity

Pattern Classification, Chapter 1

 33

Pattern Classification, Chapter 1

 34

• Data Collection

• Collect an adequately large and representative set of

examples and divide them into (70%) training and (30%)
testing the system.

• Feature Choice
• Depends on the characteristics of the problem domain.
Simple to extract, invariant to irrelevant transformation
insensitive to noise
 35

• Model Choice
Unsatisfied with the performance of one
classifier, jump to another class of models

• Training
-Use sample data to train the classifier.
- Use Many different procedures for training classifiers
and choosing models :
• Random Sub-sampling
• Bootstrap
• Cross-Validation

Pattern Classification, Chapter 1

 36

• Evaluation
– Measure the error rate (performance) using
different training methods.

– Switch from one set of features to another, or

from one model to another to improve accuracy,
i.e. to minimize error rate.

Pattern Classification, Chapter 1

 37

Performance of PR Systems

• Error rate (Prob. of misclassification)

• Speed

• Cost

• Robustness

• Reject option

• Return on investment

Pattern Classification, Chapter 1

 38

• Computational Complexity
• What is the trade-off between computational
ease and performance?

• How an algorithm scales as a function of the

number of features, patterns or categories?

Pattern Classification, Chapter 1

 39

Limitation of PR Systems

- Human have the ability to switch rapidly and

seamlessly between different pattern recognition
tasks.

- It is very difficult to design a device that is capable of

performing a variety of different classification tasks as
human.
 40

Learning and Adaptation

• Supervised learning
• A teacher provides a category label or cost for each
pattern in the training set

• Unsupervised learning
• The system forms clusters or “natural groupings” of the
input patterns
 41

Summary
• Pattern recognition is extremely useful and are now part of
many crucial computer applications:
• Pattern recognition is a very difficult problem and have
many complex sub-problems.
• Successful systems have been built in well constrained
domains.
• No single technique/model is suited for all pattern
recognition problems
• Use of object models, constraints, and context is
necessary for identifying complex patterns
• Careful sensor design and feature extraction can lead to
simple classifiers
 42

References

• Pattern Classification (2nd ed) by R. O. Duda, P.

E. Hart and D. G. Stork, John Wiley & Sons, 2000

• Pattern Recognition and Classification, An

Introduction, by Geoff Dougherty, springer Science
Business Media New York 2013