EE534: Digital Singal Processing Fall 2024

EE534 – Pattern Recognition

Fall 2024

Instructor:
Prof. Munchurl Kim
Rm #1107, LG Innovation Hall (Bldg. N24)
042-350-7419, [email protected]

Class Hours:
Monday 4th period (14:30 ~ 15:45)
Wednesday 4th period (14:30 ~ 15:45)

Prerequisites:
The course assumes previous programming experience and some familiarity with linear algebra and statistics.
 Basic concepts of probability, random variables, and probability distributions
 Working knowledge and competency of MATLAB

Prerequisite class: EE210 - Probability and Introductory Random Processes,

EE528 – Engineering Random Processes

Credit: Lecture:Experiment:Credit: 3:0:3

Classroom: Rm. 1104, Bldg. N24 (LG Innovation Hall)

Class website: Pattern Recognition site at KLMS (https://klms.kaist.ac.kr) - KAIST Learning Management System.

Class materials
The class materials such as lecture notes, homework problem sheets and solutions, and data for experiments will be
posted in the KLMS class web site (https://klms.kaist.ac.kr).

Office Hours:
Tuesday 16:00 ~ 18:00, Thursday 16:00 ~ 18:00
Teaching Assistant:
Jongmin Park (Email: [email protected])

Teaching Assistant’s Office Hours:

Monday 16:00 ~ 18:00, Wednesday 16:00 ~ 18:00

Text Book:
Richard Duda, Peter Hart and David G. Stork, Pattern Classification and Scene Analysis, John Wiley & Sons 2001.

-1-
EE534: Digital Singal Processing Fall 2024

References:
 Kevin P. Murphy, Machine Learning: A Probabilistic Perspective, The MIT Press, 2012.
 Sergios Theodoridis and Konstantinos Koutroumbas, Pattern Recognition, Third Edition, Academic Press,
2006.
 Robert Schalkoff, Pattern Recognition: statistical, structural and neural approaches, John Wiley & Sons 1992.
 Christopher M. Bishop, Neural networks for Pattern Recognition, Oxford 1995.
 Andrew Webb, Statistical Pattern Recognition, Arnold publisher 1999.
 K. S. Fu, Syntactic Pattern Recognition and Applications, Prectice-Hall 1982.

Objective:
This class provides an introduction to classical pattern recognition. Pattern recognition is the assignment of a physical
object or event to one of several prescribed categories, which is very fundamental to many research areas.
Applications include face recognition, target classification, automated speech recognition, fingerprint identification and
optical character recognition. Homework assignments will include computer assignments using Python or MATLAB.
The in-class students are required to plan and conduct their own final project using the pattern recognition
technologies being learned throughout the course. This will lead students to experience the pattern recognition
technologies with real world applications. Total grades are based on the results of two in-class exams as well as on
performance on homework assignments and your class attendance.

Grading Policy:
05% Class Attendance
15% Homework
40% Midterm Exam
40% Final Exam

Homework:
There will be several graded homework assignments that involve written solutions to problems and software (MATLAB,
Python) implementations. Homework turned in late will not be accepted.
[Very Important Note]: Each homework should be submitted to TAs via e-mail prior to the submission due dates.
Each student can work on homework by hand on papers and take pictures on his/her solutions on the papers in JPG
format. The pictures for each homework should then be included in a single MS Word file and submitted to the TAs.
The visual quality of the pictures should guarantee the readability of your works on homework, otherwise the credits
will not be given for the unreadable solutions to the problems. The submission of homework must follow the format
below:
Ex. For homework #1 of a student with the name “FirstName LastName” and student ID 20241234,
1. The e-mail subject should be “[EE534-HW1-20241234] FirstName LastName”.
2. The WORD file name for homework #1 should be “EE534-HW1-20241234(FirstName
LastName).docx”.

Attendance:

Examinations:
There will two exams, Midterm and Final exams. Exams will be closed-book, closed-notes, no reference materials or
calculators. All work performed on an examination must be an individual effort. Cheating on an exam will result in a
severe grade penalty.

-2-
EE534: Digital Singal Processing Fall 2024

Academic Integrity:
Academic honesty is taken seriously in this class during the semester. For homework problems or programming
assignments, you are allowed to discuss the problems or assignments verbally with other class members, but under no
circumstances can you look at or copy anyone else's written solutions or code relating to homework problems or
programming assignments. All problem solutions and code submitted must be material you have personally written
during this semester, except for any library or utility functions to be supplied. Failure to adhere to this guideline can
result in a student receiving a failing grade in the class. It is the responsibility of each student to follow the course
guideline.
Academic dishonesty includes but is not limited to cheating, plagiarism, collusion, and submission for credit of any
work or materials that are attributable in whole or in part to another person, taking an examination for another person,
any act designed to give unfair advantage to a student or the attempt to commit such acts.

Class Ethics:
 No eating /drinking in class (except bottled water)
 Cell phones must be kept outside of class or shut-off during class
 No exceptions! If your cell-phone rings during class / lab (or you use it in any way), you will be asked to
leave and counted as unexcused absent. It will also cause difficult-to-repair damage to “class attendance”
part of your grade.
 No web surfing and / or unrelated use of computers, when we use computers in class / labs.
 In-class discussions are welcome, and in fact encouraged, within the limits of mutual respect and courtesy.
 You are responsible for checking the class web site often for announcements, homework / exam solutions.

-3-
EE534: Digital Singal Processing Fall 2024

Course Schedule
Spring 2024

Weeks Contents
Introduction to Pattern Recognition Course (EE534)
1-1
Chapter 1 Pattern Recognition Overview
Chapter 2 Bayesian Decision Theory
1-2
- Bayes Rule, Bayes Decision Theory (continue case)
Chapter 2 Bayesian Decision Theory
2-1 - Classifiers, Discriminant functions and Decision Surfaces
- HW #1
Chapter 2 Bayesian Decision Theory
2-2
- Normal Density, Discriminant function for Normal Density
Chapter 2 Bayesian Decision Theory
- Bayes Decision Rule (discrete case)
3-1
- Logistic Regression, Linear Transformation
- HW#2
Chapter 3 Maximum Likelihood and Bayesian Parameter Estimation
3-2
- Maximum likelihood estimation, Bayesian Parameter Estimation
Chapter 3 Maximum Likelihood and Bayesian Parameter Estimation
- Bayesian Parameter Estimation
4-1
- Dimensionality Problems
- Principle Component Analysis (PCA)
Chapter 3 Maximum Likelihood and Bayesian Parameter Estimation
- Principle Component Analysis (PCA)
4-2 - Fisher’s Linear Discriminant Functions
- Singular Value Decomposition (SVD)
- HW #3
Chapter 4 Non-parametric Techniques
5-1 - Nonparametric Density Estimation
- Parzen Windows
Chapter 4 Non-parametric Techniques
5-2 - K-NN Estimation
- K-NN Rule
Chapter 5 Linear Discriminant Function
6-1 - Linear Discriminant Function
Decision Surfaces
Chapter 5 Linear Discriminant Function
6-2 - Perceptron Criterion Function
Relaxation Algorithm
Chapter 5 Linear Discriminant Function
- Minimum Squared-Error for Discriminant Function
7-1
- Ho-Kashyap Algorithm
- HW #4
8 - MID-TERM EXAM (2 hrs)
Chapter 5 Support Vector Machine
9-1 - SVM for linearly separable two-category case
SVM for linearly separable two-category case
Chapter 5 Support Vector Machine
9-2 - ν -SVM
- Kernel Methods for SVM
Chapter 6 Multilayer Neural Networks
10-1
- Introduction

-4-
EE534: Digital Singal Processing Fall 2024

- Feedforward operation and Classification

- Back-propagation Algorithm
Chapter 6 Multilayer Neural Networks
10-2
- Back-propagation Algorithm
Chapter 6 Multilayer Neural Networks
- Back-propagation Algorithm
11-1 - Relation to Bayes Theory and Probability
- Improved Back-propagation Algorithm
- HW #5
Chapter 7 Stochastic Methods.
11-2 - Boltzmann Machine (BM)
- Restricted Boltzmann Machines (RBMs)
Chapter 9 Algorithm Independent Machine Learning.
- Bias-Variance Dilemma
12-1
- Resampling for Estimating Statistics
- Resampling for Classifier Design
Chapter 10 Unsupervised Learning and Clustering
- Maximum Likelihood Estimates
12-2
- K-means clustering algorithm
- Unsupervised Bayesian Clustering
Chapter 10 Unsupervised Learning and Clustering
13-1 - Hierarchical clustering algorithm
- Component Analysis
13-2 Generative Model I - Variational AutoEncoder (VAE) 1
14-1 Generative Model I - Variational AutoEncoder (VAE) 2
14-2 Generative Model II - Generative Adversarial Networks (GANs) 1
15-1 Generative Model II - Generative Adversarial Networks (GANs) 2
15-2 Generative Model III - Diffusion Models
16 FINAL EXAM (2 hrs)
* Above schedule is subject to change.

-5-
EE534: Digital Singal Processing Fall 2024

Some Tips to read mathematical notations

 f ( x ) : “f” of “x”

 e jθ : “e to the j theta ”

 ft : “f subscript t” or, in short, “f sub t”

 g t : “g superscript t”

 g t (x ) : “g superscript t of x”

 ∂f ( x ) ∂x : “take a partial derivative on f with respect to x”

 g ′ : “g prime”

 g ′( x ) : “g prime of x”

 g ′′ : “g double prime”

 g : “g under bar”

 A ∩ B : A intersection B

 A ∪ B : A union B

∞
 X (z ) = ∑ cn z − n : x of z is equal to the summation of c sub n times z to the minus n from
n = −∞

minus infinite to plus infinite

 A ( t ) cos (ωt ) : “A of t times cosine (of) omega t”

-6-