INTRODUCTION TO

Machine Learning
 Why “Learn” ?
• Machine learning is programming computers to
optimize a performance criterion using example
data or past experience.
• There is no need to “learn” to calculate payroll
• Learning is used when:
– Human expertise does not exist (navigating on Mars),
– Humans are unable to explain their expertise (speech
recognition)
– Solution changes in time (routing on a computer
network)
– Solution needs to be adapted to particular cases (user
biometrics)

Lecture Notes for E Alpaydın 2004

Introduction to Machine Learning 2
© The MIT Press (V1.1)
 What We Talk About When We Talk
About“Learning”
• Learning general models from a data of particular
examples
• Data is cheap and abundant (data warehouses,
data marts); knowledge is expensive and scarce.
• Example in retail: Customer transactions to
consumer behavior:
People who bought “Da Vinci Code” also bought “The
Five People You Meet in Heaven” (www.amazon.com)
• Build a model that is a good and useful
approximation to the data.
Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 3
© The MIT Press (V1.1)
 Data Mining
• Retail: Market basket analysis, Customer
relationship management (CRM)
• Finance: Credit scoring, fraud detection
• Manufacturing: Optimization, troubleshooting
• Medicine: Medical diagnosis
• Telecommunications: Quality of service
optimization
• Bioinformatics: Motifs, alignment
• Web mining: Search engines
• ...
Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 4
© The MIT Press (V1.1)
 What is Machine Learning?
• Optimize a performance criterion using
example data or past experience.
• Role of Statistics: Inference from a sample
• Role of Computer science: Efficient algorithms
to
– Solve the optimization problem
– Representing and evaluating the model for inference

Lecture Notes for E Alpaydın 2004

Introduction to Machine Learning 5
© The MIT Press (V1.1)
 Applications
• Association
• Supervised Learning
– Classification
– Regression
• Unsupervised Learning
• Reinforcement Learning

Lecture Notes for E Alpaydın 2004

Introduction to Machine Learning 6
© The MIT Press (V1.1)
 Learning Associations
• Basket analysis:
P (Y | X ) probability that somebody who buys
X also buys Y where X and Y are
products/services.

Example: P ( chips | beer ) = 0.7

Lecture Notes for E Alpaydın 2004

Introduction to Machine Learning 7
© The MIT Press (V1.1)
 Classification

• Example: Credit
scoring
• Differentiating
between low-risk and
high-risk customers
from their income
and savings

Discriminant: IF income > θ1 AND savings > θ2

THEN low-risk ELSE high-risk
Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 8
© The MIT Press (V1.1)
 Classification: Applications
• Aka Pattern recognition
• Face recognition: Pose, lighting, occlusion (glasses,
beard), make-up, hair style
• Character recognition: Different handwriting styles.
• Speech recognition: Temporal dependency.
– Use of a dictionary or the syntax of the language.
– Sensor fusion: Combine multiple modalities; eg, visual
(lip image) and acoustic for speech
• Medical diagnosis: From symptoms to illnesses
• ...
Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 9
© The MIT Press (V1.1)
 Face Recognition

Training examples of a person

Test images

AT&T Laboratories, Cambridge UK

http://www.uk.research.att.com/facedatabase.html
Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 10
© The MIT Press (V1.1)
 Regression
• Example: Price of a
used car
y = wx+w0
• x : car attributes
y : price
y = g (x | θ)
g ( ) model,
θ parameters
Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 11
© The MIT Press (V1.1)
 Regression Applications

• Navigating a car: Angle of the steering wheel

(CMU NavLab)
• Kinematics
(x,y) of a robot
α = garm
(x,y)
1 1

α2= g2(x,y)
α2

α1

 Response surface design

Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 12
© The MIT Press (V1.1)
 Supervised Learning: Uses
• Prediction of future cases: Use the rule to
predict the output for future inputs
• Knowledge extraction: The rule is easy to
understand
• Compression: The rule is simpler than the data
it explains
• Outlier detection: Exceptions that are not
covered by the rule, e.g., fraud
Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 13
© The MIT Press (V1.1)
 Unsupervised Learning
• Learning “what normally happens”
• No output
• Clustering: Grouping similar instances
• Example applications
– Customer segmentation in CRM
– Image compression: Color quantization
– Bioinformatics: Learning motifs

Lecture Notes for E Alpaydın 2004

Introduction to Machine Learning 14
© The MIT Press (V1.1)
 Reinforcement Learning
• Learning a policy: A sequence of outputs
• No supervised output but delayed reward
• Credit assignment problem
• Game playing
• Robot in a maze
• Multiple agents, partial observability, ...

Lecture Notes for E Alpaydın 2004

Introduction to Machine Learning 15
© The MIT Press (V1.1)
 Resources: Datasets
• UCI Repository:
http://www.ics.uci.edu/~mlearn/MLRepository.html
• UCI KDD Archive:
http://kdd.ics.uci.edu/summary.data.application.html
• Statlib: http://lib.stat.cmu.edu/
• Delve: http://www.cs.utoronto.ca/~delve/

Lecture Notes for E Alpaydın 2004

Introduction to Machine Learning 16
© The MIT Press (V1.1)
 Resources: Journals
• Journal of Machine Learning Research www.jmlr.org
• Machine Learning
• Neural Computation
• Neural Networks
• IEEE Transactions on Neural Networks
• IEEE Transactions on Pattern Analysis and Machine
Intelligence
• Annals of Statistics
• Journal of the American Statistical Association
• ...
Lecture Notes for E Alpaydın 2004
Introduction to Machine Learning 17
© The MIT Press (V1.1)
 Resources: Conferences
• International Conference on Machine Learning (ICML)
– ICML05: http://icml.ais.fraunhofer.de/
• European Conference on Machine Learning (ECML)
– ECML05: http://ecmlpkdd05.liacc.up.pt/
• Neural Information Processing Systems (NIPS)
– NIPS05: http://nips.cc/
• Uncertainty in Artificial Intelligence (UAI)
– UAI05: http://www.cs.toronto.edu/uai2005/
• Computational Learning Theory (COLT)
– COLT05: http://learningtheory.org/colt2005/
• International Joint Conference on Artificial Intelligence (IJCAI)
– IJCAI05: http://ijcai05.csd.abdn.ac.uk/
• International Conference on Neural Networks (Europe)
– ICANN05: http://www.ibspan.waw.pl/ICANN-2005/
• ...

Lecture Notes for E Alpaydın 2004

Introduction to Machine Learning 18
© The MIT Press (V1.1)