CS446: Machine Learning

Lecture 10-11 (Concepts Relevant to ML Models)

Instructor:
Dr. Muhammad Kabir
Assistant Professor
[email protected]

School of Systems and Technology

Department of Computer Science
University of Management and Technology, Lahore
 Previous Lectures…
 Label Encoding

 Understanding and Implementing Train Test Split

 Machine Learning Process

 How Machine Learning works?

 What is Learning?

 Understanding Curve fitting in Machine Learning

 Complex Model need?

 Today’s Lectures…
 Selecting right ML model
 Cross-Validation – Statistical measures
 Overfitting in ML – concept, signs, causes and prevention
 Under fitting in ML – concept, signs, causes and prevention
 Bias-variance tradeoff in ML
 Loss function
 Model evaluation – Accuracy score, mean score error
 Model parameters and hyperparameter
 Gradient decent in ML
Machine Learning Process….
 Model Selection…
 The process of choosing the best suited model for a particular problem
is called model selection.
 It depends on various factors such as the dataset, task, nature of the
model etc.
 Model Selection…
 Model selection depends on
1. Type of Data
a. Images & videos – CNN
b. Text or Speech – RNN
c. Numerical data – SVM, Logistic Regression,
Decision Trees etc.
2. Tasks we need to carry out
a. Classification tasks – SVM, LR, DT
b. Regression tasks – Linear Regression,
Random forest, polynomial reg.
c. Clustering tasks – K-means, Clustering,
Hierarchical clustering
 Model Validation methods…
 In ML approaches, the error rate is commonly utilized to appraise the
performance of the classifiers and for this purpose, the benchmark dataset
is divided into different portions.
 Usually, two statistical methods called holdout and cross-validation are
used for the partitioning of the benchmark dataset.
1. Holdout
a. Some data is kept for testing, and the remaining is reserved for training.
b. Generally, one third for testing and two-third for the training.
2. Cross-validation
a. Dataset is partitioned into a fixed-size of mutually exclusive folds.
b. three renowned statistical cross-validation tests
c. subsampling test (k-fold), jackknife, and independent test
 Cross-validation methods…
1. Jackknife
a. Leave-one-out cross-validation (LOOCV).
b. the dataset is fragmented into n-folds.
c. One for testing and n-1 for training
d. maximum data is utilized for training
e. Time-consuming
f. Result didn’t change
 Cross-validation methods…

2. K-fold
a. Dataset is divided into
K subsets without
repetition.
b. Each time uses one of
the subsets as the test
set, and the remaining
K-1 subsets as the
training set repeats K
times to learn K-1
classification models
 Cross-validation methods…
3. Independent test
a. The independent dataset test is one of the essential tests,
employed to assess the generalization performance of model in
the field of ML.
b. The model is trained to deploy a benchmark dataset, while an
independent dataset is utilized for its testing.
c. The data, consumed in the training phase are not used for the
testing phase, which means that the data in the benchmark
dataset and independent dataset are dissimilar.
d. The main reasons to exploit an independent dataset test are, to
evaluate whether a trained model over-fits over the training
dataset, and to adequately assess the generalization capabilities
of a trained model.
 Cross-validation methods…
Cross-validation implementation
 Overfitting in ML Models….
 Overfitting refers to a model
behavior that models the training
data too well.
 Overfitting happens when a model
learns the details and noise in the
training dataset to the extent that
it negatively impacts the
performance of the model.

 Sign that the model has overfitted

– high training data accuracy & low
testing data accuracy
Example - Overfitting in ML Models….
Example - Overfitting in ML Models….
 Overfitting in ML Models….
Causes of overfitting
 Less data
 Increased complexity of the model
 More layers in Neural Network (NN)

Preventing the overfitting

 Using more data
 Reduce the number of layers in the NN
 Early stopping
 Bias-variance tradeoff
 Using Dropouts (usually in NN and Deep Learning)
 Underfitting in ML Models….

 Under-fitting happens when the model does

not learn enough from the data.
 It occurs when a ML model cannot capture
the underlying trend of the data.

 Sign that the model has overfitted – Very Low

training data accuracy & high testing data accuracy
Example - Underfitting in ML Models….
Example - Underfitting in ML Models….
 Underfitting in ML Models….
Causes of overfitting
 Choosing the wrong model
 Less complexity of the model
 Less variance but high bias

Preventing the overfitting

 Selecting the correct model appropriate for the problem
 Increasing the complexity of the model
 More number of parameters to the model
 Optimal Bias-variance tradeoff
 Bias-variance Tradeoff in ML Models….
Bias
 The bias is known as the difference between the prediction of the
values by the ML model and the correct value.
 Being high in biasing gives a large error in training as well as testing
data.
 Its recommended that an algorithm should always be low biased to
avoid the problem of Underfitting.
 Bias-variance Tradeoff in ML Models….
Variance
 The variability of model prediction for a
given data point which tells us spread of
our data is called the variance of the
model. High Variance
 The model with high variance has a very
complex fit to the training data and thus
is not able to fit accurately on the data
which it hasn’t seen before.
 high on variance - overfitting of data.
 The high variance data looks like follows.
High Variance
Bias-variance Tradeoff in ML Models….
 Bias-variance Tradeoff in ML Models….
Tradeoff
 Too simple algorithm - may be on high bias
and low variance
 Too complex - may be on high variance and
low bias.
 This tradeoff in complexity is why there is
a tradeoff between bias and variance.
 An algorithm can’t be more complex and
less complex at the same time.
 For the graph, the perfect tradeoff will be
like.
 Loss function in ML Models….

 A loss function measures how far an estimated values is from

the true (actual) values – difference between predicted and
actual values.
 It helps to determine which model performs better and
which parameters are better.
Example - Loss function in ML Models….
Example - Loss function in ML Models….
Example - Loss function in ML Models….
ML Models evaluation – Accuracy & Error….
ML Models evaluation – Accuracy & Error….
ML Models evaluation – Accuracy & Error….
ML Models evaluation – Accuracy & Error….
 Chapter Reading

Chapter Chapter 01
Pattern Recognition and
Machine Learning
by
Machine Learning
by
Tom Mitchell
Christopher M. Bishop