Scribd
Upload
66 views

Overfitting Regression

The document discusses overfitting in machine learning models. Overfitting occurs when a model learns the details and noise in the training data too well, resulting in poor performance on new data. It can happen due to insufficient data or noise in the training data. A model is overfitting if a simpler model has lower training error but higher test error. Ways to address overfitting include using more data, model selection techniques like regularization which penalize complex models, and being Bayesian in modeling assumptions.

Uploaded by

Allan Robey
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
66 views

Overfitting Regression

The document discusses overfitting in machine learning models. Overfitting occurs when a model learns the details and noise in the training data too well, resulting in poor performance on new data. It can happen due to insufficient data or noise in the training data. A model is overfitting if a simpler model has lower training error but higher test error. Ways to address overfitting include using more data, model selection techniques like regularization which penalize complex models, and being Bayesian in modeling assumptions.

Uploaded by

Allan Robey
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

Machine Learning

Overfitting

Kriti Srivastava

An adaptation of NPTEL lectures for educational purpose


Overfitting
• Learning a model that classifies the training data
perfectly may not lead to the model with the best
generalization performance.
– There may be noise in the training data
– May be based on insufficient data
• A hypothesis h is said to overfit the training data if
there is another hypothesis, h’, such that h has
smaller error than h’ on the training data but h has
larger error on the test data than h’.
Overfitting
• Learning a model that classifies the training data perfectly may not
lead to the model with the best generalization performance.
– There may be noise in the training data
– May be based on insufficient data
• A hypothesis h is said to overfit the training data if there is another
hypothesis, h’, such that h has smaller error than h’ on the training
data but h has larger error on the test data than h’.

On training

accuracy On testing

Complexity of model
Underfitting and Overfitting
Overfitting

Underfitting: when model is too simple, both training and test errors are large
Overfitting due to Noise

Decision boundary is distorted by noise point


Overfitting due to Insufficient Examples

Lack of data points makes it difficult to predict correctly the class labels
of that region
Notes on Overfitting
• Overfitting results in model that are more complex
than necessary

• Training error no longer provides a good estimate of


how well the model will perform on previously
unseen records
Model Selection & Generalization
• Learning is an ill-posed problem; data is not sufficient
to find a unique solution
• The need for inductive bias, assumptions about H
• Generalization: How well a model performs on new
data
• Overfitting: H more complex than C or f
• Underfitting: H less complex than C or f

8
Triple Trade-Off
• There is a trade-off between three factors:
– Complexity of H, c (H),
– Training set size, N,
– Generalization error, E on new data overfitting

• As N increases, E decreases
• As c (H) increases, first E decreases and then E increases
• As c (H) increases, the training error decreases for some time
and then stays constant (frequently at 0)

9
Notes on Overfitting
• overfitting happens when a model is capturing
idiosyncrasies of the data rather than generalities.
– Often caused by too many parameters relative to the
amount of training data.
– E.g. an order-N polynomial can intersect any N+1 data
points
Dealing with Overfitting
• Use more data
• Use a tuning set
• Regularization
• Be a Bayesian

11
Regularization
• In a linear regression model overfitting is
characterized by large weights.

12
Penalize large weights in Linear Regression
• Introduce a penalty term in the loss function.

Regularized Regression
1. (L2-Regularization or Ridge Regression)

1. L1-Regularization

13

You might also like