Bias-Variance Tradeoff
in Machine Learning
Overfitting
Underfitting
Bias-Variance Tradeoff
The bias-variance tradeoff is a fundamental concept in machine learning that helps us understand the performance of a
model. It describes the tradeoff between two sources of error that affect the predictive performance of a
model: bias and variance.
Bias:
•Definition: Bias refers to the error introduced by approximating a real-world problem, which may be complex,
by a simplified model.
•High Bias: A model with high bias pays little attention to the training data and oversimplifies the problem. This
leads to underfitting.
•Causes:
• Using an overly simple model (e.g., linear regression for a nonlinear problem).
• Not capturing the underlying patterns in the data.
•Impact: High bias results in poor performance on both the training and test datasets.
Bias and Training Error:
•Bias is related to the training error.
•A model with high bias is too simplistic and fails to capture the underlying patterns in the training
data.
• This results in high training error because the model cannot fit the training data well.
• Example: Using a linear model to fit nonlinear data.
•Low bias means the model is flexible enough to fit the training data well, resulting in low training
error.
Variance
•Definition: Variance refers to the error introduced by the model's sensitivity to small fluctuations in the
training dataset.
•High Variance: A model with high variance pays too much attention to the training data, including noise and
outliers. This leads to overfitting.
•Causes:
• Using an overly complex model (e.g., a high-degree polynomial).
• Capturing noise in the training data as if it were a genuine pattern.
•Impact: High variance results in excellent performance on the training dataset but poor performance on the
test dataset.
Variance and Testing Error:
•Variance is related to the testing error.
•A model with high variance is too complex and fits the training data too closely, including noise and
outliers.
• This results in high testing error because the model fails to generalize to unseen data.
• Example: Using a high-degree polynomial that fits the training data perfectly but performs
poorly on test data.
•Low variance means the model generalizes well to unseen data, resulting in low testing error.
Overfitting:
Overfitting occurs when a model learns the training data too well, including its noise and outliers,
which harms its generalization to unseen data.
Characteristics of Overfitting:
•The model performs exceptionally well on the training data but poorly on the test data.
•The model has high variance and low bias.
•The model is overly complex, with too many parameters relative to the number of observations.
Causes of Overfitting:
•Using a model that is too complex (e.g., high-degree polynomials, deep neural networks).
•Training for too many epochs (in iterative algorithms like gradient descent).
•Having insufficient training data relative to the model's complexity.
How to Detect Overfitting:
•Compare training and test performance:
• If training accuracy is much higher than test accuracy, the model is likely overfitting.
•Use techniques like cross-validation to evaluate performance on unseen data.
How to Prevent Overfitting:
•Regularization: Add a penalty for complexity (e.g., L1/L2 regularization).
•Simplify the Model: Use a less complex model (e.g., reduce polynomial degree).
•Increase Training Data: More data can help the model generalize better.
•Early Stopping: Stop training when validation performance starts to degrade.
•Dropout: In neural networks, randomly drop units during training to prevent co-adaptation.
•Pruning: Remove unnecessary parts of the model (e.g., decision trees).
Underfitting:
Underfitting occurs when a model is too simple to capture the underlying patterns in the data,
resulting in poor performance on both the training and test datasets.
Characteristics of Underfitting
•The model performs poorly on both the training and test data.
•The model has high bias and low variance.
•The model is overly simplistic, with too few parameters to capture the complexity of the data.
Causes of Underfitting
•Using a model that is too simple (e.g., linear regression for a nonlinear problem).
•Not training the model for enough epochs (under-training).
•Using insufficient features or failing to engineer meaningful features.
How to Detect Underfitting
•Compare training and test performance:
• If both training and test accuracy are low, the model is likely underfitting.
How to Prevent Underfitting
•Increase Model Complexity: Use a more complex model (e.g., higher-degree polynomials, deeper
neural networks).
•Feature Engineering: Add more relevant features or create new features.
•Train Longer: Allow the model to train for more epochs (if using iterative algorithms).
•Reduce Regularization: If regularization is too strong, it can cause underfitting.
🎯 Key Takeaways:
✅ Underfitting → High Bias, Low Variance → Model is too simple, fails to
capture patterns.
✅ Overfitting → Low Bias, High Variance → Model is too complex, memorizes
noise.
✅ Generalized Model → Low Bias, Low Variance → Balanced, best
performance.
