Scribd
Upload
7 views24 pages

Performance Tuning

This document covers performance tuning for linear regression, focusing on R-squared metrics and regularization methods. It discusses the limitations of R-squared and introduces regularization techniques such as LASSO (L1), Ridge (L2), and Elastic Net, which help simplify models and manage multicollinearity among regressors. The lecture emphasizes the importance of selecting significant features and balancing model complexity to improve predictive performance.

Uploaded by

d36078067
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
7 views24 pages

Performance Tuning

This document covers performance tuning for linear regression, focusing on R-squared metrics and regularization methods. It discusses the limitations of R-squared and introduces regularization techniques such as LASSO (L1), Ridge (L2), and Elastic Net, which help simplify models and manage multicollinearity among regressors. The lecture emphasizes the importance of selecting significant features and balancing model complexity to improve predictive performance.

Uploaded by

d36078067
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 24

Machine Learning

Theory & Practice

Module 3: Linear Regression

Lecture 2: Performance Tuning for Linear


Regression
Lecture Outline

Topic 1: R-squared and Adjusted R-squared

Topic 2: Regularization
Topic 1: R-Squared and Adjusted
R-Squared:
These are performance metrics that show how much of the
output variation is explained by a LR model
Variation in Response
(accumulated wealth)

Possible responses

(age)
Coefficient of Determination: R-Squared - R2
•
Issues with R-squared

•
dftotal=
• LR: R2 = 1-SSres/SStot •
• R2 Always increases as more regressors
are added.
• R2 is biased estimate
• R2 Does not penalize non-significant terms
• R2 Is always positive
• R2 is not suitable for statistical test of
significance of weights
Topic 2: Regularization methods
REGULARIZATION: PROCESS OF MAKING THE LEARNING
MODEL SIMPLER
Regularization with Validation
Example: Cost of House
Regressors:
1. Floor Area
2. No of bedrooms
3. No of balconies
4. Type of locality No. Regressors

5. Green cover SSres

6. Front door facing which direction


7. Educational background of neighbourhood
Regularization with Modified Loss Functions

• Augment Ordinary Least Squares with regularization


term:
• LASSO Regression L1 Regularization
• Ridge Regression L2 Regularization
• Elastic Net Regularization
Least Absolute Shrinkage & Selection Operator(LASSO):
L1 Regularization
Minimize cost function: 1) Ordinary Least Squares 2) Regularization Term

Minimize { }
Forcing

• L1 penalizes regressors by shrinking their weights


• Regressors that contribute little to error reduction are more penalized
• λ is the weighting factor for regularization to tune overfit ↔ underfit
w2
Sum of squares of Residuals

• L1 can lead to some zero coefficients.


especially if λ is large
w1

• L1 not only reduces overfitting


but also helps eliminate insignificant features
• LASSO selects only significant regressors
Multicollinear Regressors
• The following regressors are highly positively correlated and
each apparently impacts property cost.
• Total Area
• Number of rooms
• Free areas: balconies and corridors
• Size of rooms
• Without regularization, all coefficients would be inflated
• L1 retains only one of them and eliminates the rest
Ridge Regression : L2 Regularization
Minimize cost function: 1) Ordinary Least Squares 2) Regularization Term

Minimize { }
Forcing,
w2
• L2 leads to near zero coefficients

Constraint Region for L2 w1

• L2 handles multiple correlated regressors better


Elastic Net Regularization
•
Constraint Regions in
Ridge, LASSO & Elastic Net Regularization

w2
w1
Recap
•
Recap
Ridge regression employ L2 regularization by adding “sum of
squares of weights” to OLS, with weight factor λ
L2 regularization cannot eliminate any regressor but can
appropriately shrink insignificant ones
Ridge regression handles multiple correlated features by
diminishing or enhancing them simultaneously, instead
eliminating all but one (as in L1)
Elastic Net regularization employs a mix of L1 and L2 with
their own weighting factors to create a balanced model
Balance is a sense of harmony….

You might also like