Ridge regression
3 likes4,774 views
Ridge regression is a technique used for linear regression when the number of predictor variables is greater than the number of observations. It addresses the problem of overfitting by adding a regularization term to the loss function that shrinks large coefficients. This regularization term penalizes coefficients with large magnitudes, improving the model's generalization. Ridge regression finds a balance between minimizing training error and minimizing the size of coefficients by introducing a tuning parameter lambda. The document includes an experiment demonstrating how different lambda values affect the variance and mean squared error of the ridge regression model.
![Linear Regression Continued
1 In Equation 1, β0, β1 are two unknown constants, also known as
parameters.
2 Our objective is to use training data and estimate the values of ˆβ0, ˆβ1
3 So far we have discussed the case of simple linear regression. In case
of multiple linear regression, our linear regression model takes the
form
y = β0 + β1x1 + β2x2 + . . . + βpxp + (2)
4 A commonly used technique to find the estimates of the
co-efficients(parameters) is least square method [1].
Ananda Swarup Das A Note on Ridge Regression October 16, 2016 3 / 16](https://image.slidesharecdn.com/ridgeregression-161016121454/85/Ridge-regression-3-320.jpg)
![The Bias-Variance Trade Off
As stated in [1], the expected value of the residual error (yi − ˆf (xi )) is
given by
E(yi − ˆf (xi ))2
= Var(ˆf (xi )) + [Bias(ˆf (xi ))]2
+ Var( ) (4)
1 In the above equation, the first term on the right hand side denotes
the variance of the model that is the amount by which ˆf would
change if the parameters β1, . . . , βp are estimated using different
training data.
2 The second term denotes the error introduced by approximating a
may-be complicated real-life model with a simpler model.
Ananda Swarup Das A Note on Ridge Regression October 16, 2016 5 / 16](https://image.slidesharecdn.com/ridgeregression-161016121454/85/Ridge-regression-5-320.jpg)
![The Bias-Variance Trade Off Continued
Also shown in [1], the expected value of residual error (yi − ˆf (xi )) can also
be expressed as
E(yi − ˆf (xi ))2
= E(f (xi ) + − ˆf (xi ))2
= [f (xi ) − ˆf (xi )]2
+ Var( ) (5)
Notice that we have replace yi = f (xi ) + . The first part [f (xi ) − ˆf (xi )]2
is reducible and we want our estimation of parameters be such that ˆf (xi )
is as close as possible to f (xi ). However, the Var( ) is irreducible.
Ananda Swarup Das A Note on Ridge Regression October 16, 2016 6 / 16](https://image.slidesharecdn.com/ridgeregression-161016121454/85/Ridge-regression-6-320.jpg)
![What do we reduce
1 Reconsider the Equation 4,
E(yi − ˆf (xi ))2 = Var(ˆf (xi )) + [Bias(ˆf (xi ))]2 + Var( ), the expected
value of MSE cannot be less than Var( ).
2 Thus, we have to try to reduce the variance and the bias for the
model ˆf .
Ananda Swarup Das A Note on Ridge Regression October 16, 2016 7 / 16](https://image.slidesharecdn.com/ridgeregression-161016121454/85/Ridge-regression-7-320.jpg)
![The Significance of the choice of λ
1 Stated in [1], for every value of λ there exists a constant s such that
the problem of ridge regression coefficient estimation boils down to
minimize
n
i=1
(yi − β0 −
p
j=1
βj xi,j )2
(6)
s.t p
j=1 β2
j ≤ s
2 Notice that if p = 2, under the constaint p
j=1 β2
j ≤ s, ridge
regression coefficient estimation is equivalent to finding the
coefficients lying within a circle (in general a sphere) centered at the
origin and is of radius
√
s, such that the Equation 6 is minimized.
Ananda Swarup Das A Note on Ridge Regression October 16, 2016 10 / 16](https://image.slidesharecdn.com/ridgeregression-161016121454/85/Ridge-regression-10-320.jpg)
Ridge regression
- 1. A Note on Ridge Regression Ananda Swarup Das October 16, 2016 Ananda Swarup Das A Note on Ridge Regression October 16, 2016 1 / 16
- 2. Linear Regression 1 Linear Regression is a simple approach for Supervised Learning and is used for quantitative predictions. 2 Assuming X to be a quantitative predictor and y to be a quantitative response and the relationship between the predictor and the response to be linear, the linear relationship can be written as y ≈ β0 + β1X (1) 3 The relationship is represented as an approximate one as it is assumed that y = β0 + β1X + where is an irreducible error that might have crept in while recording the data. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 2 / 16
- 3. Linear Regression Continued 1 In Equation 1, β0, β1 are two unknown constants, also known as parameters. 2 Our objective is to use training data and estimate the values of ˆβ0, ˆβ1 3 So far we have discussed the case of simple linear regression. In case of multiple linear regression, our linear regression model takes the form y = β0 + β1x1 + β2x2 + . . . + βpxp + (2) 4 A commonly used technique to find the estimates of the co-efficients(parameters) is least square method [1]. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 3 / 16
- 4. How good is our Estimation of the parameters 1 In the regression setting, a technique to measure the fit is mean-squared error which is given as MSE = 1 n n i=1 (yi − ˆf (xi ))2 (3) Here, n is the number of observations, yi is the true response and ˆf (xi ) is the response predicted by our model defined by the co-efficients estimated by the training data. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 4 / 16
- 5. The Bias-Variance Trade Off As stated in [1], the expected value of the residual error (yi − ˆf (xi )) is given by E(yi − ˆf (xi ))2 = Var(ˆf (xi )) + [Bias(ˆf (xi ))]2 + Var( ) (4) 1 In the above equation, the first term on the right hand side denotes the variance of the model that is the amount by which ˆf would change if the parameters β1, . . . , βp are estimated using different training data. 2 The second term denotes the error introduced by approximating a may-be complicated real-life model with a simpler model. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 5 / 16
- 6. The Bias-Variance Trade Off Continued Also shown in [1], the expected value of residual error (yi − ˆf (xi )) can also be expressed as E(yi − ˆf (xi ))2 = E(f (xi ) + − ˆf (xi ))2 = [f (xi ) − ˆf (xi )]2 + Var( ) (5) Notice that we have replace yi = f (xi ) + . The first part [f (xi ) − ˆf (xi )]2 is reducible and we want our estimation of parameters be such that ˆf (xi ) is as close as possible to f (xi ). However, the Var( ) is irreducible. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 6 / 16
- 7. What do we reduce 1 Reconsider the Equation 4, E(yi − ˆf (xi ))2 = Var(ˆf (xi )) + [Bias(ˆf (xi ))]2 + Var( ), the expected value of MSE cannot be less than Var( ). 2 Thus, we have to try to reduce the variance and the bias for the model ˆf . Ananda Swarup Das A Note on Ridge Regression October 16, 2016 7 / 16
- 8. Certain Situations Provided the true relationship between the predictor and the response is linear, the least square method will have less bias. 1 If the size of the training data, n is very very large compared to the number of predictors that is n >> p, the least square estimates tend to have less variance. 2 If the size of the training data, n is slightly larger than p, then the least square estimates may have high variance. 3 If n < p, least square method should not be applied without using dimension reducing techniques. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 8 / 16
- 9. Ridge Regression 1 In this presentation, we will deal with the second situation where n is slightly greater than p using Ridge Regression which has been found to be significantly helpful in dealing with variance. 2 In the least square method, coefficients β1 . . . βp are estimated by minimizing Residual Sum of Squares(RSS) RSS = n i=1(yi − β0 − p j=1 βj xi,j )2. Notice that β0 = y , the mean of all the responses. 3 In case of Ridge Regression, the minimization function changes to n i=1(yi − β0 − p j=1 βj xi,j )2 + λ p j=1 β2 j . The λ is a tuning parameter which constraints the choices of the coefficients, but decreases the variance. To minimize the objective function, both the additive terms are to be minimized. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 9 / 16
- 10. The Significance of the choice of λ 1 Stated in [1], for every value of λ there exists a constant s such that the problem of ridge regression coefficient estimation boils down to minimize n i=1 (yi − β0 − p j=1 βj xi,j )2 (6) s.t p j=1 β2 j ≤ s 2 Notice that if p = 2, under the constaint p j=1 β2 j ≤ s, ridge regression coefficient estimation is equivalent to finding the coefficients lying within a circle (in general a sphere) centered at the origin and is of radius √ s, such that the Equation 6 is minimized. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 10 / 16
- 11. Ridge Regression Coefficient Estimation B1 B2 Figure: The Residual Square of Sum(RSS) n i=1(yi − β0 − p j=1 βj xi,j )2 is a convex function and when p = 2, the contours look like a set of concentric ellipses. The least square solution is denoted by the innermost marron dot. The ellipses centered at that dot have constant RSS thats is all points on a given ellipses share the common value of RSS which is equal to the Var( ). As the ellipses expand away from the least square estimate, the RSS increases. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 11 / 16
- 12. Ridge Regression Coefficient Estimation B1 B2 Figure: In general, the ridge regression coefficient estimates are given by the first point at which the ellipse contacts the constraint circle,the green point in the above Figure. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 12 / 16
- 13. A Small Experiment 1 I am using Python scikit-learn for the purpose of the experiment and in this context, it must be mentioned that the book by Sebastian Raschka, Python Machine Learning, PACKT Publishing is a good book to understand how to use scikit-learn effectively. 2 The data set that is used for the experiment can be found from https://archive.ics.uci.edu/ml/datasets/Housing. 3 The data set comprises of 506 samples and 14 attributes. I have used 11 attributes as predictors (column number: 1,2,3,5,6,8,9,10,11,12,13 ). I have used column number 14 as the responses. 4 Since 506 >> 11, and we are trying Ridge regression for the setting where n is slightly larger than p, I have randomly selected 20 observations from the data set of which 14 has been used for training and 6 has been used for testing. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 13 / 16
- 14. A Small Experiment 1 0 1 2 3 4 5 6 7 8 values of lamda 1 0 1 2 3 4 5 6 7MSE Train Mean Squared Error Test Mean Squared Error Ananda Swarup Das A Note on Ridge Regression October 16, 2016 14 / 16
- 15. A Small Experiment 1 Notice that when λ = 0, the minimization function which is minimize( n i=1(yi − β0 − p j=1 βj xi,j )2 + λ p j=1 β2 j ) is equal to minimize( n i=1(yi − β0 − p j=1 βj xi,j )2), the case of least square estimation. Notice the differences between the MSEs of test data vs training data. A sharp/large difference denote significance variance of our model. Notice the difference between the MSE of the test and the train data at λ = 0. As the value of λ increase, the variance decreases up-to λ = 4. 2 In general, the choice of λ can be done through grid search using the inbuilt module linear−model.RidgeCV from scikit-learn. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 15 / 16
- 16. Citations G. James, D. Witten, T. Hastie, and R. Tibshirani. An Introduction to Statistical Learning: with Applications in R. Springer Texts in Statistics. Springer New York, 2014. Ananda Swarup Das A Note on Ridge Regression October 16, 2016 16 / 16