What is a Regression Model?
A regression model estimates the relationship between one dependent variable (target) and one
or more independent variables (features). It helps answer questions like:
How does the number of hours studied affect exam scores?
How does advertising budget influence sales?
Common Types of Regression Algorithms
Algorithm Description When to Use
Models a straight-line relationship
Simple relationships; fast and
1. Linear Regression between dependent and independent
interpretable.
variables.
2. Polynomial Fits a polynomial equation (e.g., When data shows a curved
Regression quadratic, cubic) to the data. trend.
Linear regression with L2
3. Ridge Regression When multicollinearity exists.
regularization to reduce overfitting.
Linear regression with L1
For feature selection and
4. Lasso Regression regularization; can shrink coefficients
sparse models.
to zero.
Combines L1 and L2 regularization When you want benefits of
5. Elastic Net
(Ridge + Lasso). both Ridge and Lasso.
Used for classification, not actual Binary classification problems
6. Logistic Regression
regression. (e.g., spam or not spam).
7. Support Vector Uses support vector machines to Complex relationships with
Regression (SVR) predict continuous values. high-dimensional data.
8. Decision Tree Predicts outcomes by learning decision Non-linear relationships, easy
Regression rules from data. to visualize.
9. Random Forest An ensemble of decision trees; reduces High accuracy, non-linear
Regression variance. problems, handles outliers.
10. Gradient Boosting Builds models sequentially to correct When accuracy is top priority
Regression previous errors. (but slower).
11. K-Nearest
Predicts based on average of k nearest No assumptions on data
Neighbors (KNN)
neighbors. distribution.
Regression
12. XGBoost / Advanced boosting algorithms with Kaggle-level solutions; very
LightGBM Regression high performance. fast and accurate.
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from [Link] import mean_squared_error
# Example data
X = [[5], [10], [15], [20]] # Hours studied
y = [50, 60, 70, 80] # Exam scores
# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)
# Model
model = LinearRegression()
[Link](X_train, y_train)
# Prediction
y_pred = [Link](X_test)
# Evaluation
print("MSE:", mean_squared_error(y_test, y_pred))
Polynomial Regression – Python Code
import numpy as np
import [Link] as plt
from sklearn.linear_model import LinearRegression
from [Link] import PolynomialFeatures
from [Link] import mean_squared_error, r2_score
# Example data: hours studied vs exam score (non-linear relationship)
X = [Link]([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]).reshape(-1, 1)
y = [Link]([20, 25, 30, 38, 48, 60, 75, 90, 105, 120])
# Create polynomial features (degree = 2)
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
# Fit model
model = LinearRegression()
[Link](X_poly, y)
# Predict
y_pred = [Link](X_poly)
[Link](X, y, color='blue', label='Actual')
[Link](X, y_pred, color='red', label='Polynomial Fit (Degree 2)')
[Link]('Polynomial Regression Example')
[Link]('Hours Studied')
[Link]('Exam Score')
[Link]()
[Link](True)
[Link]()
mse = mean_squared_error(y, y_pred)
r2 = r2_score(y, y_pred)
print("Mean Squared Error:", mse)
print("R² Score:", r2)
Ridge
import numpy as np
import [Link] as plt
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from [Link] import mean_squared_error, r2_score
# Split the dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# Ridge Regression with alpha = 1.0
ridge_model = Ridge(alpha=1.0)
ridge_model.fit(X_train, y_train)
# Predict
y_pred = ridge_model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print("Coefficients:", ridge_model.coef_)
print("Intercept:", ridge_model.intercept_)
print("Mean Squared Error:", mse)
print("R² Score:", r2)
import numpy as np
from sklearn.linear_model import Lasso
from sklearn.model_selection import train_test_split
from [Link] import mean_squared_error, r2_score
# Split the dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# Train Lasso model (alpha = regularization strength)
lasso_model = Lasso(alpha=0.1)
lasso_model.fit(X_train, y_train)
# Predict
y_pred = lasso_model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print("Coefficients:", lasso_model.coef_)
print("Intercept:", lasso_model.intercept_)
print("Mean Squared Error:", mse)
print("R² Score:", r2)
