ML ASSIGNMENT – 1

Bayes Linear Regression Technique

Name: Karthik Nivedhan A

Roll number: 221225
ECE Third year

Sources: Bayesian linear regression (alpopkes.com)

Introduction To Bayesian Linear Regression | Simplilearn
Chapter 6 Introduction to Bayesian Regression | An Introduction to Bayesian
Thinking (statswithr.github.io)

1.Introduction
Bayes linear regression, a statistical modelling technique, provides a
probabilistic approach to modelling relationships between variables. Unlike
traditional linear regression, which treats parameters as fixed values, Bayes
linear regression views them as random variables, allowing for a more nuanced
understanding of uncertainty and enabling more robust inferences.

2. Motivation
The primary motivation behind Bayes linear regression is to address the
limitations of traditional linear regression. By incorporating prior information
about the model parameters, Bayes linear regression can:

 Improve prediction accuracy: The prior can help to regularize the

model, preventing overfitting and improving its generalization
performance.
 Provide uncertainty quantification: The posterior distribution of the
parameters allows us to assess the reliability of our predictions.
 Handle small sample sizes: The prior can provide additional
information when the sample size is limited.

In Bayesian linear regression, the mean of one parameter is characterized by a

weighted sum of other variables. This type of conditional modelling aims to
determine the prior distribution of the regressors as well as other variables
describing the allocation of the regressand) and eventually permits the out-of-
sample forecasting of the regress and conditional on observations of the
regression coefficients.
The normal linear equation, where the distribution of display style YY given by
display style XX is Gaussian, is the most basic and popular variant of this model.
The future can be determined analytically for this model, and a specific set of
prior probabilities for the parameters is known as conjugate priors. The
posteriors usually have more randomly selected priors.
When the dataset has too few or poorly dispersed data, Bayesian Regression
might be quite helpful. In contrast to conventional regression techniques, where
the output is only derived from a single number of each attribute, a Bayesian
Regression model's output is derived from a probability distribution.

3. Methodology
The goal of the Bayesian Regression Model is to identify the 'posterior'
distribution again for model parameters rather than the model parameters
themselves. The model parameters will be expected to follow a distribution in
addition to the output y.

The posterior expression is given below:

Posterior = (Likelihood * Prior)/Normalization
The expression parameters are explained below:
 Posterior: It is the likelihood that an event, such as H, will take place
given the occurrence of another event, such as E, i.e., P (H | E).
 Likelihood: It is a likelihood function in which a marginalization
parameter variable is used.
 Priority: This refers to the likelihood that event H happened before event
A, i.e., P(H) (H)
This is the same as Bayes' Theorem, which states the following -
P(A|B) = (P(B|A) P(A))/P(B)
Where, P(A|B) is the likelihood that event A will occur, provided that event B has
already occurred.

According to the aforementioned formula, we get a prior probability for the

model parameters that is proportional to the probability of the data divided by
the posterior distribution of the parameters, unlike Ordinary Least Square
(OLS), which is what we observed in the case of the OLS.

If 'y' is the expected value in a linear model, then

y(w,x) = w0+w1x1+...+wpxp
where, the vector "w" is made up of the elements w0, w1, ... The weight value is
expressed as 'x'.
w= (w1…wp)
As a result, the output "y" is now considered to be the Gaussian distribution
around Xw for Bayesian Regression to produce a completely probabilistic
model, as demonstrated below:

p (y|X, w. 𝛼) = N (y|Xw, 𝛼)
where the Gamma distribution prior hyper-parameter alpha is present. It is
handled as a probability calculated from the data. The Bayesian Ridge
Regression implementation is provided below.
The Bayesian Ridge Regression formula on which it is based is as follows:
p(y|λ) =N (w|0, λ^-1Ip)
where alpha is the Gamma distribution's shape parameter before the alpha
parameter and lambda is the distribution's shape parameter before the lambda
parameter.

5. Example problem
Predict the closing price of a stock based on its opening price.

Date Open price Closing

1 may 100 102
2 may 102 105
3 may 105 108
Solution:
Let the model be:
Closing Price = β₀ + β₁ * Open Price + ε

Where,
βn are the parameters
ε is the error value
open price is independent variable
closed price is the dependent variable

Assume the priors to be normally distributed, then only we can apply Bayesian
linear regression.
The likelihood for each data point is:

P (Closing Price | β₀, β₁, σ²) = N (Closing Price | β₀ + β₁ * Open Price, σ²)
Where sigma squared is Standard deviation.

Assuming σ² = 10, we can calculate the likelihood for each data point as
follows:

For the first data point:

P(102 | β₀, β₁, 10) = (1 / √(2π * 10)) * exp(-(102 - (β₀ + 100β₁))² / (2 * 10))
For the second data point:
P(105 | β₀, β₁, 10) = (1 / √(2π * 10)) * exp(-(105 - (β₀ + 102β₁))² / (2 * 10))
For the third data point:
P(108 | β₀, β₁, 10) = (1 / √(2π * 10)) * exp(-(108 - (β₀ + 105β₁))² / (2 * 10))
 Overall likelihood = P(102 | β₀, β₁, 10) * P(105 | β₀, β₁, 10) * P(108 | β₀,
β₁, 10)

Now we can be values of β₀ and β₁ obtained using MCMC methods which can be
used to get new samples from a probability distribution.
We get open price value of 112. So, we can write the closed price for each point
as,

Predicted Closing Price = β₀ + β₁ * 112

5. Python code.
import pandas as pd
import numpy as np
import pymc3 as pm

# Assuming data is in a pandas DataFrame named 'data'

X = data['Open Price'].values.reshape(-1, 1) # Reshape for
matrix multiplication
y = data['Closing Price']

with pm.Model() as model:

# Priors
intercept = pm.Normal('intercept', mu=0, sigma=1)
coef = pm.Normal('coef', mu=0, sigma=1)

# Likelihood
mu = intercept + coef * X
sigma = pm.HalfCauchy('sigma', beta=1)
likelihood = pm.Normal('likelihood', mu=mu, sigma=sigma,
observed=y)

# Sampling
trace = pm.sample(draws=1000, tune=1000)

# Extract posterior samples

intercept_samples = trace['intercept']
coef_samples = trace['coef']

new_data = np.array([110]).reshape(-1, 1)
predicted_prices = intercept_samples + coef_samples * new_data
6. Merits and Demerits
Merits
1. Handles Uncertainty Well:
- Bayesian linear regression gives you a range of possible values for the model
parameters, not just a single estimate. This helps you understand how confident
you can be about the predictions.
2. Built-in Regularization:
- By using prior information, Bayesian methods automatically control for
overfitting (where the model fits the training data too closely). This can be
especially helpful when you have a small dataset.
3. Flexible with Prior Knowledge:
- You can include your own expertise or previous knowledge about the
parameters in the form of priors. This can improve the model if you have useful
information before seeing the data.
4. Probabilistic Predictions:
- Instead of giving a single prediction, Bayesian linear regression provides a
range of possible outcomes, which helps in understanding and preparing for
different scenarios.
5. Natural Model Averaging:
- It naturally combines different possible models based on their probabilities,
which can often lead to better predictions than using just one model.

Demerits
1. Computationally Intensive:
- Bayesian methods can be slow and require a lot of computing power,
especially with large datasets or complex models. This is because they often use
advanced techniques like Markov Chain Monte Carlo (MCMC) for calculations.
2. Choosing Priors Can Be Hard:
- The results can depend heavily on the choice of prior distributions. Finding
the right priors can be tricky and may need expert knowledge.
3. Scalability Problems:
- Bayesian methods might not scale well with very large datasets. While there
are approximate methods to make it more feasible, they might not be as
accurate.
4. Complex to Implement:
- The process can be more complicated than traditional linear regression,
which might be a barrier if you're looking for something straightforward.
5. Interpreting Results Can Be Tough:
- Understanding the probabilistic outputs and communicating them effectively
can be more challenging compared to straightforward estimates from
traditional linear regression.

In summary, Bayesian linear regression is great for managing uncertainty and

incorporating prior knowledge, but it can be complex and demanding in terms
of computation and interpretation.

7. Conclusion
Bayes linear regression offers a robust and flexible approach to modeling
relationships between variables by incorporating Bayesian principles. Its ability
to manage uncertainty and integrate prior knowledge makes it a valuable tool in
various fields, from finance to environmental science. While it presents
challenges such as computational complexity and sensitivity to prior choices,
the benefits of comprehensive uncertainty quantification and adaptability to
new data make Bayes linear regression a powerful technique in modern
statistical and machine learning practices. As data science continues to
advance, Bayes linear regression will remain a key method for tackling complex
modeling challenges and enhancing decision-making processes.

THANK YOU, MAM.