What is Logistic Regression:

Logistic regression is a statistical method used to analyze a data set

where there are one or more independent variables that determine
the outcome. Outcomes are measured as dichotomous variables
(only two possible outcomes). It is used to predict the probability of a
categorical dependent variable. In logistic regression, the dependent
variable is binary, meaning it only contains data coded as 1 (yes,
success, etc.) or 0 (no, failure, etc.).
Understanding Logistic Regression:

In logistic regression, we are essentially trying to find the

weights that transform the input data (independent
variables) into predictions (dependent variables).
Using a logistic function (also called a sigmoid
function), we ensure that these predictions are in the
range 0 to 1. This function maps each real number to a
different value between 0 and 1. In the case of logistic
regression, it converts the output of linear regression
into probabilities.

The logistic function has an S-shaped curve, defined by

the following formula:

Event probability = 1 / (1 + e^(-y))

where y is a linear combination of input features,
weighted by model coefficients.
"e" is the base of the natural logarithm, and "y" is the
equation of the straight line (y = mx + b in simple linear
regression).

It is used to predict a categorical dependent variable using a

specific set of independent variables.

Logistic regression predicts the output of a categorical

dependent variable.
Therefore, the result must be a categorical or discrete value.

It can be yes or no, 0 or 1, true or false, etc., but instead of

giving exact values of 0 and 1, it shows a probability value
between 0 and 1.

Apart from its usage, logistic regression is very similar to linear

regression.

Linear regression is used to solve regression problems, while

logistic regression is used to solve classification problems.
In logistic regression, instead of fitting a regression line, we fit
an "S" shaped logistic function to predict the two maximum
values (0 or 1).

The logistic function curve gives the probability of whether a

cell becomes cancerous, whether the mouse is obese (based
on body weight), etc.

Logistic regression is an important machine learning algorithm

because it can provide probabilities and classify new data using
both continuous and discrete data sets.

Logistic regression can be used to classify observations based

on different types of data, and the most effective variables for
classification can be easily determined.
Logistic Function (Sigmoid Function):

The sigmoid function is a mathematical function that

maps predicted values to probabilities.

It maps any real value to another value in the range 0

and 1.

The values for logistic regression must be between 0

and 1 and cannot exceed this limit, forming a curve like
an "S" shape.

The S-shaped curve is called a sigmoid function or

logistic function.

In logistic regression, we use the concept of threshold

to define the probability of 0 or 1.

For example, values above the threshold tend to 1 and

values below the threshold tend to 0
Differences b/w Linear and Logistic Regression:
Terminologies involved in Logistic Regression:

Independent variables: Input features or predictor variables that are

applied to the prediction of the dependent variable.

Dependent variable: The target variable we want to predict in the

logistic regression model.

Logistic function: A formula used to express how independent and

dependent variables relate to each other.

Logistic functions convert input variables into probability values

between 0 and 1, representing the probability that the dependent
variable is 1 or 0.

Odds: The ratio between what happens and what doesn't happen. It is
different from probability because probability is the ratio of what
happens to what could happen.

Log Odds: Log odds, also known as the logit function, is the natural
logarithm of odds. In logistic regression, the log odds of the dependent
variable are modeled as a linear combination of the independent
variables and the intercept.
How does Logistic Regression work?

Logistic regression models convert the continuous-valued

output of a linear regression function into a categorical-
valued output using the sigmoid function, which assigns each
real-valued set of independent variable inputs to a value
between 0 and 1. This function is called a logical function.

Let be an independent input feature

And the dependant variable is Y having only binary value i.e 0 or

1
Then apply the multi-linear function to the input variables
X

Here xi is the ith observation of xi Wi=(W1,W2,W3…….Wn)

Is the weight or coefficient, and b is the bias term also
know as intercept. Simply this can be represented as the
dot product of weight and bias.

Whatever we discussed above is the linear regression

Sigmoid Function

Now we use the sigmoid function where the input will be

z and we find the probability between 0 and 1.i.e
Predicted y.

As shown above, the figure sigmoid function converts

the continuous variable data into the probability i.e.
between 0 and 1.
Where the probability of being a class can be measured
as:

Logistic Regression Equation

Probability is the ratio of something happening to

something not happening.

It is different from probability because probability is the

ratio of what happens to what could happen. would be
weird

Applying natural log on odd will be

Then the final regression equation will be:

Likelihood function for logistic Regression

The predicted probabilities will p(X;b;w) = p(x) for y = 1

and for y = 0 predicted probabilities will 1-(X;b;w) = 1-p(x)

Taking natural logs on both sides

Gradient of the log-likelihood function

The assumptions of logistic regression are as follows:

Independent Observations: Each observation is

independent of other observations.
This means that there is no correlation between the
input variables.

Binary dependent variable: Assume that the dependent

variable must be binary or dichotomous, which means it
can only take on two values.
SoftMax function is used for more than two categories.
Linear relationship between the independent variable

and the log odds: The relationship between the

independent variable and the log odds of the dependent
variable should be linear.

No outliers: The dataset should not contain any outliers.

Large sample size: The sample size is large enough
Type of logistic Regression:

On the basis of the categories, Logistic Regression can

be classified intro three types:

1. Binomial: In binomial logistic regression, the

dependent variable can only be of two possible
types, such as B. 0 or 1, pass or fail, etc.

2. Polynomial: In multinomial logistic regression, there

can be three or more possible unordered types of
dependent variables, such as "cat", "dog", or "sheep".

3. Ordinal: In ordinal logistic regression, there can be

three or more possible ordinal types of the
dependent variable, such as B. "Low", "Medium" or
"High".

Code implementation for logistic Regression

Binomial Logistic regression:

The target variable can only have two possible types: "0"
or "1", in this case "win" vs. "lose", "pass" vs. "fail", "dead"
vs. "alive", etc. can represent. Us the Sigmoid function,
which has been discussed above.
Import necessary libraries according to the requirements
of the model.

This Python code shows how to implement a logistic

regression model for classification using a breast cancer
dataset.

Multinomial Logistic Regression:

A target variable can have three or more unordered

possible types (that is, the types have no quantitative
meaning), such as "Disease A" vs. "Disease B" vs.
"Disease C."

In this case, use the SoftMax function instead of the

sigmoid function. The SoftMax function of class K is:

Here, k represents the number of elements in the vector

z, and i, j iterates over all the elements in the vector.
Then the probability will be:

Ordinal Logistic Regression:

It deals target variables with ordered categories. For

example, a test score can be categorized as:
“Very poor”,” poor”, “good” or” very good”, Here, each
category can be given a score like 0,1,2 or,3

What is an activation function:

In an artificial neural network, a node's activation

function defines the output of that node or neuron for a
specific input or set of inputs.

That output is then used as input to the next node, and

so on until the desired solution to the original problem is
found.
It maps the resulting value to the desired range, e.g., B.
between 0 and 1 or -1 and 1 etc.

This depends on the choice of activation function. For

example, using a logistic activation function maps all
inputs to a real number range from 0 to 1.

Example of a binary classification problem:

In a binary classification problem, we have an input x,

say an image, and we have to classify it as having a
correct object or not.

If it is a correct object, we will assign it a 1, else 0.

So here, we have only two outputs – either the image

contains a valid object or it does not.

This is an example of a binary classification problem.

When we multiply each of them features with a weight
(w1, w2, …… wm) and sum them all together, node output
= activation (weighted sum of inputs).

Types of activation Function:

The Activation Functions are basically two types:

1. Linear Activation Function-

Equation: f(x)=x
Range: (-infinity to infinity)
 2. Non-linear Activation Function-

It makes it easy for the model to generalize with a variety

of data and to differentiate between the output.

By simulation, it is found that for larger networks ReLUs

is much faster.

It has been proven that ReLUs result in much faster

training for large networks.

Non-linear means that the output cannot be reproduced

from a linear combination of the inputs.

The main terminologies needed to understand for

nonlinear functions are:

Derivative: Change in y-axis w.r.t.

change in x-axis.

It is also known as slope.

Monotonic function: A function which is either entirely

non-increasing or non-decreasing.
In conclusion, logistic regression is a powerful

statistical technique that allows us to model

the probability of a binary event based on a

set of input variables.

It is widely used in machine learning and data

analysis, and its interpretability makes it a

valuable tool for understanding the

relationship between input variables and

output probabilities.