MAT6007 Deep Learning
Gradient Descent
Prakash P VIT , Chennai
Road Map
Understanding the Mathematics
behind Gradient Descent
Gradient Descent
Prakash VIT Chennai
Understanding the Mathematics behind Gradient
Descent
Agile is a pretty well-known term in the software development
process.
The basic idea behind it is simple:
build something quickly, ➡️get it out there, ➡️get some feedback ➡️make
changes depending upon the feedback ➡️repeat the process.
The goal is to get the product near the user and guide you
with feedback to obtain the best possible product with the least
error.
Also, the steps taken for improvement need to be small and should
constantly involve the user
The idea of — start with a solution as soon as possible, measure and iterate as frequently as possible, is
Gradient descent under the hood.
Prakash VIT Chennai
Understanding the Mathematics behind Gradient
Descent
Objective
Gradient descent algorithm is an iterative process that takes us to the minimum of a
function.
The formula below sums up the entire Gradient Descent algorithm in a single line
Prakash VIT Chennai
Understanding the Mathematics behind Gradient
Descent
A Machine Learning Model
arbitrary line in space that passes through
Consider a bunch of data points in a 2 D some of these data points
space. Assume that the data is related to
the height and weight of a group of
students.
We are trying to predict some
relationship between these quantities to
predict the weight of some new
students afterward.
This is essentially a simple example of a
supervised Machine Learning technique
Prakash VIT Chennai
Understanding the Mathematics behind Gradient
Descent
Predictions
Given a known set of inputs and their corresponding outputs, A machine learning model
tries to make some predictions for a new set of inputs.
This relates to the idea of a Cost
The Error would be the difference between the function or Loss function.
two predictions.
Prakash VIT Chennai
Understanding the Mathematics behind Gradient
Descent
Cost Function
A Cost Function/Loss Function evaluates the performance of our Machine Learning
Algorithm
The Loss function computes the error for a single training example, while the Cost
function is the average of the loss functions for all the training examples
Let’s say there are a total of ’N’ points in the dataset, and for all those ’N’ data
points, we want to minimize the error.
So the Cost function would be the total squared error
The goal of any Learning Algorithm is to
minimize the Cost Function.
Prakash VIT Chennai
Understanding the Mathematics behind Gradient
Descent
How do we minimize any function?
Cost function is of the form Y = X²
To minimize the function above, need to find
that value of X that produces the lowest value
of Y which is the red dot in the above figure
It is pretty easy to locate the minima here since it is a 2D graph, but this may not
always be the case, especially in higher dimensions
Devise an algorithm to locate the minima, and that algorithm is called Gradient Descent
Prakash VIT Chennai
Understanding the Mathematics behind Gradient
Descent
Consider that you are walking along with the graph below, and you are currently at the
‘green’ dot. You aim to reach the minimum, i.e., the ‘red’ dot, but from your position, you
are unable to view it.
Possible actions would be:
You might go upward or downward
If you decide which way to go, you might take a bigger step or a little
step to reach your destination
Essentially, there are two things that you should know to
reach the minima, i.e. which way to go and how big a step
to take.
Understanding the Mathematics behind Gradient
Descent
The Minimum Value
Tangent at the green point, know that if we are
moving upwards, we are moving away from the
minima and vice versa.
Also, the tangent gives us a sense of the
steepness of the slope
The slope at the blue point is less steep
than that at the green point, which means
it will take much smaller steps to reach the
minimum from the blue point than from
the green point
Understanding the Mathematics behind Gradient
Descent
Mathematical Interpretation of Cost Function
• Let us now put all these learnings into a mathematical formula.
• In the equation, y = mX+b ‘m’ and ‘b’ are its parameters.
• During the training process, there will be a small change in their values.
• Let that small change be denoted by δ.
• The value of parameters will be updated as m=m-δm and b=b-δb, respectively.
• Aim here is to find those values of m and b in y = mx+b
• For which the error is minimum, i.e., values that minimize the cost function.
The idea is that by being able to compute the derivative/slope of the function, find the minimum of a function
Understanding the Mathematics behind Gradient
The Learning rate Descent
This size of steps taken to reach the minimum or bottom is called Learning Rate.
Derivatives
Use derivates to decide whether to increase or decrease the weights to increase
or decrease any objective function
Two concepts from calculus
Chain Rule
Power Rule
Understanding the Mathematics behind Gradient
Calculating Gradient Descent Descent
apply these rules of calculus in our original equation and find the derivative of the Cost
Function w.r.t to both ‘m’ and ‘b’.
Calculate the gradient of Error w.r.t to both m and b
m¹,b¹ = next position
parameters;
m⁰,b⁰ = current position
parameters
Example
Find the local minima of the function y=(x+5)² starting from the point x=3
Step 1 : Initialize x =3. Then, find the gradient
of the function, dy/dx = 2*(x+5). learning rate → 0.01
https://
gist.github.com/rohanjoseph93/
ecbbb9fb1715d5c248bcad0a7d
3bffd2#file-gradient_descent-ip
ynb
Prakash VIT Chennai
References
https://www.youtube.com/watch?v=jc2IthslyzM
Reducing Loss: Gradient Descent | Machine Learning Crash Course (google.com)
Understanding the Mathematics behind Gradient Descent. | by Parul
Pandey | Towards Data Science
https://
towardsdatascience.com/implement-gradient-descent-in-python-9b93ed7108d1
Prakash VIT Chennai
Thanks
Prakash VIT Chennai
