Gradient descent is a very important concept in AI and mathematics. This algorithm is used to minimize the errors during the learning phase of the ai model. The gradient is calculated is over the cost function.

Cost function plots the difference between the predicted value and actual expected value of a model. ie, it is a function representing error of the model.

What is Gradient

Gradient is the derivative of the fuction at different points in the curve. For example if there is a function represented by

f(x) = x ^2
y = f(x)  

Then by definition it’s derivative is 2x. ( Exponent rule)

Gradient Descent Process

Hence if we increase the x value the y value will increase and this means we are moving farther away from the minimum of the function. This is clear if you look the curve of the function shown below.

If we increase x on the positive side ( right side of the curve) then the gradient increases and moves away from the minimum point. But the reverse is true for the left side of the curve. Hence if you move in the reverse direction of the gradient, that direction is towards the minimum. This logic is used in AI to find weights such that the gradient descents to the minimal point.

if x = 2 , then the gradient is 4

if x = 3 , then gradient is 6

Clearly moving away from 0 or the bottom of the curve

if x = -2 then gradient is -4

if x = -3 then gradient is -6

Clearly moving away from bottom.

Hence if you want to find a value of x that moves towards the lowest point then the formula is

This formula is used to calculate the weight and bias values such that the cost reduces. w = w – learning rate * derivate of ( cost function wrt to w)

This process is iterated many times with different values of w and b to predict and match against training data until the error reduces to a reasonable value.