In this lecture we are going to understand the concept behind how neural networks actually learn.
Still this lecture we have called what the neural network is now we are starting with.
How does a neural network work.
Here is a quick recap.
A neural network is a network of cells in our network.
We are going to use sigmoid neurons because these learn in a more controllable manner in a sigmoid neuron.
Two things happen.
We first multiply the input features we deviate that are represented by W and then add a bias to be
this value.
We name as Z or Z.
The second step is the application of sigmoid or the logistic function that is the cell calculate one
upon one plus it is the part minus the.
This value is the output of the cell and is always between 0 and 1.
This output becomes the new input for the next layer.
This continues till the last layer till we get the final output.
As per our network now the problem for which we are solving is this we want to find out the ways and
biases of all the cells in the system so that the final output of this network is as close to the actual
value of the variable to be predicted
for better understanding.
Let us just calculate the number of variables we need to calculate for this small network head we have
two neurons in the head and left and one neuron in the output layer and there are two input features
x1 and x2.
So for the first neuron we are trying to calculate W1 into X1 plus W2 into x2 plus B1 is equal to Z.
This V will be put into the activation function that is the sigmoid function and that will be the output
of this neuron.
US say that the output of this neuron is represented by A1 for the second neuron.
We have two new Newgate W3 and w4.
We calculate this value W3 X1 plus W for x2 plus B2 this B2 is the bias of this neuron is equal to z2.
We apply activation function on the z2 to get it to these A1 and A2 are the inputs to this final output
neuron for these two inputs.
We need two new rate W5 and W six.
So the equation at this output neuron is W5 into even plus W 6 into A2 plus b 3 gives z3.
When we apply activation function on this Z3 we get the predicted output from this output neuron.
So if we look at the variables that we need to estimate for weights we have W1 W2 W3 w for w 5 and W
6.
So we are estimating 6 wait for biases we have B1 B2 and B3 3 biases.
So for this small network we need to establish the values of nine variables to make this neural network
ready for predictions.
Now how do we find out the values of WS and beats the technique followed for.
This is called gradient descent gradient descent is just another optimization technique to find minimum
of a function.
There are other optimization techniques also such as ordinarily squared which is used in linear regression
but for a large number of features and complex relationships gradient descent shows much better computational
performance than any other technique.
This means that if you have a large number of input variables and a very complex relationship between
input and output gradient descent will train the model in a much faster way as compared to other optimization
so let's first discuss the process followed in gradient descent in a stepwise manner.
We start by assigning a random weight and bias values to all this is in our network.
Since all the weights and biased values are available that is we have randomly assigned all the weight
and biases.
Our model is ready to give out output.
The second step is the input one training example.
We use the x values of the training example and calculate the final output of the network using these
weights and biased values.
Third step is that we compare the predicted values vs. the actual values and we know the difference
between these two using some error function.
E will come back to this error function later.
Remember that we have the actual y value because this was a training observation so these actual values
are being used to give feedback to our network that how bad it is performing
the fourth step is we try to find all those weights and biases changing which we can reduce this error.
Lastly we update the values of the weights and bias and repeat this process from step two.
This loop goes on the low for the reduction and edit function can be achieved and these steps.
The first step is called initialization head.
We just give some random initial values to base and bias.
The second step is called forward propagation.
This is because in this step we start with input values process them in Layer 1.
Then we take the output of Layer 1 and process it and layer 2 and so on.
Then we get one final predicted output.
We are simply moving forward in terms of the layers of the network.
So this is forward propagation.
The third and fourth step are quite backward propagation in these steps.
We already have the final edit function and we look backwards in our network to find out which weeds
and biases have maximum impact on this error function.
Once we establish which weights and biases have maximum impact.
We update these weights and biases slightly to reduce the edit.
So this is the process we follow to implement gradient descent and neural networks but we are still
not discussed the concept behind gradient descent
gradient descent is a mathematical technique which is used to find out the minimum of a function.
Let's see this example in the graph on the left on the x axis.
I have this variable and on the y axis I have a function applied on this variable.
This is the plot of this function.
Now if you want to find out the value of X at which the function has minimum value there are two ways
to it.
One is if you know the exact relationship between X and the function you can use calculus to find the
minimum of this function.
But as you know in r machine learning problems we do not have this exact relationship.
So we use a second technique which is an attractive technique in this technique.
We start at a random point on this block saying we have this value of x and ethics.
Now instead of focusing on the whole graph we focus only on this small part of the graph and try to
find out what happens if we slightly increase the value of X or decrease the value of x.
In other words we are trying to find out which way is this slope.
If this globe is negative that is like this we increase the value of X a little bit and then we will
see that if X will also decrease.
Similarly if the slope is positive we decrease the value of x which will slightly decrease the value
of Epix
we continue taking these small small steps till we reach the final minimum point when we are at this
point moving either side only increases the value of the function so we stop our process here.
This outdated technique of finding instantaneous slope also known as gradient and slightly moving down
that slope that is descent is called gradient descent.
If you want to picture this you can think of yourself being on top of a hill.
You cannot see anything around you because it is dark and foggy.
Now you want to come down the hill as fast as possible.
What do you do.
Ideally if you could see you would spot the closest down hill point and run do it.
But since you cannot see and you know the gradient descent technique you take a step in each direction
see which direction is more down and then you move from your current position to that new position.
Then you again Take out your right foot check which is the direction of steep slope and move.
Did you keep doing this and eventually you will come downhill.
This is the concept behind gradient descent.
In the next lecture we will merge the two ideas.
The first is the process that neural networks used to implement gradient descent.
And the second idea was what gradient descent is.
Mathematically we will merge these two and understand how gradient descent is helping us achieve the
minima and neural networks.