In this lecture, we are going to understand the concept behind how neural networks actually learn.
So tell this lecture, we have code what that neural network is.
Now we are starting with how does a neural network work?
Here's a quick recap.
A neural network is a network of cells in our network.
We are going to use sigmoid neurons because these learn in a more controllable manner in a sigmoid neuron.
Two things happen.
We first multiply the input features with the words that are represented by W and then add a bias.
Don't be disvalue.
We name is Z or Z.
The second step is the application of sigmoid auditoria state function.
That is, DeSales calculates one upon one.
Plus it is two but minus the.
This value is the output of the cell.
And is always between zero and one.
This output becomes the new input for the next list.
This continues till the last layer, till we get the final output as per our network.
Now the problem for which we are solving is this.
We want to find out DVDs and biases of all the cells in the system so that the final output of this
network is as close to the actual value of the variable to be predicted.
For better understanding, let us just calculate the number of variables we need to calculate.
For this small network here, we have two neurons in the hidden layer and one neuron in the output layer.
And there are two input features, x1 and X2.
So for the first neuron we are trying to calculate W one into X1 plus W2 into X2 plus B one is equal
to Z.
This zie will be put into the activation function.
That is the sigmoid function.
And that will be the output of this neuron.
Let's say that the output of this neuron is represented by even for the second neuron.
We have two new weights, W three and W four.
We calculate this value w3 x1 plus W4 x2 plus B2.
This B2 is the bias of this neuron is equal to Z2.
We apply activation function on the Z2 to get it to these A1 and A2.
Are the inputs to this final output neuron for these two inputs.
We need two new weights, W5 and W6.
So the equation at this output neuron is W5 into even plus W6 into A2 plus B3 gives Z3.
When we apply activation function on this Z3, we get the predicted output from this output neuron.
So if you look at the variables that we need to estimate for weights, we have W1, W2, WTW for W5
and W6.
So we are estimating six wait for BIAS'S.
We have B1, B2 and B3 three Bias's.
So for this small network we need to establish the values of nine variables to make this neural network
ready for predictions.
Now how do we find out the values of WS and beats the technique followed for.
This is called gradient descent.
Gradient descent is just another optimization technique to find minimum of a function.
There are other optimization techniques also, such as ordinarily squared, which is used in linear
regression.
But for a large number of features and complex relationships, gradient descent shows much better computational
performance than any other technique.
This means that if you have a large number of input variables and a very complex relationship between
input on output, gradient descent will train the model in a much faster way as compared to other optimization.
So let's first discuss the process followed and gradient descent in a stepwise manner.
We start by assigning a random vade and bias values to all this is in our network.
Since all the words and biased values are available, that is, we have randomly assigned all weight
and biases.
Our model is ready to give out output.
The second step is we input one training example.
We use the X values of the training example and calculate the final output of the network using these
weights and by its values.
Third step is that we compare the predicted values vs. the actual values and we know the difference
between these two using some air function.
E will come back to this function later.
Remember that we have the actual Y value because this was a training observation.
So these actual values are being used to give feedback to our network that how bad it is performing.
The fourth step is we try to find out those weights and biases changing, which we can reduce this error.
Lastly, we have daily values of debates and bias.
And I repeat this process from step two.
This loop goes on the law.
Further reduction and edit function can be achieved.
And these steps, the first step is called initialization head.
We just give some random initial values to bits and bytes.
The second step is called forward propagation.
This is because in this step, we start with input values.
Process them in layer one.
Then we take the output of layer one and process it.
And layer two.
And so on.
Then we get one final predicted output.
We are simply moving forward in terms of the layers of the network.
So this is forward propagation.
The third and fourth step are quite backward propagation in these steps.
We already have the final error function and we look backwards in our network to find out which weeds
and biases have maximum impact on this error function.
Once we establish which weights and biases have maximum impact.
We update these Weirton biases slightly to reduce the error.
So this is the process we follow to implement gradient descent in neural networks.
But we are still not discussed.
The concept behind gradient descent
gradient descent is a mathematical technique which is used to find out B minimum of A function.
Let's see this example in the graph on the left on the x axis.
I have this variable on the Y axis.
I have a function applied on this variable.
This is the plot of dysfunction.
Now, if you want to find out the value of X at which the function has minimum value, there are two
ways to it.
One is if you know the exact relationship between X and the function, you can use calculus to find
the minimum of this function.
But as you know, in our machine learning problems, we do not have this exact relationship.
So we use a second technique, which is an attractive technique.
And this technique.
We start at a random point on this plot.
So if we have this value of X and affix now, instead of focusing on the whole graph, we focus only
on this small part of the graph and try to find out what happens if we slightly increase the value of
X or decrease the value of X.
In other words, we are trying to find out which way is the slope.
If the slope is negative, that is like this, we increase the value of X a little bit.
And then we will see that X will also decrease.
Similarly, if the slope is positive, we decrease the value of X, which will slightly decrease the
value of ethics.
We continue taking these small, small steps.
They'll release the final minimum point when we are at this point moving either side only increases
the value of the function.
So we stop that process here.
This iterative technique of finding instantaneous slope, also known as gradient and slightly moving
down that slope that is descent is called gradient descent.
If you want to picture this, this, you can think of yourself being on top of a hill.
You cannot see anything around you because it is dark and foggy.
No, you want to come down the hill as fast as possible.
What do you do?
Ideally, if you could see, you would spot the closest downhill point and run to it.
But since you cannot see and you know, the gradient descent technique, you take a step in each direction,
see which direction is more down, and then you move from your current position to that new position.
Then you again take out your right foot, Jack, which is the direction of steep slope and move.
Did you keep doing this?
And eventually you will come downhill.
This is the concept behind gradient descent.
And the next lecture we will must be two ideas.
The first is to process that neural networks used to implement gradient descent.
And the second idea was what gradient descent is.
Mathematically, we will merge these two and understand how gradient descent is helping us achieve the
minima and neural networks.