Hello and welcome back to the course on Deep Learning.
In today's tutorial we're talking about gradient descent.
What we learned previously was that
in order for a neural network to learn, what needs to happen is backpropagation.
And that is when the error, the difference or
the sum of squared differences between y hat and y is back
propagated through the neural network, and the weights are adjusted accordingly.
So we saw that and today we're going to learn exactly how these.
Weights are adjusted. So let's have a look.
this is our, very simple version.
Of a neural network, a, perceptron or a single.
Layer feedforward neural network.
And what we can see here is.
this whole process in action where, we've got, some input value,
then we've got a weight, then a, activation function is applied.
We have, we get y hat and then we compare it to the actual value.
We calculate the cost function.
So how can we.
Minimize the cost function.
What's, can we do about it.
Well one approach to do it is a brute force approach where we just take.
all. Lots of different possible weights.
And look at them and see which one works best.
And what we. Would do is, for instance.
Would, try out for a, let's say, for example, a thousand weights
and we try them out, we'd get something like this, for the cost function.
And this is a chart,
of, on the y axis, we have cost function on the vertical axis.
On the horizontal axis you have y hat.
And because you can.
See the formula is y hat minus y a square, this is what the cost function
would look something like that.
And basically.
You'd find the best one is over here.
so very simple, very intuitive approach.
why. Not do. This brute force method?
Why not just try out a thousand different,
cost for a thousand different, parameters.
Or inputs for.
Weights and see which one works best.
You'll find the best one that way. Well, if you have.
Just one way to optimize, this might work.
But as you increase the.
Number of weights, increase the. Number of.
Synapses in your network, you have to face the curse of dimensionality.
And so what is. The curse of dimensionality?
the best way to describe this or explain it is to just look at a practical example.
So remember this example we had when we were talking about how.
Neural networks.
Actually work, where we were, building or.
Running a neural network.
For, a for profit evaluation.
So this is what it looked like when it was trained up already.
Well, when it's not trained before, it's trained before we know which
one of the weights.
The actual neural network looks like this, right.
Because we have. All these different.
Possible, synopses, and we still have to train up the weights.
And here we have a total of 25 weights.
So four times five at the start, plus five more.
from the hidden layer.
To the output layer, 20. Five weights total.
And let's see, how we could possibly brute force
25 weights says it's a very simple, neural network right here.
Very simple. Just one hidden layer.
And how could we, brute force our way through a.
Neural network of this size?
Well, let's do some.
Simple mathematical calculations.
We have 25 weights.
So that means if we have a thousand combinations
that we're going to test out for every weight, the total number of combinations.
Is a thousand to the power of 25, or 1000 or 10 to the power
of 25 different combinations.
now let's see how Sunway head to
tail lights the world's fastest supercomputer.
As of, June 2016.
what?
How would it approach this problem? Right. So.
Sunway Tahu light, it looks like
this is a whole huge, building.
Pretty much for this one supercomputer.
And, it got the Guinness World.
Record for being the fastest supercomputer.
Right now, it is the fastest supercomputer in the world.
And Sunway.
Tiger Light can operate at a speed of 93.
Petaflops.
flop stands for, floating operation per second.
So it can do 93 to the power.
Well, times ten to the power of 15 floating operations per second.
That's how quick it is.
in comparison, average computers right now,
they do like just over several gigaflops and so on.
So like kind of those ranges.
Weigh less. Than Thai Sunway tail light.
So Sunway tail light is in the forefront of technology.
And let's say hypothetically that it can do one.
one test, one combination
for our neural network in one, flop basic and one floating operation.
That is not possible.
That is not practical because you need multiple.
Floating operations to test out.
A single weight in your neural network.
But even let's let's give it a head start.
Let's say that it can do in a ideal world, it can do.
That in one, floating operation.
It can do one, test per one floating operation.
That means it will still require ten to the power of 25 divided by 93.
Times ten to about.
15 seconds to come, to run
all of those tests to brute force through that network.
So that means one or approximate ten to the power of 58.
Seconds, and that is the same as ten to the.
Power of 50 years.
That is a huge number that is longer,
than the universe has existed.
And that is definitely not going to simply this number is so huge.
It's just definitely not going to work for us, at all in our, optimization.
So there we go.
This is a no no.
Even on the world's fastest supercomputer, Sunway Tail Light.
So we have to come up with a different approach.
How are we going to. Find the optimal weight.
By the way.
This our neural network was very simple.
What about if the neural networks looks like something like this, right?
Or even greater than that then?
yeah, it's just not going to happen at all.
Ever.
So the method we're going to be looking at is called gradient descent.
And, you may. Have heard of it already.
if not, we will.
Find out what it is right.
Now. So, there's our.
cost function, and now we're
going to see how we can faster come up with a faster way to.
Find the best option.
So let's say we start somewhere. You got to start somewhere.
So we start over there.
And from that point, in the top left.
What are we going to do? Is we're going to look.
At the angle, of our cost function at that point.
So we're just going to basically that's
why it's called gradient because you have to differentiate.
We're not going to look at the mathematical equations. I, we will.
Provide some tips on additional reading at the end of the.
Next lecture.
but. Basically you just need to differentiate.
Find out what the slope is in that specific point,
and find out if the slope is positive or negative.
If the, if the slope is negative, like in this case, means that
you're going downhill.
So, to the right is downhill, to the left is uphill.
And from there it means you need to go, right.
Basically you need to go downhill.
And that's what. We're going to do.
Vroom takes a step. Right to the ball.
Rolls down.
again same thing. You calculate the slope.
This time the slope is positive, meaning right is uphill.
Left is downhill, and you need to go left.
And you roll the ball down.
And again you calculate the slope and your old ball.
Right.
There you go.
So that's how you find in in simple terms.
That's how you find the best.
weights, the best, situation that minimizes your cost function.
Of course, it's not going to be.
Like a ball rolling is going to be a very zig zag type.
Of approach, but it's easier to.
Remember and kind of, it's is more fun to.
Look at it as a ball rolling. But in reality, yes, you.
Just, it is going to be like a step by step approach.
So it's going to be a zigzag type of method.
Yeah.
And also there's, there's lots of other elements to it.
So there's.
things like for instance, why like why does it go down.
Why does it not like go away over the line.
So it could have jumped out of this, going upwards instead of downwards
and things like that.
So there are parameters. That you can. Tweak.
And again we will mention where you can find out more on that.
And plus we'll have this in the practical application.
But in the simplest intuitive approach this is what is happening.
We are getting to the bottom. by.
Just understanding which way we need to go instead of brute forcing through.
Thousands and thousands and millions and billions.
And quadrillions of combinations, we can just simply.
Every time have a look.
At where is where, which way is it sloping. So.
Right, like you.
Imagine you're standing on a. Hill, which way does it feel.
That it's going downwards? And whichever way it is going downwards.
You just keep.
Walking that way you like take 50 steps that way and then you, you assess again.
Okay. Which way is it going? Downwards.
This way.
Okay.
Now take 50 steps or let's take 40 steps that way.
So it gets less and less and less as you get closer.
So here's.
An example. Of gradient.
Descent applied in a two dimensional space.
So that was a one dimensional example.
here we have a two dimensional space for the gradient descent.
As you can see it's getting. Closer to the minimum.
And it's also called gradient descent because you're.
Descending into the minimum of the cost function.
And finally here's the gradient descent applied in three dimensions.
This is what it looks like if you projected onto two dimensions,
you can see it zigzagging its way into the minimum.
So there we go. That was gradient descent.
In next tutorial we'll talk about
stochastic gradient descent is we'll be continuation of this tutorial.
And. I look forward to seeing you there.
Until next time, enjoy deep learning.