In this lesson we will talk about some fundamental concepts of machine learning namely over fitting
and regularization.
The reason we're talking about it now is because we've encountered this problem in our last lesson there.
We saw that after one hundred and fifty epochs our validation era started to creep up as we continue
to train our model.
So why did this happen and why is it a problem.
Well all of this has to do with a phenomenon called over fitting over fitting happens when your model
learns the training data too well meaning the model starts to learn all the little quirks that are present
in your training data and it becomes better and better at fitting itself to the training data.
It not only learns the relationships that are present in your training data but it also learns all the
noise that's there too as a consequence the model becomes unable to generalize well.
In other words the model becomes unable to predict future observations outside of the training data
said reliably so a consequence of over fitting is poor performance on the validation data set on the
test data set and any future observations that you collect.
The easiest way to understand this problem of over fitting is visually.
So I'm going to show you a series of chance to build out this intuition.
Suppose we've got some sample data here we've got some x's and some Y's and our goal is to plot some
regression line.
Now if we try to draw a line that connected as many of these points as possible then you're going to
end up with a really squiggly line.
Maybe the line would end up looking like this.
And yes this line minimizes the distance between itself and all the points but it's really looking quite
complex.
That's the first thing.
And the second thing is that it's not looking terribly realistic.
And this lack of realism actually becomes clear once more data points are collected and added to this
plot.
If we do that this line starts to fit the new data rather poorly meaning this model is a clear example
of over fitting.
But conversely what this also means is that collecting more data is actually a good way to combat or
fitting in the first place.
The thing is over fitting is actually a problem that is present across many machine learning techniques
not just neural networks but neural networks are actually particularly prone to over fitting.
Why.
Well neural networks tend to have many many parameters.
Our model alone had around 400000.
Compare that with a very complex regression with say 100 variables or 20.
In effect the more complex the model the more prone it is to over fitting.
Neural networks are actually able to learn the training data especially well so if this is an example
of a model over fitting the data what does the opposite look like.
What would under fitting looked like in the regression context.
Well it would look something like this.
In this case the model was not really learning the relationship present in the training data well enough.
This is a good analogy to what we had when we only train our neural network for one epoch.
What you would want ideally is actually a balance you would want your model to learn the relationships
that are present in the data but you don't want the model to over fit.
Now just to show you another example Suppose you had a classification problem instead of a regression
problem what would overfishing look like in this case with a classification problem.
It's actually the decision boundary that starts to look rather strange and very esoteric when over fitting
is present the Green Line is a model that's over fitting to the data.
It's trying very hard to divide up those two groups and as a consequence the model is overreacting to
the noise that's present in the training data set a much better decision boundary would be something
like the black line the black line is likely to generalize much better and gives better results on the
validation and the test data sets.
So now that we know what over fitting and under fitting is the question is how do we diagnose the problems
and how do we prevent them.
Well we've already talked about one way of diagnosing over fitting is looking at the performance on
the validation data set is the loss on the validation data set still decreasing or is it no longer decreasing
or is it even increasing.
These are the kind of signs to watch out for.
Also what's the performance on the validation data set.
How is the accuracy changing as we're training the model.
And is there a big difference between the accuracy on the training data set versus the accuracy on the
validation data set.
These are the kind of signs that you need to keep an eye out for.
Now suppose that we've diagnosed this problem for a fitting.
How do we fix it.
Well the techniques for preventing over fitting are called regularization and regularization actually
encompasses a variety of techniques all of them tend to impose some sort of constraint or some sort
of penalty on complexity.
The theoretical justification for penalizing complexity is that a simpler solution is preferable to
a complex one.
And this is actually very much in line with the Zen of Python remember.
Simple is better than complex and complex is better than complicated.
Looking at the validation loss on tensor board one regularization technique already seems obvious.
If the problem is is that we're training our model over too many epochs then all we need to do is to
stop overtraining it right.
Case closed.
Why not stop if the validation accuracy is no longer going up and our validation error is no longer
decreasing.
And the answer is that yes this is actually a completely valid technique.
This technique even has a name.
It's called early stopping and if you wanted to implemented carries actually gives us the option to
do so in the callback section.
You can actually search for early stopping and there you can find that you can stop the training when
a monitored quantity like see the validation loss has stopped improving.
So all we would need to do is at early stopping as another callback to our list of callbacks right here
but in addition to early stopping there is another very very powerful technique that I'd like to show
you and this regularization technique is called dropout the dropout technique was actually published
by a team of researchers in 2014.
They found that if you randomly ignore some of the neurons during the training then you can reduce over
fitting.
In other words during each training step some random neuron either in the input layer or in the hidden
layers is not considered.
Say this is our neural network.
If we apply the drop out technique to the input layer then we can specify a chance for every single
one of these neurons to not be considered during the training.
So if there's a 20 percent chance that each of these neurons can drop out then during the first training
step maybe the first neuron and all of its connections will be ignored.
If this neuron and all of its connections drop out then the network shrinks it becomes a less complex
network because there are fewer connections during the next training step.
A different neuron might drop out the neuron that dropped out during the first training step will come
back and a different one might drop out with a 20 percent probability.
Now this technique might seem rather strange right.
Why does this work.
Well it means that the connected neurons all the neurons that are downstream in that first hidden layer
now don't want to rely too heavily on any single input.
If a random neuron drops out during the training step every time all of the connected neurons will try
to hedge themselves in order not to weigh any particular input too heavily and this will help prevent
over fitting.
But let me ask you this with a network that implements the drop out technique learn differently than
a network that has no dropout.
What about speed.
Will it learn just as quickly or will it take more time to answer these questions let's dive back into
our code.
Let's create a second model in Jupiter and find out because we can use tensor board to compare our second
model with our first model and see how they learn differently.
Let's go to this section here in our Jupiter notebook where we're defining our model.
I'm going to add another cell below and here I'm going to create our model underscored two.
And this is also going to be a sequential model.
Now previously we've added all the layers inside of these square brackets of the model.
But this isn't the only way we can do things.
In fact when we go to the getting started guide we see that Chris also has this add method to add the
different layers to the model.
So after you've created the model you can call dot add and then supply the layer between the parentheses
that you want to add to the model.
So this isn't an alternative way to write your code in Caris and let's try this out since we might often
see this on blog posts or on stack overflow.
The first thing I'm actually going to do is recreate all of these layers in Model Number Two.
So I'll say model underscore to dot and and I'll add my first dense layer.
This is going to be 128 units with the activation being redo.
Now I'm going to give this layer a name.
I'm going to call it m to underscore hidden one.
Now when I had the other layers very quickly.
So just copy this line paste paste paste and I want to change the inputs here to 64 16 and 10.
The last activation should be soft Max and the layer names of course have to be updated as well.
So this is my output layer output.
This is hidden 3.
This is hidden too.
And there we go.
I've added all my layers as before.
So this code and this code is actually equivalent.
The only thing that's left to do is to add my drop out.
Now here's how I'm going to do it.
First I'm going to check the caris documentation.
So under layers I can see dents which I've used before.
And I can scroll down and there I will find dropout my dropout layer will require some probability between
zero and one as the dropout probability and the convention is that numbers between 20 percent and 50
percent work rather well we're going to try out 20 percent if you'd like to have the same random neurons
drop out as me.
Then we're going to set the same seed.
So let's implement this back in our notebook.
The first thing we're gonna do is we're actually going to input dropout backup on our import statements
under Kerry's start layers.
We're going to import dense activation and crop out.
So shift enter on the cell.
Now we can scroll down to where we created our second model and write model underscore to dot ad and
then drop out.
Now our dropout probability which we said was gonna be zero point two and our seed is gonna be equal
to say forty two but there is one more thing we have to specify because this will kind of act as the
first layer in our model.
We have to specify the input shape for our dense layer the parameter name was input underscore DRM four
dimensions but for our dropout the name is actually can I read input on a score shape.
And this will be equal to a tuple so parentheses and then total on a score inputs and then a comma afterwards.
And that is how we're going to specify the total number of inputs for our dropout.
Having added this code we're going to apply our dropout to our input later.
The only thing left to do is to compile our model.
Right.
And we can actually do this by copying this code.
We're going to use the same optimizer can use the same loss when we use the same metrics so I kind of
pasted in here and I going to say model to dot compile fantastic let's head shift into here.
And now we can come down here and we can copy the cell paste it and we can change model one to model
two can change the name for tensor board from model 1 to model 2 and then we can finally fit our model
and see how it will behave compared to our model without the dropout.
So let's try it out.
I'm very curious already.
That's head shift enter on the cell and switch over to tensor board and let's play the waiting game.
If I scroll down here in the bottom left corner then I can see model to add 16 0 8.
So here we go.
We can see that the loss is coming down.
This is still an epoch.
14 and as the model continues to run the validation loss continues to decrease and then it has a bit
of a wobble here and then it's slowly ticking up again by around epoch.
One hundred and thirty.
So what we're seeing here is that the drop out does indeed reduce over fitting but it will not eliminate
it completely.
We do have to use some combination of early stopping stopping to train our model at a certain point
and the drop out technique to get a better result.
Now let's take a look at our validation accuracy.
One thing that's quite interesting is that we actually haven't lost out on much accuracy.
Model number two with the dropout and model number one without the dropout are around 30 percent accurate
by the end of the training.
One thing that is very noticeable is that the dropout model tends to learn a bit more slowly.
It tends to learn the training data set slower than the model without the dropout and this is exactly
what we intended.
By the way if there are too many crafts on him then you can toggle them down here on the left hand side.
Suppose I just want to see model number two then I can hit this radio button here and if I want to do
a little horse race with Model 1 and model 2 then I can take the individual tick boxes here to get them
side by side like so on the chart remember high said that we could apply the dropout to both the input
layer as well as to a hidden layer.
Well I've now got a challenge for you.
What I'd like you to do is create a third model.
Call it model on a score three that has to dropout layers.
The first dropout layer should be the same as with Model Number Two.
So it should be applied on the inputs but the second dropout layer should be added after the first hidden
layer.
And have a dropout rate of 25 percent.
I'll give you a few seconds to pause the video and give this a go.
Ready.
Here's the solution.
So I'm just going to take this cell copy it and paste it move it down and I want to change all the references
from Hoddle to to model 3 and after I've done that allowed the second dropout layer here model on a
school three dot and dropout zero point two five.
Now you're free to use the same seed as before but in this case you don't actually have to add anything
afterwards because Chris is smart enough to infer the number of inputs on this dropout layer because
it is not the first layer and that's it.
It's all done.
Now we're going to get to the very interesting part we're actually going to train our model on our entire
training dataset and no longer this extra small training dataset that we've had before.
We're going to see what a difference it makes when we're moving from a training data set of 1000 samples
to 40000 samples so check it out.
I'm going to copy this cell pasted below.
I'm going to take this one here and comment it out.
I'll keep it around for reference but I'm going to take model one and we're going to reduce the number
of epochs we're gonna train this model on.
We're gonna go down from one hundred and fifty to one hundred and then we'll change our training dataset
to X on this train and Y underscore train and we're going to call this model model 1 x l That way we
know we're dealing with the large dataset I'll copy the cell I'll paste it again and here we're gonna
change this to model 2 and Model 2 right here and copy that paste it again and where to change this
to Model 3 and when change in this model name to Model 3 as well.
This last cell here I'm going to comment out and I want to move this up slightly and now I should be
left with these three cells I'm very very quickly going to come up here and recompile Model 1 and 2
before I run my code so that I know that they're all gonna be starting from scratch and the last thing
I want to do is I'm going to go into tensor board and here I'm actually only going to keep these two
lines around I basically want to delete all these other runs that we've had the way I want to do this
is I wanted to leave my files directly so the two models that I want to keep around are model one at
eleven fifty six and Model 2 at 16 0 8 and the other ones I'm no longer that interested in these are
all legacy runs and then I'm going to come back into Jupiter and I'm going to run this cell run the
cell and run this cell now my computer is really working hard it's kind of crunching a lot of numbers
and we can go into tenths aboard and we can refresh the entire thing so that it will redraw charts and
we can watch but I think it's actually much more advisable to get a coffee at this point because this
could really take a while one of the things though that's gonna be obvious almost immediately is the
massive improvement in the validation accuracy that you're going to be getting from your model as soon
as you add more data when we only had 1000 training examples we were doing very very poorly on the validation
data set and the number of epochs that we're using to train our model wasn't able to even make that
much of a difference but look at what happens with version one of our model when we went from 1000 examples
to 40000 examples this is really why people say that data is the new oil the more data the better.
All right now all my three versions of my model have finished training and you can see that the first
one took four minutes the next one took about six minutes and the last one took me about eight minutes.
Model Number Two had dropped out on the input layer and model number three had dropped on both the input
layer and the first hidden layer.
Let's compare how these models did in 10 support the first thing that we're looking at is the accuracy
on the training data set.
And here what we see is actually quite interesting.
So the red and orange lines are the models on the small training data set and the light blue pink and
dark red lines are on the larger training data set.
What we see is that model number three with the most amount of drop out learned the training data said
the most slowly.
This is especially obvious if you look at what was happening from epoch number 50 onwards that pink
line is far below the other two I think model number two got off to a very strange start.
Initially it seemed not to learn very much at all.
And then the accuracy improved dramatically within a few epochs.
Now I suspect this might be due to the weight initialization at the start.
There is some randomness to the starting point for these algorithms with the images.
So if we were to fit this model again we might get a slightly different line but nonetheless we see
that models with more drop out learn more slowly the next jaunt intensive board shows us the training
loss.
So here we see that for every single run that we've done all of the models reduce the training loss
the more they train.
This is expected and also not that interesting.
So let's go down to the validation accuracy.
This is much more interesting because here we see the dramatic difference between the models that were
trained on a small data set of a thousand samples and the models that were trained on the larger data
set of 40000 samples.
Now mind you this is the validation accuracy on the 10000 samples in the validation data set.
And what we see is that model 1 2 and 3 all end up between 47 and 49 percent accuracy on the validation
data set.
But what we also see is that the accuracy didn't really change much towards the end.
And this can indicate overtraining and over fitting.
Let's scroll down to the validation loss and see if this is true and what we see here is that model
2 and 3 which have the dropout feature have a validation loss which continues to decrease although very
very slowly almost right up to the end.
Model number one on the other hand has a validation loss that's pretty much static after epoch number
50.
And this to me suggests that we should probably stop training model number one a little earlier so where
does this leave us.
Well there's a couple of things that we found out.
One is that the amount of data makes a massive difference increasing the amount of data in the training
dataset can help reduce overfishing dramatically and it will also boost the accuracy on data that the
model hasn't seen before.
The second thing that we've seen is that early stopping is a useful technique for preventing over fitting
training our models are more and more epochs is not necessarily a good thing.
The third thing that we've seen is that drop out is a useful technique for reducing over fitting models
that do implement the drop out technique learn a little bit more slowly but their validation errors
creep up much later than those without drop out and the fourth thing that we saw is a little bit more
on the magnitude of these impacts increasing the amount of data 40 fold had a much bigger impact than
adding one or two drop out layers to the structure of our model judging by the validation accuracy.
At the moment our models are getting approximately 50 percent of the classifications correct.
Now if this was a spam filter this would be terrible because in that case an email is other spam or
not spam.
But in this case we're actually differentiating between 10 different things and getting 50 percent of
those classifications correct doesn't actually seem to be such a bad thing especially if this is our
first go in the next lessons.
We're going to be evaluating these models a little bit more closely.
We're going to take a hard look at the predictions these models make on individual images and we're
going to evaluate how our favorite model fares on the test data center for all of that and more.
I'll see you in the next lesson.
Take care.