All right.
Welcome back to talking about k fold cross-validation, a
very important tool in your toolkit for assessing,
how well your model is working with the data that you have.
Okay.
So here is what we normally do.
We have a data set, and we usually split it into a training
set and a test set.
And from here we're going to talk about k fold cross-validation.
But first I wanted to make a quick note that,
so there is there are two schools of thought.
Well, basically one school of thought is that when you're doing KS
fold cross-validation, you don't need the test set.
It is enough to do k fold cross-validation in the second, school of thought,
you still do the test set and you do k fold cross-validation on the training set.
and then you still use the test set later on.
So those are two different approaches.
We are going to talk more about that at the end of this tutorial.
In throughout this tutorial we're going to use a second school
of thought because it's more, it's more general.
And then we'll be able
to, like simplify it to make it, appropriate for the first goal
thought at the end of this tutorial when we discuss it.
So for now, let's, stick to breaking
the data set into a training set and a test set.
Now, once you've broken down what you do next for k fold cross-validation.
So normally what you would do is
you would just train your model here and then test your model here.
Right.
And from that you would get like a result. Yes.
The model hasn't seen this data.
So you would be able to tell how well it performs on this test set.
But what if you just get lucky on this test set?
What if it just so happens that it does well in test set,
but then on future data, it's not going to do well at all.
So that's what k fold cross-validation is for.
It's here to combat that scenario where you just got lucky on the test set
to ensure with more certainty that your model is doing well.
So what are we going to do is we're going to take the training set.
We're going to split it into, ten folds.
It's actually k folds.
But for our tutorial for simplicity's sake,
which is going to assume k equals to ten.
So it's playing into ten folds.
A fold is just a fancy word for saying we're going to split ten parts.
Each part is about this.
They're all about the same in size and they don't overlap.
then what we're going to do is we're going to train the data
on, nine of these folds and keep one fold
as an unseen, fold for validation.
So that's, that's going to be our training data
and that's going to be our validation data.
Think of it as, basically training data, testing data.
But we're going to use validation.
So we don't confuse with this taste testing testing data.
So we're going to train it on this, this data of these nine folds.
And then validate or find our metrics and calculate whatever
we need to calculate of how well our model is performing on this validation
set of validation fold, because it has not seen it before.
Great. Then we're going to do that again.
But now we're going to shift the validation fold.
The validation fold becomes this fold.
So now we're going to train the data
on the on this data set right on these nine folds.
And as you can see it's slightly.
So the training data has slightly changed and the fold is completely new.
All the validation folds completely new.
And again it's not going to be seen during this training.
So we're going to get a new model as a result of this training.
a new trained model.
And we're going to validate on this fold.
And note every time we do this for every fold or every like
combination of folds.
So here and here we have to use the same hyperparameters.
Very important.
So we've decided on our hyperparameters.
And now we're just training the model again and again on slightly different
training data and validating it on the validating fold, which is changing,
which is shifting, as you can see.
So here's our six training.
So we train it on all of this data
and then validate it on this fold which is not seen during training.
So we keep doing that and we keep shifting shifting.
So if we have ten folds we're going to have to do train
ten train train ten models.
And each time we're going to use the same hyperparameter.
So the model hyperparameter model and the hyperparameters
are the same during training.
Of course, it will result in it'll be a different slightly different result.
And then we'll validate it on the validation fold.
And as a result we will have ten sets of metrics.
Remember, if we just did the training set and the test set just the normal two,
then we would have one set of metrics. Then we could have gotten lucky.
Whereas here we're going to have ten sets of metrics.
It's much less likely that we got lucky ten times.
Right.
So it's much more reliable now that we're going to have,
ten sets of metrics and we can look at them in aggregate.
And that's exactly what we're going to do.
So let's make some space.
And here we're going to assess these metrics and look at them in aggregate.
And if these metrics look good in aggregate
then the modeling approach is valid.
So the model you've selected
and the hyperparameters you selected are good for this data.
And then what we're going to do is we're going to train the model again
to go and train the model one more time, one last time.
This time we're going to train an all of the training data,
and then we're going to test it on the test set as usual.
That's our final step.
On the other hand, if the aggregate metrics don't look good
then something's wrong then.
Otherwise we need to. If so, they don't look.
If if they don't look good, we need to adjust hyperparameters of the model.
Or we have to change the model entirely
and repeat this whole process of k fold cross-validation.
So that's what k fold cross-validation is.
And that's how it works.
As we discussed at the beginning, there's a few schools of thought.
This was a second school or they don't really have numbers.
This is one of the schools of thought that says we should have the training set.
Apply k fold cross-validation to the training set,
says the metrics, and then retrain the model on the training set.
Once we're happy and then still test it on test set, the other school of thought
says, let's get rid of this testing step, right?
So we've already, run the model here many times,
and we've tested it on these validation metrics.
So we're just going to train the model on the training set.
And we will no need to test it anymore.
We've tested it. We know it works.
then there's a modification of that school of thought as well
where you don't even train it on this training set anymore.
So you just say, okay, well, we've, train the model ten times here.
We've got ten different models. As a result.
We've tested all of them, so we don't need to do this test.
And we're not going to do the training.
We're just going to pick one of these models.
that is kind of like a little bit challenging in my view.
Like how do you pick which one to go with?
You're going to just take
the one of the the best metrics or them the closest metrics to the average.
So it's it creates a little bit of extra work to pick the model out of these,
because they're all going to be slightly different models,
because the underlying training data was slightly different.
So that's also an option.
And then there's yet another
modification of how to think about k fold cross-validation.
what you could do is you could take,
this part and do it first, you know, do the classic part first.
Split your data into a training set and a test set.
Train your model on the training set, test it on the test set.
And then if you're happy with the results of this classic approach,
you could go an extra step and apply k fold cross-validation.
So do all of this.
But after you've done the training and testing.
And so then once you've done all of this, if your aggregate metrics
still look good, then you can confirm and say, well, I'm happy,
that even like, like I, I didn't get lucky on the test set.
Basically it was indeed the fact that it works on the test set,
is is not just a chance in reality, which I tested it later.
I tested it with k fold cross-validation still works.
So I'm just going to keep my original model
that I trained in this first approach.
So in this case
k fold cross-validation is acting as like an add on to your classic method.
So it's kind of the same thing as we discussed.
as that this the what's it called?
The general, the most general k fold cross-validation.
When you do all this first and then you train and then you test,
it's kind of the same thing, but it's just doing it backwards.
So you can do that to, what whatever you're happy with, whatever works for you.
As long as, you know why you're doing it
and, you know, like, what results you're aiming for.
You know, how to assess, the, indications
that k fold cross-validation is giving to, the rest of the details.
There's no, like, one hard and fast way that you have to do it, as long as you get
the outcome. We get the benefits of k
k fold cross-validation that you're aiming to get.
And of course, you don't let, the, the validation
data leak into the training data
so you don't let the model see the validation data during training
or the test set during training set if you're doing those okay.
So that's k fold cross
congratulate on adding a new powerful tool for model assessment to your toolkit.
And I look forward to seeing you back here next time.
Until then, enjoy machine learning.