Hello and welcome to this auditorium.
And mostly
welcome to the last part of this course, 14 model selection and Boosting.
So in this part we will do two things.
First evaluating our model performance.
And second, improving our model performance.
And then there will be this bonus section about one of the most powerful
algorithm in machine learning, which has become more and more popular.
And that is called XGBoost.
But first, we want to be able to improve the model performance
of all the machine learning models we've built in this course,
and improving
the model performance can be done with a technique called model selection
that consists of choosing the best parameters of your machine
learning models.
Because you know, remember, every time we build a machine
learning model, well, we had two types of parameters.
The first type where the parameters that the model learned that is, the parameters
that were changed and found optimal values by running the model.
And then the second type of parameters are the parameters that we chose ourselves.
For example the kernel parameter in the kernel SVM model.
And these parameters are called the hyperparameters.
So there is still room to improve the model
because we can still choose some optimal values for these parameters.
But since these parameters are not the parameters learned by the model,
then we need to figure out another way to choose
these optimal values for these parameters for the hyperparameters.
And that's one of the powerful thing we'll do in this part ten.
And that will be through a very efficient technique called grid search.
But before we start grid search,
we need to optimize our way to evaluate our models, because so far,
what we did is split our data set between a training set and a test set.
And you know, we trained our model on the training set
and we tested its performance on the test set.
That's the correct way of evaluating the model performance.
But that's not the best one because we actually have the variance problem.
The variance problem can be explained by the fact that when we get the accuracy
on the test set, well, if we run the model again and test again
its performance on another test set, well, we can get a very different accuracy.
So judging our model performance only on one
accuracy on one test set is actually not super relevant.
That's not the most relevant way to evaluate the model performance.
And so there is this technique called k fold cross-validation
that improves this a lot because that will fix this variance problem.
And how will it fix it.
It will fix it by splitting the training set into ten folds when k equals ten.
And most of the time k equals ten.
And we train our model on nine folds and we test it on the last remaining fold.
And since with ten folds we can make ten different combinations
of nine folds to train the model and one fold to test it.
That means that we can train the model and Tesla model
on ten combinations of training and test sets,
and that will already give us a much better idea of the model performance,
because what we can do afterwards is take an average
of the different accuracies of the ten evaluations,
and also compute the standard deviation to have a look at the variance.
So eventually our analysis will be much more relevant.
And besides we'll know in which of these four categories will be.
Because if we get a good accuracy and a small variance
will be on the lower left one, if we get a large accuracy
and a high variance, we will be on the lower right one.
If we get a small accuracy and a low variance,
we will be on the upper left one.
And eventually, if we get a low accuracy
and a high variance, we will be on the upper right one.
So this k fold cross validation is very useful.
And besides our performance analysis is much more relevant.
So let's start with this k fold cross-validation.
Our first technique of model selection.
So since we already built a lot of models we're not going to build another one.
We are going to use one of the model
we built and apply k fold cross-validation on it.
And so the model we're going to use is this kernel SVM
we made in part three classification.
And that remember we used to predict if the customers are going to click
on the ads on the social network to buy yes or no the SUV.
So the model is already built and we already have everything.
So what we're going to do is take the whole model
and we are going to add a new section
code inside of it that is going to be, of course,
the section code that will implement k fold cross-validation.
So before we start doing it let's pick the right folder as working directory.
So we go to machine learning a to Z.
We are now in the last part of this course.
Congratulations for reaching it.
Part ten model Selection and boosting and section 48 model selection.
All right.
Make sure that you have the social network at CSV file.
And if that's the case you're ready to go.
All right so now where do we apply the k fold cross validation code section.
Well since that consists of evaluating the model performance.
Well the most relevant location to put
it is right after we build our kernel SVM model.
That is right after we built the model.
And actually in this code here we have the predictions
of the test results and the confusion matrix.
That is actually a first way of evaluating the model.
But as I said at the beginning of this tutorial,
this is a correct way of evaluating the model, but not the best one.
And in today's tutorial,
we are introducing a much better way to evaluate our model.
And so let's put it right after this section.
As in a more advanced performance evaluation method.
And so we're going to call this section applying
k fold cross validation.
All right.
So now the first thing that we have to do is to install the carrot package.
Because this contains a very practical tool
to create the ten folds of our training set.
So let's start with this install dot packages
and in parentheses and in quotes carrot.
All right. So mine is already installed.
We can check it out.
Check this same on your list of packages.
Here it is. Carrot.
So I will just put that in comment.
But don't forget to install it.
And then let's not forget the library command
to import automatically the carrot package.
All right.
And now let's start coding k fold cross-validation.
So first we're going to create the ten folds that will divide our training set.
And to do this it's very simple.
We're going to use the create folds function by the carrot
package to create these ten folds very efficiently.
So let's do it.
We're going to call these folds.
Folds will actually be a list of ten different test folds
composing our training set.
So let's use this create capital F folds function.
Here it is.
And inside the parenthesis we just need to specify the training set.
So here I'm adding the training set.
And then we take our dependent variable column by which we want to make the split.
You know it's exactly like when we split the data
set between the training sets and the test set.
We need to specify the dependent variable to make it split so that the training set
and the test set are well distributed according to the dependent variable.
Well here that's the same.
We creating ten folds of the training set.
And we are specifying the dependent variable to make sure
are well distributed according to the dependent variable.
So that's why here we need to specify our dependent variable which is purchased.
All right.
So that's the first argument of the create false function.
And of course as you might have guessed the second argument is the number of folds
you want to divide your training set into.
And really a good choice of the number of false is ten
because by creating ten fold, we will eventually get ten accuracies.
And ten accuracies is a relevant way
to measure the accuracy through the mean of these ten accuracies.
So we will take ten folds, and I recommend to do that in practice.
So here we just add k equals ten.
All right.
Now we're going to implement k fold cross-validation.
Because what we just did here is just to create the folds.
But now we need to implement the algorithm itself.
And to do this well there are several ways of doing it.
But we're going to use a very practical function in R
which is called the l apply function.
And that consists of applying a function to the different elements of a list.
So this list is going to be our folds list that contains the ten test folds.
And the function is the function that is going to compute
the accuracy for each of these ten test faults.
So let's start by creating a new variable that we're going to call CV.
And then let's use here this L apply function.
All right. And you're going to understand what's going to happen.
So in this L apply function we need to input two arguments.
The first argument is the list of the elements to which
we are going to apply the next function, which is the next argument.
And so as I just said this list is false.
The list of our ten test false.
And then the next argument is the function.
So a function in R can be written this way.
Function.
Then in parentheses we need to input the argument which we will call x.
This is a local argument so far, but x
is actually going to be each one of the ten test folds.
So x here and then a pair of brackets.
Here we go.
And inside these brackets we are going to implement this function
that will compute the accuracy of the model on each of these ten test folds.
So basically in this function we are going to implement k fold cross-validation.
So what do we need to implement k fold cross-validation.
Well first we need the training fold.
The training fold is the whole training set to which we withdraw the test fold.
So basically training fold here
I'm creating a new local
variable actually that I'm call that I'm calling training fold.
And so as I just said this is the whole training set.
Here we go.
But to which we will draw the test fold that is minus x.
Because, you know, x is actually each element of this folds this here.
So by putting minus x here we are taking the whole training set
but without the test fold.
And therefore that's actually the training fold.
And then come up to take all the columns.
All right.
So we got our training fold.
Now let's get our test fold.
So our test fold try to guess what it's going to be.
Test fold equals training set.
And inside the square brackets.
Well where do we need to put here.
Well that's actually x because you know x represents
all the observations for each one of the ten test folds.
So we got our test fold.
And then what do we need to do now.
Now what we need to do is train
our kernel SVM model on the training fold.
And then we will test its performance on the test fold.
So basically what do we need to do now.
We need to add our model which is our kernel SVM classifier.
So what we can do now is just take this code section here.
Because that's where we build the model.
And we need to include this model in the function.
That's why we're taking it.
So copy and let's add it here paste.
And here we go.
We have our model but we're not training
this kernel SVM classifier on the training set.
We're training it on the training fault
because that's the principle of k fold cross-validation.
We are training a classifier on each one of the ten training folds.
So that's why here we're taking the training fold.
And that we create here inside this function that we're making right now.
And then we keep the same argument.
All right.
Then what do we need to do.
Well that's executive same as what we're doing
when we make a model that is predicting the test results.
That's the next step because that's from this test
that results that we will then compute the confusion matrix
and therefore the accuracy which is exactly what we need
that is which is exactly which will be returned by the function
we are making right now to implement k fold cross-validation.
So same let's copy this line to predict the test result.
And let's copy it here.
And is that all?
Of course no because we are not testing or classifier on the test set.
But we are testing it on the test fold.
Right.
Because you know, we are training a model on the training fold
and testing its performance on the test fold.
So now that's good.
And now let's move on to the next step
which is to compute the confusion matrix.
So still let's take this line here
and let's paste it below right here paste.
And now of course we need to change test set
and replace it by test fold.
All right.
And this will give us the confusion matrix of this classifier
SVM model of this kernel SVM classifier.
And that is trained on the training folds and test it on the test fold.
And therefore this line of code
will give you the confusion matrix for the observations of the test fold.
All right.
And now last step we need to compute the accuracy because we
are doing all this to get the accuracies for all the ten test folds here.
So let's compute the accuracy.
The accuracy is
we've calculated this accuracy many times.
We take the number of correct predictions
which is CM one come at one because this corresponds to the number
of correct predictions of the first class plus cm two comma two.
Because this corresponds to the number of correct predictions
of the second class, and since we have two classes,
this sum corresponds to the total number of correct predictions
and then we divided by the
total number of observations in the test set, and therefore
that's the number of correct predictions which is this sum.
Here,
to which
we also need to add the number of incorrect predictions.
And therefore, you know, we can copy this
and take the first number of incorrect predictions
that corresponds to the first class and this second
number of incorrect predictions that corresponds to the second class.
And so here we are actually taking all the elements of this confusion matrix.
That is the number of correct predictions plus the number of incorrect predictions.
And so now with this line of code we get the accuracy for one fold.
But since we're using this L apply function, this will do all this.
Compute the accuracy for each of the ten test folds.
And therefore we will get ten accuracies.
And we will compute its mean, which will give us a much more relevant
accuracy than just a single one.
We obtained earlier with our previous method
of evaluating the model performance.
All right.
So now we have everything, but we just need to specify that
we want to have this accuracy returned because this is a function.
So we need to specify what we want this function to return.
And to do this we just add return parentheses cis and accuracy.
And now everything is ready K fold cross-validation is well implemented.
All right.
So now we are ready to get the ten accuracies that will result
from this ten fold cross-validation technique.
So we are going to select everything from here up to the top
because we haven't imported the dataset yet.
So let's press Command Control plus enter to execute the whole thing.
Here we go.
Everything was correctly executed in less than one second, so that's perfect.
Let's have a look at the results.
So first let's put that down.
All right.
So here we get all the results.
First the data set was well imported.
We split it into the training set.
And the test set at this section here.
And then we build our classifier which is our kernel SVM classifier.
And of course we get our CV list
that we built through K-Fold cross validation.
That is this CV list, which is the list
of the ten accuracies that result from k fold cross validation.
And so let's check it out.
Let's have a look at what these ten accuracies are.
So we're going to look at it from the console.
So I'm pressing CV here and pressing enter.
And here we go.
That's the results.
That's the ten accuracies.
So for the one we get 93% accuracy.
That's very good.
Full to 87% for three 100%.
So no incorrect prediction.
Full for 86%.
Full 596% 90% on full 690%
and full 793% on full, 890% full nine
and eventually 83% on full ten.
So that clearly illustrates what I told you about this variance
problem that can occur when we rerun the model several times, because indeed
we get different accuracies and sometimes
a large difference in accuracy from one fold to another.
So from here to here that's fine.
But here, for example,
from fold two to fold three, we get a 13% difference of accuracies.
So that's why it's not that relevant
to compute the accuracy on one single split.
And it's much more relevant to compute the accuracies on ten splits,
because then we can take the mean.
And that's exactly what we are going to do right now.
We are going to compute the mean of the ten accuracies that we obtained here.
So to get this mean it's actually very simple.
We're just going to use the mean function.
So parentheses here and inside this mean we of course input CV
because CV is a list of our ten accuracies that we obtain here.
However just to make sure we get the values of the accuracies
of each of the ten fold, we need to specify here as dot numeric
and in parenthesis we include CV to make sure we take
the mean of these values here that are the accuracies.
And let's put this average of the accuracies
into one variable that will appear in the values here.
And let's call this variable simply accuracy.
Because the mean of these accuracies is just the ultimate relevance.
Accuracy.
So accuracy equals mean of the accuracies in this CV list.
All right.
So let's compute it and we'll get
let's see an accuracy of 91%.
And that's the relevant accuracy.
We are looking for.
So overall we can say with more credibility
that our model our kernel SVM classifier is pretty performance.
So that's pretty good.
And now congratulations
you have a much more advanced way of evaluating your model performance
in your data science toolkit, which you'll see in the next tutorial.
We'll see a very powerful technique that will help us choose the optimal
hyperparameters of any machine learning model we built.
So I look forward to doing that in the next tutorial.
And until then, enjoy my machine learning.