Hello and welcome to
this art tutorial and in previous tutorials, we already
implemented a multiple linear regression model that we fitted to the training set.
But when we take a step back, do you think it's actually
the optimal model that we can make with the data set that we have here?
You know, because when we built this model,
we actually used all the independent variables.
But what if
among these independent variables there are some
that are highly statistically significant, that is have great impact
or great effect on the dependent variable profit,
and some that are not statistically significant at all.
That means that if we removed this non statistically significant variables
from the model, well we would still get some amazing predictions.
So the goal in this tutorial is to actually find a team,
an optimal team of independent variables, so that each independent
variable of the team has great impact on the dependent variable profit.
That is, each independent variable of the team
is a powerful predictor that is highly statistically significant
and definitely has an effect on the dependent variable profit.
And this effect can be positive.
That is, for an increase of one unit of the independent variable,
the profit will increase or it can be negative.
That is, for an increase of one unit of the independent variable
the profit will decrease.
And so we're going to divide this final step of building
this optimal model using backward elimination into two tutorials.
In this first tutorial that we are having right now, I'm going to walk you
through the backward elimination algorithm or without completing it up to the end.
That means that at the end of this tutorial, you'll get a homework
which will consist of completing what we started with backward elimination.
So I'm sure you will have no problem, because I'm going to walk you through
the introduction of backward elimination so that you can understand everything
and have all the tools to complete the homework.
And then in the next tutorial, I'll give you the solution of this
homework and we will complete together the backward elimination.
So I hope you're excited.
Let's start right now.
The backward elimination.
So for those of you who follow the Python tutorial,
you will notice that it's actually a little more simple here,
because in the Python tutorial we had to use another library and another
multiple linear regression model to implement backward elimination.
And this time we will simply take the model that we created here,
and we will use this amazing function summary of R that returns
a great deal of statistical information that can help make our model more robust.
And then same as in Python, we are going to do some copy paste
very simply until we get to the final team of our independent variables.
So let's do this.
You're going to see it's really quick and easy.
We're going to do that very efficiently.
And the first step of doing that is to actually take our model
because as I just mentioned,
we are going to use this same model to implement backward elimination.
So here I just copied the model and I'm going to paste it here.
And now we're just going to change two things.
The first thing is instead of having this dot here,
you know this dot that represents all the independent variables,
we we're going to write all the independent variables separated by a plus
because you know, the principle of backward elimination
is that we will remove each independent variable
that is not statistically significant one by one.
So we need here to write each of the independent variables
so that when we copy paste this model here we will just need to remove
the non statistically significant variable from this equation here okay.
So let's first do this
I'm going to take our data set here to look at the independent variables.
So the first independent variable is already spent.
So let's add it here.
So r dot d.
So you know as a reminder there is a dot here
because the original name for this independent variable is r space
d and r just replaced the space by a dot here.
So it's good to know that if you're working with R
and if you have some data sets with spaces in your column names,
and then okay, so what was the name.
Dot another dot.
So that means another space dot and spend.
All right. So that's the first independent variable.
And now let's add the second one.
So we need to separate them by a plus okay.
And what is the second one.
The second one is administration okay.
So here there is no dot.
Everything is fine just as it's spelled right.
Administration plus.
Marketing spend.
Plus.
And we have one one last independent variable which is the state.
So here we don't need to create the dummy variables as we did in Python.
Because remember we used here this amazing
factor function that encoded this state categorical variable
into factors that are one two, three.
And there is no relational order between those categories.
So everything is fine.
We don't need to create any dummy variables.
So that's the beauty of R.
And so here same we don't need to sum two separate dummy variables.
We can take the original independent variable state okay.
So as I mentioned there are two things that we would like to change here.
The first thing was to replace the dot by all this independent
variable separated by a plus.
And now
the second thing that we would like to do, but it's not compulsory, is just because
I would like to use all the data set to see the correlations
is to replace here training set by
simply our data set.
So that's not compulsory.
We can actually do backward elimination using the training set.
But we're just taking the whole data set in order to have complete information
about which independent variables are statistically significant
and which independent variables are not.
Okay. And now actually we're almost ready.
We just need to use the summary function, which we actually already used before.
And there is nothing more simple than using the summary function.
We just need to take the summary function here.
And then in parenthesis we input our regressor.
Here it is. And now that's actually ready.
We're actually ready to start the first steps of our backward elimination.
Well speaking of backward elimination let's have a look at the slide.
You saw with Kirill in the intuition tutorial.
And here is the slide.
So let's have a quick reminder about the five steps here.
So the first step is to select
a significance level that is a threshold for our p value
such that if the p value of an independent variable is below
the significance level, then this independent variable stays in the model.
And if the p value of the independent variable is higher
than the significance level, then it will not stay in the model.
We will remove it.
So. But the first step is just to select a significance level.
We don't have to do anything here with the independent variables.
We just need to choose one.
And we're going to choose 5% 0.05 okay.
And now step to step two is to fit the full model with all possible predictors.
So that's actually what we've just done.
You know by taking all our independent variables
in our regressor using the LM function that actually fits the full model
with all the possible predictors, that is with all the independent variables.
Okay. So done.
And now what is step three.
Step three is to look at the predictor that has the highest p value.
So we will find it thanks to our summary function.
And if the p value is higher than the significance level
that is if it's higher than 5%, then we'll go to step four.
And if that's not the case our model is actually ready.
But don't worry, it will not be that quick.
So actually.
So let's suppose
we found the highest p value higher than the significance level of 5%.
Then we need to move on to step four.
And the step four is actually to remove
this independent variable that has the highest p value.
And once we remove the predictor we're ready to move on to step five
which is to fit the model without this variable.
So that's why, you know, we wrote all the independent variables
one by one separated by a plus.
Because, you know, once we reached step five here, we will just copy paste
the regressor and the summary function and remove this independent variable
that had the highest p value from the regressor
to build a new regressor without this variable.
And that will fit this model without the variable.
And once it's done we go back to step three here
to repeat this same pathway that is.
Once again we're going to look for the independent variables
among the new team of independent variables
without the independent variable that we just removed.
So we're going to look for the independent variable that has a highest p value.
And same
if the p value is higher than the significance level we'll go to step four.
And otherwise our model is ready.
So let's do this.
We already completed step one by choosing a significance level of 5%.
And same for step two.
We actually fitted the model with all possible predictors.
Well we need of course to execute the code.
And now we will move on to step three which will consist
of looking for the independent variable that has the highest p value.
So let's do this right now
okay.
So as I just mentioned we need to execute this
to build our regressor with all the independent variables.
Well actually we don't really need to execute this because we actually
executed this code section here that builds exactly the same regressor.
But just to complete all these steps in this tutorial, let's execute that again.
That will not cause any issue.
So I'm going to press Command and Control plus enter to execute.
And here we go.
Same regressor created again with a different syntax.
And that's because we want to remove the non-significant independent
variable one by one.
Great.
So that actually completes step two.
And now let's move on to step three which was to look
for the independent variable that has the highest p value.
And to do this we are going to select this
summary function with the regressor inside and press Command and Control Plus.
And to execute.
Let's move that up a little bit.
So these informations are very important
informations when we want to build a robust model.
And it's not only thanks to the informations about the P values here
that will help select the optimal team of independent variables.
Because below we also have this multiple r squared.
And this adjusted r squared.
That will help us build even more robust model than the one
we are going to make in the next two tutorials.
Because at the end of this part there is this section called Evaluating models
performance to actually improve the model performance.
And in this part we will actually use the multiple r squared and the adjusted
R squared to finalize our journey towards the most robust model.
And you will perfectly understand why at the end of this part.
But for now, let's focus on the p values.
So the p values are actually in this column.
And in R there's actually a shortcut to look at the statistical significance.
It's this last column here.
Well this last column doesn't have a name.
But you need to look at the stars here because as a reminder,
the more the p value is below 5% our significance level,
then the more the independent variable will be statistically significant.
For the dependent variable profit, and the more the p value
is higher than the significance level 5%,
then the less statistically significant the independent variable will be.
So in short, the lower is the p value, the more your independent
variable will have high impact on your dependent variable,
and the higher is the p value, the less effect.
In fact, your independent variable is going to have on the dependent variable.
And there is this reminder here that says that if the p value
is between zero and 0.1 percent,
then it's three stars, meaning that it's highly statistically significant.
Then if the p value is between 0.1% and 1%,
then it's two stars, meaning that it's very statistically significant
but less significant than when there is three stars.
Then if the p value is between 1% and 5%,
then it's simply statistically significant.
That is, your independent variable
still have some good impact on the dependent variable.
And then if the p value is between 5% and 10%, then it's a dot, meaning
that there is definitely a certain level of statistical significance.
That is, your independent variable has a certain effect on your dependent variable,
but definitely not as much as your other independent variables
that are in these categories here, especially for this one.
And finally, if your p value is between 10% and one,
well, there's absolutely no statistical significance.
So that means that with what we first observe here,
well we can see that the R&D spend is highly statistically significant.
But the rest seems to be not significant.
But let's wait for the backward elimination to be completed to find out
if our final team is actually only composed of already spent,
because by removing some independent variables here, that will remove
some possible bias, that once some independent variables are removed,
we can actually find an independent variable that is more
statistically significant than what it appeared to be at the beginning.
That is at the first step here.
So let's find out about that.
And actually you will find out
about that yourself because this will be the subject of the homework.
But don't worry I'm going to walk you through the first steps of
backward elimination and you will complete it yourself.
And in the next tutorial, of course, we'll have the solution
and we'll complete it together.
So I look forward to seeing if we get the same results.
Okay, so now let's carry on with backward elimination.
So remember we are at step three.
And step three is actually to look
for the independent variable that has the highest p value.
And we can find it very easily.
It's actually this one because indeed its p value is 0.999.
That is 99%.
So that's actually a very high p value.
And we are way above the significance level of 5%.
So this dummy variable for state state
two is definitely not statistically significant at all.
It has absolutely no effect on the dependent variable profit.
And by the way we also observe that state three here has a 94% p value.
And there is no way that if we remove state two
well this p value will decrease below the 5% significance level.
So we can actually remove this state three independent variable as well,
because clearly the state has no effect or impact on the dependent variable profit.
So we will actually make some kind of a shortcut here.
And instead of removing the independent variable that has the highest p value
that is state two, we will actually remove both these dummy variables for state
because definitely the state is not statistically significant.
So let's do this.
Let's remove the state variable from our equation.
So I'm going to put that down
okay.
So as I told you it's very simple.
We're just going to copy this and paste it here.
And then so what do we have to do. Now.
We just need to remove the state independent variable from our equation.
Here.
And by doing that we complete step four.
Because if we go back to our slide, step four is to actually remove the predictor.
Great.
And now we can move on to step five which is to fit the multiple linear
regression model without the independent variable state that we just removed okay.
So we removed state.
So step four completed.
And now step five is to actually fit the model
without this independent variable state that we just removed.
So let's do this.
We just need to select this press command and control.
Press enter to execute.
And here it is.
Our new regressor is ready without the state independent variable.
So now we have a team of three independent variables.
Wait and see which independent variable is going to be kicked out of the team.
And speaking of that, this is where I'm going to leave you alone for the homework.
But don't worry, you will have the solution in the next tutorial.
But really try to implement this yourself.
Complete backward elimination up to the end,
and it will be fun to see if we get the same results.
And there is actually kind of a decision
to make at the end of backward elimination.
So I'm curious to see how you make that decision, make that call,
because both solutions are actually great
and we will talk about that in the solution.
So good luck for the homework you're going to see.
It's going to be fine.
So basically what you only have to do is to follow this backward elimination slide.
And so together in this tutorial we went up to this step five here.
And now.
As you can see you have to go back to step three and redo the steps
three four, five exactly like we just did.
Until you find that the highest p value is not higher than the significance level,
and when it's the case, your model will be ready.
Okay, so I look forward to seeing you in the next tutorial.
I look forward to comparing with you your results to mine,
and I'm sure everything will be okay.
Backward elimination is very practical
and it will actually be fun and easy to complete this.
So thank you for watching this tutorial and I look forward to seeing you
in the next one for the solution.
Until then, enjoy machine learning.