Hello and welcome back.
Today we're continuing our saga of model evaluation.
And we're talking about adjusted R squared.
Previously we discussed r squared which is defined
as one minus the residual sum squared divided by the total sum of squared.
And r squared is a goodness of fit for our model's greater or is better.
just a quick reminder.
We mentioned that r squared is between 0 and 1 and that
there is no overarching rule of thumb.
The values really depend on the industry and use case.
So for some industries and use cases, 0.9 might be a great R squared and 0.4
might be a terrible R squared for the R for other industries
and use cases, 0.4 might be a great R squared.
So it really depends.
However there is an overarching problem across the board
which is to do with adding new independent variables.
So let's say we have a linear regression with two independent variables.
And we decide to add a third one.
For example, we got new data,
new column of data or we got, we're just trying to explore
and see what other variables might be, helpful in the explanation in our model.
So what happens when we add another variable is that the total
sum of squares doesn't change because it only depends on the average
of the y actual values, and doesn't depend on the y hat values.
But the residual sum of squares will change
and in fact it will only either decrease or stay the same.
The problem we're facing is that the residual
sum of squares will never increase when we add when we try to add
another variable, and this might not be intuitive at first.
So let's talk about it a little bit.
the main reason for this is that we are using the ordinary
least squares method to build our models, and what the ordinary least squares
method does is.
It aims to minimize the residual sum of squares.
So let's try see this in action when we add this new variable x3.
the ordinary least squares method is going to look for coefficients
b3 that improve the y hat predicted values.
As long as it finds, a coefficient b3 where
the y hat values are better than they were before, closer to the actual values
than the sum of the residual sum of squares will improve
it could improve by a lot if the the prediction is much better.
Now, what can improve by a tiny little bit even,
if the prediction is a little bit better?
Now, in the situation where the ordinary least squares
method cannot find a coefficient b3 that improves the predictions
like all possible coefficients, B3 make the predictions worse,
then the ordinary least squares method is just going to be
very, smart or sneaky, you can call it.
And this is going to put turn B3 into zero.
It's just going to say, okay, we're going to set B3 at zero.
And that means even though we technically added an extra variable,
it's not participating at all in the predictions
because its coefficient is zero.
So in that case the residual sum of squares won't
change, will be exactly as it was before.
So we end up with a situation where
we can just keep adding more and more variables
that maybe have even nothing to do with our problem at hand.
But by virtue of some random correlations, our
r squared in some cases will be improving.
Improving it never gets worse.
So the residual sum squared will decrease.
That means r squared will increase.
And that's a problem because we don't want to end up
with models that have lots of, variables
that have nothing to do with that model, that are not really adding a lot of value,
but in general are just increasing r squared.
So what is the solution?
The solution is, a new version of r squared and adjusted r squared.
And that's exactly what it's called.
It is calculated with this scary looking formula.
But as you'll see
from the practical tutorials we've had learn, it's not scary at all.
and you'll be able to recreate this, actually manually.
So here there are a couple of new, parameters.
K is the number of independent variables that are in our model,
and n is the sample size.
And the important thing here is to look at k.
So if we
if k increases over
here then the denominator decreases.
That means the whole ratio increases.
And because it's being subtracted that means adjusted r square decreases.
So that's the important point here that this new formula
penalizes us for adding additional variables.
So basically it's only worth adding an extra variable if r squared.
This original r squared if it increases substantially
enough that compensates for this penalty.
So you know, adding new variables
becomes something that has to be justified.
If it's not justified, then the new variable is not worth adding.
And that's what adjusted r squared is about.
It's about making sure that we only add variables
when they bring substantial improvement to our model.
So we go that's adjusted r squared.
Hope you enjoyed this tutorial and I'll see you next time.
Until then, enjoy machine learning.