In this lesson what we're gonna do is we're going to look at our regression coefficients, our thetas, in more 1
detail. 2
So far we've discussed how to interpret the value of our coefficients in both the original linear model 3
and the log linear model 4
after our data transformations. The thing is if we're looking to predict house prices we're not just 5
going to be interested in the sign and the size of our coefficients, 6
we're also going to be interested in their significance. Just because there's a number next to a particular 7
feature does not mean that this feature has much explanatory power. 8
Just because there's a number here doesn't mean that this feature is actually significant. 9
Remember how we talked about how a doctor checks your vital stats? A regression coefficients vital 10
stat that tells you about its significance 11
is called the p value. 12
Now most academic research papers that you come across analyze the significance of their findings using 13
this metric of p-values. 14
And here's how this metric is typically used. 15
If the p-value is less than a certain threshold, that threshold being 0.05, then the result 16
is deemed statistically significant and when the p value is greater than 0.05, then this 17
result is considered to be not statistically significant. 18
This threshold of 0.05 is kind of where the consensus amongst academics is regarding 19
significance, and for better or worse, there's a little bit of a cult around this particular value. When 20
it comes to calculating the p values, 21
the scikit-learn's linear regression model isn't actually much help. 22
So what we're gonna do is we're going to look beyond the scikit-learn module to calculate our statistics 23
for our regression. 24
We're gonna go beyond the simple linear regression provided by our machine learning module. In these 25
lessons we're gonna be looking at some detailed statistics of our module. 26
And this is why we're gonna be importing a different Python module to take us further here and this 27
Python module is called Statsmodels. 28
Now let's add a section heading. So I'm going to to change this cell to markdown and I'm going to put a section heading here 29
that reads "p values and Evaluating Coefficients". What we're gonna be doing next is we're gonna be using 30
the Statsmodels module to run our linear regression and we will run it so that we get the same results 31
as we would with scikit-learn. 32
However we will be able to use this Python Statsmodels module to pull up detailed statistics that we 33
can't easily get with scikit-learn. 34
The first thing that we need to do of course is import our Statsmodels. 35
So we're gonna go to the top of our notebook and in our first cell, we're gonna say "import statsmodels. 36
api as sm" and then I'm going to hit Shift+Enter on the cell and what I see when I do this is there 37
is a deprecation warning here. 38
This deprecation warning refers to my statsmodels module and what it's saying is that at the time of 39
recording this module is still using a component that is outdated, so it's using this Panda's datetools 40
component which is outdated or deprecated. Now I'm really not too concerned about this for two reasons. 41
One is that we're not gonna be using any functionality to do with dates from the statsmodels API and 42
two is that the statsmodels API will also be updated by the people who created it and what they will 43
do is they will make sure that it doesn't break and that it's maintained in good working order. 44
So by the time you're running this, you may or may not see this deprecation warning. 45
Now let's run our regression with this new module. 46
The thing to note is that in order to make our regression tie out with scikit-learn, we're going to have 47
to add an intercept, because as you can see there is an intercept here from our regression with scikit- 48
learn. 49
So what I'm going to do is take our features from the training data set and add an intercept. So I'll write 50
"sm.add_constant(X_train)" and what I'll do is I'll store this 51
modified dataframe in a new variable. 52
So what I'm gonna say is "X_incl_const = sm.add_constant( 53
X_train)". 54
Now what we can do is call on the statsmodels OLS function, which will give us back a model object 55
which we can then use to fit our regression. 56
So I'm going to say "model = sm.OLS(y_train, X_ 57
incl_const) " 58
What we're doing here is we're calling the OLS function, OLS stands for Ordinary Least Squares 59
and just like scikit-learn, 60
this gives us a linear regression model which we're storing here. As arguments 61
we've provided our target values and our features and these features include this constant that we've 62
added. 63
Now we can use the statsmodels API to fit our regression. Fitting our regression will give us some results. 64
So I'll create a new variable called results and set that equal to "model.fit()". 65
In other words, calling the fit method on the model will return to us our regression results. 66
So the question is how do we take a look at these results? 67
For starters, let's see if we can print out the coefficients like we have here. To get these coefficients, 68
we'll use the results params attribute. 69
So "results.params" and Shift+Enter will show us the coefficients. 70
So here they are, and these values tie out with what we saw in scikit-learn. I'm going to comment this out 71
and now we can take a look at the p-values that I keep harping on about. To show these, 72
we also use our results object, put a dot after it and access the pvalues attribute. So "results. 73
pvalues" and Shift+Enter will show us the p-values for all our coefficients. 74
I don't think this is formatted particularly nicely, so what I'm going to do is I'm going to comment this 75
out again and both of these two series into a data frame. 76
So with "pd.DataFrame({'coef': results.params, 77
'p-value': results.pvalues})". 78
We can look at our coefficients and their p values formatted nicely side by side. 79
Check it out. 80
Now this is starting to look pretty good, but you know what? 81
I find these p-values really, really hard to read in scientific notation. 82
So what I'm going to do this I'm going round them. I'm going to come up here where I'm creating our data 83
frame and I'm going to call the round function, so "round()", closing parentheses before the 84
curly brace and then comma, and then a number of decimals that we should round to. 85
So I'm going to go with 3 and refresh our output. 86
Now let's talk about how to interpret these results. 87
Well, remember the rule of thumb that any p value over 0.05 is not significant? 88
In our case two of our features failed this test, namely the INDUS feature and our AGE feature. 89
These two features do not appear to add much additional information. 90
All the others are indeed statistically significant. 91
Now let's make a note of this for later, because maybe maybe we could remove the INDUS and the AGE features 92
from our model. In the next lesson, 93
we're going to discuss a potential problem in our regression. 94
Remember how we had high correlations between our features? In the next lesson, 95
we're going to check formally if our regression suffers from the problem of multicolinearity.