So now that we've plotted our actual prices versus our predictions and we've generated this chart here, 1
let's move on in our residual analysis to the sister chart of this one, because in this chart we can't 2
actually see the residuals explicitly, right? 3
We can see kind of how far the data points are from the cyan line here, 4
but our residuals are actually not on one of the axes. 5
Let's make this much more explicit and plot our residuals versus the predicted values. I'm going to come up here, 6
add a comment that reads "Residuals vs Predicted values". 7
I'm going to take these lines of code, copy them, paste them below 8
and now I'm going to modify them a little bit. 9
So I'm going to get rid of this line on the x axis, 10
I want the label to read "Predicted log prices" and then "\hat y_i", "fontsize = 14". On the 11
y axis 12
I want it to read just "Residuals". And for the title I want it to read "Residuals vs Fitted Values". 13
Now let's update the arguments in the scatter plot. 14
So for the x axis, you've already guessed - it's gonna be "results.fittedvalues" and for the y axis, 15
it's going to read "results. 16
resid". For the color, 17
I'm going to go with "navy". For the alpha, 18
I'll leave it at 0.6. 19
Now let me hit Shift+Enter. 20
So this is the chart that you can compare to the charts that you've seen in the slides earlier on in 21
the lesson. 22
How do we interpret what we're looking at here? 23
Are you seeing any obvious patterns? 24
I think it's actually pretty ok, the residuals look fairly random for the most part and the residuals 25
are actually centered around zero. 26
So looking at the y axis here, we can see that a lot of the residuals are centered around zero. The residuals 27
are also fairly symmetric. 28
They don't seem to be systematically high or systematically low. 29
So the model is kind of correct on, on average. 30
But in this chart we also see the issue with the high price bracket homes filtering through. 31
No surprise there. 32
Can you spot them? 33
Which data points on this chart correspond to all the fifty thousand dollar homes? 34
They're actually right here. 35
You see how this almost traces out a line? These data points on this chart correspond to these data points 36
on this chart. Those fifty thousand dollar properties that we're bad at predicting seemed to be lining 37
up. 38
Now let's do our next check on the residuals that we talked about. 39
Let's check for normality. 40
Let's see if our normality assumption is satisfied or close to satisfied, because in the beginning of 41
the lesson we said, well we kind of want these residuals to be normally distributed. 42
Let's check out if they really are or not. 43
In the cell below, I'm going to add a comment and that's going to read "Distribution of residuals (log prices) 44
- checking for normality'. A normal distribution if you remember has a mean and a skew of what? Zero, 45
Right? 46
The skew should be zero and the mean should be zero. 47
How do we print out the mean and the skew? 48
We'll take our results objects, "results.resid", get the residuals and then we can chain a method 49
onto this. For the mean, we would use ".mean()". Let me hit Shift+Enter. What's printed out here 50
is in scientific notation. 51
So let's round this, let's say "round(results.resid.mean())", comma and then let's 52
round to 3 decimal places and have the closing parentheses at the end, Shift+Enter, and we see here 53
that the mean of our residuals is indeed very, very close to zero. 54
Let me store this in a variable called "resid_mean". 55
And now let's print out the skew, so "results.resid.skew()" should be our skew. 56
See what that is. 57
Huh. 58
Okay. 59
0.12 approximately. I can also round that, round to three decimal points. 60
And I'm also going to store this in a variable, "visit_skew" is equal to this whole thing. 61
So fair enough. 62
The mean is equal to zero. 63
The skew is not equal to zero, but it's not too far off. 64
Now looking at these two numbers is helpful but it's even better if we complement this with a plot, with 65
a graphic. 66
So I'm going to use seaborn here. "sns.distplot()", distribution plot parentheses and then as the arguments 67
we'll provide our residuals, 68
so "results.resid". 69
And for the color we'll go with the "navy" again. 70
I think every plot needs a title so "plt.title()" and then as a title we'll say 71
'Log price model: residuals'. 72
Now let's go with "plt.show()" and see what this looks like. 73
Here we go. 74
Here we see the distribution of our residuals using seaborn's distplot function. 75
I can come back up here, make this an f-string by putting the little f in front and add our residuals 76
mean and the skew into the title, so I'll go with "Skew ({resid_skew})" 77
and "Mean ({resid_mean})". We didn't calculate the mean and the skew and rounded it 78
for nothing after all. Let's show it in our chart. 79
There we go. 80
So how are we doing? 81
Well, the mean is equal to zero, but that's no surprise. 82
That's actually by design. 83
That's how the regression model's best fit line is calculated. 84
No matter how bad your regression line, the mean is gonna be equal to zero by design, but I think the 85
skew being close to zero is a result of our data transformation and I'm going to prove this to you shortly. 86
Looking at this histogram and the estimated distribution for the residuals by seaborn, what's really 87
comforting to see is that the residuals are fairly symmetrical, right, and they have a fairly constant 88
spread throughout the range. 89
So I think we're doing pretty ok. 90
The thing that you do notice however is that this distribution in contrast to a normal distribution 91
has much longer tails. 92
So there's more values in the extreme left and the extreme right than what you would see with a normal 93
distribution. 94
You've got a bigger peak in the middle and then you've got longer tails on either end. 95
So this is where the similarity to the normal distribution is much, much weaker. 96
Okay. 97
So we've looked at three charts of our residuals, but I think what we really, really need to do is, we 98
need to compare how these charts looked like for different models, because if these three charts are 99
all we've ever seen we don't really have much context, right? 100
And so on that note I'd like to pose a challenge to you. 101
I want you to generate these three plots, right. 102
So this distribution, the residuals vs the fitted values and the fitted values vs the observed 103
values for the original model that we had. 104
So this was the model with all the features using normal prices not the transformed log prices. 105
And after you've generated those charts, I want you to analyze and interpret the results that you're 106
getting back, so I'll give you a few seconds to pause the video and give this a shot. 107
OK, ready? 108
Here's the solution. 109
Use the lazy man's approach and copy this entire cell, I'm going to then come here and paste it in and 110
I'm going to modify the code a little bit. I'm going to change my comment here, 111
say "Original model" 112
"normal prices & all features". To use normal prices, 113
I have to, not just get rid of this comment, but I'm going to have to get rid of this "np.log()" here and to use 114
all the features, I'm going to delete "INDUS" and "AGE" from the arguments under the drop method. Scrolling 115
down, 116
don't need this comment anymore. 117
Don't need these comments anymore and then for the scatter plot, 118
I'm gonna go with a different color. 119
I'm gonna go with Indigo. For the labels on this chart, 120
I'm going to say "Actual prices 000s", "Predicted prices 000s". For the title, 121
I'm going to say "Actual vs Predicted prices". Coming down, 122
I'm going to delete this line of code, which we don't need. For our second chart, 123
I'm also gonna go with indigo, and I'm going to update the labels and now all I have to do is add the distribution 124
graph. 125
So that's gonna be a "Residual Distribution Chart" which I'm going to grab from up here. 126
I'm going to grab these lines of code here, 127
copy them, put them down here, paste them in. Again, 128
change the color to indigo to set them apart a little bit, 129
update my title, let's have it read "Residuals" and print out the skew and the mean in the title. 130
And that's pretty much it. 131
The coding side of this challenge is pretty trivial because we're reusing a lot of the code. 132
But let's take a look at what the charts look like and see what the differences are between what we 133
are doing here and what we did earlier. 134
First up, our actual versus our predicted prices. 135
Now visually the first graph and this one here actually seem quite similar. 136
And that's no surprise given that the correlation between the fitted values and the observed values 137
is around the same. 138
Yes, it's a bit was we know that from the r-squared that we calculated and it has a little bit lower 139
correlation but it's not super dramatic on the differences. The predicted and the actual values are actually 140
fairly close to the cyan line as they were with the log prices. Now coming down on the second chart here. 141
This one is much more interesting. Here we're definitely starting to see a little bit of a difference. 142
Compared with the log prices, 143
the cloud of residuals looks almost like it's got a little bit of a parabolic shape to it. 144
It's kind of subtle and you almost have to kind of squint a little bit. 145
But what we're seeing here doesn't look entirely random. 146
This provides further justification that the log transformation for the target values that we did was 147
indeed appropriate. 148
Now, what about the third chart? 149
What about the histogram and the distribution of the residuals? Coming down 150
we see that 151
yeah, 152
the mean is equal to zero. 153
But what about the skew? 154
And here we need to see that with a skew of 1.5 approximately the distribution of the residuals 155
is actually fairly lopsided. 156
This makes this distribution a lot more dissimilar from a normal distribution, because the skew of a 157
normal distribution is zero 158
and we've got 1.5 approximately. A distribution of residuals, 159
that's not close to a normal distribution makes things much more difficult when it comes to making predictions 160
and making forecasts, which is ultimately what we wanted to do, right? 161
This is the assignment that our boss gave us in our imaginary real estate office. 162
So I hope this was a helpful contrast to what we saw earlier and provides a bit more context. 163
But I want to show you one more example, because before we finish this lesson I want to show you the 164
kind of pattern that you could see in your residuals when you're missing important features or omitting 165
kind of key variables in your regression. 166
So let me come back up here, copy this, paste it and then I'm going to update my comment here. 167
I'm going to say "Model 168
Omitting Key Features using log prices", 169
and now what I'm going to do is start dropping quite a few features from our dataset. 170
I'm going to drop, not just INDUS and AGE but I'm also going to drop LSTAT, I'm going to drop 171
RM, I'm going to drop NOX and I'm going to drop Crime. 172
Now we said we'll use log prices, 173
so I'm going to add "np.log" back here where we're getting our prices and then just as a review, 174
you don't actually have to stick to the named colors that are in matplotlib, 175
you can actually specify any color you want, any shade you want. 176
If you go to a web site like flatuicolors.com you can grab a particular hex code that identifies 177
a particular shade of a color. The hex codes always start with this pound symbol and then there are six letters 178
or numbers following that. 179
So I'm going to take Alizarin here, which I can then paste in here where I've referenced Indigo. 180
So "c=#e74c3c". This is this shade of Alzarin that we've copied from the other website. 181
Coming down, 182
I'm also gonna replace the color in our second chart, so that way each of our models has a certain theme 183
going on, and I'm also gonna delete this block of code at the bottom. 184
Finally just gonna update the title here. 185
So I want that title to read "Actual vs Predicted prices with omitted variables". And the very 186
last thing we have to do on the labeling front is change our x and y labels. 187
So these are gonna be back to log prices, xlabel is gonna read "Actual log prices" and our ylabel is gonna 188
read "Predicted log prices". 189
Let's take a look at our charts. 190
There we go. 191
So this is interesting, right? 192
As before, we see this banding here on the top right with our very expensive properties at fifty thousand. 193
We also see that, as expected, the correlation between our fitted values and our observed values is much, 194
much lower because we're leaving out a lot of information, a lot of explanatory features from our model 195
but not only that, we see this kind of like banding here. 196
So you've got all these data points lining up here and here and even inside this cloud here. Scrolling 197
down, we see that this is even more extreme when we look at the residuals vs the fitted values. 198
Here you can see the banding very, very clearly in the residual chart. Instead of a completely random 199
distribution of residuals, 200
what we see in this chart here are clusters. 201
This is a very, very clear pattern and it's telling us that there's some important information that's 202
missing from our model and this information has somehow found its way into the residuals. 203
And this kind of brings me to my final thoughts on the banding that we see with the fifty thousand dollar 204
homes. 205
My hypothesis as to why we see these properties lining up like this is because there's something maybe 206
missing from our model, maybe there is some feature that these are homes all have in common or there 207
was something in the way that the data was collected or there is some sort of interaction between a 208
feature of these homes that we're not capturing in our model. 209
If I wanted to kind of dig into this further and improve this model that we have further, this would 210
be one of the things I would be looking at. 211
This would be something I could dig into. 212
But we have more important things to do. 213
You and I we're gonna be moving on to bigger and better things. 214
We're gonna be moving on to making predictions from our regression model. 215
This is what we ultimately set out to do, right? 216
I'll see you in the next lesson. 217
Take care.