Before running our regression analysis there's another thing that we should seek to understand 1
prior to feeding our data into our machine learning algorithm. As part of the data exploration we need 2
to understand to what extent our variables move together. 3
We should look both at the correlation of our features with our target, but also at the correlation between 4
our different features. 5
So what is correlation and why should you give a ***? Excuse me, 6
that was my inner Bostonian coming out, but let's talk about what correlation is first. 7
Correlation is the degree to which things move together and I think you can see this best through some 8
pictures. 9
Now, it's no coincidence that a lot of these pictures might look quite similar to what we saw on our lessons 10
on univariate linear regression. 11
So here in our first picture on correlation we have a chart showing two variables, how much sun I get 12
and how much ice cream I eat. 13
These variables clearly have a relationship. 14
The amount of sun and the amount of ice cream tend to move together. If one of these values is high then 15
the other value tends to be high too. 16
If one of them is low then the other one tends to be low as well. 17
This is called a positive correlation. 18
The amount of sun and the amount of ice cream I eat are positively correlated. 19
Okay, so what about a negative correlation? A negative correlation 20
looks like this. The resulting graph will be downward sloping. 21
What we're looking at here is the amount of time I spend on a packed train carriage hunched over with 22
my face in somebody else's armpit on the London Underground, 23
and how happy I am. And looking at this graph, 24
The more time I spend on the London Underground the less happy I am. 25
If one of these variables has a high value than the other one tends to have a low value. 26
The variables tend to move in opposite directions. 27
Now the emphasis id is on the words "tends to" - when x is high y tends to be low. 28
You see correlations have a like a strength to them, a magnitude - correlations can be strong or they can 29
be weak. If the value of x is high then the value of y can be low but it doesn't have to be. 30
But you know what? I'd say this graph that we're looking at here is not an accurate representation of 31
how I actually feel on the London Underground. 32
The correlation with my happiness is actually much stronger than what I've represented here. 33
Wanna have a guess as to what this graph would look like if these two variables were in fact perfectly 34
negatively correlated? At the moment we kind of see a bit of a band, a cloud of data points as the correlation 35
gets stronger. 36
This cloud of data points actually becomes more and more narrow. In the extreme, 37
all the data points would line up to pretty much form a straight line. 38
Now this is a much more accurate representation of how I feel about British transport infrastructure 39
from the days of Queen Victoria. 40
Okay, so we've looked at a picture of perfect correlation, perfect negative correlation at that. 41
What do you think the graph would look like if the reverse was true? What would this graph show if there 42
was no correlation between the two variables at all? 43
It would actually look like this. 44
Remember how I said that the cloud of data points narrowed as the strength of the correlation increased? 45
If there is no correlation then we get a plot that looks somewhat like this. 46
We get a cloud of points with no clear pattern. 47
In other words, the number of divorces in the state of Maine are uncorrelated to the amount of margarine 48
consumed. 49
Now so far I've been explaining the intuition behind correlation in a very visual manner. 50
However, you'll typically see correlation written about in mathematical notation and the notation for 51
correlation tends to look like this. 52
We tend to use the Greek letter rho for correlation. 53
Also the correlation is actually calculated as a single number -1 one and 1. -1 54
is a perfect negative correlation and positive 1 is a perfect positive correlation and a correlation 55
of 0 means that there's no correlation at all. The variables are uncorrelated. 56
Okay so I hope that makes sense. 57
Now we know that the correlation is a statistical measure of a linear relationship between two variables. 58
Why does correlation matter? 59
Why should we care about correlation? 60
Why should we look at the correlations of our features during the data exploration stage? 61
The answer is is that we primarily care about two things. 62
One, the strength of the correlation and two, the direction. Since our goal is to be able to predict house 63
prices and build a valuation tool, our model should include features that are correlated with house prices. 64
We want to include features in our model whose movement is associated with a big movement in the house 65
prices. 66
We want magnitude. 67
We want a correlation that is not close to zero. 68
So strength is important because it tells us how much correlation there actually is. 69
What's the extent of the movement? 70
The other thing that we want to know of course is the direction. 71
If there is an increase in x does y go up or does y tend to go down. The direction is important because 72
it tells us if the moves are in the same or in the opposite direction. 73
In other words, is the correlation positive or negative. 74
But enough talking. 75
How do we actually find these correlations? In two ways - by calculating them and by visualizing them in 76
Jupyter notebook. Let's do that now.