The next feature I want you to investigate is called RAD. 1
This is a measure of the accessibility to highways for the property, and I want to challenge you to use 2
matplotlib to generate a meaningful histogram for this RAD feature. 3
This might be a little tricky and require some thought, so pause the video, play with the Python code 4
and have a think about what this feature is actually telling us. 5
Oh and, for the histogram pick a beautiful royal purple color while you're at it. 6
I'll give you a few seconds to pause the video. 7
Here's the solution. 8
Let's check what would happen if we took this code here, 9
pasted it in, 10
changed RM to RAD, changed the x label to "Accessibility to Highways" and changed the hex code to 11
a nice purple from materialpalette.com, change that here, 12
paste it in and hit Shift+Enter. 13
We get something like this. Now, 14
this looks a little strange to me. 15
It seems like the histogram's bins are hiding some information from us, because the bins for this histogram 16
seem a little too broad. If I look at the Python code, 17
"plt.hist()", 18
we haven't supplied any bins as an argument to this function call and this means we're using automatic 19
binning. 20
We're letting matplotlib decide on how to show us the histogram. 21
Maybe what we need to do is investigate what RAD actually is and how accessibility to highways is actually 22
measured. 23
For example, what are the units in RAD? 24
Perhaps we should try to understand this before creating our visualization, so let's output RAD to our 25
Jupyter notebook, so I'm going to say "data['RAD']", all caps, and hit Shift+Enter. 26
And I'm going to scroll down and just take a look at this. 27
So I've got 506 entries and all of these seem to be whole numbers. 28
So starts out with 1, 2, 3, 29
some of them have 5, 30
some of them have 24. 31
Hmm, 32
okay, so this is a contrast to the house prices, RAD is a bunch of distinct integer values and all the 33
values seem to be whole numbers. 34
A better way that we can see this and just look at how many unique values there are in the series is to 35
use the value_counts method on this series. 36
So I'm going to put a dot after "data['RAD']" and write "value_counts()" 37
and hit Shift+Enter. 38
So this gives me a beautiful summary of how many observations in this column, in RAD, have a particular 39
value. 40
So, for example, we can see that 17 observations, 41
yeah 17 properties, in the dataset have a RAD value of 7 and there's 132 dwellings 42
that have the highway accessibility value of 24. 43
So keeping this in mind and scrolling back up to the description, RAD actually refers to an index 44
of accessibility to radial highways. 45
So that's what we're looking at. 46
We're looking at an index. 47
In other words, accessibility to highways is ranked from 1 to 24. 48
1 is the value for low accessibility and 24 is the value for high accessibility. 49
In other words, a property with poor accessibility to transport scores low on this index; and a property 50
that has good accessibility to transport has a high value on this index. 51
So looking at our histogram again. 52
So what we probably want is we want this histogram to reflect these index values instead of this automatic 53
binning. 54
We want to show these index values and we don't want to bin several of the indexed values together, 55
and that's because the data in this RAD feature already has pretty much our bins mapped out for us. 56
So we're gonna use these. 57
I can modify the histogram code right here to take this into account simply by adding the bins argument 58
and setting it equal to the value 24. 59
Now let me refresh my histogram. Voila! All right. 60
So that completes the challenge, we plotted our histogram for the RAD feature and what we can see is 61
that there's quite a few properties between the 1 and 7 range on the index. 62
And there's also a whole bunch of properties for the value 24 on the index. But, you know what this 63
histogram kind of looks like? It looks like a bar chart and bar chart is a histogram's cousin. Histograms 64
and bar charts can be used to pretty much show the same information. 65
So let me show you the Python code for creating a bar chart using matplotlib as well. 66
This is another data visualization technique that's really handy to have in your tool belt. So I'm going to 67
come down here, add a few more cells and I'm gonna make use of this values_counts method, So I'm going to copy 68
this line of code and I'm going to store the output, the result from this code, in a variable called frequency. 69
"Frequency = data[ 70
'RAD'].value_counts()". 71
Now, frequency is also a pandas series. 72
You can see this if I write the code "type(frequency)", hit Shift+Enter, so data['RAD'] 73
is a series, but the return value of this value_counts method is also a series. 74
And the reason I'm showing you this is because I want to draw your attention to something. I'm going to comment 75
this out, and what I want to do is I want to access these values right here. 76
I just want to access the labels for these unique index values. 77
I can do this in one of two ways. 78
Check it out. If I say frequency.index, 79
then I'll get a collection of all these index values in my series. 80
So this is one way of doing it. I'm going to comment this out and I'll show you the second way. 81
"frequency.axes[ 82
0]". If I hit Shift+Enter, then I get exactly the same result. 83
The axes attribute of the series can also be used to retrieve the row axes labels. And the reason I'm 84
interested in these in the first place is because we're going to use these to label the x axis on the 85
bar chart that we're going to create. 86
So check it out. I'm going to comment this out 87
and then to create the bar chart I'm going to take my matplotlib object, "plt.bar()", 88
and then I have to supply two things. 89
The first is what I want on the x axis. 90
And this is gonna be "frequency.index". 91
And the second thing I have to supply for the bar chart is the height of the individual bars. 92
So this will be an argument called height and I'm going to set that equal to, 93
well this would just be the values inside my frequency variable. 94
That'll be these values here. 95
So I'm going to say "height= 96
frequency" and then we put "plt.show()" afterwards and scroll down and hit Shift+Enter. 97
And this is what we get. As it is, there's no labels on the axes and there's also the default color 98
being used. 99
So what I'm going to do is I'm going to make this bar chart a little larger. 100
Let me grab this code up here that we have, come down here, 101
paste it in. 102
I'm going to delete this line here and then I'm going to leave my x label and y label as they are and 103
hit Shift+Enter. 104
There we go. 105
So this is a bar chart, but I want to draw your attention to one thing. The neat thing about the code 106
we've just written is that we haven't had to specify the number of bins ahead of time, 107
we haven't had to write "bins=24". 108
We haven't had to hard code the number 24 for the number of bins. 109
Instead we wrote some Python code using value_counts which figured out the best way to draw the x and 110
y axes for our bar chart for us. 111
So this is a technique that you can apply to other types of indexed data as well. 112
It makes the code that we've just written a lot more flexible than hard coding particular integer values. 113
And that's a good thing. 114
You're also gonna be looking at this chart here and you might be thinking: Hmmm this looks a lot better 115
than the histogram just because it's got these spaces in between the bars. 116
Because if we look at our histogram, it kind of looks like this at the moment - all the bins all the bars 117
are jam packed together. 118
So let me give you a little challenge so you can familiarize yourself with the histogram function 119
a little better as well. 120
I want you to modify this histogram so that it's also got some spaces between the bars. 121
The trick will be to look at the documentation by say pulling up the quick documentation in the notebook 122
and looking for the right argument to supply to the function call. You can pull up the quick documentation 123
by pressing Shift and then Tab on your keyboard and hitting this little plus sign and scrolling down 124
and taking a look at this 125
here. I'll give you a few seconds to pause the video so you can find the parameter that you have to modify 126
and give the bars a little bit more of a breathing room. 127
How did you get on? Did you solve it? Here's the solution. 128
The argument that we need to specify in this method call is "rwidth". By default, 129
this has the value none. 130
But let's check out what the description says for rwidth. 131
If I scroll down in the quick documentation I can see that rwidth is an optional argument and that 132
it is a number that specifies the relative width of the bars as a fraction of the total bin width. And 133
the first time I read this, 134
that didn't make a whole lot of sense to me. 135
So what I had to do is try out a couple of different numbers and see how the chart turned out. 136
So if we write "rwidth = 1" and hit Shift+Enter and see what we get, no change. But if we change 137
that to say 0.5 and hit Shift+Enter, our histogram starts looking like this. 138
So what this rwidth argument is doing if it's set to 0.5, our bar width will be approximately 139
0.5 and on either side of the bar we'll have a space of 0.25. If we make this 140
0.7, the gaps will get smaller and if we make this 0.3 then the gaps will 141
get wider. So, in essence, you can add a value between 0 and 1 to this rwidth argument and you'll get 142
different results. 143
If I put in the value 10 then I get exactly the same as if I put in the value 1. All good? I'm going to 144
leave it at 0.5. Cool. 145
So we've looked at the average number of rooms per dwelling, we've looked at access to radial highways 146
and we've looked at the property prices in our visualizations. So both the number of rooms and the house 147
prices were quite easy to understand, right? 148
Measuring how good the transport links were on the other hand was a little bit more complex given that 149
it was measured as an index value with accessibility to radial highways. But there's actually another 150
very nifty technique that the researchers are using to capture some information about these Boston Properties. 151
You see, there's a river running through Boston and this river is called the Charles River and it looks 152
something like this. Imagine for a second that you were conducting the original research and collating 153
the Boston housing data. 154
You want to be able to differentiate between the houses that are located right on the river and those 155
that are located elsewhere. 156
How would you go about doing this? 157
And this brings us to our next challenge. And for this challenge, 158
I want you to answer a very, very simple question. 159
Tell me, out of the 506 properties in the dataset, how many properties are located on the Charles River? 160
This challenge isn't going to be about data visualization. 161
I just need a cold hard number from you. To solve this challenge, 162
take a close look at the description of the features and then write a single line of code that will 163
spit out the answer for you. 164
And also while you're at it, have a think 165
if you expect that the properties on the river will be worth more or less than properties that are away 166
from the river. Is living next to the Charles River 167
a good thing for house prices? Because we'll find out later. 168
In the meantime, I'll give you a few seconds to pause the video so you can solve this challenge. 169
Did you get it? Here is the solution. 170
So the trick was looking for the feature description that would likely contain the answers and you maybe 171
discovered that there is a feature called CHAS and this is the Charles River dummy variable 172
which equals 1 if the tract bounds the river and 0 otherwise. 173
In other words, CHAS captures whether the property is on the river or not. 174
Now let's scroll back down and write the Python code. 175
We're going to be using our old friend value_counts to solve this. 176
If I write "data['CHAS'].value_counts()" and hit 177
Shift+Enter I'm going to get the following output. 178
I can see here that CHAS only has one of two values, 0 or 1, which ties out exactly with what they've 179
said in the description. 180
0 means not on the river and 1 means located on the Charles River. 181
So the answer to the challenges question is there are 35 properties on the river. This type 182
of feature is called a dummy variable and you'll find researchers using dummy variables to capture binary 183
information. 184
So this is a good example - is the property on the river or not on the river? 185
Are we dealing with a man or a woman? 186
Are the unemployed or employed? 187
Is it a homeowner 188
or are they renting? 189
This is the kind of information that you can capture with dummy variables. 190
In other words, working with dummy variables is actually very similar to working with an index, except 191
that a dummy variable can only have one of two values. 192
Good stuff. 193
So we're really getting into the nitty gritty. In the next lessons 194
we're gonna be looking at descriptive statistics, outliers and scatter plots. 195
I'll see you there.