In the previous lessons we've imported our Boston house price data set into our Jupyter notebook. 1
Now that we've successfully gathered our data, it is time to take a good hard look at it. 2
And this is really the next step in our workflow. 3
We've formulated our question. 4
We've gathered our data. 5
Now it's time to explore our data set in depth. 6
And oftentimes we're going to be exploring, visualizing and cleaning this data more or less at the same 7
time, simply because the problems that the data has only becomes apparent after you start digging into 8
the data set. Okay, 9
so imagine that you're back at your real estate job and that the office intern has just plonked down 10
a big data set on your desk. 11
What are the first things that you'd want to understand about a fresh dataset? 12
What's a good starting point to get to grips with a data set that you've never seen before? 13
What are the kind of questions that you'd want to ask when you're first starting out with your work? 14
Let me show you my own starting point. 15
Let me show you the first six questions that I ask myself when I first start working with a new dataset 16
that I haven't seen before. 17
The first question I'm going to ask myself is: Where does the data come from? 18
What's the source of the data? 19
The second question is: Can I find some sort of short description of what's in the data set? 20
And this is important for understanding the all important context under which the data was collected 21
and also how the data was collected. 22
Third - how big is the data set actually? 23
How many individual data points are there in the data set? 24
Am I dealing with an enormous data set or a small one? 25
This is important from a practical point because working with a dataset with 10 million data points 26
will require very different techniques than working with a data set of say 10 data points. 27
For starters, my aging laptop will totally struggle to crunch like a huge data set so it's important 28
to figure out what sort of beast you're gonna be dealing with. 29
But this aspect of data set size it's not just important from a practical point of view, but it's also 30
important from a theoretical perspective because many statistical tests that you're going to be using 31
will become a lot more powerful as the sample size increases. The fourth question is: how many features 32
are there in the dataset? 33
Now what I mean by features? Well, for each data point, how many aspects were measured, how many entries 34
are there for each row in the table, how many columns are there - this is what I mean by features. You and 35
I are going to be looking at house prices shortly. 36
And each house is gonna be in a row in this dataset. 37
The number of features will tell us how much information we have about each house. The number of features 38
will help us figure out how many characteristics we're going to be basing our prediction of the house 39
value on. The next two questions we're going to ask ourselves are: "What are the names of the features?" 40
and "What are the descriptions of each feature?". 41
The reason these questions are very crucial is because we need to understand what the dataset is actually 42
measuring. 43
So sometimes you'll get datasets with pretty unintuitive measurements so it's important to kind of dig 44
in and understand what exactly is contained in the data. 45
For starters, you'll probably want to check the units that are being used in each column. 46
So for example, is our price given in dollars or in thousands of dollars? 47
And these are just some of the basics to get right. Okay, 48
so now that we've got our to do list, let's return to the Python code and see if we can answer these 49
initial questions and check them off one by one. 50
There is a handy little Python function called "dir" that we can use to look at a Python objects attributes. 51
Check it out - "dir(boston_dataset)" and Shift+Enter 52
will bring up the following output. 53
They'll show us a list of attributes. What we're looking at here is actually a list of attributes for 54
this Python object. The first attribute here is a shorthand for I'm guessing description. 55
So let's pull this out. 56
Let's print this out. 57
I'm going to say "print(boston_dataset.DESCR)", all caps, and let's take a look at what we 58
see here. 59
So using this Python attribute 60
We do indeed get a description of this dataset. 61
This description was included in the Python object and we were able to access it with this attribute 62
that we discovered through the dir 63
function. Okay, so let's take a look at these notes. 64
We've already seen that there are 506 instances or rows in this dataset and there are 65
13 attributes, 13 categories or columns. 66
Now these 13 columns are as follows; 67
we've got per capita crime, 68
we've got concentration of nitric oxides - so this is a proxy for pollution, 69
we've got the average number of rooms per dwelling, 70
we've got a pupil - teacher ratio by town, 71
so this is a proxy for the quality of the schools and a whole bunch of other attributes. 72
Scrolling down a little more we see that the two researchers that collated this dataset are Harrison 73
and Rubinfeld and that this is based on a research paper. 74
In fact we can actually see the title of the original research paper here 75
"Hedonic prices and the demand for clean air" and this was published in The Journal of Environment Economics 76
and Management Vol. 5 in 1978. 77
So this already answers a lot of the initial questions about our dataset. 78
It's actually fairly interesting that the original purpose of the researchers was to figure out how 79
high the demand was for clean air in Boston. 80
That's what they were trying to accomplish. 81
They were trying to figure out how much more people are willing to pay to be able to breathe clean air 82
in the city. 83
The other thing that's important is that this housing data dates back to 1978 and we're working with 84
506 different entries. So let's check off the questions on our list. 85
We've figured out the source of the data. 86
We've read a brief description of the data set. 87
We've also managed to figure out the number of data points in the dataset which was 506 88
and the number of features which was 13. 89
And lucky for us the descriptions of the features was also given. 90
They call them attributes, if you remember, and the names of these attributes were given in all caps, they 91
were abbreviations. And finally we had a very brief description of all the features as well. 92
So I think for starters this is pretty good going, 93
but let's crack on. Now as an aside if you're curious on how this dataset was originally used you can 94
actually pull up the original research paper that is mentioned in the description. 95
So I just googled it and I got sent to the University of Michigan library Web site and there I was able 96
to pull up the PDF for the original paper for free. 97
And I think that's because Daniel Rubinfeld was actually at the University of Michigan while his co-author 98
David Harrison was at Harvard at the time. 99
Now if you Google this as well, let me show you how you can embed a link in your Jupyter notebook very, 100
very easily. 101
So I can copy the URL here, 102
go back to my Jupyter notebook and then going into one of the markdown cells, say the gathered data cell 103
I've got here, I can use some square brackets and some parentheses to insert my URL. So the URL 104
goes between the two parentheses. 105
So I'm gonna paste it in here. 106
This is the URL that I copied from the other tab in my browser, and then in the square brackets I can 107
include the text that I want to display instead of this long and unwieldy URL. 108
So I'm going to write "Source: Original research paper" and when I hit Shift+Enter it's gonna be displayed 109
like so and this is an active link that we have now in our Jupyter notebook. 110
Now you might not always get a nice description along with your data set like this. 111
So let's have a think about how we might look at the number of data points and the number of features 112
manually in case it wasn't presented to us on a silver plate like this. Going down to the bottom, 113
I'm going to insert another markdown cell and I'm going to add a subheading here and this is going to read 114
"Data points and features". To look at the number of features in this dataset, I'm going to first access 115
the Bunch object's data attribute. 116
I remember seeing this up above 117
when we use that the dir function. So we can write "boston_dataset.data". 118
And this is what this would look like. From the output we can see that it's an array and we could verify 119
this by writing "type" and then surrounding "boston_dataset.data" by parentheses. 120
And here we can see that it is in fact a numpy n-dimensional array. 121
This is the type of object that we would be accessing like so. Now if we wanted to see the number of 122
rows and columns, the easiest way to do this is by writing "boston_dataset.data.shape"; 123
shape is an attribute of a numpy array 124
if you recall. So hitting Shift+Enter, we can see that this array has 506 rows and 13 columns, which is 125
good because it ties out with the documentation that we read earlier. 126
The thing that you'll also notice in this line of Python code is that we are chaining our attributes. 127
I'll just add this as a comment here on the right hand side. 128
Now this is something quite important to understand, because this is a good example that shows how objects 129
and data can be nested inside one other. Scrolling to the very top, 130
we see that when we are working with the boston_dataset, we are in fact working with an object 131
of type Bunch, and this bunch has a number of attributes, including that data attribute. This data attribute 132
is in fact an object of type ndarray, a numpy n-dimensional array. And the n-dimensional array in 133
turn also has attributes, including a shape attribute. 134
And when we call on an ndarray's shape attribute, we get back a tuple. So when you see this dot notation 135
being used to chain things together, you can think of it almost like a Russian matrioshka doll, where 136
each doll contains another object. And if that's not your kind of thing then think of it maybe like the 137
movie Inception where you had dreams within dreams, just with less gunfire. And on that note I'll see you 138
in the next lesson.