In this lesson we're going to talk about two more types of data structures for storing a lot of data 1
at the same time. 2
The first one that we're going to look at is called a data frame. 3
In this lesson's resources you'll find included some fresh data. 4
Let's upload this data to our Jupyter notebook. 5
So change the tabs to your MLProjects folder and then click "Upload" and then pick the 6
LSD_math_score_data.csv file. Again, you'll find the CSV file in the lesson 7
resources. After you've chosen this file, click "Upload" and then head back into your Jupyter notebook. Here 8
we're going to write the following code - we're going to write 9
"import pandas as pd", and then we're gonna use pandas to read our CSV file and we're going to store that 10
information in a variable called data. So we'll write 11
data = pd.read_csv('LSD_math_score_data.csv') 12
As you're writing this make sure you don't have any typos in the file name; all the capitalization 13
and the spaces matter, of course. Let's hit Shift+Enter and see what we get. What we're looking for at this 14
stage are no errors. Now that we've done that, we can take a peek at our data variable. So let's print 15
it out. Let's print out data. And we can do that by writing print and then within the parentheses we'll 16
put data. 17
And what you should see at this stage are seven rows and three columns. Now, just like our lists and our 18
arrays, our data variable here is holding onto this collection. However, the super neat thing here is the 19
structure of this data. We've got our data structured in both rows and columns. Rows and columns, boys 20
and girls, are what spreadsheet monkey's dream about at night. 21
Now, as a challenge, can you find out what the type of this data variable is? 22
I'll give you a couple of seconds. Here's the solution. We'll write "type", provide our variable, hit Shift + 23
Enter, and then we get the full name data is of type pandas.core.frame.DataFrame - so the 24
type of this variable is not int, and it's not float, and it's not an array nor is it a list - it is of type 25
DataFrame and that's how we'll be referring to it. We'll be referring to it by the short name and the 26
shortening is the last bit at the end of this long name. In terms of lingo no programmer would say it 27
is of type data frame they'll simply say this variable is a data frame. This is the kind of language 28
that you'll hear people use when they're referring to types. As we've said before you can think of a 29
data frame as a collection, but with a clearly defined structure - the data inside a data frame is structured 30
in rows and columns just like an Excel spreadsheet and data frames are super common in Python and you'll 31
see data frames being used in many, many places, so it's a good idea to learn a couple of the tricks and 32
a couple of the things that we can do with data frames. For example we can grab a single column by providing 33
the column name where before we were providing the index for a list or an array. So if I write data[], 34
I can put the column name between some single quotes inside these square brackets. Say 35
I take the third column and I write Avg_Math_Test_Score, when 36
I hit Shift+Enter Jupyter notebook will display to me that data inside the single column. 37
I guess I want to bring your attention to the fact that the Python syntax is very very similar between 38
lists and arrays and data frames. 39
However, instead of providing the position or the index between the square brackets, here we're specifying 40
a column name. 41
But again typos are 42
something that we have to be very much aware of when we're doing this, because if we have a typo in our 43
column name and we're trying to fetch a column that doesn't exist. 44
Say if I delete the E and I press Shift+Enter we'll get a error. In this case, 45
it is a key error and you can see that this key error brings up a whole bunch of other errors. 46
In short, Python can't find this location in our data frame. 47
This is why we have to pay a lot of attention to our spelling. 48
We get the same error when we try to retrieve data from a data frame 49
if we treat it like a list or like an array. 50
So if we were to put an index here for, say data[1], we also get a key error. 51
So even though Python handles the data types behind the scenes and they're never really explicit in 52
your face with the syntax, this is another example where you want to be aware of what type of data you're 53
working with, what type is my variable, because that will determine what kind of instructions you can 54
give to your code. 55
Let me fix this error now so we can bring up our column. 56
Just gonna hit Control+Z or Command+Z to undo and hit Shift+Enter. 57
Now let me show you how to save this data that we're extracting, the single column in a variable. To store 58
this column in a single variable, 59
all we have to do is provide a variable name, say onlyMathScores, and set it equal to 60
data[], 61
and then the column name. If we hit Shift+Enter at this point, our output will disappear. 62
But, to prove to you guys that this data is indeed stored in this variable, we can print it out. 63
So I'll write print(onlyMathScores) and hit Shift+Enter again. 64
There we go. Extracting data from a data frame is pretty useful and we've seen how to get a single column 65
out of a data frame. 66
But what happens when we want to, say, I don't know, 67
add a column instead. 68
This is a good thing to know since very often you'll be combining different kinds of data frames or 69
different kinds of data in your python code into a single table if you will. 70
Remember how we selected a column bytes name from the data frame? We used the name of the column between 71
the square brackets. 72
Let me copy this entire line and paste it in the cell below. 73
Now you'll also remember when we tried to grab a column that didn't exist, we got an error. 74
So if I tried to store the values of Test_Subject inside OnlyMathScores and hit Shift+Enter, we 75
get our key error. But if we change things around in the cell and we move data['Test_Subject'] 76
to the left hand side of the equals sign and we provide a value on the right, 77
we are giving Python a completely different instruction. If I hit Shift+Enter now, our Python code runs 78
without a problem. 79
And that's because we are saying "Add a new column with the name Test_Subject and set all the rows equal 80
to Jennifer Lopez" 81
Let's print out our data frame and see this in action. 82
Here you go. 83
Now we have four columns and all the rows in the fourth column have been set equal to the value. 84
Jennifer Lopez. 85
So this is how you can add a new column to an existing data frame. Let's talk about how to manipulate 86
the values of a column. 87
This is very, very useful. 88
If we were to do calculations on all the values in a single column at the same time - for example, let's 89
create a new column called "High_Score" and then set the values of that column equal to 100. 90
So I'll write data which is the name of our data frame. 91
data['High_Score'], and set it equal to the number 100. I'm going to hit Shift+Enter. 92
I can print out my data frame. 93
Take a look at it now. 94
And here we see that High_Score on my 13 inch screen here shifts down and is displayed a little bit 95
below. 96
But it's still just the fifth row in the data frame. 97
Now, as a challenge, see if you can figure out how to add all the values in the average Test Score column 98
to the values in the High Score Column? 99
In other words, overwrite the values that are currently stored in the High_Score column so that they equal 100
100 plus whatever is inside the column for the average test scores. 101
And here's the solution. Using the notation that we know so far, 102
we would set the existing High Score column equal to 103
the current value stored in High Score plus 104
the value stored inside the Average Math Test Score. 105
I'm going to add a print statement below this as well so that we can see what it looks like. 106
Hit Shift+Enter and here we go. 107
All the rows inside the High Score column have been updated to be equal to 100 plus whatever was stored 108
inside the Average Math Test Score column. 109
So when we look at this piece of code right here, we can see that this pattern is actually the same one 110
that we've encountered previously in the fourth cell down from the top 111
when we set myAge = myAge + 1. 112
In this case we were also using the current value of myAge, doing a calculation with it, and then overwriting 113
the value stored inside the variable with this new value. And this is exactly what's going on in this 114
line too. 115
So now that we know how to add two columns together, what if we wanted to, say, square the values inside 116
this high score column? As a challenge, 117
can you figure out how to update the data frame so that the values inside the High Score column are 118
squared? 119
In other words, we'll want to multiply 178 by itself and then do the same thing for every other value 120
in each row in this column. 121
I'll give you a few seconds to figure this out. And here's the solution. 122
We simply set data["High_Score"] = data["High_Score"] * data["High_Score"]. 123
If we print our data frame out now, we'll see the values updated in this column as follows. Now, there's 124
other ways you can do this calculation, of course, we don't have to stick to this particular syntax. You 125
can also write the Python code in this way - 126
so instead of writing the name of the column at the very end you could have written it with two times 127
signs and then the number 2. 128
And this raises the values inside the rows of this column to the power of 2. 129
If you had a single multiplication sign it would just be multiplying all the values by 2, 130
but if you have two multiplication signs, it would be raising them to the power of 2. 131
So now our data frame has five columns. 132
It's got the time delay in minutes. 133
It's got LSD parts per million. 134
It's got the average math test scores. 135
It's got a test subject and a high score. 136
Previously we've extract that a single column and stored it in a variable called onlyMathScores. In 137
these lessons, 138
I've been harping on and on about data types. 139
Would you like to venture a guess what the data type is for onlyMathScores? 140
What category does this variable belong to? 141
Well, let's check it out. 142
Let's write type(onlyMathScores), hit Shift+Enter, and there we see the type of this variable. The full 143
name of the type, 144
specifically this variable, is of type pandas.core.series.Series. 145
Now you might look at this and you might think it's a little odd, right? 146
Because the type of our data variable is DataFrame, and previously we were working with lists and even 147
arrays and yet when we extract a single column from this data frame we end up with something of data 148
type Series. 149
Now there is no need to panic. 150
A series is actually very, very similar to an array. 151
But there are a few differences which is why a series is a different category from an array. 152
For example, the key difference is that a series is always always only one column. 153
It only has a single dimension. 154
It cannot be a matrix like an array or a list. 155
A series is much, much more restrictive. 156
Also, a series can have an attribute, like a name. You'll actually see this attribute when we print out 157
onlyMathScores. Down here, 158
you'll see that the name is basically the column heading. 159
Now some of you might be asking themselves - why are you telling me this? Why is this interesting? And 160
why does it matter? 161
Well by checking up all these data types we've actually just made a discovery - we've made a discovery 162
about the nature of data frames. A pandas data frame is essentially made up of a collection of series. 163
Each column in the data frame is a series; Average Math Scores is a series, Test Subject as a series - every 164
single column is a series and together they make up a data frame. And this brings us to a point where 165
we've talked about quite a few different kinds of data structures. 166
We've introduced you to arrays, lists, data frames and series and we know that a data frame is made up 167
of series and we also know that a series can only have one column of data, while a data frame in contrast 168
has two dimensions because it has both rows and columns. Now, say instead of pulling out a single column 169
as a series from our data frame, say we want to extract another data frame from our data frame. 170
Say we want to create a smaller data frame from our existing data frame. 171
How would we do that? At the moment, 172
we've got data inside five columns and we want to create a data frame that only consists of, say, two 173
columns. 174
Say we're only interested in the LSD parts per million and the Average Test Scores. 175
How do we construct this subset? 176
Well first, let's create a list of the columns that we care about. 177
Do you remember how to do that? As a challenge, 178
can you create a list called columnList and put two pieces of data inside of it? 179
Put the LSD parts per million header and the Average Math Score Column header inside this list variable. 180
Here is the solution. 181
We'll write columnList = ['LSD_ppm', 182
'Avg_Math_Test_Score']. And that's it. 183
We've just created a list consisting of two strings. 184
Two column heading names. 185
Now we're gonna use this list to create a new data frame. 186
I'm going to call this data frame. 187
I don't know. 188
cleanData and set it equal to data[] 189
and then inside the square brackets I'm going to pass the columnList so instead of writing the name of 190
every single column that I care about inside these square brackets I just provided a list of column 191
names. And if I print out my cleanData data frame I can see what it looks like. 192
It's just a data frame with two columns. 193
Now, we've actually written some Python code in two lines that we could have done in a single line. 194
We've split out the steps where we created a list and then created a data frame using that list. 195
Oftentimes you'll see both of these steps done in a single line. 196
So we could theoretically copy this piece of code here, 197
our list of column headings, and just put it inside here, put it inside the square brackets of our data 198
frame. 199
Now I can comment out this line because we don't need it anymore. 200
And if I press Shift+Enter, we'll actually get exactly the same result. 201
So what we've done here is simply nested a list inside another piece of code. 202
The reason I'm showing you this is because oftentimes when you see two square brackets just next to 203
each other like this it can look really really scary but all it is is a list inside of something else. 204
When we're writing our code like this, 205
we're not creating an extra variable, 206
we're not creating this column list variable. 207
We've accomplished the same thing by providing the list of strings directly. Now, to prove to you that we 208
have indeed created the data frame, 209
let's print out the type of cleanData. 210
And here we see that it is indeed a data frame. 211
Now, what if we wanted to create a single column as a data frame? A data frame, 212
after all, doesn't need to have many, many columns. It could have a single column just as well. 213
And this is actually something that's very, very useful when running regressions with scikit-learn. 214
For that we actually want to work with data frames instead of series. 215
We're gonna be interested in predicting the math test scores. 216
So in this case we can write y = [[]], 217
because we're going to be supplying that list and then all we have to do is write the name of the column. 218
We're still passing in a list here, but in this case, it's a list with only one item. 219
And when do we check the type of y by writing type(y), we can see that y is indeed a data 220
frame. 221
If we weren't passing in a list and instead only had one pair of square brackets, we are passing in a 222
string. 223
And if I re-evaluate this cell, then the type for y would be a series. 224
So this is an important point - when we provide a list to our data frame, 225
we get out a data frame and when we provide a string to our data frame we get out a series. 226
So this is another example when running Python code, that it's important to keep in mind the data types 227
that you're working with, 228
even though it's happening in the background. As a quick exercise, can you create a variable called capital 229
X and set it equal to the LSD parts per million values? 230
Also make sure that X is indeed a data frame; print the values of X and show the type. 231
I'll give you a few seconds to figure this out and pause the video. And here's the solution. 232
You'd write X = data[[]], so that we get 233
a data frame out, and then we provide the column name which was LSD_ppm. To print the 234
value, 235
we simply write print(X) and to show us the type we'd write to type(X). 236
Hitting Shift+Enter, 237
you should see this. 238
We should see that X is a data frame that consists of a single column, namely the LSD parts per million. 239
Excellent. 240
So we've done a lot of work with data frames at this point. 241
We've seen how to add columns, extract columns and manipulate the data inside a column. 242
Let's talk now about how to delete a column that we added to a data frame. After all, 243
having read this scientific study from 1968 I discovered that Jennifer Lopez did not in fact sit for 244
any arithmetic tests. To delete a column from a data frame, 245
we use the python keyword "del", short for delete and we follow this by the name of the column that we 246
want to get rid of. 247
In this case the column name is Test_Subject, so we'll write 248
"del data['Test_Subject']", and then below, 249
let's print out our data frame just to see if we have indeed gone from five columns to four. 250
And as you can see our Test_Subject column has been removed. 251
So as a quick exercise can you delete the High_Score column from our data 252
data frame. You've probably guessed it - it's the same pattern as in the cell above. We write 253
del data['High_Score'] 254
Let's print out, I'll printout data below as well so we can 255
see that the column has indeed been removed. 256
Good work. 257
I'll see you in the next lesson.