In this lesson we're going to learn about Data Visualization and Data Exploration. 1
So, why visualize our data? Well, graphs and charts are incredibly powerful and very useful in being able 2
to understand what it is that we're actually dealing with because there's actually only so much that 3
you can learn from looking at a table of numbers. 4
Very often you want to put it up on a chart so that way you can spot problems, you can spot outliers, 5
you can spot trends and you can spot patterns. 6
Pictures are incredibly informative, so let's create this chart in something called Jupyter notebook. 7
Head on over to a website called jupyter.org. 8
If you scroll down here then there's a button where it says "Try it in your browser", and this will take 9
you to jupyter.org/try. 10
I want to put a link to this very site in the course resources so you'll find it there as well. 11
Once you're here click on "Try Jupiter with Python". 12
And after a little loading time then you should be dropped off on a screen like this. 13
Now if you didn't end up on the page that I showed you just now and instead you got this message then 14
you can do one of two things: either you can skip ahead and see how we can install Jupyter or you can 15
wait a few minutes and just refresh. 16
And that should drop you off somewhere like this. 17
So, why do I like Jupyter and why are we using it? 18
Well this is an incredible tool. 19
It's very powerful. 20
It's very easy to use and it provides instant feedback. 21
When we're writing our Python code Jupyter works on something called cells. 22
And you can execute the code in these cells individually. 23
Let me show you what I mean. 24
So with this little plus sign here you can insert your cells and if you type something in one of these 25
cells, say 5+5 and hit Shift+Enter on your keyboard, then this cell will execute and you'll 26
see the output below the cell. 27
Alternatively, if you don't wanna use the keyboard shortcut there's also this Run button up here. 28
So let me just change this to 5+6 and press Run then you'll see this cell being executed and 29
the output being updated. 30
The other reason I really like Jupyter notebook is because it's fantastic at allowing you to present 31
information because the cells are actually all not created equal. 32
Some of these cells only present you with information like these ones and these are so-called markdown 33
cells and other cells are actually cells that expect Python code like these ones and you can see this 34
in the difference in the formatting. 35
Now let me show you how to start a notebook of our own. 36
If I click up here what says Jupyter just on the logo, I could take into something that looks a little 37
bit like a Windows Explorer or Mac finder. 38
I've got folders and files on him. 39
And here I can click on New and say New Notebook and I can specify that I want a new Python 3 notebook 40
and this will open a new tab in my browser with a fresh notebook that we can use for this project. 41
So let's get this notebook a name instead of it being called Untitled let's call it Linear Regression 42
and then we'll hit Rename. 43
And if you change tabs back to the first tab here, you can see this notebook appear in green right here. 44
So, how do we get our data from our spreadsheet into our Jupyter notebook? 45
Well there's a small button here called Upload and if you click on it then you can upload a CSV file 46
to Jupyter. 47
Now you can either download your data from the Google Sheet in a CSV or a comma separated value format. 48
But what I'd recommend is to download the cost_revenue_clean.csv file that 49
we've provided in the course resources. 50
This way we can ensure that we're all working from the same clean data. 51
And there's no accidental typos. 52
So just select this file and hit Open and then confirm your Upload. 53
You should then see the CSV file appear right here. 54
So now we've uploaded it successfully, heading back into our Jupyter notebook. 55
We now have to get the data from the CSV file into our notebook. 56
In other words, we need a way of accessing the data in the notebook itself. 57
The problem is is that our notebook is currently blank. 58
It doesn't know how to do anything yet. 59
So we have to give our notebooks some sort of functionality and we can do this by accessing code that 60
other people have already written and taking advantage of that other people's Python code is bundled 61
up in something called a module. 62
And in order to use a module in our notebook we have to import this module. The first module that I'd 63
like to introduce to you is called pandas. 64
So if you write import pandas in his very first cell and hit Shift+Enter on our keyboard or press this 65
little Run button, then we've accomplished just that. 66
Now our notebook has all the functionality that the pandas module has. Pandas is one of the key modules 67
for Data Science and we'll be using it a lot on this course. 68
It's incredibly useful from manipulating and analyzing data and the pandas module can already help us 69
because it knows how to read CSV files. 70
So let's use it to get the data into our Jupyter notebook. 71
If we write pandas.read_csv and then we open a pair of parentheses, 72
then we can supply the file name of our CSV file between these two single quotes. Our file name, 73
if you recall was cost_revenue_clean.csv and you can see that right here. 74
So I'm just going to copy this and then paste it in between these two single quotes. 75
The key thing to understand here is that we're accessing the ability to read CSV files from the pandas 76
module using this dot notation, but now that we've read our CSV file we need to store the data from this 77
file somewhere. 78
So we need some sort of object that holds onto our data like a container and we can create this simply 79
by writing data = on the left hand side and then having pandas.read_csv on the right hand 80
side. 81
What this will do is it will create an object called data which then gets a value data that's contained 82
inside the CSV file. 83
And if we run the cell we can actually look at it. 84
So let's click Run and then write data and then just hit Run again and here what we should see is all 85
the data points and all the information that was previously stored inside our CSV file. 86
The great thing about this is now we have an object in our notebook that we can use to analyze and explore 87
our data. 88
Let me show you what I mean. 89
So instead of printing out all these 5000 rows below the cell what we can do is we can put a dot after 90
our data object and tell it to describe itself. 91
So with data.describe and two parentheses at the end, and hitting Shift+Enter we get the following 92
output : we get some summary statistics on the data that's in the two columns - our production budgets and our 93
worldwide gross, we get to see how many entries were in our data frame, we get to see how many entries 94
were in these columns. 95
In this case we have 5034 entries. 96
The reason that these numbers look a little bit funny is because they're in scientific notation. 97
So for example 5.034e3 equates to 5034 and 98
3.29e7 equates to 32 million so 3.29e7 or 3.29 million 99
is actually the average production budget of all the films in our dataset and this is what 100
we see here labeled as the mean. But not only do we get the number of entries and the average of the 101
values but we can also see a number of other statistics like the minimum value the smallest value to 102
the smallest production budget was and one hundred dollars and we also see the smallest amount of worldwide 103
gross revenue which was 2600 dollars. 104
So now we've got all our data in one place. 105
We have it all stored inside an object called data but let's separate it out let's separate out what 106
we're using to make our prediction our production budget from what we're going to predict, namely our 107
gross revenue. 108
And the reason we're doing this is because we're preparing our data for the next steps. 109
We want to be able to create some charts and we want to be able to create our linear regression. 110
So it's going to be very, very nice to stay organized and have our features separated from our target. 111
So I come down here to a new cell and I'm going to create two more objects. 112
One is going to be called capital X. And the other one is going to be called lowercase y. 113
Now our X is going to hold on to our production budgets and our y is going to hold on to our revenue. 114
So we're using that equals sign again just as we did up here. 115
So now we have to think about what should these two things be equal to? What kind of object should our 116
X and our y be at this point? 117
I'd like to introduce you to one of the main workhorses of data science: 118
the pandas data frame. You can think of a data frame as a very powerful spreadsheet, like a spreadsheet 119
on steroids. 120
So the question is - how do we get hold of a pandas data frame? 121
Well we can come back up here where we're importing pandas and we can import the data frame from pandas. 122
So from pandas import DataFrame. 123
Now if you hit tab on your keyboard then Jupyter notebook should actually autocomplete some of the code 124
for you. 125
That way you won't make a typo here with say a lowercase f or something. Autocomplete is incredibly useful 126
for avoiding these small mistakes. 127
Now we can come back to our X and our y. Here, what we're gonna do is we're going to create a new data frame 128
and this data frame is going to contain our production budgets so X is going to equal DataFrame open 129
parentheses data comma columns is equal to square brackets and in the single quotes we're going to 130
provide the column name. The column name for X is gonna be production_budget_usd so you can just copy 131
this from the spreadsheet or you can type it out as long as you make sure that the spelling matches 132
exactly and then your line of code should look like this. 133
So, what are we doing here? 134
We're creating a new data frame from our raw data which we've got here and we're selecting a single column 135
from this data, namely the production budgets and that's how we're splitting up our data frame here for 136
our X's. 137
Now at this point I'd like to throw our challenge to you. 138
I'd like you to pause the video and split up the data for our y's. 139
Can you create a data frame just like this but instead of grabbing that first column can you grab the 140
other column? 141
The worldwide gross and then this information as our y. 142
I'll give you a few seconds to pause the video and give this a go. Alright, 143
Here's the solution. 144
You can actually copy the line above and paste it below and then just replace the column name with the 145
column name for the worldwide gross revenue. 146
So in this case you need to take this column name here and add it between the single quotes inside the 147
square brackets. 148
Having done this, we've separated our data. 149
We've created two new data frames one called X which holds onto our features and one called 150
y which holds onto our target. 151
So let's execute the code in the cell and see what happens. 152
So if we click Run here, then what you see is that we actually get an error. 153
We get something that says: "name 'DataFrame' is not defined". 154
And the reason we're getting this error is because our notebook actually doesn't know what a data frame 155
is. 156
But, why is that? 157
If we go up here we can very, very clearly see that we're importing the DataFrame from pandas. 158
The reason we're getting this error is because we've modified the cell, but we actually have not run 159
this cell yet. 160
In other words, we have to select this cell, run the cell and it's at this point that our notebook knows 161
about the data frame. 162
So if you come down here to where we're creating our X and our y and run the cell again this arrow will 163
disappear, the code will now execute successfully. 164
So now that we've successfully separated out our data let's visualize it and for that we need some functionality 165
from another module. That module is called matplotlib. 166
Matplotlib has a whole bunch of graphing and plotting functionality and this is the module that will 167
help us create our charts. 168
So let's import matplotlib into our notebook going to the very top where we've got all our import 169
statements nicely organized. 170
Let's add another one. 171
Let's say import matplotlib. 172
Now the specific plotting functionality that we actually need is only a small part of matplotlib 173
and this plotting functionality is called pyplot. 174
So if we import matplotlib.pyplot and hit Shift+Enter on our keyboard to execute the cell then 175
we can start creating our charts. Coming down to the bottom of our notebook, in a new cell, 176
we can now create our first chart. Let's create a scatter plot. So let's get a hold of 177
matplotlib.pyplot.scatter(X, y). 178
What this line of code will do is it will create our scatter plot for us to show our chart in our notebook. 179
We actually have to tell matplotlib to show our plot and we can do this with matplotlib.pyplot.show 180
and then parentheses at the end. Let's hit Shift+Enter and see what this looks like. After 181
our cell executes we should see something like this. 182
Now I want to do a couple of things. 183
First off, I want to simplify our code. 184
And second I want to prettify our chart so that it becomes a lot more informative. 185
The problem with the code is that it's a little bit long. 186
If you go online and you take a look at some of the documentation you rarely see code written like this. 187
What people do is they shorten this bit of the code. 188
So what we can do is we can go to our import statements at the very top and we can create something 189
called an alias for our pyplot and we can do this by simply writing as plt and this as keyword gives 190
our pyplot this short name. 191
So we're importing matplotlib.pyplot as plt and this means we can come down here and instead 192
of writing all of this and then scatter we just write plt.scatter and here we just write plt.show 193
and just to prove that this works. 194
I'm going to have to execute this cell first because I've made some changes here and then I can come 195
down here and execute this cell and what we should see is the same result as before. 196
So what can we do to make this chart a lot more informative? 197
Well we can add a title to it or we can add labels to the axes to make it a lot more clear what's going 198
on. 199
To do that we can adopt a very, very similar approach to what we did up here. 200
We had a; we had some sort of object. 201
Then we put a dot after it and then we told it to do something. Up here we we're using our data object, 202
but with our chart we're going to use plt. 203
So to add a title to our chart we simply write plt.title and then in the parentheses we provide 204
the title. 205
So I'm going to call this chart "Film Cost vs Global Revenue" and when I hit Shift+Enter to refresh 206
the charts then the title will appear just above. 207
And similarly, we can add labels to the axes. So with plt.xlabel we can add production budget in dollars 208
to the x-axis and with plt.ylabel we can add worldwide gross to the y-axis. 209
Brilliant. 210
Now, what else can we do? 211
Well first, let's make our chart larger. 212
So if I come up here and I write plt.figure I can change the size of our chart and I can do that 213
by setting the figures figure size with figsize as equal to and then parentheses and I can supply a 214
width and a height here. 215
So I'm gonna go with ten by six. 216
And if I hit Shift+Enter then my chart will update and become much larger than before. 217
And at this point what's becoming very clear is that our y-axis and our x-axis are actually going below 218
zero. 219
So there's some space here to plot anything that is a negative number. 220
But if we look up here we can actually see that for both our production budgets and our revenue 221
we don't go anywhere near zero. 222
So what we can do for our chart is we can set a range for what should be displayed on the x-axis and 223
the y-axis to customize our chart. 224
In this way we would write plt.ylim, 225
and then in the parentheses we set a minimum and a maximum value. 226
So for our y, I think we can go between 0 and 3 billion just so that we capture this data point up here 227
in the top right corner. 228
So I'll add plt.ylim and then in the parentheses 0, 3 and then billion; billion 229
has nine zeros I believe. 230
So one two three...six seven eight nine. 231
Let's hit Shift+Enter on the cell and refresh that chart. 232
And let's take a look at our changes. 233
Now that empty space below has disappeared. 234
So let's do the same thing for the x-axis. 235
And actually I want to leave that over to you as a challenge. See if you can figure out how to add some 236
code to the cell to set the limit on the x-axis to have a minimum value of zero but a maximum value 237
of 450 million. 238
That way when you refresh you should get rid of this whitespace here and we should see our chart much 239
more clearly. 240
I'll give you a few seconds to pause the video before I show you the solution ready. 241
Here we go. 242
It's very similar to what we did with our y-axis. 243
In this case we have to modify one thing in our code - instead of ylim, 244
we're gonna go for xlim and instead of three billion we're gonna go for 450 million which is 450 followed 245
by six zeros. 246
If I hit Shift+Enter on my cell then I should see my chart update like so. 247
But we can actually improve the styling a little bit further. 248
One thing we can do for example is we can add transparency to our dots here, so we can go to our scatter 249
plot and similar to what we did here with setting the figure size for the figure, 250
so the overall size of the chart, we can actually set the transparency for the dots in the scatter plot 251
and we can do that by putting a comma after the y and setting something called the alpha value. 252
So by setting alpha equal to 0.3 and hitting Shift+Enter then we'll see that our dots have 253
become a lot more transparent and this will give us a good appreciation for the number of data points 254
that we have in certain areas of the chart. 255
So down here we have loads and loads of movies and loads of data points and as we go to the top right 256
there's fewer and fewer films. 257
So where are we at now at this point? 258
We've done some excellent work visualizing our data and this has helped us spot some of the clear outliers 259
in the film world. 260
For example this film right here has had an enormous production cost and also made a lot of money. 261
This movie is called Avatar by the way. 262
If you're wondering and looking at this chart you can see just how out of the ordinary this film was 263
relative to other movies. 264
But there are also some films that have made very very little money but cost a fortune. 265
And these films are down here. 266
That said, what's very clear from this chart is that most films are actually smaller productions less 267
than say 1 million. 268
And we can see that from this very dense cloud of data points in the lower left hand corner. 269
Looking at this chart we can also start to understand more about the relationship or the potential relationship 270
between movie budgets and movie revenue because there indeed seems to be some sort of upward trend that's 271
visible in the chart and higher budget films do indeed appear to have higher revenue. 272
But how much higher exactly? But how much higher exactly and how strong is the relationship? 273
To answer these questions we will need to turn to our linear regression. Our linear regression can help 274
us quantify this relationship. 275
But before we do that in the next lesson let's save our work. 276
We've written quite a lot of code and it would be a shame to lose it. 277
And I can tell you one thing trying Jupyter online is prone to get you disconnected if you leave it 278
alone for too long. 279
So go to File and then go to Download as > Jupyter notebook. 280
And this is the .ipynb extension. That way, if you do get disconnected, your work is saved and 281
you can come back here to upload your Jupyter notebook if need to continue where you left off. 282
I'll see you in the next lesson.