In this lesson we're going to talk about Input statements, modules and packages. In the previous lessons, 1
we've actually seen and used three different input statements already. 2
For example, we had "import pandas as pd", we had "import matplotlib.pyplot as plt" and we also had 3
"from sklearn.linear_model import LinearRegression". 4
These are the three import statements that we're going to talk about in detail in this video. Each of 5
these import statements imports a Python module that we can then use in our Jupyter notebook. If you've 6
already got some experience in programming, perhaps in other languages, 7
the easiest way to think about Python modules is as frameworks or as libraries. 8
This is what they're going to be called in other languages. 9
That said, the simplest way to think about a Python module is that every file that ends in ".py" is 10
a Python module and by importing a module we get access to the variables and the functions inside the 11
file that was imported. 12
In other words, importing gives us access to a .py files contents. 13
Now, so far we actually haven't seen any files with a .py extension. 14
And why should a file end in .py in the first place? 15
Well a .py file is a file that contains Python code and to see how this all works we're gonna create 16
a file it with a .py extension together. 17
To do that we're gonna open a text editor. In fact, every Windows machine comes with a default text editor 18
pre-installed. That text editor's name is notepad and it looks like this. 19
Now you're probably not going to have the shortcut on your taskbar, 20
so go to your Start menu and then go to find all your programs and search for "Notepad" and you should 21
also find it hard. And remember, not Microsoft Word - Notepad. 22
That's what we're going with. 23
If you have a Windows machine, then open Notepad now. 24
Now on the other hand if you're on Mac, the default text that are that you've got pre-installed is called 25
TextEdit and you can find this program in your applications folder. So open this program now, 26
and create a new document. 27
If your new document looks like this then go to "Format" and click on "Make Plain Text". This is what 28
you want your text editor to look like for the next steps. 29
Now of course there are lots of other, 30
and some might say superior, text editors that are available, too. My current favorite is actually called Atom 31
and this is made by the team behind the GitHub Web site I've included the link to atom.io in the 32
lesson resources if you want to download it. It's available for both Windows and for Mac and once you 33
start using it you'll never go back to Notepad again, I promise. 34
Now that you've got your text editor open create, a variable called "theAnswer" and set it equal to the 35
number 42, then go to "File" and click "Save As". Now, navigate to your desktop 36
and instead of saving this file with a ".txt" extension, what we're gonna do is we're gonna call it 37
"life" and then append a ".py" extension. 38
This is because we want to save a ".py" file. 39
The important thing on Windows is that you have to change the Save As file type from text documents 40
(.txt) to 41
this other item in the dropdown. 42
You want to change it to "All Files" and only then can you click "Save". 43
After you've done that, check your desktop to make sure that the file that you saved really does read 44
.py at the end and not something else like .py.txt, 45
Okay? This is what we're looking for "life.py". Now, if you're on Mac and you're using TextEdit we're 46
going do the very same thing as the Windows crew. 47
We're gonna create a variable called theAnswer and set it equal to 42 and then we're gonna go to "Save". 48
You're also gonna go to desktop and then you should be able to name your file "life" and give it the extension. 49
".py". 50
Now before you click "Save" just double check that it says "Plain Text Encoding: Unicode" down here. 51
If it says something else like Rich Text you haven't gone to format and made the file plain text. So 52
after you've verified this, 53
just click "Save" and then you can quit your text editor and verify that there is indeed a life.py 54
file on your desktop. 55
And just like that we've created our first Python script file. 56
Now let's upload this file to our Jupyter notebook. 57
Go to your MLProjects folder in Jupyter and click "Upload" then navigate to your desktop where you've 58
saved your file, choose it and click "Open" and then afterwards confirm your upload here. 59
If you've been successful. you should be able to click on this .py file and verify it that it says 60
"theAnswer = 42". Jupyter notebook will open this in a new tab for you. 61
The exciting thing is that now that we've uploaded our Python script as a .py file to our project 62
folder, we can start using it in our Jupyter notebook. 63
Check this out. Back in our Python intro Jupyter notebook, 64
We can write "import life" and hit Shift+Enter. 65
If you see no error messages at this point, the import was successful. 66
Now because I have an irrational love of data types, let's check out what the type of this thing called 67
life actually is. 68
Going to write type and put life between the parentheses and hit Shift+Enter. Aha! 69
So here we find out that our life.py file when we import it is called a module. 70
Now here comes the really cool part. 71
We're going to access the code inside our life module. Right, 72
"life.theAnswer" and hit ShiftEnter. 73
Now remember, what we see now is the value inside our the answer variable that is shown below the cell. 74
How cool is that? 75
I think this is super cool. 76
It's like our Jupyter notebook has made a new friend. 77
But let's take a second to think about what this line of code actually means. 78
It literally means fetch the value of the name 79
theAnswer that lives inside the module called life. 80
The thing to note here is the Python syntax. 81
We've accessed the variable inside that life.py file using the dot notation. We've accessed 82
theAnswer by writing 83
life.theAnswer and this notation will come up again and again as we progress throughout the course. 84
Okay so now that we've just uploaded our own file and then imported it into our notebook. let's draw 85
a link to the previous lessons. Now, 86
you'll probably remember that to 87
import pandas 88
we didn't have to upload any .py files - we just wrote 89
"import pandas". 90
And then it just worked. 91
How is it that we can actually do that? 92
See I used to think it was magic but then sadly I discovered that it was not in fact magic. 93
It turns out that when we installed Anaconda, a whole bunch of modules were already included in our installation. 94
And this is why there was no need to upload any pandas.py to our project's folder. 95
Notebook already knew about pandas and we could just write "import" pandas. But this should make you suspicious, 96
right? 97
It begs the question - what kind of other stuff does Jupyter already know about? 98
So let's take a look together at what modules we have access to out of the box from installing Python 99
Anaconda. We can get hold of this information in various places. 100
The first place that we're going to check is the Anaconda Navigator. 101
When you open the Anaconda Navigator, you'll notice that on the left you have this tab called "Environments". 102
And when you click on it, under "root", you get a long, long list of all the modules that you have access 103
to. We can even filter on installed versus not installed and we can also filter by name. 104
So if I type in pandas I can see that there's a checkmark next to it. 105
So pandas is one of the installed modules, which means that we can access it easily by writing an import 106
statement. 107
Now these filters in the Anaconda Navigator are actually super handy because we can quickly figure out 108
if something was installed or not installed. But if we look closely at this window, you'll notice that 109
it's not referring to the items in this list as modules. Instead, it's referring to them as packages. 110
And it says I've got 195 packages available and this brings me to my next 111
point. 112
When we were writing our import statements and we looked at the type of thing that we imported it was 113
called a module. 114
However when people talk about pandas or scikit-learn they'll use another word, they'll use this word 115
package, just like the Anaconda Navigator. Even on the Anaconda web site, they talk about Anaconda having 116
1000+ data science packages. Since you and I both installed Anaconda on our own computers, 117
we actually see what these packages actually look like because pandas has to live somewhere on our hard drive 118
after, all right? 119
I mean, don't tell the World Wildlife Fund, but you and I are not gonna go on a hunt for pandas. When we 120
installed Anaconda on our own machine, 121
it would have created a folder somewhere. On a Mac, 122
that folder will be under your home folder, so we can go to "Go" > "Home" and then get take to our home folder. 123
There we'll have a folder called "anaconda" and then inside the anaconda folder with all the anaconda related 124
stuff, 125
we have a folder called pkgs - pkgs, 126
of course stands for packages. On Windows, that Anaconda folder will be hiding elsewhere. 127
So you'll have to go to your local disk. 128
This will probably be your C drive and then you'll have to go to Users then you'll have to go to your 129
account. 130
So mine's called "P.D." and inside there you'll have a folder called Anaconda3. Once inside your Anaconda 131
folder, 132
you're gonna have to venture deeper to find the pandas. 133
You've got to open your lib folder and then you'll have to open the site-packages folder and there you're 134
going to get the list of all the packages that came with Anaconda once you installed it. 135
Now this is exactly the same list that you saw in the Anaconda Navigator. And looking at this list, you'll 136
probably realise that even though the Anaconda team brags about 1000+ packages in Anaconda, we're 137
actually super lucky that by default it doesn't install all of them on the machine from the get go. 138
So now that we've successfully hunted down our pandas folder, let's return to the original question - why 139
is pandas referred to as a package? 140
Well, as you can see when we look inside the pandas directory, pandas is not a single .py file. 141
Instead, pandas is actually a whole bunch of .py files that are grouped together into a single bundle. 142
In other words, a package is just a collection of files. 143
And when we refer to pandas as a package, we are referring to this bundle of files that is living on 144
our computer. Because, after all, the sad fact of life is that there just isn't any magic. 145
And, to be honest, we don't actually deeply care about what all of these individual files do exactly. 146
That's all under the hood. 147
But I wanted to show it to you guys nonetheless. At the end of the day all we need is an easy way to 148
refer to all of this code inside this pandas directory. 149
And this brings us full circle to the Jupyter notebook, because our Python code with that one import 150
statement allows us to gain access to all the files in that pandas directory with a single line of code. 151
But this story doesn't end here. 152
The packages that we downloaded as part of Anaconda aren't the only modules that we have access to actually. 153
Some modules are built into the Python language itself. 154
And these built in modules include everything on this list. 155
This list is the so-called standard library. 156
And there are over 200 modules here. 157
So it's a pretty massive list. Scrolling around on here, 158
we see that there's one module called math. 159
Let's take a closer look at this math module. The math module is something that we can use to do all 160
sorts of calculations. 161
And this module actually has information on two things, namely the value of pi and the number E. 162
So, as a challenge, as an exercise, can you import the math module into your Jupyter notebook; 163
and after you've imported the math module can you write the Python code to print out the value of pi 164
below the cell? 165
And then, after that, write some more code to print out the value of e as well. 166
I'll give you a few seconds to figure this out and pause the video. And here's the solution. 167
And just like we've imported pandas we simply write "import math". And to show the value of pi below the 168
cell, 169
I can write "math.pi", hit Shift+Enter and there it is. There's the value of pi as per the math module. 170
And looking at the official Python documentation, we can see that we can access the value of e using 171
this code right here. In our Jupyter notebook, 172
this means that I can simply write math.e and hit Shift+Enter to complete the final part of the 173
challenge. 174
So that's pretty cool, right? 175
Using the module and the dot notation we can access any of the variables inside the module through their 176
name. 177
This is exactly the same syntax that we used when we uploaded our own .py file and accessed the 178
answer by writing "life. 179
theAnswer". And this goes to show that we can access a lot of code from other modules like the math module 180
using the same dot notation. 181
Now let's take another look at this pandas import statement. Because this import statement that we wrote 182
with pandas was a little more complex than the one we wrote with "import math" or "import life", the pandas 183
import statement had something a little extra we wrote "import pandas as pd" and Jupyter is highlighting 184
the as in green. 185
So the syntax highlighting is telling us that the as in the line is a Python key word. 186
So what does this 187
"as pd" part do? 188
Why is it in there? 189
If you remember, we actually saw this pattern again with the other import statement that we used further 190
down "import matplotlib.pyplot as plt". 191
The answer is is that all this "as pd", "as plt" code is doing is giving an alias to the import. 192
Why do we need an alias? 193
Well, when you think about it the module math only has four letters. 194
It's not a lot of typing it's pretty short, but this module matplotlib.pyplot has about 17 characters. 195
And the thing you'll learn pretty quickly about programmers is that programmers are lazy, and a programmer 196
doesn't want to type out 17 characters every time they're referring to their plot. 197
So instead we wrote "import matplolib.pyplot as plt". 198
This created a variable called "plt" which referenced our 199
matplotlib.pyplot and allowed us to use a short name instead; plt became the alias for 200
matplotlib.pyplot. And, 201
we didn't have to call it plt by the way. We could have called it anything we wanted to. 202
We could have called it 203
Jennifer. But you'll pretty quickly discover that there are certain conventions in Python programming 204
and people like to stick to convention. 205
For example pandas is usually given the alias pd, 206
for some reason, don't ask me why. 207
So now that we've talked a little bit about this as keyword to create an alias let's put it into practice 208
so that it really sinks into memory. 209
We're gonna use this as keyword with our life module because we can give that module an arbitrary alias 210
too. Check this out if we write "import life as hitchhikersGuide" 211
and hit Shift+Enter, 212
then we also have to update the cell below. 213
And the reason is that we can no longer refer to this module as life because, well, we've given it an 214
alias. If we were to re-evaluate this entire notebook by going to "Kernel" and going to "Restart & Run 215
All" and scrolling down we see that we can no longer refer to our module by the name of life. 216
We've given life an alias, so we have to use that alias. 217
We can no longer use the other name, meaning we have to replace this with hitchhikersGuide. 218
And if I hit Shift+Enter now, this arrow disappears. 219
The same is true with the cell below. 220
Because our program crashed, these cells were actually never evaluated. 221
If I hit Shift+Enter here again, we get this name error and we have to replace life.theAnswer 222
with hitchhikersGuide.theAnswer. 223
Now our Python code will run without crashing. So I hope this exercise made it a bit more clear as to 224
how this as keyword works in practice. 225
Once we have an alias, we have to use the name of the alias to refer to the module. 226
Now let's take another look at the final import statement from the previous lessons. This import statement 227
read "from sklearn.linear_model 228
import LinearRegression" and the syntax highlighting here is telling us that this from is also a Python 229
keyword. 230
So without me showing you this in the video right away, can you as a challenge structure our life.py 231
module to import the answer using the from keyword? 232
I'll give you a few seconds to pause the video. 233
And here's the solution. 234
The pattern is exactly the same - we would use "from life 235
import the answer" and hit Shift+Enter. Now a very, very reasonable question that you will probably want 236
to ask at this point is "How is this different from the other import statements?" 237
I mean, why are there three ways of importing things? 238
Can we just keep things simple? 239
And the answer to that is "Well yeah". 240
So the three import statements are all doing very, very similar things. 241
And the reason I'm showing you these three ways of doing things is because people out there in the wild 242
are using all these three ways of writing their own Python code. But there's also a subtle difference 243
with this from keyword and that difference to the other import statements that we've used so far is 244
that in this case we're copying out the answer from our life.py file. 245
The answer is now its own variable inside our Jupyter notebook, because we've just made a copy of it 246
and this means we can do stuff with it. 247
For example, we can create another variable and set it equal to theAnswer. 248
So if I write "myFavouriteNumber 249
= theAnswer" and then print out myFavouriteNumber 250
and hit Shift+Enter, it prints out 42. But I can also manipulate the answer itself. 251
So if I write "theAnswer = theAnswer+1", hit Shift+Enter and then print out the value, 252
we can see that this variable now updated to 43. The value, by the way, inside the .py file is still 253
unchanged, but the copy that we've made inside this notebook has updated to 43. So, in summary, the from keyword 254
copies out the answer from our life module and makes it accessible as a variable inside our current 255
notebook. 256
And this is different from the other import statements where this doesn't happen, because if I scroll 257
up and we look at the cell up here we have to access the answer by writing "hitchhikersGuide.the 258
Answer". 259
So we have to use this dot notation to get hold of our variable. 260
Now let's compare this "from life 261
import theAnswer" and "myFavouriteNumber = theAnswer". 262
Let's put this in context of the code that we've written previously. 263
When we were estimating the movie revenues we had "from sklearn import LinearRegression" and then we 264
set regr to LinearRegression. 265
In other words what we did was we copied an object called LinearRegression over into this notebook 266
and then because it was too long to type every time we stored linear regression in a variable with a 267
short name, namely rigour. 268
Now I also want to come full circle. 269
This is a really good point to connect the dots between the lesson on data types, 270
the lesson on collections and this lesson on modules. 271
Remember when we check the data type for our regression intercept and when we read the full name of 272
the data frame. 273
Well, guess what? 274
Looking at the full name of the data types, you can actually see the module name where this data type 275
came from. 276
So for example, the data frame came from the pandas module and our ndarray came from a module called 277
numpy. 278
In other words numpy was the module, where this ndarray was defined. 279
Okay, so we're slowly coming to the end of this lesson on modules, packages and imports and we've covered 280
quite a few important concepts here. 281
So let's review them before we move on. 282
For starters we learned that Python scripts live inside files ending in .py and we also saw how 283
we can import existing code as a Python module into our Jupyter notebook projects using these import 284
statements. 285
We saw that there's a couple of different ways we can import a module. 286
The simplest way being just to write "import" and then the name of the module. 287
For example, "import math". The next level up was using the "as" keyword in the import statement. 288
So in this case we import the module but we create an alias - 289
we create a new name to give the module, and we can use this name to refer to it later on in the code. 290
This is what we did when we wrote "import life as hitchhikersGuide". 291
Every time we wanted to access the code inside one of these modules we had the module name, 292
then the dot, and then whatever we wanted to access inside the module. 293
In this case we had hitchhikersGuide.theAnswer and later on we had math.pi. 294
Finally, we saw how we can use the from keyword to copy out a specific piece of code from another module. 295
And this is what we did with "from life 296
import the answer" and this is also what we did previously with "from sklearn import LinearRegression". 297
So at this point you might be asking yourself - why does any of this matter? 298
Why is this a big deal? 299
And I suspect that you're probably already starting to see the power of these import statements and 300
how they're a massive game changer. Because by importing a module we can use other people's code 301
and this saves us the work from having to write the code ourselves. 302
In the previous lessons we accessed the linear regression capability through a module called "scikit- 303
learn". 304
We didn't actually have to write the code that implements the specifics of linear regression. 305
All we did was import the functionality and use it. 306
The people who created scikit-learn essentially created a reusable software component - their linear 307
regression code can be imported and used by everyone around the world for free. 308
So hats off to the authors of scikit-learn. 309
These guys and girls worked their butts off to create the code that we are using every time we import 310
scikit-learn. When we imported our life.py file, 311
well we only had a single line of code in it. 312
In contrast check out the code that we imported when we wrote "from sklearn import LinearRegression". 313
This code is actually open source so you can look at it and read it on github.com or you can actually 314
open the .py file itself from the scikit-learn package on your computer. 315
Now this file alone has about 500 lines of blood, sweat and tears and that's just a single .py file 316
inside the sklearn package. 317
In other words there's loads and loads more files just like this that are part of scikit-learn. I guess 318
the reason that I've climbed onto this soapbox is that I want to get this idea across that we can do 319
all this cool stuff so effortlessly because we're standing on the shoulders of giants. And on that note, 320
let's set the stage for analyzing the effects of drugs on math test scores and let's import our plotting 321
module matplotlib and our linear regression to draw our best fit line. Let's write "import matplotlib 322
.pyplot as plt" and "from sklearn.linear_model import LinearRegression" and 323
then hit Shift+Enter. 324
No need to import pandas by the way we've already imported it as part of a previous lesson. So I'll see 325
you in the next tutorial. Upwards and onwards!