Ok, so we've talked about the model we're gonna use 1
and we've talked about where the term regression actually came from. 2
Now it's time to write some Python code. 3
One of the habits that I want us to get into when training our algorithms is to split up our dataset 4
into two parts. 5
We're going to shuffle all our data and divide our dataset into a training dataset and a testing dataset 6
because we want our algorithm to learn those theta parameters based only on the training dataset. 7
And this means that we can use the other part of our dataset that hasn't been used for testing, because 8
with the testing dataset we can see how our algorithm performs out of sample, how it performs on a data 9
set that it hasn't seen yet. 10
Back in Jupyter notebook I'm going to add a markdown cell with a section heading that reads "Training 11
& Test Dataset Split". There we go. And in the cell below, 12
I'm going to create two more variables, 13
the first one is gonna be called "prices" and I'm going to set that equal to "data[' 14
PRICE']", so prices is going to hold on to our series of price data and then we'll have "features" which is 15
gonna be equal to "data", 16
so our entire data frame, then dot, then I'm going to say "drop()" and then in the parentheses I'm going to have 17
"'PRICE', 18
axis = 1". 19
What did I just do here? 20
I took our entire data dataframe and I've dropped one column from it, namely our target values and to 21
accomplish this I've used the drop method. 22
So this method will return a new dataframe which I've stored in a variable called features. 23
But this data frame will not include the PRICE column. 24
The second argument here "axis = 1" is there to specify that we're looking to drop a column as 25
opposed to a row. For a row you'd have "axis = 0", for a column, 26
use "axis = 1". To split up our dataset in our notebook 27
we're going to make use of scikit's learn capabilities once more, but we're going to have to import this 28
capability, 29
we're going to have to add an import statement at the very top. 30
I'm gonna add "from sklearn.model_selection 31
import train_test_split", 32
so that's all lower case "from sklearn.model_selection import train_test_ 33
split". 34
Let me Shift+Enter on this cell and then go back to where we left off. 35
Now this train_test_split function will actually return to us four values - a 36
training and a testing data set for both our features and our targets. 37
So we'll have X_train, X_test, y_train and y_test - four values that are going to be returned from this function. 38
When a function returns multiple values, the Python syntax we'll use to store those values is called tuple 39
unpacking. 40
So let's create the variables that will hold onto the output from this function. 41
It's gonna be "X_train, 42
X_test, y_train, y_ 43
test" 44
and that's gonna be set equal to "train_test_split()", so the function 45
itself. 46
And now all we have to do is provide some arguments to this function call. 47
So what arguments does this function need? 48
Well, it has to know which features and which targets to shuffle and to split up. 49
So the first argument is gonna be our features variable which we've created above. 50
The second argument is gonna be our prices variable which we've created above. 51
The third argument is gonna be what kind of split that we want to make we want to make. Do we want to make a 50/50 split? 52
Do we want to make a 60/40 split? 53
What kind of split do we want this function to make? 54
I'm gonna go with an 80/20 split, so I'm going to say "test_size = 0.2". 55
What this means is that our test data set is going to be 20% of the total. 56
Now I could leave this function like this, but I'm going to add one more argument. 57
And the reason is is that this function will shuffle and split the data, 58
however this shuffling is random, right? 59
So my shuffle will be different from your shuffle. 60
If we want to get comparable results, then we have to shuffle our data set in exactly the same way. And 61
to ensure that you and I can do that, 62
let's supply another argument, 63
the random state. So I'm going to say "random_state = " and then we can pick a number. As 64
long as we pick the same number, 65
we're gonna be good, 66
we're gonna get the same shuffling. So I'm gonna pick 10, "random_state = 10". 67
Let me come over here and hit Enter on my keyboard, so that my line doesn't get too long and wraps a 68
bit more nicely like this. Okay, 69
so there actually quite a lot going on in this line of code because we are shuffling all our data and 70
then splitting it up. 71
The thing about the shuffling is that there is a random number generator which will generate this randomness 72
and do the shuffling. 73
The last argument that we supplied here, this random_state argument basically draws a line 74
in the sand to kind of set the starting point for the random number generator that does our shuffling. 75
If you and I both have the same starting point, 76
then we get the same shuffle. 77
Now a question you might ask is: why shuffle the data? 78
What's the point? 79
And the answer to that is that sometimes when you get a fresh dataset straight out of the box the data 80
is actually in a certain order. 81
A good analogy is a deck of playing cards. 82
Did you ever buy a new deck of playing cards? Or did you ever watch a magician perform a con trick on 83
stage with a new deck? 84
When you take off the plastic wrapper on a deck of playing cards and you take the cards out of the box 85
and you look through them, you'll notice that all the cards are in a certain order. 86
Now obviously this means that before playing a card game with this deck, you have to shuffle the cards, 87
otherwise you end up with a very, very terrible game of poker. 88
In other words, the cards dealt have to be random, 89
otherwise it defeats the purpose. And the same holds true for this training. 90
and our test datasets. The rows or the data points that are allocated to these datasets have to be 91
random as well. 92
There can't be a clear pattern in how the data points are allocated. 93
And that's why it's customary to shuffle any dataset as well. So customary in fact that our scikit 94
learn's split function does the shuffling of our rows in our dataframe automatically and why we use 95
this random state argument in the function call. Cool! And just like that we've created an 80/20 split 96
with our data from our dataframe. 97
But you don't have to take my word for it. 98
You can verify the 80/20 split yourself. 99
The percent of the training set will be the number of rows in X_train divided by the total number of 100
rows in the dataset as a whole. 101
We can calculate this using the length function, so "len(X_train)/len( 102
features)" will do this calculation and that is equal to 0.7984. 103
So very, very close to 80%. 104
If you wanted to calculate the percentage of the test dataset, 105
so "% of test data set", 106
you could do the same thing, 107
you could get the number of rows in the test dataset and divide by the number of rows in the feature 108
dataset. 109
Another way to get the number of rows is to say "X_test.shape" and grab the first element 110
in the shape attribute, 111
that's the number of rows. 112
So "X_test.shape[0]/features.shape[0]" 113
and that's equal to 0.2. Brilliant!