Hello everyone and welcome to the linear regression with Python lecture.
Now that we've learned about the theory behind linear regression Let's go ahead and use sikat learned
to implement our own linear regression models.
We'll start off by working with a housing data set trying to create a model to predict housing prices
based off of existing features.
Just linear regression is our first machine learning algorithm will work with some artificially created
data sets later on we're going to progress to using real and mesir data sets from Kaggle and other sources
.
But right now we don't want to focus too much on cleaning or data we'll focus on that aspect later on
.
Right now we just want to get the basics of creating the model correct.
Let's go ahead and Jump's a notebook to get started.
All right.
Here I am at the Jupiter notebook as a quick reminder you can find all the notebooks and all the data
for this section of the course on your machine learning sections folder under the linear regression
folder.
We're going to be working with the USA Howsabout CXXVI file and the lecture notes are under Leerie Russian
with Python.
This linear regression book is the one I just created and I'm working with now.
I would also recommend that if you're working along with this lecture you create your notebook in the
same folder.
Let's go ahead and get started.
First thing we need to do is our imports.
What's going to happen is we're going to say import pandas as PD and we're going to say important umpires
and P and then we need to do our visualization libraries go out and say import map plot lived high plot
as Piazzi and import seaborne as Asinus and then in order to see these visualizations in the notebook
I need to specify map plot lib in line.
OK.
Let's go ahead and check out our data.
I'm going to create a data frame called DMF and say PD read cxxviii and call us say underscore housing
and then let's go ahead and check that the head of the data frame and this data frame we have a couple
of columns we have every row representing a particular house at a certain address which is the last
column and the house sold for a particular price.
And then the other columns represent statistics for the city or area that the house is located in.
Such is the city's population.
The average number of bedrooms for houses in that area the number of rooms on average for other houses
in that area the average house age for houses in that area and then people's average income for the
houses in that area.
So again these first couple of rows to start with average area are corresponding to the location or
City of that house.
And keep in mind that the state is artificially created which is why we have such significant digits
after the decimal point usually are not going to be able to have that much accuracy.
Let's go ahead and explore this data just a little bit.
What I would always recommend you do is check out the info method on your data frame and that's going
to inform you of the total number of columns and the total number of entries as well as give you hints
or statements on what type of objects are in the data frame.
The other thing I would also recommend you do is to called the scribe method off your data frame in
order to get a quick account lips.
Describe a quick count of some statistical information of your data frame such as the mean minimum maximum
certain deviation etc..
Of all the columns at least in America ones notice here the address column is not available because
that was a string value and then another quick note.
If you ever need to reference with the column names are you can just say DFA columns and return back
a list of the column names and notice here that there's spacing in between some of the words here.
Let's go ahead and do some simple plots to check out data.
One very simple plot you can do if the data isn't very very large is seaborne spare plots and then you
can just passen the entire data frame.
And this will automatically create not just histograms of all the columns but also correllation scatter
plots.
And this is what that looks like.
You can see the histograms looks like everything's more or less normally distributed except for the
average number of bedrooms.
And you'll notice that the average number of bedrooms is segmented into two three four five and six
which kind of makes sense.
There's some noise around there but you can't have usually 2.5 bedrooms which is why this sort of behavior
shows up.
OK let's go ahead and just check out a distribution of one of the columns.
You'll probably usually want to check out the distribution of your target column or what you're trying
to predict in this case.
We're trying to predict the price of the house.
So let's go ahead and check out the distribution of the price OK it looks like the average price is
falling somewhere around the 1 million mark somewhere between 1 and 1.5 million and everything looks
to be normally distributed from there.
That won't always be the case as far as pricing goes or any other target variables they may not be normally
distributed.
But this is a nice case since we're not really working of cleaning this data.
And then something else you may want to do for a data set is to do a heat map of the correlation between
each of the columns.
You can say DFT c o r r to get the correlation matrix.
Let me go and just show you what that looks like.
The c o r r the shoji the correlation between all the columns and then you can plot that out as a heat
map by passing that in through s s the heat map.
And here we can see you'll usually see a diagonal of full correlation which makes sense because every
column should be perfectly correlative itself.
And then you also see some information based off of other columns.
Right now we get a lot of gray and low correlation amounts and you can actually see this numerically
by saying anote equals true.
And this will pass and the actual numbers for the correlation.
OK let's go ahead and shift our focus now to actually using sikat learn and training a linear regression
model.
OK.
The first thing you need to do is split our data into an X-ray that contains the features to train on
an a y array with the target variable.
In this case the price column which is what we're trying to predict will go ahead and toss out the address
column because right now it only has text formation that the linear regression model can't use later
on throughout the course.
We'll learn about natural language processing which will allow us to work with text information.
I'm going to go ahead and say DMF columns to grab my column names and then I will set X equal to DMF
and then pass and a list of the column names.
I can just go ahead and actually grab this from here and pasted it.
And those are my features.
And then why is going to meet my target variable.
What I'm trying to predict in this case is the price column.
Now that we have are x and y data.
Our next step is to do a train test split on our data.
We want to split our data into a training set for the model in a testing set in order to test the model
once it's been trained.
Let's go ahead and use sikat learn to do this sikat learn comes with a trained test pilot function and
you can grab it by saying from S-K learn da cross-validation and you can just actually do auto complete
this.
And so I cross underscore validation import.
And you want to do train test pilot.
That's something I always like to do is whenever you're working if a new function such as train to split
it's a do shift tab in the Jupiter notebook to read the documentation string for it.
You can actually expand this by clicking this plus button and scrolling down.
Basically we're going to be doing is passing an R x and y data and then specifying the test size.
You can scroll down and see a nice example here where you say extreme ex-best White trying white test
is equal to train to split X Y test size and then also a random state which we can also add in.
Let's go ahead and copy this line.
Nego and copy it.
And I'm going to paste it here and put it all on one line.
All right.
Let's go ahead and set our test size two point four and so our random state 2 1 0 1.
Let me go and explain what's going on here.
This is essentially is using tuple and packing to grab your training set and you're testing set and
you're going to have your X train and your y train and then you can have your X test and your Y test
.
Then in the train test split function you pass on your X data your y data and you could optionally also
passen your test size and test size.
It's just a percentage of your data set that you want to be allocated to the test size.
In this case we're going to go and allocate point 4 for 40 percent.
A lot of times it's also really common to do 30 percent or 0.3 then this random underscore state equals
101.
You can just pass any number here that's optional if you want it or not.
But this essentially ensures a specific set of random splits on your data because this train split is
going to occur randomly in order to make sure that your random split is the same as my random split
.
Go ahead and specify random and or score stay as equal to 101.
That way we get the same random splits.
Otherwise your results may not match up exactly like mine will still be correct.
You just won't have the exact same results.
All right let's go ahead and run this now that we have our training and testing data.
We need to create and train the model.
So the next step is saying from Eskay learn dots and linear regression is found under the linear underscore
model family.
So we say linear and you can use time to kind of autocomplete this import.
And we want the linear regression model.
OK.
Once if imported the linear regression model we go ahead and instantiate an instance of the linearly
Russian model I'm going to go ahead and make a variable called L M equals and then say linear regression
and instantiate is basically creating linearly Russian objects.
And then I'm just going to say Elham dots and then click tap to check out all the available methods
.
So all these available methods on my model what I want to do first is use fits in order to train or
fit my model or my training data and fit.
If we look at shift tab takes in an x and y and it goes ahead and fits the linear model.
In this case I only want to fit my model to my training data.
So instead of just passing x and y what I'm going to do is say X train y train and know that I get an
output telling me that the linear regression model has been trained.
Notice also that I don't set this equal to any other variable object.
This is already taking effect onto the object itself.
So I don't mean to say L-M equals L-M dot fig.
You can just say LM that fit.
OK.
Now the next step is to evaluate our model by checking out its coefficients and then seeing how we can
interpret them.
Let's go ahead and print the intercepts.
I can grab the intercept by calling my model L-M and then saying I and CRC Peetie and then at the very
end underscore or you can just use tab autocomplete to see this.
The next thing you can check out are the coefficients and the coefficients are going to relate to each
feature in our dataset.
You can grab coefficients by saying l m c o f underscore and this will return the coefficients for each
feature.
Let's go ahead and create a data frame based off these coefficients in order to kind of organize this
a little better.
So each of these coefficients relates to the columns in x or x train.
Soon go out and check that out is the same columns in X and X train.
What I'm going to go in and do is create a data frame off of these.
So NSA PD data frame passen the coefficients as my data passen the columns as my index and I'm going
to go in and say columns is equal to then it's going to say Kohath.
Let's go ahead and set the sequel to F for co-efficient data frame and whip's this needs an underscore
.
There we go.
And let's check the head of.
Actually it took the whole CEF.
OK.
So here we have a coefficient for each feature.
And if again if you want the more mathematical reasoning behind these coefficients and how they work
Mixy do the reading and I'm going to briefly explain how to interpret these coefficients.
Basically what this means if we take a look at this first one if we hold all other features fixed a
one unit increase in average area.
Income is associated with an increase of $21 and 52 cents in price.
And then for the second one we could also say something like a one year increase in the average area
house age is associated with an increase of $164 and a.
Or excuse me a hundred sixty four thousand eight hundred eighty three dollars and you can do a similar
analysis for each of these features and the coefficients.
Now you may be wondering does this actually make any sense.
Probably not because this is artificially created data but if you want to do a real data analysis you
can actually repeat this project on the Boston data set.
That is the same housing sort of data but it's actually real.
It's from a 1970s study and you can find it in sikat learn let me go ahead and show you how you can
go ahead and grab that Boston data set what you can do is you can call from S-K learn dot data sets
import load Boston and there's a link in your notebook wups load Boston near princes.
There's a link in your notebook for the documentation on the state of frame.
But basically what it does say Boston is to load Boston and then you call it you actually call that
function load Boston and then if you check out what Boston is it's essentially a dictionary with a bunch
of information on it and you can check out the description of the data set by saying Boston the C R
and this will print out the Boston House's price data set.
Now this is real housing data that you can go ahead and repeat our analysis once word of this lecture
.
If you want to do a similar analysis for house price data on a real dataset.
Keep in mind this data is quite old.
And then if you want to grab the data itself you can just say data and you'll see here you data and
then you can also grab the feature names
which are right here essentially the columns or data frame and then you can also grab the target.
And these are the target prices in thousands.
All right.
So that's just a quick aside in case you ever want to repeat this if you're not satisfied with this
conclusion which is really not realistic that as her house gets old you will get an increase in price
.
If these coefficients were negative then you would say for each increase in unit you get a downsizing
in price for the house.
OK we'll go ahead and end our first part of our analysis here.
In part two we'll discuss getting predictions from our model.
But before we go to part 2 let's do a quick review of everything we just did.
First we had to actually grab the data using pandas and he did not read CXXVIII.
We did a little bit of exploratory data analysis but since his artificial data we're not going to focus
too much on that later on we'll see a lot more in-depth exploratory data analysis but then we went ahead
and separated our data into X and Y where X is the features and Y is what we're trying to predict.
We said from S-K learned that cross-validation import train test split we went ahead and split our data
specifying a test size and also specifying a random state.
So your results are the same as what's shown in this video.
Then we said from S-K learn that linear model import the actual model linear regression we create a
linear regression object basically an instantiation of a linear regression model.
We fit that model to the training set and then we explore the intercept and the coefficient.
And you can check out the Jupiter notebook for this lecture if you want a larger breakdown how to interpret
these coefficients and basically just formatted the coefficients into a nice data frame in order to
read them and then at the end we showed you how I can access a real dataset from sikat learn itself
.
S-K learned that data sets import load Boston in case you want to repeat the sort of analysis on a real
dataset.
So we're not quite finished yet.
In part two we'll discuss had actually GRABBE predictions from our model based off our test set.
Thanks everyone and I'll see you at the next lecture.