Hello everyone and welcome to part two of the logistic regression of Python lecture series.
In part one we focused on exploring the data through visualizations and there are exploration we actually
found that we had some missing data.
What we want to do now is switch gears to cleaning our data cleaning our data will allow us to transform
our data into an acceptable form for our machine learning algorithms.
Let's go ahead and jump right back to the Jupiter notebook where we left off last time.
OK here I am at the Jupiter notebook if you remember from last time we went ahead and used a little
trick with train is not method.
And as soon as heat map in order to quickly create a mapping globalization to where we were missing
data we want to fill in missing data for the age column instead of just dropping all the missing age
data rows.
One way to do this is by filling in the mean age of all the passengers.
This is known as imputation and that means you could just take the average age and fill in the missing
values with the average age.
However we can even be a step smarter about this and check the average age passenger class.
Let me go ahead and scroll down and show you what I mean by this.
I'm going to go ahead and create a box plot using seaborne with X equal to the class column and y equal
to the age column in my data is going to be equal to train.
And let's go ahead and run this and check out our bark's plot.
What I'm going to go ahead and also do is set the LTE the figure 6 size is equal to.
Let's go ahead and make this let's say 10 by 7 just so it's quite a bit larger.
All right.
Now we can kind of zoom in on the center points of the sexual box plot.
We can see here that when we separate by class the wealthier passengers in the first class and second
class tend to actually be a bit older than passengers in the third class.
Now you may think that this kind of makes sense because maybe wealth in order to reach the first or
second class tears takes age or time to accumulate.
So if you have enough time to accumulate enough wealth you'll be older and more likely to be in the
first class.
Well we can do is use these average age values in order to impute the age based off of the past year
class.
Now if you wanted to you could create an entire model to try to predict age based off of all the features
.
You'll go ahead and use the sort of simple imputation which is a little better than just imputing the
average age in by imputation just mean filling in.
Let's go ahead and create a function in order to apply that function I'll say imputes underscore age
and it takes an argument.
C O L S calls.
And then I'm going to go ahead and do is essentially say age we'll go ahead Agis say capital.
Age is equal to C o S 0.
The first item there and then we'll say P class is equal to C S calls.
The second item at index 1 and I'm going to say if pd is not all of age
and then I'm going to have some if statements here.
If the class is equal to one of them in the first class I'm going to go ahead and return the average
age for passengers in the first class that occurred at 37.
And if you wanted to get the exact value what you could do is just go ahead and actually get the average
by class using pandas.
And just like say something like that mean or if you wanted to you could create a box plot using Watley
and just do it that way.
So I'm going to say return to 37 and then you'll If the class is equal to 2 meaning there in the second
class I'm going to go ahead and return 29 and then finally.
Else.
Meaning they're not in first class or not in second class but they still have is NULL for age and going
to go ahead and return twenty four which is the average value here or the third class and then else
on my outer if re here.
So if they're at the ages no I'm going to go ahead and return to class.
Excuse me an age based after class.
Otherwise I'm just going to return their age since I know it already.
OK that's the function and we're going to go ahead and apply that function.
So when I say train age is equal to train and now I'm going to an age ends the class column.
So a list of these columns apply.
And I pass in the name of the function I want to apply and then say x is equal to 1.
So I want to apply this across the columns and their ego.
It ran correctly.
Let me go in and break down real quick what's actually happening here.
I've created a function here that takes an argument calls and notice here that actually grab based off
my index values.
The age in the class and I'm able to do that because I'm using to apply these two columns and then when
I say if PD is null that age.
So at that particular point then we certify or excuse me we check by passenger class if it's first class
we return the average age for first class passenger if it's second class passenger return 29 otherwise
or the third class or average averages 24.
If they're not anal than we actually know the age and we just return age and then they went ahead and
applied that function to these two columns here.
And then I said x is equal to one to apply it to the columns.
Now let's go ahead and check the heat map again.
So say seaborne or S and S heat map
train is null and set Y tick labels equal to false.
Otherwise you'll see a ton of white labels and then go ahead and set C by equal to false.
And if we run that we get this color mapping scheme.
But what I do is set my color map to this because it's a little easier for me to see.
OK.
And it looks like now we are no longer having any missing information for the age column.
We successfully put in values that were reasonable guesses for people's age based off of their class
.
Now the issue we have is the cabin column and just honestly there's too many missing points in order
to actually do something useful of this column here.
Well you could do is try to create a new feature something that says that we know the cabin 0 or 1.
But there's just so much missing information here that it's just easier to go ahead and drop that cabin
column in order to do this.
I'm going to say trained up drop
cabin state that the x is equal to one and then go ahead and say in place is equal to true.
And if I check out train head on notice that I'm no longer going to scroll so the right have that Cabane
column and if I go ahead and run this again notice here.
Now we only have that missing value embarked since this is just essentially one row of missing information
or one or two very few rows.
I'm going to go ahead and just drop any more missing values so I'll say train but drop an A and set
in place is equal to true.
And now I should have no missing values so I go ahead and run this one more time.
Boom.
Now we have everything one solid color.
Nothing is missing.
We are good to go.
OK we've taken care of the first step of cleaning data which is dealing with missing values.
We went ahead and filled in some missing values and we went ahead and dropped some other missing values
.
Now we need to do is deal of categorical features we'll need to convert categorical features into what
is known as a dummy variable using panderers.
Otherwise our machine learning algorithm won't be able to directly take in those features as inputs
.
And what I mean by that is something like this.
If you see the sex column here we have a categorical feature of male or female.
A machine learning algorithm isn't going to be able to take in just a string of male or female will
have to create a new column in The King a zero or one value for if someone is male or not in order to
encode that information in a way that a machine learning algorithm can understand it.
And this is known as creating a dummy variable it will be a similar thing for something like the embarked
column.
The Embarq column here is just a letter representing a city and again a machine learning algorithm can't
just take that letter and immediately know what to do with it.
They'll need something in a dummy variable format which is going to indicate a zero or one value with
a column as a actual value for the city.
Let me go ahead and explain this further by showing examples of how we can actually do this with Pandurs
.
I'm going to go ahead and say PD and we're going to use the get underscore dummies method and you can
go in and click shift tab.
You want to explore his method a little more but it basically does is convert categorical variables
into dummy or otherwise known as indicator variables.
That we pass in the column that we want to convert.
So for instance the sex column male female if we just do this.
Notice what we get out.
Now we have a data frame with a column for every single category and a zero or one value which is essentially
just a boolean zero if they were not female 1 if they were female.
But likewise for a male.
Now there's one slight issue of this in that one column here is a perfect predictor of the other column
meaning if our machine learning algorithm gets fed both columns the machine learning algorithm will
immediately know that hey if it's zero female I can predict perfectly that it's going to be one male
and this is going to be an issue known as multi-collinearity and it basically will mess up the algorithm
because a bunch of calls will be perfect predictors of a another column in order to avoid this.
What you do is as another argument here you go ahead and say drop first equals true and then it will
go ahead and drop the first column.
And now this is exactly what you want to feed in to your machine learning algorithm as a column.
We'll go ahead and set this equal to sex sex.
Is this date a frame of just whether or not this person was male.
Now we're going to do a similar thing for the Embarq column and we create a data frame called Embarq
said equal to the get dummies train say Embarq.
Well go ahead and say drop first is equal to true and run this and then I'm going to go ahead and check
out the Embarq data Fringe's the head of it and look here I went ahead and dropped the C column and
now I have indicators of Q and s.
But notice one call or all the columns are not perfect predictors of each other because he went ahead
and dropped that C column.
Now what we can do is go ahead and say train is equal to the dot com cat and we're just going use concatenation
to add these new columns to our data frame.
So pass and a list of the data for want we want train and then the new Sex and Embarq data frames and
will say access is equal to 1 because we want to add them in his new columns and we check out the head
of train.
We get something that looks like this are normal columns women going to make two rows here repassing
and two our no more columns and then we go over to the right.
Notice we have these new columns.
Male Q and s and these are going to be a replacement columns for our machine learning algorithm.
These are indicator or dummy variables.
What that means is we don't really need a lot of these other columns anymore.
We don't need this Embarq column because we've already encoded it to Q and S..
We don't need the sex column anymore because we already encoded it to this male column.
Other columns are not going to be all to use are things such as the ticket column or the name column
.
There's not too much information there that we can possibly use if you want.
You can try feature engineering new features such as grabbing the first letter of the ticket maybe or
the title of someone's name but we'll go ahead and drop the column so we're not going to use.
So say train drop and I'm going to pass in the columns and news and I'm going to use sex because we
already encoded it to that mail column.
Likewise for embarked.
I'm not going to use that column because it already encoded it's that Q and S column not going to use
someone's name because there's no way to easily pass that into a machine learning algorithm.
Maybe we can feature engineer some stuff off of that.
But it's quite complicated since we're dealing with text data.
So we'll go ahead and wait so we get to something like natural language processing that gets some ideas
of how to work that sort of data and then filing it drop ticket because there's not an easy way to grab
information from a machine learning algorithm with that.
And finally I'll just say access is equal to one because these are all columns and all states in place
is equal to true.
There we go.
And now let's check out the head of our data frame.
And this looks perfect for a machine learning algorithm.
All the data is numerical everything is concatenated together and we have our columns working perfectly
.
The last thing we want to look at is this first numerical column was this passenger Id noticed this
passenger ID is essentially just an index that starts at 1.
We go ahead and check the tail of this.
It's the exact same thing just an index that starts at 1 and that means that even those numerical it's
not really useful to us essentially as an index or an act to predict whether passengers survive just
based off their position index.
We'll go ahead and say drop passen passenger ID
and say x is equal to 1 in the we'll say in places equal to true.
Now let's check the head of the state of frame one more time.
All right.
Now let's take a quick look at the columns one last time to verify everything's OK.
We have these survived column which is 0 or 1.
That's good because that's our label column.
We have the class column as the numerical indication of what class they were in.
We have the age column of how old the passenger was.
We have the sibbe espie.
How many siblings or spouses they had on board.
We have the pair C.H. how many parents or children they had on board the fare they paid whether or not
they were male whether or not they embarked.
Hugh and whether or not they embark as something interesting to know about the class column is that
it's actually a categorical column meaning it can only have three distinct categories one two or three
.
And that means we could have also done PD get dummys on that column.
I'm going to go ahead and leave that to you as an exercise to do after you've already run the model
with it as a normal column just by itself.
And the reason for that is because I want you to explore the differences between leaving it as a column
by itself of one to three versus using the PD Dockett dummies of the separation of those categories
and then you're going to be able to explore in your own how a machine learning algorithm will react
to the column being treated as a category versus a continuous variable.
All right.
That's it for this part 2 what we did in part two is we went ahead and just cleaned the data up got
rid of the missing data filled in some other missing data and we also learned how to use get dummies
to create and deal if categorical columns coming up next is the final steps we're actually going to
create and train and build our machine learning classification model using logistic regression.
All right thanks everyone and I'll see you at the final part three of this lecture series