Hello everyone and welcome to the logistic regression of Python lecture I'm really excited for this
lecture because we're going to be working with a titanic data set.
This is a very famous dataset and it's often a student's first step in machine learning reclassification
.
We're going to be trying to predict the classification survival or the CCed for passengers who were
on the Titanic and we're going to be using a semi clean version of this titanic data set in the dataset
.
Original source is actually from Kaggle dot com.
Let me go ahead and briefly explain what Kaggle dot com is.
I'm going to go ahead and jump to that Web site in my browser.
OK.
So here I am on Kaggle or Kaggle dot com pending how you pronounce it but it's K a G G L E dot com.
And basically this is a data science hosting Web site.
It has two or three main aspects to it.
The first aspect is when we're actually already familiar with which is the data sets aspect.
Basically this is a Web site that can host data sets for instance if you're interested in one of these
data sets you can kind of scroll down here.
Check out some of these data set such as airplane crashes since 1938.
You can go ahead and click on this link and load up the data set a way to download it and some information
on that data set.
For instance here you can go ahead and click valid data set and read a description of the data set.
And sometimes even has open ended questions free to answer just as a curiosity.
The other aspect of this Web site is the competition's page.
You go ahead and click on the competition page.
It'll take you to a list of all the active competitions and basically this is the place where companies
can post their own data sets with their own problems and ask either a single person or a team of people
to compete for a monetary prize in order to solve their own real company problems.
Sometimes they're offering money other times or offering jobs at their corporation.
So you'll see things such as Facebook posts a challenge on this Web site and if you do well in the challenge
they'll actually ask you to interview at their company and other challenges are just for knowledge.
So you can see these simpler challenges here actually just knowledge based challenges and the one we're
going to do is the Titanic machine learning from the Zastrow challenge.
And this is the basic source of where we're actually going to be getting our data from.
Now this data is actually already been downloaded for you and it's part of the whole package you downloaded
Rafal and notebooks.
Let me go in and show you where to find it.
If you go ahead and jump to the machine learning sections folder under logistic regression you'll see
some files here.
And the Titanic to see V and Titanic trained CACP files are the data sets we're going to be working
with.
So go ahead and use these two CNC files that have already been downloaded for you and the notebook we're
going to be using is logistic regression of Python.
Later on we'll show you how to do the project and then we'll show the project solutions.
But right now let's go ahead and jump straight to a new notebook.
Or you can go ahead and follow along with the lecture notebook but I'm going to go ahead to get started
and do some machine learning with this titanic dataset.
All right.
I went ahead and clear the toolbar and the header just give us some more room.
Let's go ahead and get started with our imports.
I do Panas the pie as MP and then let me go ahead and do my visualization imports.
I must say map plot lived up high plot as Piazzi I like importing seaborne as asinus.
And then since I'm using the note book I'd say map plot lib in line.
Perfect.
Well we're going to go ahead and start doing is by reading in the Titanic train c a c file into a panda
state of frame and created data from Coltrane and say PD read cxxviii and I can actually just start
typing Titanic and if I click tab here it will auto complete to the options available for me.
So in this case I want to do this titanic underscore train dot CSV run that and then I'm going to go
ahead and check the head of the data frame just to get an idea of what's in there.
It looks like we have a passenger write the column A survive which notice it's a 0 or 1 so 0 if they
did not survive what if they did survive.
P class which is the passenger class so the passengers can be in first class second class or third class
depending on where they are in the ship.
There is also the name of the passenger the sex or gender the passenger of the string male or female
.
The age of the passenger if they happen to know it.
And then this sibb s.p that indicates the number of siblings or spouses aboard and then parse S.H. indicates
the number of parents children aboard.
And then there's a ticket number a passenger fare how much they paid for their ticket.
And if we go ahead and scroll over to the right we'll see a couple more columns.
There is cabin which is the cabin they were in.
If it's known and then embarked and it's a port of impartation so see for Cherbourg.
Queue for Queenstown and as for Southampton.
All right.
Those are the basic columns.
Let's go ahead and begin some exploratory data analysis.
Now a lot of times you're going to have missing data.
We're going to go ahead and do as you see Born to create a simple heat map to see where we're missing
the most of our data.
I'm going to go ahead and do this.
I have my training data and if I say is null as a method on it let me go ahead and just show you what
this looks like at least what part of the output looks like that is the data frames too big.
But you basically get booleans of true and false so you'll get a false if is not null.
Or a true if it is null.
So remember back to that very first cabin it was.
And so it was unknown it was null.
So it went ahead and put true there with this sort of data frame.
I can actually make a heat map so I'm going to say this and I say Asinus that heat map and then I'm
going to add a few more arguments so we don't get a bunch of tick labels.
I will say Why tick labels is equal to false.
I'm going to go ahead and say C bar is equal to false because we're not doing an actual color bar.
And then I'm going to finally pass and see map as this.
Let's go in and see what this gives us.
All right.
So notice what I've created here.
I've created a heat map of those boolean values those true and false statements and due to this.
See map does color mapping every yellow dash here basically stands for a true point where true it was
null.
So we can just glimps now order data from a very far bird's eye view and check out that yes we're missing
some age information are missing a lot of Cabane information and we're missing just essentially one
row of embarked roughly about 20 percent of that age data is missing and the proportion of age missing
is likely small enough for a reasonable replacement of some form of imputation meaning I can actually
use the knowledge of the other columns to fill in reasonable values for that age column.
Looking at the cabin column however it looks like we're just missing too much of that data to do something
useful with it at a basic level.
We're going to go ahead and probably drop this later or change it to send up some other feature like
Cabane known 1 or 0.
Let's go ahead and continue on.
By visualizing some more of this data will come back to this problem of dealing with the missing data
.
Later on we're going to go ahead and do it.
A couple of more plot to just do some exploratory data analysis at a visual level.
I'm going to go ahead and say Asinus set style to white grid just because I personally prefer that style
I think it looks a little better.
First thing I want to do is just get a count of who survived who didn't survive for classification problem
.
It's always a good idea just to see the ratio of the actual target labels.
Now we can do this for simple counterplot.
I can say Asinus stock counterplot and then I'm going to just go ahead and set x to the survivors column
set my data to train and then I'll go ahead and just run that.
And here he can see I have quite a few more people that did not survive versus one which is survived
.
But it looks like around 550 of these passengers in their dataset did not survive.
And it looks like about this particular training set we have about 350 ish maybe a little less that
actually survived.
So looks that we have more or excuse me less survivors than non survivors or deceased.
We're going to go ahead and do is create another count plot of seeing survival with the Heugh of sex
and I can actually just add the sense I'm going to say Hugh is equal to sex and this is going to add
some coloring so it can actually tell by gender the survival rates.
OK and I'm going to go ahead and add a palette argument to make this a little easier to see.
I'm going to do RD.
B You underscore are lower case are.
Now you don't have to do this just for coloring.
But that way mail is blue and female is red or a light pink color.
This makes it a little easier to read.
OK looking at this plot now I can really tell there's a trend here.
It looks like people that did not survive were much more likely to be male and people that did survive
were almost about twice as likely to be female.
And we're going to see that trend come up later when we're looking at the factors and coefficients of
what led to someone's survival.
Let's go ahead and continue exploring the data by changing this hue to the passenger class so I'll go
ahead and say it's sort of sex here P.E. class.
So for the passenger class and I'll go ahead and remove this pallet or go ahead and just use the full
palette which is blue green and red.
OK.
Again looking at this we can kind of tell the trends here on survived versus not surprised.
It looks like the people who did not survive were overwhelmingly part of the third class or the lowest
class that was the cheapest to get onto.
And it looks like more or less the people that did survive were leaning a little more towards the higher
classes.
One and Two will also have to see how many people are actually in the passenger class or per passenger
class.
Well we're doing that analysis.
Let's go ahead and just get an idea of the age of people on the Titanic to do that.
I can go ahead and do it.
Distribution type plot.
I'll go ahead and say S.A. this plot or passen the age column.
Remember this plot just takes in an array of values.
And for now I'll go ahead and say drop N.A. since I don't want to have those Novis messing up my distribution
plot.
Also Katie equals to false because I just want to see a histogram of those values and let me go ahead
and set bin's to something a little higher such as 30 minutes from that.
OK we had an interesting what appears to be almost by modal distribution where you have quite a few
young passengers towards between ages 0 and 10.
So there's quite a few children in that zone.
But then after that you can see it starts to get an average age towards maybe twenty or thirty somewhere
around there.
So it is quite maybe skewed towards younger passengers.
And as the older you get the less representation you have on board.
OK let's go ahead and show you another way you could have made this plot if you wanted to if you weren't
a huge fan of Seaborn could have also done this directly off of the age column doing something like
that plot twist and Panda's own visualization can do this for you and you can also add more bends if
you want to see an Sabin's 35 just to get kind of a similar looking plot here.
OK.
Depends what you want to try to do.
It's up to you.
OK let me go ahead and go back to seaborne real quick.
Let's go and explore some of the other columns just to get a reminder of what those columns are.
I want to go ahead and call info.
OK we've already checked out the survived column quite a bit we've seen P.E. class the sex column age
column.
Let's go ahead and explore that Sid S P That was the number of siblings or spouses on board.
Let me go ahead and do an S and ask counterplot set X equal to that column s and tell seaborne my data
that I'm going to use is that train data frame.
All right.
Now looking at this plot I can basically immediately tell that most people onboard did not have neither
children or a spouse on board.
And if we look at the second most popular option it's just one which is actually most likely to be probably
a spouse versus just having a single parent when children onboard.
So those numbers pretty much make sense you have a lot of single people on board.
They don't have a spouse and how many children are onboard.
Probably those are the men in the third class.
And after that you have the second most common type of passenger in our data set is some of just one
either sibling or spouse.
And that's probably going to be their spouse.
Couples on board.
OK.
In other column we haven't actually explored yet.
Is the Fair column.
Let's go ahead and take a look at how much people paid as a distribution.
We're going to go ahead and do this with pandas.
I'll say train fare.
And remember if we take a look at that it's basically just their fares were prices they paid.
I'll say just call a histogram and let's go ahead and just run this and see what we get.
OK immediately we see that there's quite a wide range of values.
Most distribution seems to be somewhere between 0 and 100 to get a little more detail.
I'm going to increase my bin size to 40 and then we can also kind of stretch it out by passing in Fig's
size to maybe 10 by four.
OK.
Now it can get higher distribution or higher level of detail in this distribution.
Now it looks like most of the purchase prices are between 0 on a 50.
So it makes sense that things are distributed towards the cheaper fare tickets because as you already
seen most passengers are actually in the cheaper third class.
And keep in mind these are fares back when the Titanic was around and that was in 1912.
So these prices haven't been adjusted for inflation.
OK let me go ahead and take a quick moment to show you an example of how we could have done this sort
of plot with cufflinks in case you really like the interactive plots.
Could have said import cufflinks as C-f and then told cufflinks that we want to operate off line
and then we go ahead and essentially do the exact same thing.
So grab the fair call them off a train and then we just say I plot in order to see this in our plot
and say kind of a sequel to this.
You can specify some more bends and then see this run.
Now I can go ahead and zoom in and check out these bins myself and I can add more bins.
So I want to put in 50 adman's.
Zoom in.
See where I'm most interested and get the actual counts and exploit that data that way just another
tool free to use if you want in a more interactive experience with this data.
Lots of stuff you can do with Python.
So very powerful language and there's lots of great libraries for it.
OK I'm going to go ahead and comment that out.
So let's go ahead and end this lecture here as finishing our exploratory data analysis.
So we just went ahead and kind of explored the data without really knowing too much about it beforehand
just to try to grab some contextual information from it.
We went ahead and saw the distribution of fair prices saw the accounts of siblings versus spouses on
board made the assumption that most of them were probably single based off this count or just had one
spouse.
We went ahead and check that the age distribution and then we also looked at survival versus passenger
class and survival versus the sex or gender.
Go out and take a look at that again.
Whip's should be capitalized and we can see that again that mostly men are in the non-survival group
and mostly female are in the survival group by ratio of two to one female to male and the survival group
.
And then we also saw that we have quite a bit of missing information and we want to figure out how can
we deal with the age column and how can we build this cabin column.
All right.
Well go ahead and tackle how to deal with that missing data in part two of this lecture series.
Thanks everyone.
Go ahead and explore that a little more on your own and if you're still interested in making some more
visualisations it all go ahead and see in the next lecture