Hello everyone and welcome to part two of the day to cap some project solutions lecture for the night
.
When one calls capstone project.
Let's go ahead and jump right where we left off and go to the Jupiter notebook.
OK here I am at the notebook right where we left off we discovered that the months were actually missing
a couple of values versus nine 10 and 11 were missing and we want to do is now work with pandas to try
to create maybe a line plot or some other visualization that tries to take that into account.
So maybe we can fill in those missing data points.
OK.
We're going to go ahead and do is create a group by object called by month and say DMF group by passing
the month column and use count method for aggregation as asked for the directions.
And then it was go ahead.
Just check out the head of.
By month.
OK and here we can see the head of.
By month you'll notice that basically what happened was we counted every instance of the columns by
the month or we're going to go ahead and do is notice that some of the counts are different depending
on the actual column value.
And that's because some values are essentially missing.
When you aggregate it by month or we're going to go ahead and do.
Nor is even everything out.
It's just choose the latitude column as our value for the aggregate count.
Well go ahead and assume that if we had the latitude of the call the call actually took place.
Now we can create a simple plot of the data frame indicating the count of the cost per month and the
way we can do that is actually just by plotting out the columns we say by month call the column for
instance latitude and then say plot.
And here we have our month and now we've actually been able to fill in some of that missing information
through a essentially a fit between points eights and points at 12.
And then you can see the down curve maybe a little better than you would have been able to.
Had you done it with just a bar plot and to show you that maybe in a little more detail I'm going to
go in and copy this code paste it here and I'm going to go ahead and remove the hewe And then if I run
this we will see the code.
Here we have month.
Don't worry about this little warning.
But this essentially shows the same information.
However without the 9 10 and 11 month.
So you can see the drop down from month 7 two months eight and 12 hour.
Maybe it's a little more clear.
The drop down between 8 and 12 using this sort of line plot.
And you can actually play around with which columns you choose to plot such as maybe the township column
or the zip column etc to see how pronounced this drop is just kind of the area of exploratory data analysis
or maybe there isn't exactly correct answer unless you want to show something very specifically through
quantitative methods and in that case you probably would show this sort of bar plot.
But it's also nice just to get an idea of the trend using just this simple plot command.
All right.
Let's go ahead now and see if we can use see mourns L-M plot to create a linear fit on the number of
calls per month.
And keep in mind you may need to reset the index to a column in order to do that.
What I'm going to go out and do is say this.
I'm going to say S.A. thought L.M. plot for that linear model plot from Seaborn and I want my X if we
go out and check this out to be month and we can go in now select maybe township's for the y axis.
You can also select latitude or really any other column if you want to kind of experiment around with
this so essentially just counts.
And then the data is going to be the by month.
But notice here I said X is the month column.
If we take a look back at the by month we don't actually have a month column the month was the index
.
So what I'm going to go to do is reset the index to month in order for this to work
and way to make it even more clear.
Let me just show you what this actually looks like.
If I reset the index or the by month I will make month into a new column and that allows Seaborn to
actually access this information instead of setting it to the index.
And that way I can actually pass and month here's a string.
Otherwise Seaborn would get confused.
And that's what we had to use this recent index to the column and then we get the linear model fit and
we can see that generally as far as what Seymore it kind of plots out in this model that the number
of calls goes down from 0 to month 12.
Now keep in mind seaborne doesn't have enough information to realize that this is by month.
And so that 13 and 14 shouldn't actually exist or continue.
This would actually be seasonal data and in that case probably a linear model is actually the best fit
and you can see that seaborne countries show this with this shaded area indicating error and the error
basically grows as you go into these months where there's less information known which actually makes
a lot of sense to have information about nine 10 or 11 months.
And you can see this trend where it goes down starts to come up and then crashes that month eight that's
something you may want to explore later on in the data.
Well let's go ahead and just leave it at that for now and explore a little bit more about the date information
.
OK.
We wanted to do next was creating a new column called date that contains the date from the time stamp
column and you'll need to use apply along with the date close parentheses method.
Let me go in and show you what this actually means by dates by showing you an example of the time stamp
column.
I'm going to go ahead and just grab the first instance of the time stamp column and call one of those
objects.
All right.
This is a time stamp object and it's a little different than actually a date object.
And the reason for that is if I call t dates off of it I now get a date object.
And this just shows the dates information.
It no longer has the time information and that's good because what we're going to want to do later on
is create a plot that looks like this just the number of calls per date.
We don't want to flood this information with number of calls her every minute every hour.
All right.
That means let's go ahead and say ZF date is equal to DMF of time stamp.
And just like we did before I'm going to go in and apply this Fil-Am the expression I will take that
time see and return to the dates.
I go ahead and apply that.
Now if I take a look at ZF head and go all the way to the right here.
Now I have this nice date column which is driven off of this time stamp column and then when I want
to go ahead and do is grouped by the state column with the count aggregate.
So I'm going to say diff date.
Whoops.
Actually I want to say DFA group by passen date.
That's what I want to buy.
I'm going to go ahead and counts and then let's just go in and see the head of the state of frame what
this looks like.
OK.
And this essentially counts all the instances her date for the actual date that occurred.
Again notice that there's some difference between some of these column values and that's just basically
indicating that you were missing some information for instance out of the 300 in 96 calls on December
11th are the penny how you want to read that.
Whether it's yet to be December 11th you only had 333 with the zip code versus 396 we had the latitude
.
Let's go ahead and just choose one of these columns.
It's kind of arbitrary since they're actually all pretty close to number but I'm going to go ahead and
grab the latitude column and you can experiment by grabbing different columns how they compare.
And here you have the latitude column which now basically gives me a date and the count of number of
calls where we knew the latitude and then I can go ahead.
Just plot this out and it gets me a plot that looks like this.
And notice we get some of these labels overlapping and what we can do is say peel t Titely out as we've
shown before in order to actually kind of fix that index and get something that looks a little nicer
.
All right.
And then when we actually notice here is it looks like there are some significant spikes in February
and a little bit in March and then we get some downturns or some bottom out spikes in May and maybe
late August.
A little hard to tell here.
And you can go ahead and mess around with this plot or extend the plot to try to get a little more information
about the actual dates.
Well we're going to go ahead and do now is recreate this plot to create three separate plots with each
plot representing a reason for the 911 call.
Back to the reason data frame or excuse me call Mishu what I mean by that.
So if I scroll down here I want to recreate this plot.
So here we have one plot with only the traffic calls only the fire calls and then only the M-S calls
and we we do is actually a very similar process to what we did here.
We're going to copy and paste this but instead of the entire data frame I'm going to use conditional
selection to actually do the group by.
So I'm going to say DSF where ATF reason
is equal to the first one we can just go ahead and say traffic to us there and then we go ahead and
plot this and there we have the plot for traffic and if you wanted to you could have also said Appeal
t the title and indicated you want the title to be traffic and now we can actually just use the same
code to perform the other charts.
So if we wanted fire we just replaced this with fire as a reason.
And now we're able to take advantage of all the work we just did previously to create these three separate
plots.
And you can go ahead and kind of explore them on your own and see if you can find differences between
maybe a number of calls or a frequency of calls during the dates.
For instance it might be interesting to see how weather would affect fire based off the month or the
day.
You can see there's some interesting spikes there.
OK.
Moving along.
Let's finally go to creating heat maps with Seaborn and our data.
And first we're going to need to do is restructure that data frame so that the columns become the hours
and the index becomes the day of the week.
There's actually lots of ways to do this but what I would recommend trying to do is using group by with
a new method that I wanted you to discover called the unstacked method.
And there's a link here to the documentation and you can refer to the solutions if you get stuck on
this.
Let me go ahead and break down how to actually make something that looks like this.
This heat map remember back to the cluster map and the heat map lectures.
We need our data to be in what's known as matrix form where every single cell converts back to a value
representing the value at the column and the index itself.
Meaning our zero on Fridays.
This is how many calls occurred.
OK let's go ahead and learn how to use the unstacked method.
Then there's some examples here in the documentation which hopefully you may have discovered on your
own.
But he got stuck on it.
No worries because I'm going to show you how to do it right now.
We're going to go ahead and say DF group by.
And I'm going to have to group by a list.
In this case because I don't want to just group by day of week I also want to group by hour
and then when I'm going to go ahead and do is say count this just go ahead and take a look at what this
looks like and this basically allows me to group by and create a multilevel index.
Day of week and then the hour and then just the counts for each column here.
So I have this multi-level index what I can do now is just call a singular column off of it maybe reason
we can go ahead and do that
and now I have this multilevel index.
And the key to the unstacked method is it basically allows you to unstacked this and make this into
a matrix form.
And now I have this matrix form.
So the steps were grouping by multiple columns creating that multilevel index and then being able to
use and stack to essentially unstacked that index and create one of them to be the columns and one of
them to be the index.
All right.
Don't worry if you got stuck on this or had trouble with this this is actually very kind of advanced
method in order to do this.
There's other methods you could have tried such as using a pivot or a pivot table of Pandurs kind of
up to you but again don't worry too much if you are able to do this.
It's kind of comes a lot with just practice experience and being able to look stuff up in the documentation
.
Well will go ahead and call this entire data frame day hour and now that we have that data frame I can
easily create heat maps and this part will actually be pretty straightforward.
We just call S and S heat map passing day hour and that's essentially all we have to do.
Now we get day of week versus the hour and then the number of calls as the color indicator and we can
continue to try to make this look more like the solutions by saying See map passing in viridis.
And now we get the same color scheme as we do in the solutions and we can make the figure a little bigger
by saying Piel tief that figure and adding in Fig size as an argument.
And now we have basically what was in the solutions.
And we can also create a cluster map with this data frame in very much the same manner by saying s and
s cluster map.
And again just passing in the data frame we quit created our See map for this
and that creates the same cluster map.
If you're confused on how to actually what's actually going on on this line Feel free to post to the
Q&A forums.
And I'm happy to explain this a little more.
But just keep in mind this is kind of advanced and if you get stuck Don't worry if you reference the
solutions moving along.
What we want is to do is repeat these same plots and operations for a data frame that shows the month
as the column I'm going to go ahead and just copy and paste this from the solutions because it's essentially
the same code as we just did above.
But in this case instead of saying for instance the hour we just set the month and if we go ahead and
check that out.
Now we have day month and we essentially just make the same plots again.
So we just go up here call it map scroll down a bit instead of day hour.
I want to use day month
and that gets me.
This heat map and if I want to create a cluster map off of this I can do the same thing except just
call cluster.
And this creates a cluster map.
So now I can actually explore the data and see maybe what hours of the week during what days of the
week are the most popular for night one calls to occur.
And if we just take a kind of a brief look at this it makes sense that not a whole lot of calls are
happening between hours of 0 and 5 because that's essentially midnight to 5:00 in the morning.
So everyone's asleep.
Most of the criminal activity or reasons for and I don't one calls such as traffic accidents m mess
or fire is not really that much criminal activity in this dataset.
It's mostly fire M-S and traffic reasons.
Those all tend to occur during the daytime daytime.
And what's interesting is that not too many of them tend to occur on Sundays and Saturdays and you can
actually see this a little clearer with the cluster maps so this will actually create clusters where
it thinks there's low amounts of calls versus high amounts of calls and sometimes what's nice for cluster
maps is to choose something like the cool warm color mapping in order to really see this.
And sometimes this is a little more clear of a color scheme for people to actually get the groupings
in their minds.
It looks like we have high calls during one's day Monday and Tuesday and maybe Thursday and Friday during
the hours 8 16 and 17.
So a little more towards the evening it tends to look like versus low call amounts are during Maybe
Saturday and Sunday in the morning.
OK.
And then we just repeated those operations per month and you can kind of compare summer months versus
winter months here and you can continue exploring the data how you see fit and that's basically all
there was to this project.
I hope you enjoyed it.
Hope you found it interesting.
Again don't feel bad if you had to look up the solutions for this kind of complex one liner here.
You can always feel free to post in Q&A forums if you need any clarification on what was actually going
on there.
All right.
Feel free to post for any questions whatsoever.
And I will see you at the next lecture.
Thanks everyone.
Hope you enjoyed the project.