Hello everyone and welcome to the K means clustering project solutions lecture and this lecture we're
going to begin going through the solutions for the Kamins clustering portfolio project.
Let's go ahead and get started by jumping back to the notebook.
All right.
Here I am at the No but I'm going to go ahead and get started by answering the questions or completing
the tasks you wanted to import libraries.
We usually use for data analysis.
Well we always have Pandurs PD and maybe we need non-pay So go in and say no umpires and P and then
we also have our data visualisation libraries so we have mapped plot lym pipe plot as PLDT will import
seaborne as Asinus And then since I'm using that chip notebook map plot plotted in line.
All right then we want to read in the college data file using read CISC method.
So I'm going to go ahead and say ZF is equal to PD.
Read cxxviii and we can just type in college.
C O L L.
And then we can actually just tab autocomplete to get college underscore data.
Note that it doesn't have the CSP ending and we also wanted to figure out how to set the first column
as the index.
You can do that by passing the parameter index underscore c o l equals and then you can actually specify
what column index you want to be the actual index.
So we want the very first column index 0 to be the index column and then with that we can go ahead and
check the head of the data and it should look like this.
So we have our various columns here.
And if you want a reference to what these numbers actually represent or what they mean you can always
just read here.
Things such as the accepted number of applications or how many new students were enrolled et cetera
.
All right.
We can also use the info and describe methods.
So seeing that info we see how many entries we have and we have 18 columns and describe is a nice way
to quickly see the statistics behind these column values.
You can see the max the men's standard deviations etc..
Let's go ahead and move along to creating some exploratory data analysis visualizations visualisations
We'll start off by creating a scatterplot of the grad rate versus the room board and we'll have the
hewe be colored by the private column.
We'll go ahead and say asinus.
And to do a scatterplot it can either do peel teeth that scatter.
But since we want to use seaborne a tricky can do is use s n s l m plot and then say X is room board
and then have grad rate to be Y or our data is going to be T.F. set hew to private's.
And if I just run this you'll begin to see starts get these sort of scatter plot we want but I actually
don't want these fit lines and I can remove them by specifying that underscore e.g..
Or that fit regression equal to false.
And now I begin to see the plot I wanted we can also do other arguments or parameters such as saying
The palette is equal to cool warm size equal to 6 an aspect equal to 1 if you want to kind of change
the size of the actual plot and make it a little larger for it's easier to see.
But that's the basic idea it's a little tricky of seaborne to do this sort of scatter plots but the
idea was doing L-M plot for a linear model and then set fit underscore Regg for fit the regression line
equal to false.
And then you'll have a nice cup of scatterplot colored by whatever column you want the huge value to
be.
In this case private.
Moving along now we want to create a scatterplot of EFTA undergrad versus outstate where the points
are also colored by this private column.
That's actually really easy we can just copy and paste or we have a purity.
We're going to remove Palette cool warm
and I'm going to change room up board to this outstate column
and then and grad that rate is going to be F dot undergrad we'll go ahead and run this.
And we have this plot and we can already tell just by the out-of-state tuition column versus the number
of undergrads private versus public schools tend to be very different from each other.
So the tuition tends to be much higher for these private schools versus There's kind of a clear cut
off point for the tuition of a public school.
And you can also see these public schools tend to have much higher rates of undergrads versus these
private schools make sense.
Private schools are usually smaller more expensive versus public schools a little cheaper and much larger
student body.
Now we want to do is create a stacked histogram showing the out-of-state tuition based on the private
column.
And I also said try using seaborne Fassett grids do this.
And if that's tricky you can also do it just by combining two instances of pantless plot with the kind
being equal to histogram.
Let me go ahead and show you how you can do this with S and S.
Fassett great.
We're going to say G is equal to S and S Fassett grid pass in my data frame and I want the hewe to be
again private.
So the private column to match the coloring I have here in the notebook in the same palette is cool
warm.
And then what I do here is I say map and then a map map plot lives histogram realty that hissed on the
outstate column and then I can keep putting in parameters here such as the number of bins.
And what I want my alpha values to be.
For some see through and then we can go ahead and create a very similar histogram.
Now this is quite small.
So in this Fassett grid I'm going to add some arguments such as size equal to 6 and aspect is equal
to 2.
And then we get a very similar plot here and you can mess around the values are the alpha values or
the palette Pennino what you want the spot to really look like.
Again this is quite tricky to do with Fassett grid.
We've already shown previous examples and other projects that have do this with two stacked versions
of a Pandurs visualization off of the panda's column.
So we're going to just leave it at this for now.
It'll create a similar histogram using the Rattray rate column.
This is again just basically grabbing this.
Copying it pasting it and instead of outstate we're just going to do grad rates and run that.
OK.
Taking a quick look at these histograms we can again see that out of state tuition for these public
schools tend to be much less than the out-of-state tuition for these private schools in the second history
.
And we can compare the graduation rate frequency count for public schools versus private schools and
both tend to look to be normally distributed.
You'll notice that there's here in the private school in blue there looks to be a graduation rate that's
actually higher than 100 percent.
So that doesn't make any sense you can't have more than a hundred percent of students graduate.
We want to do is find out what is the name of that school.
So what does that bar actually representing.
Well that's actually quite easy to do if panderers we can say fine DMF a greater rate
greater than 100 and if you run that you'll get back this entire row.
And this is the Casanova college which if we go all the way over here to the grad rate we can find that
column it's 118 percent so that doesn't make sense.
So we're going to set that school's graduation rate to 100.
So it does make sense.
And a quick note you make it a warning not an error when doing this operation.
So use data from operations or just redo the histogram visualization to make sure it actually went through
when we're going to show you what I mean by this.
When we set off Grad rate that I'm going to grab the college name if we go ahead and take a look at
it again.
Whoops.
It was the Cazenovia college and I'm going to set that equal to 100 and then run this.
And no I do get a warning value is trying to be set on a copy of the slice of the data frame so we can
do this warning is basically just telling you hey you're actually affecting your original data frame
.
So if I copy this and paste it here and run it again.
Now note that they no longer exist.
And we can check that with another method just by doing this visualization and so I copy this visualization
and paste it here again notice now it stops right at 100 where we expect it to all right now it's time
to actually do the Kamins cluster creation's first thing I want to do is import Kamins from sikat learn
will say from S-K learn cluster import Kamins and then we're going to create an instance of a Kamins
model with two clusters which makes sense because we know we expect two groups private versus public
.
So say a number of clusters and they make sure Im actually typing this correctly.
Should be k means and the number of clusters is going to be equal to 2 and I want to fit this model
.
So say k means that it isn't going to fit into all the data except for the private label.
And notice we dont do a train to split here because that wouldn't make sense on an unsupervised learning
algorithms as we don't actually have labels to split.
And then I'm going to say access is equal to 1 so fitted on all the features.
And now I want to know what are the cluster center vectors we remember from the lecture.
We can just grab those by saying k means that cluster underscore centers underscore and it'll return
back the centers.
And notice the number of dimensions on this.
This is going to be the exact same number of dimensions as you have features in your data set.
Now as far as evaluation like I mentioned before there is no perfect way to evaluate clustering.
If you don't already have those labels.
However since this is just an exercise we do have the luxury of the label so we're going to take advantage
of this just to compare how well Kamins clustering on its own with no labels put in can detect the differences
between these two school types in order to do this.
I'm going to create a new column for the data frame called cluster which is one free private school
and zero for a public school.
There's lots of different ways you can do this.
But just to show you a reminder of how to use the apply method for data frame and then create a function
called converter that takes in a cluster.
And if cluster is equal to the string yes we're going to return one else return zero.
And just to make it clear we actually just convert this to private instead of the word cluster.
So it's a better choice of words here.
All right then I'm going to say DMF of cluster make a new column called cluster and set it equal to
the private apply my converter function.
And now if I take a look at the head of my data frame it looks like this.
We go all the way to the end.
We have a cluster which is ones and zeros and it lines up perfectly with this column right here.
Private yes or no.
Now all we have to do basically is just convert this private column from yes and no strings to ones
and zeros.
Now we're going to create a confusion matrix and classification report to see how well the Kamins clustering
work without being given any labels.
First thing I want to do here is save from S-K learn metrics.
Import the confusion matrix and the classification report and then let's go ahead and print those out
.
I'll say prints confusion matrix pasand the cluster column versus what we thought were the labels using
Kamins and I can do that just by saying k means that labels will do the exact same thing.
Let me go in and print a new line here and they will do the exact same thing for the classification
report.
Let's go in and run this and see how he did.
OK.
So taking a look at this we do have quite a few errors quote unquote.
But remember this not so bad considering the algorithm is using pure features to cluster the universities
into two distinct groups.
And the precision and recall are actually not so bad.
Again keep in mind in most settings you're not going to be able to do a confusion matrix or classification
report on Kamins clustering because that implies that you already have the labels and if you already
had the labels you wouldn't use an unsupervised learning algorithm.
OK before we end this let me go ahead and just review everything we did as we went along through this
project.
We imported the libraries we needed.
We got the data did some very basic checking of the data and then we did a little bit of exploratory
data analysis.
Now this exploratory data analysis was definitely harder as far as the functionality we had to grab
from seaborne than what you've seen before.
So it was quite a challenge for you.
Don't worry it was meant to be that way than if you had a reference solutions.
Definitely don't feel bad because this was pretty challenging.
Visualisations using just the Seaborn library the scatterplot the trick to this was using See morn's
l plot and setting the regression line fit to false.
Or you could have just gone with map outlives own Piazzi that scatter.
And then we also create another scatterplot of the undergrad vs. out-of-state.
And here we can see there is definitely some separation here between the two classes.
Then we create a stacked histogram using Seaborn Fassett grid.
Again a very tricky visualization to pull off but don't feel bad if you weren't able to do it.
Referencing the solution.
And then we also do the same thing for the graduation rate column.
And that's where we discover there was some error with within our data that we had a school with a graduation
rate that was higher than 100 percent.
We went ahead and got that school fixed it so that it was 100 percent.
We got a warning just telling us that hey you're actually setting a real value on your data frame.
And then we checked this by calling that again to see that it was no longer there or just by repeating
the actual histogram that we saw earlier and seeing that the X-axis now capped off at 100.
We went straight to Kamins cluster creation.
Import came means algorithm set the number of clusters you want fit it to all your data.
Remember we had a drop the private because it's not going to take in a label because it's an unsupervised
learning algorithm.
You can check out the cluster center vectors which are centers in essentially a 17 dimensional space
or however many features there were.
And then we did a little bit of evaluation remember it's unsupervised learning algorithm.
So these things are very hard to evaluate if you don't already have those labels.
OK.
Hope you enjoyed the project and I'll see you at the next lecture