Hello and welcome to this art tutorial.
In the previous tutorial, we imported on mobile data set
and we prepared our data correctly by taking the two columns
we are interested in the annual income and the spending score.
So we created this variable x that contains these two columns.
And now things are going to get much more interesting
because in this tutorial we are going to build our dendrogram.
And we will use it to find the optimal number of clusters
exactly like we did in the K-means section, where as you remember,
in step two, we use the elbow method chart to find the optimal number of clusters.
Well, here in Hierarchical Clustering step two,
we will also look for this optimal number of clusters.
Only this time we're not going to use the elbow method.
We aren't going to use the dendrogram.
So let's do that right now.
The very cool thing about it
is that we only need one line of code to build this dendrogram.
So let's write it.
Let's write this line of code.
We start by creating our variable dendrogram.
Then equals.
And then we're going to use the class H cluster.
So let's type h class here.
And then let's press F1.
And here we have all the info of this H class class.
So let's look at the arguments here.
We only need the first two arguments.
The first argument is a dissimilarity structure as produced by this.
And in our case this parameter is going to be
the distance matrix of our data set X, which is a matrix
that tells, for each pair of customers, the Euclidean distance between the two.
So that means that
for each pair of customers, we take the two coordinates annual income
and spending score, and we compute the Euclidean
distance between the two based on these coordinates.
Okay.
So that was just to explain the first parameter of the h class class.
And so let's import it.
In our code we input dist
and in parenthesis x comma.
And then method equals Euclidean.
So that specifies that we want to compute
the Euclidean distance matrix for our data x okay.
So that's the first parameter this distance matrix.
And now the second parameter is the method.
So this method is simply the method used to find the clusters.
And like in Python we're going to choose the most common method
which is the word method.
Here it's called word 30.
And it's actually a method
that is trying to minimize the variance within each cluster.
Kind of like what we did in K-means when we were trying to minimize the
within cluster sum of squares.
Well, here it's based on the same idea.
But instead of trying to minimize the within cluster sum of squares,
we are trying to minimize the within cluster variance to find our clusters.
So here we write method equals words.
So that is the end of the line to build this dendrogram.
And now we just need to plot it.
So just below we are going to write plot.
Then in parentheses is dendrogram.
Then let's give it a title by typing main equals
paste parentheses dendrogram in quotes.
Then let's give a name to the x axis
by adding x slab equals customers, because in the dendrogram
all our customers are going to be on the X axis.
And then finally let's give a name to our y label.
We're going to call it Euclidean distances.
And that's because in the dendrogram the vertical lines that we're going to see
are actually the Euclidean distances of the clusters.
That is between the centroids of the clusters.
Okay.
So we are good to go with our plot.
So and our dendrogram actually.
So let's select all this code section here execute.
And here is our dendrogram.
So let's have a look at it I'm clicking on zoom
here to make it bigger okay.
And now let's try to find the optimal number of clusters
thanks to this dendrogram.
So as Kirill explains in the intuition section
to find this optimal number of clusters, we need to find the largest
vertical distance that we can make without crossing any other horizontal line.
And then we just
need to count the number of vertical lines at this level okay.
So let's start by finding the largest vertical distance.
So it's not here obviously.
Then maybe this one.
It's quite a large distance.
That would actually give us three clusters because as you can see here
I'm crossing three vertical lines.
Definitely not this one.
And here we have another large distance.
You see from this point to this point is quite a large distance.
And then below obviously we don't have any large distance.
So now the question is what is the largest distance
between this distance and this distance.
Well if you have a better look at it we can see that
the largest distance is actually this distance.
And how many vertical lines do we have at this level.
Let's see 1234 and five.
So that means that our optimal number of clusters is five clusters.
And that's of course a relief because that's what we obtained
with the K-means algorithm using the elbow method.
So everything is fine.
Everything is perfectly coherent.
So we will completed our second step.
And now we are ready to move on to the next step, which is to fit
our hierarchy called clustering algorithm to our data X.
And that's what we will be doing in the next tutorial.