Hello and welcome to this art tutorial.
So in the following tutorials we are going to implement the apriori algorithm.
And as usual, we are going to make this machine
learning model to create some added value in some specific business.
And in this part this business problem is going to be about
optimizing the sales in a grocery store, a grocery store in the south of France.
And you're going to perfectly understand how the apriori algorithm
is going to do a perfect job at doing this, optimizing the sales.
Because recently a lot of stores considerably created some added value
thanks to machine learning and especially association rule learning,
by using it to optimize the sales of their products.
And how did they do that?
Well, they just used association rule learning
to know exactly where to place the products in the store.
You know, for example, I'll give you a very simple example.
If someone buys some cereals, well,
the same person is very likely to buy some milk as well.
So by placing the cereals close to the milk,
the store is very likely to put these two products into the same basket,
even if the buyer originally intended to only buy cereals.
Or I can give you a more general example.
Suppose a person wants to buy a specific product.
Let's call it product A, and this product A can be associated
very well to another product B
and the person who wants to buy the product A might not think of this
good association between the product A and the product B well,
if you place the product A and the product B next to each other,
well, the association can suddenly pop up in the buyer's mind
so that, you know, the buyer can tell, hey, that's actually a good combination.
Why don't I try these two for my next lunch or something?
And so again, even if the buyer was originally meant to buy only the product,
a well, due to this association that popped up in its mind
thanks to the placement of product A and product B next to each other,
well, the buyer finally buys the two product A and product B, so that's
the idea of how we can create added value for retail stores or grocery stores.
And so what we'll make in this future tutorial to optimize the sales
can be also apply to any other store that is selling some different products.
You can think of an online store.
You know these recommendations.
People who bought this also about that.
Well, these recommendations are based on association rules as well.
But not only it can be also the result of a recommendation systems
like collaborative filtering or content based item based collaborative filtering.
But association rule learning has a role to play.
So now let's make our first association rule learning algorithm,
which is the apriori model for this specific store in the south of France.
So let's do it.
As usual, we're going to set the working directory by going to our
new part folder which is the folder part five Association Rule Learning.
And we are starting with the apriori algorithm.
So that's the folder we want to set is working directory.
Make sure that you have the market basket optimization CSV file.
And you can click on this more button here and then set as working directory.
All right. We're in the right folder now.
So the first thing we're going to do is to import the data set.
So the data set is market basket optimization.
So as usual we are going to call it data sets.
And then of course we're going to use the read dot CSV function.
And now we simply input the name of the CSV file.
So here we go to market
basket
optimization dot csv.
All right.
So let's execute that.
And let's explain what the data set is about.
Here we go. Data set will import it.
It has 7500 observations and 20 variables.
So let's check it out I'm clicking on data set here.
All right.
And that is the data set.
So the first thing that we can see here is that this line here contains
some products as these products here.
And of course these are not the titles of the different columns here.
So to improve this what we can first do is to add this header argument here
header equals and then simply false this way.
And that tells are that the first line of our data
set doesn't contain the titles of the columns.
So let's check it out now let's select this line and execute.
Here we go.
let's close this and click again on the data set.
And here we go.
We don't have any titles for the columns, but you know, this
first observation is no longer seen as the titles of the columns.
But that's the real observation itself okay.
So better.
And now let's describe the data set.
So as I told you, we are making this apriori
model for a store in the south of France.
And so we want to find out the association rules of the different products
of the store, to see how the manager of the store can optimize
the placement of its different products to optimize the sales.
Okay.
So the first thing to say now is that this store is located
in one of the most popular places in the south of France.
So a lot of people go into the store.
And so, you know, this place is a very convivial place,
a very friendly place where people love to hang out, relax, talk to each other.
And so these people come very often to this store because
even if it's not to buy something, it's at least to meet their friends.
And therefore, the manager of the store noticed and calculated that on average,
each customer goes and buy something to the store once a week.
So this data set here contains the 7500 transactions
of all the different customers
that bought a basket of products during a whole week.
Indeed, the manager took it as the basis of its analysis
because since each customer is going on average
once a week to the store, then the transaction registered over
a week is quite representative of what customers want to buy.
So based on all these 7500 transactions,
our machine learning model, our apriori model is going to learn
the different associations it can make to actually understand
the rules, such as if customers buy this set of products,
then they're likely to buy this other set of products.
So that's what we want to figure out.
And that's what our apriori model will tell us.
Okay.
So each observation line here corresponds
to a specific customer who bought a specific basket of product.
So for example if we look at line two here, that corresponds to one customer
who bought burgers, meatballs and eggs at a specific time of this week.
And that's the same for all the other observations
that correspond to other customers, or maybe the same customer
who went back to the store another day or another time.
So that's what the data set is about.
But actually, this is not the data sets we're going to use
to train our apriori model.
And the reason is that the package we're going to use to build our apriori model,
which is, by the way, the rules package, doesn't take a data
set like this as input.
It doesn't take a CSV file that we imported thanks to the readcsv
function.
What it takes as input is called a sparse matrix.
And so what is a sparse matrix?
It's actually a matrix that contains a lot of zeros.
In machine learning you will encounter a lot of times the word sparsity
that corresponds to a large number of zeros.
So a sparse matrix is a matrix containing a very few number of non-zero values.
So what we're going to do now is transform this data set here into a sparse matrix.
And can you guess what we're going to do.
Well what we're going to do is take all the different products of this data set.
And actually I already know that there are 120 products.
And we're going to attribute one column to each of these 120 products.
So that means we'll get 120 columns.
So for example.
So for example we'll have the column shrimp
the column almonds the column avocado the column vegetables.
Mix cortices energy drink tomato juice up
to the 120th product that there is.
We're going to see all the products then on a plot.
But in this data set there are 120 products
which are, by the way, the 120 products of the store.
So there's going to be one column for each of these products,
and that's going to be the columns.
And then the lines are still going to be the different transactions
corresponding to each of the 7005 hundred customers
that bought a basket of products during the whole week.
But instead of having the list of the product they bought,
we will have in each of the 120 columns here, a 0 or 1,
and it's going to be a one.
If the product is in the basket of the customer during its transaction,
and a zero if the product is not in the basket.
So for example, let's take the second customer here,
the second customer, but a basket of three products burgers, meatballs and eggs.
Okay.
So in our sparse matrix we'll have one
burgers column, one meatballs column and one x column.
They're not necessarily going to be next to each other.
You know burgers can be the fifth column and meatballs can be
the ninth column and X can be the 12th column.
That depends on how the rules package is going to make a matrix.
But we will have a column for each of these three products.
And so in these columns, since the customer number two
bought some burgers, meatballs and eggs, there will be a one
in each of these columns.
There will be a one in the burgers column or one in the meatballs column,
and a one in the X column.
And all the rest of the columns are going to have a zero value.
And that's because all the other products were not in the basket of this customer
number two.
So you can guess, you can imagine that we are going to have a lot of zero values.
And that's even more true considering the fact
that we have a lot of customers that bought baskets of only one product.
For example, this customer number ten here bought some French fries,
this one bought some cookies.
This one bought some mineral water.
So, you know, for these three customers who bought only one product,
we're going to have only one column
that contains a non-zero value and all the other columns.
That means all the 119 columns will contain zeros.
So you can see that we're going to have a lot of zeros in this matrix.
And so for those of you who are discovering sparsity,
I'm happy to introduce you to sparse matrices.
So let's build this sparse matrix right now.
You'll see that it's going to be very easy.
So let's go back to our code and let's create this sparse matrix.
So to create this sparse matrix we're going to use a package of course.
And this package is the rules package.
So we're going to install it and import it.
So as usual we're going to take the function install dot packages.
And then in parentheses we just input the name of the package in quotes.
And that's the rules package.
All right.
So let's check to see if I have it.
Well I already know I have it.
It's actually already here and already imported.
So that's the package for which the description says that
it's mining association rules and frequent item sets.
Okay.
So mine is already installed.
So I'm not going to execute this line.
I'll just put in comments.
And so if you don't have the package here in the packages list,
you need to select this line and execute.
And this will install the package without any issue.
And as far as I'm concerned, I'm just going to put that in comment right.
And to make sure that the rules package is well imported,
we need to add the line here.
Library and in parenthesis a rules already there.
Perfect.
And that makes sure
that if you execute the whole script, the rules package will be imported.
And now we're ready to create our sparse matrix.
So since our data set has no use here
because we're not going to use it to build and train our apriori model,
we will call our sparse matrix again data set.
Okay.
And to create this sparse matrix, it's actually almost the same as importing
a CSV file, because instead of writing here read dot csv,
we simply need to type read dot transactions,
read that transactions and then it's the same in parenthesis.
We need to input the name of the CSV file.
So we'll copy that and paste it here.
That's the first argument.
But then we need to specify to this function that the separator
of our CSV file is actually comma.
So we need to add here set equals in quotes comma.
And why do we need to do this.
It's because you know our CSV file.
If you open it with a text editor
you will see that the different products are separated by a comma.
And we actually didn't have to specify that the separator was a comma here,
because that's the default separator of the readcsv function.
But that's not the default separator of the read transactions function.
So that's why we need to specify two here.
So set equals comma.
And actually we could stop here.
But since I promised you to give you
real life data sets I added on purpose and reality in the data sets.
And this reality is about having some anomalies in the data.
And these anomalies are actually some duplicates.
Indeed, when this manager registered all the different transactions,
well, he might have been very likely to make some human
mistakes to put some duplicates in the data.
So for example, if we go back to our data set.
So that's the whole data sets import of that CSV with the red dot CSV function.
And so for example, when this transaction of the 31st customer was registered,
one can make some mistake of putting twice light cream here, for example.
And to train the apriori algorithm we need to have no duplicates.
So there is actually a good way to handle these duplicates.
It's actually very simple because we just need to add an additional argument.
If we look at the reader transactions function here by pressing F1,
you can see that are empty duplicates argument.
And as you can see, it's a logical value
specifying if duplicate items should be removed from the transaction.
So since the apriori algorithm is trained
on transaction data set, they're supposed to have no duplicate values.
We need to add this argument,
removed duplicates, and set it to true,
and that will remove all the duplicates in each of the transactions.
Maybe your data sets won't have any duplicates, but
it's very common to have a few anomalies in data sets, such as some duplicates.
But here we will be fine thanks to this remove duplicates argument.
All right, so we're now actually ready to create our sparse matrix.
So let's do this.
Let's execute this line. And here we go.
All right so now the sparse matrix is created.
Unfortunately we can not have a look at it because as you can see if I click
on this data see here the new data set sparse matrix is not appearing here.
That's actually the old one. So we can close this.
But we can actually get some info about this sparse matrix.
But before getting all this detailed information, well we can see
that we already have some information about the duplicates itself.
When you execute this line to create your sparse matrix
with the reductions actions function, including the duplicates argument,
you will automatically
have this message distribution of transaction with duplicates.
And here we see that we have one five.
That means that there are five transactions containing one duplicates.
And for example, if in your data set you have some triplicate duplicates
that appear twice in any transaction, well,
you will have a two here and you will have the number of triplicate here.
So that just gives the distribution of transaction with duplicates.
And anyway now they're removed.
So we can actually get some more detailed info about this sparse matrix.
And to get this info, we, as we already did many times
before, need to use the summary function.
So summary.
And here we input data set.
All right.
And that will give us some info about the sports matrix.
So let's execute this line. And here we go.
So what do we see here.
First we are reminded that this data set contains
transactions as item matrix in sparse format.
So that exactly means that's a sparse matrix.
And we can see that we have 7000 and 501 rows.
And we have 119 columns.
And we can see that we have a density of 0.03 in this sparse matrix.
And what does that mean.
That means that the proportion of non-zero values is 0.03.
We have 3% non-zero values and 97% zero values.
Okay. Then we have the most frequent items.
So the item that is the most part is mineral water.
Yes, it can be very hot in the south of France.
And it's a good French tradition to have a bottle of water during meals.
Okay then.
The French love very much eggs.
They love spaghetti, French fries, chocolate
and that's all the other products.
And then then we have some interesting information about the distribution
of the baskets of all the 7500 transactions.
So for example here, this one associated to 1754,
means that there were 1754 baskets containing only one products.
And then we have 1358 baskets containing two products,
1044 baskets containing three products, etc..
And we also have the quantiles
of this distribution with the minimum value, the maximum value.
So the minimum value is of course the basket of one product.
The maximum value is a basket of 20 products, and on average,
people put four products in their basket when they go to the store.
All right. So that's interesting information.
But of course we will get some even more interesting informations afterwards.
And speaking of these more interesting informations, we can already have one.
Now it's actually
going to be a visual information because we're going to make a frequency
plot of the different products,
but by the different customers in the store during this whole week.
And so to get this plot very easily, we can use one function
of the arrows package, which is the item frequency plot function.
And in this function
we just need to input two arguments, which is going to be the data set.
So that's the sparse matrix. So that's the first argument.
And the second argument is sub n.
And that's the number
of the most sold products you want to have in this frequency plot.
So for example if I put top n equals
100 I will get the 100
most purchased product by the French customers in this French store.
So let's check it out.
I'm going to execute this line.
Here we go. And that's the plot.
Don't worry I'm going to zoom on it so that we can see better the products.
And here we go.
So that's the first 100 products most purchased by the customers.
So that's kind of interesting.
And if you want to have less products in this plot, you can just look at the top
ten and you'll get actually the first ten products
purchased by the customers, which are of course the same first ten products.
Okay, so this plot is actually going to be interesting for us,
what's coming next, because we will have to choose a value
for the support according to the Priore algorithm itself,
and we will be able to actually use this plot to look at different
supports of the product, to choose a good value for our support.
So that's what we're going to do in the next tutorials.
We're going to start training our apriori model on our data set,
which is going to be the sparse matrix here that we just built.
And so I look forward to building the apriori model with you.
And until then enjoy machine learning.