Hello and welcome to this art tutorial.
So that's it.
We did the first step
of natural language processing, which consisted of cleaning the text
we're working with.
And now it's time to create the sparse matrix of features
containing all the different reviews
and the rows and all the different words of the reviews in the columns.
So as a reminder, what we're about to build is a huge table
in which the rows are the 1000 reviews.
So we're going to have one row for each review.
So we're going to have 1000 rows.
And then the columns are going to contain all the words
we can find in the 1000 reviews in this corpus.
That is the 1000 cleaned reviews.
And so basically what this means is that we are going to take
all the different words in the 1000 cleaned reviews in the corpus,
and we're going to create one column for each word.
So suppose we count in total 1005 hundred words in this corpus of reviews.
Well, that means that our huge table is going to contain 1005 hundred columns.
And then for each cell in this huge table, well, then each cell will correspond
to one review corresponding to the row and one word corresponding to the column.
And so the value that will contain
the cell is the number of times the word appears in the review.
So, as we explained earlier, since most of the words don't appear
in the reviews, well, most of the cells will contain a zero.
And then of course we'll get a few ones because each review is composed
between 5 to 10 words.
So in each row we're going to have 5 or 10 cells having a one,
and all the other cells will have zero, and the cells that have
a one will be in the columns corresponding to the words that are in the review.
And maybe sometimes, but very rarely, we'll get a 2 or 3.
That happens when the word appears twice or three times in a review.
I can give you a simple example.
Let's imagine that we have
a super positive review saying, I love this restaurant very, very much.
Well, in this review the word very appears twice.
So for this particular review in the table, that is, let's say it's
the row 100.
Well, in this cell that belongs to this row
and that belongs to the column corresponding to the word.
Very well we'll get a two because very appears twice in this review.
So that could happen. But it's very rare.
And what's most important to understand is that in this huge table
we'll get mostly zeros a few ones and very few twos or threes.
And we'll get so many zeros that we call this table a sparse matrix.
A sparse matrix is a table that contains a lot of zeros.
It contains very few non-zero values.
And that's exactly
what we're about to obtain because of what we've just explained.
And using the information of the table we build that we have the word sparsity.
And sparsity refers to that situation where we have a lot of zeros.
And speaking of sparsity, let's also keep in mind that if we clean
all the reviews here in this first step of natural language processing, it's
in order to reduce as much as possible this future sparsity
that will occur in this huge table that we're about to build.
So that's the whole point behind this first step here.
Cleaning the text.
It's to avoid having too much sparsity.
That is it's to avoid having a table too big with too many columns.
Because remember one column is created for each word in the corpus.
So by doing all these steps here, we removed a lot of words
and a lot of characters, punctuation, numbers, etc.
so that in this final huge table we get the minimum
number of words and therefore the minimum number of columns.
And one last quick reminder we are creating this table
in order to have the framework of classification models.
That is, you know, having several independent variables and one dependent
variable.
We haven't created the dependent variable yet.
It's actually in this data set here we will just take
the second column of this data set because this contains the outcome.
Whether the review is positive or negative we can see that here.
It's the second column light one.
If the review is positive and zero of the reviews negative.
So that's the dependent variable column.
And the independent variables are going to be nothing else than these columns
corresponding to each one of the words in the cleaned reviews of the corpus.
Because for each review that has each observation, we can link
the review to each of the columns, because for each of the review,
we can associate a value for each of the columns, and this value is
the number of times the word corresponding to the column appears in the review.
So that's how we create our independent variables.
And then we'll create our dependent variable.
And therefore we'll get the classification model as we used to work with.
And eventually we went because we will have everything.
We will have our independent variables, we will have our dependent variable.
And we already have all our classification models.
That's the models we made in part three.
So we will just need
to apply these models on our new data set that we were about to create
that contains the independent variables as the words and the dependent variable
as the light column in our original data set.
All right.
So let's do it. Let's create this table.
And in R we can do it very efficiently using a function
a function that is called document or matrix.
And it's super easy because this function will only take one argument.
And as you might have guessed, it's going to be the corpus.
And that's it.
This will create this huge sparse matrix with all the 1000 reviews in the rows,
and with all the words of the reviews in the columns.
So let's do it.
Let's call this sparse matrix of features DTM,
because the function we're about to use is document or matrix.
So so far we'll call it DTM.
So equals.
And then we use this super function document
term matrix.
Here it is. I just need to press enter.
And as I just said we just need to input one argument which is our corpus.
All right.
And that's done. Here it is corpus.
This will create our sparse matrix of features.
So I'm going to select this line and execute
and done the sparse matrix of features is created.
It appears right here DTM.
We can click on this button here to have some info.
And actually what's interesting to see now is the total
number of words counted in the corpus to create all the columns.
And we can see this total count here 1577.
So that means that the number of columns indicated by end
call here in our document or matrix
or sparse matrix is 1577.
So that means that this huge table has 1000 rows.
So we expected this because of course we have 1000 reviews,
but we didn't expect the number of columns in total because simply
that was the total number of words in the reviews.
So we can count them.
But we can see this number here 1577.
So that's already a big table.
But be prepared if you're working with more complicated
text or longer text like articles or books.
Well, you might get a lot more columns here
because you will get a lot more words.
So what you'll have to do
and you can ask me any questions about that in the Q&A, is reduce
even more to sparsity by filtering the words in your text.
And speaking of filtering, that's what we're going to do right now.
We are going to apply a filter to clean even more the reviews
by only considering the most frequent words
that means that it's like we're going to add a step in this text
cleaning process,
which will consist of only taking the words that are the most frequent.
For example, the words that appear in only one review,
well, they might be removed because they're not frequent,
they only appear once.
Only one cell in the matrix contains one, because these words only appear
in one review.
And these words, of course, are not very relevant
because since they only appear in one review, well, our machine learning
classification model will not be able to establish any correlation
between this word and the outcome, whether the review is positive or negative,
because indeed, to understand such correlations,
the word would need to appear in at least two reviews.
So that's the kind of words we're going to remove.
And again this is in order to reduce sparsity.
And speaking of sparsity I will show you something very interesting.
Right now if we go to the console and type here DTM,
then we'll get other information about this sparse matrix of features.
And the information that I want to
highlight here is of course this sparsity information.
And as you can see the sparsity is 100% right now.
And that's because there are a lot of zeros in the matrix.
And also because we haven't filtered any non frequent word yet.
So that's what we'll do right now.
We will filter all the words that appear only once.
We will filter all the words that are not frequent in the reviews.
All right.
So let's do it.
To do this we are going to update our document term matrix.
So we're taking again DTM here.
All right.
Because we're updating our sparse matrix and equals.
And now we're going to use a function a very practical function
that will filter the non frequent words of our sparse matrix
which so far is nothing else than DTM.
So DTM will be one of the inputs.
And we will filter all the non frequent words by specifying a proportion
of non frequent words that we want to remove from the sparse matrix.
And this proportion of non frequent words will be obtained
thanks to the second input of this function.
Because the second input is
the percentage of the most frequent words we want to keep in the reviews.
So let's say we want to keep 99%
of the words in the review that are the most frequent words.
Well, this second input will take the value of 99%.
So let's use this function.
This function is remove sparse terms.
Here it is remove sparse terms.
So pressing enter and ready to input the two arguments.
So the first argument
is of course the sparse matrix on which we want to apply this filtering.
So of course it's DTM.
All right.
And the second input is the proportion of words that are the most frequent words.
And that will be kept in this sparse matrix.
So let's say we want to keep 99% of the most frequent words.
Well we would need to input here oh point 99.
And therefore we will build the same sparse matrix,
but this time containing 99% of the words
that are the most frequent in this sparse matrix of features.
And therefore, you know, we're not looking at the corpus containing all the words
and counting the most frequent words of this corpus, where this function remove
sparse terms will do, is to look at all the columns of the sparse matrix here,
and then keep 99% of the columns that have the most ones in the columns.
Because each column corresponds to a word, and therefore, when there are very
few ones in the columns, that means that this word appears in very few reviews,
and therefore these are the words
that are non frequent in the reviews and accordingly not relevant.
And that's why we can remove them.
All right.
So let's do it.
Let's apply the filter to be cautious.
Let's maybe take a higher proportion of frequent words we keep.
Because actually with 99% we might remove a lot of words.
You can try it on your studio to see.
But here since we don't have many reviews, you know, we have 1000 reviews.
That is not much compared to other texts.
We can work with in natural language processing.
Let's be careful here and apply a 99.9% proportion of frequent words.
So here I'm going to add a nine
and you'll see that it will already remove quite a lot of words.
So let's try it I'm going to select this and execute.
And indeed as you can see we now have
691 columns in the sparse matrix.
That is we only kept 691 words.
So clearly we can see that
by keeping 99.9% of the words that are the most frequent.
Well, that's already removes almost 1000 words,
because originally we had remember more than 1005 hundred words.
So be careful with this.
Be careful not to apply a too low
proportion of frequent words you want to keep, and to choose that.
Remember to look at the total number of words
that is counted when you build this first sparse matrix.
And of course, you can also choose this number by considering
the total number of reviews you have in your original data set.
And you know, since we only had 1000 reviews,
well, that's why we take such a high proportion here.
And let's see by how much we reduce the sparsity.
So we have to type DTM again because our document sparse matrix
that is our sparse matrix was just updated with all these words removed.
So pressing enter here and the sparsity now became 99%.
So better.
But anyway that was fine because we didn't have too much columns.
You will see that if you work with larger text
you will get a lot more words and therefore a lot more columns.
All right. So that will be all. For this tutorial.
We built our Bag of Words model.
Congratulations for that.
And now it's time to make the classification model.
So that's what we'll do in the next and final tutorial of this section.
And until then enjoy machine learning.