Hello and welcome to this art tutorial.
So, so far we've been doing a great deal of simplifications
for our corpus and therefore for a future sparse matrix of features.
But we can still do better, and doing better is what we'll do in this tutorial.
And it's a new step of the cleaning process that is called stemming.
So what is stemming?
Well, stemming is about getting the root of each word.
For example, if we look at the first review, we have this loved one here.
And the root of this word would be love.
So what is the purpose of getting the root of the word?
Well, it's of course still related to our goal to reduce the total
number of words that will be in our future sparse matrix of features.
And we can do this by taking the root of the words.
Because whether we have loved or love or will love or loving,
well, this actually means the same thing for our algorithm.
And not only it means the same thing, but also it gives the same hint
whether the review is positive or negative.
So we don't really need to have some different tense of one same verb.
And we don't really need to have derivative words.
We just need the root of the words.
And that will be perfectly enough for our machine learning classification
model to train on the future sparse matrix of features
that therefore will contain only the roots of the words.
And you can imagine how we will considerably reduce
the final total number of words.
That is, the final total number of columns in the sparse matrix of features.
Because of course, by only keeping the roots
of the different versions of the same word.
Well, of course that simplifies it very well and therefore
considerably reduces the final total number of words.
So that's stemming.
That's also a very important step in natural language processing.
You will most of the time apply stemming to your text
whether you are working with reviews or articles or books or HTML pages.
Well, for any kind of text, it's really help your machine learning
algorithm to do an even better job for your classification problem.
So let's do it for our reviews.
And it is still going to be very simple.
We will do another copy paste here.
So I will actually copy this line because we only need two parameters
the corpus and a function that will perform the stemming.
So based here and I will replace
remove punctuation by the appropriate function to proceed to the stemming
which is the stem capital D document.
Here it is.
That's the function we use to perform stemming on all the reviews of our corpus.
So let's check it out.
Let's select this line right now.
Our first review is well left place.
And you'll see that after stemming left becomes love.
All right. So let's execute now.
Press command and control list.
Enter to execute.
Here we go. New corpus updated.
And now let's have a look at the first review of this new corpus.
So I'm pressing the up arrow here to get this line of code.
And now pressing enter.
And here we go.
Wow love and place.
So loved was replaced by love.
Because the root of love is love.
All right. So.
And that's the same for all the reviews
and all the other reviews, the words were replaced by the root.
So that's done for this new step.
And actually we are almost done with the cleaning process.
We have one final step and we will do this final step in the next tutorial.
Until then, enjoy machine learning.