Hello and welcome to this art tutorial.
So we're still trying as hard as we can to simplify
the corpus in order to reduce the future sparse matrix of features.
As much as possible.
And so far we put all the words in lowercase.
We removed all the numbers and removed all the punctuation.
And in today's tutorial
we are going to remove all the non relevant words of our reviews.
So what is the non relevant word.
It's for example this word here.
Indeed if we know that this first review well love this place is a positive review.
It's not thanks to this word.
It's of course thanks to the loved word.
So that means that this word here is not a relevant word.
And so it's totally non relevant for us and not useful to include it
in the future. Sparse matrix.
That will be nothing else than the matrix of features
and the outcome whether the review is positive or negative.
So that's what we'll do in this tutorial.
We will remove all these non relevant words.
And we will update our corpus of reviews by removing all these words.
All right. So same as before.
Very simply we take this line copy and paste it below.
And in this line we're going to replace remove
punctuation by remove words.
But that's not all this time.
It's not as simple as before.
We need to add something
which is a something that will specify which words we want to remove.
And actually there is a built in list of non relevant words
that is called stopwords.
And that basically contains all the non relevant words like this.
Although the articles prepositions like and or well you know all these words
that don't help the machine
learning algorithm figure out if they reduce positive or negative.
So this list of non relevant words in the Stopwords list is practically
always used in natural language processing, because indeed, these words
will never help you or your algorithm to classify your texts.
So you will most of the time use it.
And therefore that's a very important step because of course
that simplifies the corpus and reduces the future sparse matrix very much.
So as I just said, this is not what we only need to input in this map function.
We need to input a third parameter.
And that corresponds to the words we want to remove.
That is all the non relevant words.
And these words are in this stopwords
function.
All right.
So basically this returns all the words that are not relevant for our model.
And therefore thanks to this function here we will remove
all the words returned by this Stopwords function.
All right.
So that's all for this line of code.
But we need to add a little something.
Well especially for you
if you're doing natural language processing for the first time,
which is a library that we need to install and import
to be able to use the stop words function, because this function is not included
in the default package of R,
so we need to install the required package to use this function.
And this package actually has a funny name.
It is called snowball C.
And so let's now install this package so we can, you know, copy this line,
paste it here.
And in this quote here in the parenthesis we input snowball.
So it's spelled this way. Snowball.
And then C.
All right.
So right now in some comments but you know check in your package this list
if you already have this snowball package we never know.
And if you don't have it well you can execute this line
without the command to install the package.
All right.
And now of course, as usual we will import the package
automatically thanks to this library function.
So we will same copy this line, paste
it below and replace time by actually snowball
C. All right snowball.
See now the required package are installed
and imported to be able to use this Stopwords function.
So everything is all good.
And now let's try it on the first review.
Because the first review contains some irrelevant words like this.
Here, it might be the only word that is removed because you know
well might not be a word of this stopwords list because, you know,
the Stopwords list is a list of common words like, articles and prepositions,
common words, but irrelevant words and well, is not common.
So it might not be removed.
But definitely this will be removed because this is a common
and non relevant word.
So let's check it out.
We will select this line and execute.
Here we go.
And now let's have a look at the first review by pressing the up arrow.
Here it is. Let's press enter.
And as I just said this was removed.
So now the first review becomes well loved place.
And you know even if we really simplify the review no.
Now it doesn't look the same as the original review.
Well we can still understand that it's a positive review.
And especially our machine learning model will understand it very well.
And that will be thanks to this loved word here,
which is a word that, of course, might be present in some other reviews,
which themselves will be positive reviews as well.
So that's how our machine learning algorithm will understand
that love indicates a positive review, and therefore
that's all it needs to establish that kind of correlations.
And this is totally unused form.
And we were right to remove it.
All right.
So that's done for this step.
That was another very important step.
But that's not all.
In the next tutorial
we'll do another very important step which is the stemming step.
And so I'll explain you what it is
and how we perform this new cleaning step in the next tutorial.
Until then, enjoy machine learning.