All right.
Are you ready.
To do some cleaning?
Yes. Good.
Because now we're about.
To do a deep. Cleaning of the. Text. Right.
So far, the texts have punctuations.
Different characters.
Other than. Letters that have capital letters.
Lowercase. The verbs are conjugated differently.
Well, we're going to simplify all this.
And that's indeed an essential step when doing natural language processing.
We need to clean the text as much as we can in order to.
Ease the. Learning process of the.
Future machine learning model.
Which we will.
Train to understand the reviews, understand English.
Basically.
And. Predict if the reviews are positive or negative.
All right, so let's do this.
Let's first. Import the libraries meaning the tools.
Which will. Allow us to. Clean these. Texts.
So the first one and that's the main one the most essential one.
It is the library.
All right.
So let me just. Import it first.
It's simply called r e.
And that's the library. You know we'll use two.
Yes. This one the first one that's.
The library we'll use to simplify basically the reviews.
But this is not the library that will allow us to do the.
Stemming, which I will remind and explain later.
On. Okay. So r e.
Then we will import of. Course, the.
MLK library.
Very classic library.
In natural language processing which.
Will allow us to download.
The ensemble. Of Stopwords.
So what are the stopwords?
These are, you know, the. Words.
We don't want to include in our reviews.
You know, after cleaning the texts, which, you know, are.
Words that. Are not relevant.
To help. The predictions of whether.
A review is positive. Or negative.
And these words include, you know, the simple ones like the, you know.
All the articles like the, And you know, all these.
Words which.
Don't give any hint of.
Whether a review is. Positive or negative.
So we will remove all these words, you know, all the words that are not.
Helpful to predict if a review is positive.
Or negative.
And speaking. Of these, stopwords.
Well, now that we import it and like we can call Nltk
and from which we're going to download.
Well, all this upwards.
And to specify this
we need to enter here in quotes inside this download function from the Nltk.
Library.
Stop. Words.
And this will. Get all the stopwords. And.
You'll see. Later on how.
We use this.
To indeed not include these non relevant words in our.
Reviews. Okay. So good.
Now we're not done with analytic.
Yet because. From.
NLT okay.
And then from the.
Corpus. Module of the Nltk. Library we.
Will. Import.
All these.
Stopwords that we just downloaded before.
There we go.
So basically this line of code downloads them.
And this line of code imports them into our notebook.
Okay.
So all these stopwords.
And finally steal from. Nltk.
And from.
The. Stem module from the Nltk.
Library.
And then once again from the Porter submodule of the stem
module of the NLG library.
Well, we're going to import.
A class which is the Porter.
Stemmer class. Perfect.
And this is a class we'll use of course.
To apply.
Stemming on our. Reviews.
So now let me remind what this is about.
Stemming consists of taking only the.
Root of a word.
That indicates enough about what this word means.
So for example, let's say.
There is a review that.
Says, oh, I love this restaurant. Okay.
And let's say we want to apply stemming to.
The word loved.
Well, what it will do is that it will.
Transform love into.
Love, just to simplify.
The review, because whether we say, oh, I.
Loved this restaurant or, oh, I love this.
Restaurant, well, you know, that means the same.
That means that the review is positive.
So we can totally remove all the conjugation of the verbs,
you know, just keeping the present tense so that we can indeed.
Simplify the reviews.
Because remember at the end, you know, after cleaning the text, when creating.
Actually the bag of words model, we will create a.
Sparse matrix where in each column we will have all the different words.
Of all our. Different reviews.
And therefore, in order. To.
Optimize or, you know, minimize the dimension of this sparse matrix
where the dimension is. Exactly.
The number of columns, well, we need to simplify as much as we can.
The words.
And if we don't apply stemming, well, you know, in the sparse matrix
we will have one column for love and one column for loved.
And since that means the same thing, that would be redundant
and that would make. The.
Sparse matrix even more complex, you know, with a higher dimension.
So that would be wrong.
So that's exactly what the.
Stemming is about. It consists. Of.
Reducing.
The final dimension of the sparse matrix so that we can indeed have not too much.
Trouble to learn the text.
From our machine learning model. All right.
So that's what this poor Stemmer class will do.
And now there you go, my friend.
We can start cleaning the text.
We have all the tools we need.
So the. First thing we'll.
Do is. Create a new list.
Which we'll call. Corpus. All right.
And we will. Initialize this list.
As an empty list.
And what. Will this list be. Exactly.
You know, what will it contain?
Well, it will simply contain.
All our. Different reviews.
You know, all the different reviews from our data set.
But all. Cleaned and all into.
This list. Corpus.
So what we'll do actually.
Is, you know, we will make.
A for loop to iterate through all the different reviews of our.
Data set.
And for.
Each of these review, we will apply.
A cleaning process. You know, by.
Putting all the letters in lowercase.
And removing the punctuations and removing the.
Stopwords. All these things.
And we will do that one review.
After another, and each time we clean a. Review well we will add it to this.
Corpus.
So this corpus will.
Only get in the end all the cleaned reviews.
Okay.
And we do this of course.
Because then the future functions will use in the next steps.
Expect our reviews.
In the list and all cleaned.
Okay. So corpus.
And now now that we initialize.
This list well. We're going to.
Populate it with the clean reviews through a for loop.
For loop which will iterate,
you know, with this classic looping variable I.
In the range from.
Zero to well. Guess what.
Guess what is the upper bound?
You know, we will simply iterate.
Through all the reviews. And since we have one.
Thousand. Reviews in our. Data set, well.
I will simply go from.
Zero to. 1000.
Right? As simple. As that.
It will iterate
through the indexes of the reviews, which go effectively from zero to. 999.
Okay.
So for loop ready.
And now we can go inside the for loop.
And there we go. Now we're going to apply different.
Steps to.
Clean each and every single review.
Of our data sets.
So first of all.
We will create a new variable which we will call review.
And that. Variable will exactly be that.
Clean review. But you know we will clean it step by step.
So we will update review.
Each time we proceed to a new kind of cleaning.
And the first kind of cleaning will. Do will be.
To remove all punctuations.
In other words, it will be to keep only the.
Letters in our reviews. All right.
And to do.
This we're going to call our r e library.
From which.
We're going to call this sub.
Function, which is a function that can.
Replace.
Anything in a text, you know, in a string. Actually.
By anything else you want.
And what we're going to.
Replace actually.
Is. Any element that is not. A letter.
You know, from A to Z.
By a space.
So that every. Punctuation like.
Quotes, double quotes, commas or.
Collins or anything you want will.
Be replaced by a space.
And it. Has to be replaced by a space, because otherwise.
We can have two. Words that stick together.
So we need to make sure we.
Replace the.
Punctuations. By spaces so. That we can indeed.
Still. Separate the words. All right.
And a way to. Do this.
Thanks to.
This sub. Function is to enter here in the. Parameters.
First what we want to replace.
And the trick.
To say that.
We want to replace anything that is not a.
Letter is to do it this way.
You start with a pair of square brackets here just like that.
So that's what's. Inside.
This pair of square brackets.
Will be what will be replaced.
You know, by. The spaces.
And the trick to say that what we want to replace or anything but.
Letters. Is to include a hat. Here.
And now we'll explain what this means and then add.
A. So you have to do it like that actually.
Double hat a. Okay a.
Dash z.
So all the lowercase letters from A to z, but also then.
All the capital.
Letters from A to z.
All right.
And what this hat means. Is exactly.
Not, you know, this symbol
in mathematics or computer science means not meaning not.
All the. Letters from A to Z. In lowercase.
Nor. The capital letters from A to Z, which.
Is exactly. What we want.
We want to replace anything that is not the letters from A to Z in lowercase.
Or capitals.
By spaces.
And the. Way to specify that.
We want to.
Replace all these by spaces is, well, exactly.
What we have to enter here as a second parameter
and which we will enter in, you know.
Quotes. But inside. A space.
Right. What's inside. These quotes is exactly.
What we want to. Replace.
Those non. Letters here by.
All right. So we're going to replace everything.
That is not letters meaning all the punctuations.
By this space okay.
And then finally we have to enter one final argument.
Which is of course.
Where we want to do all these.
Replacements, you know, inside what's inside.
Which review.
Right inside which text. And so very simply.
The third. Parameter we have to enter here.
Is the review. In which we want to.
Do all these replacements.
And to. Access the review.
Well that's very easy.
We need to take our data set.
There we go. Then we need to take. The right.
Column of the data set which contains the reviews. And that's.
Of course the.
First column which we can either access with the dialog function
and then specifying the index. Zero.
Or I'll.
Show you another trick by adding here pair.
Of square brackets.
And then just entering in. Quotes.
Well the name of the column which is.
Read view right.
If we go back.
To our data set, you will see that.
The name of the first column is. Review. Okay.
So as you want I look works very well too.
And now do we need to add something here.
Well of course because this. Only gets.
The first. Column containing. All the reviews.
But now we were dealing with a specific review.
The one of index I.
And therefore to catch that specific review.
Which we want to clean right now inside is for loop.
Well is simply to add in a new.
Pair of square brackets.
I. All right.
So this will get the review.
Of index I in this first column review.
Of the data. Set.
And that's exactly. What we want.
All right.
First cleaning done.
Now we. Will proceed to two other. Types of cleaning.
Then we will take a little break. And then we will. Proceed to.
Stemming which will consist of, you know, simplifying the words
in order to get only the root and therefore simplifying eventually.
The. Sparse matrix.
All right.
So that new step will be to. Transform.
All the capital. Letters into. Lowercase.
And that's very.
Easy.
To do this we just need to take our review variable.
From which we can now call.
A. Specific function. Because, you know, we. Created this variable.
As an. Output of this step.
Function by the. Re library.
And therefore like objects.
It now has some attributes and functions.
Or you know methods that you can call and that function.
We want now to.
Simplify all the letters into lowercase is the lower.
Function. And that's it.
You just have to enter it like. That.
Review dot. Lower.
But this will return.
The review with only lowercase. Letters.
And since we want to.
Update our review variable, well we simply need to add here review.
Equals write equals the result of.
Applying the lower function to.
Our previous. Version of the review.
Okay. So very. Easy.
And then one final cleaning.
Before we proceed to.
Stemming in the next. Tutorial.
Well actually what.
We have to do now is something.
To prepare. For the. Stemming.
And that's something is to. Split the.
Different elements of. The reviews.
In different words.
Actually, because the different. Elements are now. Words.
So we're going to.
Split the review.
Into its different words so that then we can apply stemming.
To each of these.
Words, you know, by simplifying them.
By their root okay.
So the way to do this. Is very simple.
Once again you know, I'm going to copy.
This. And.
Paste it here.
And instead of calling the lower function I'm simply going to call.
The split.
Function. as simple as that.
This will split your.
Review into its different words.
And now now, my friends, we're. Ready.
For the stemming.