Hello and welcome to this art tutorial.
So in the previous tutorials we imported the data set and we started
with the big first step of natural language processing,
which is about cleaning the texts we are working with.
So this first step consisted of creating a corpus that basically is a new data set,
but this time only containing the reviews, the text of the reviews.
And basically this is in this corpus that we were going to clean
all the 1000 reviews, and we are going to clean them step by step.
And in this tutorial we're going to do the first cleaning step.
All right so let's do it.
This first cleaning step will consist of putting all the reviews in lower cases.
And what is the purpose of doing that.
Well we are doing this so that in the final sparse matrix
containing all the words of our 1000 reviews, we don't get twice
the same word, you know, with one word starting with the capital letter
and the same word, but not starting with the capital letter.
And of course, we only want to have one version of this same word,
and therefore we'll keep the one with lowercase.
So that's why right now, in this first step of the cleaning process,
we will put all the words of our 1000 reviews in lower cases.
So let's do it.
To do this we are going to update the corpus this way.
Then equals because we are updating it.
That is it will contain new reviews
which are going to be the same reviews but with lower cases.
And to put other words of the reviews in this corpus, we will use the t
m underscore map function that will do the job for us.
So function.
So we need to add some parenthesis.
And now we need to add two parameters.
The first parameter is actually the corpus itself.
But you know the old version of the corpus that is the corpus that we have here
which contains the original versions of the reviews,
that is the 1000 reviews we have in our data set.
And this corpus here will be the new updated version of the corpus,
that is the corpus containing all the reviews in lower cases.
So that's the first parameter, the old version of the corpus.
And the second parameter is a function that is some kind of a transform function.
And that will simply transform each word of the corpus
by replacing the capital letters in lowercase.
So this function is content underscore transformer.
Here it is. Let's press enter here.
And actually this content transformer function can perform
several transformations.
As we can see in the yellow rectangle here the parameter of this function is fun.
And so we need to add the function that we want
to put all the words of the reviews in lower cases.
And this function is called two lower.
All right.
So that's the function parameter that we need to input
in this content transformer which is like a transform function.
Having several transformation possibilities as input here.
And the possibility that we choose is this two lower function
which will put all the words in lower cases.
And basically this tmp function is used so that we can apply this content
transformer to lower function for all the words of the 1000 reviews of the corpus.
Great. So that's actually done.
That's actually all
we need to put all the words of the 1000 reviews in lower cases.
So I'm going to show you now what it does.
So what we'll do before selecting this and executing this.
What we'll do is have a look at, you know, one review of the corpus.
Let's take the first review and then we'll run this line of code
and you'll see what it does to this first review okay.
So let's access to the first review.
And to do this we need to use the as that character.
And then in parentheses we input the corpus.
But then since we want to look
at the first review of this corpus well we need to add some double brackets.
Actually.
And one because this is the index of the first review because indexes in
are started one.
So this way we will have a look at the first review.
And you know, since the corpus is kind of a complicated object,
we need to use these double brackets here to access to the written review.
And besides we need to use this as dot character
function to have this written review displayed.
All right.
So I'm going to press enter here.
And as I just told you we get the written review.
Well love this place.
Which is of course
the first review as we can see in our data set while love this place.
All right. So that's the first review.
That's the original version of the first review.
And now we are going to apply the first step of the cleaning process,
which is to put other reviews in other cases.
So let's do it.
Let's select this line and execute.
All right.
So as you can see it was very fast.
All the 1000 reviews were just transformed in lower cases.
So let's check it out.
Let's check it out for the first review.
So we just need to, you know, press the up arrow
to get the previous command which is this one.
And you know, since
our new corpus is also called corpus, we just updated the corpus.
Well, we can run this and hopefully we'll get
the same review written in lower cases.
So let's check it out.
Let's press enter here.
And here we go.
As you can see the capital W became this little W.
And this capital L became this lower L.
Perfect. So first simplification.
Now in the final big table the final sparse matrix,
we will get two versions of the same word, one in capital letter
and one in lowercase.
We'll get one unique version of the word.
And therefore we did the first simplification
of our future sparse matrix.
So that's the first good thing done.
And now we will proceed to the next step of the cleaning process,
which will be to remove all the numbers of the reviews, because indeed the numbers
are not very relevant in telling if a review is positive or negative.
Well, we need to be cautious actually, because maybe some reviews are,
you know, on a scale of 1 to 10, I give it ten.
Well, that's definitely a number that is fully correlated to the outcome,
whether the review is positive or negative.
So we should pay attention to that.
But we could have other numbers that are totally irrelevant,
like, you know, some addresses that contain numbers or phone numbers.
Well, that would be a little weird in a review, but we never know.
Well, in general, when we are dealing with text, when we're working with text,
we want to get rid of the numbers because these are most of the time
not very relevant.
And you know, this could add a lot more columns.
So in general it's better to remove the numbers.
So that's what we'll do in the next tutorial.
And until then enjoy machine learning.