Hello and welcome to this art tutorial.
So we did the big part the cleaning process.
We started by creating this corpus here.
Then we put all the words in lower cases.
Then we removed all the numbers, all the punctuation
and we also removed all the non relevant words.
And finally we took the roots of all the words in the 1000 reviews.
So all these steps considerably simplified the words in the reviews.
So thanks to all these steps here, all the simplifications.
Well, our final sparse
matrix of features will get a much smaller number of columns.
So that's good for algorithm because we reduced sparsity.
And now we have one final little step to do
that will not be done to simplify even more of the corpus.
But that just consists of removing the extra spaces.
Because by doing all these simplifications here, well,
you know, we removed some things, and those things that were removed
could actually have been replaced by a space,
and that would cause an extra space in the review.
And therefore, if we want to have
perfectly clean reviews, we must remove these extra spaces.
If we remove all the extra spaces, well, the columns in our final sparse matrix
and features will only contain the words that are relevant
and will not contain any space or anything else.
So that's what we'll remove in this final step of the cleaning process.
So let's do it again.
It's going to be very simple.
Will copy this line here.
Copy and paste it below and replace
stem document here by strip whitespace.
Here it is pressing enter and all good final step ready to be executed.
And therefore the whole cleaning process is ready to be completed.
So before we execute this line, let's have a look
at a review that now has an extra space.
I remember that when we removed the number in the review 841.
Well, we got an extra space. Here it is.
You know, before the review was for 40 bucks ahead.
I really expect better food.
And when we applied the remove numbers here to the corpus, well,
the number 40 here disappeared.
But it actually didn't disappear.
Was just replaced by this extra space here.
Because indeed we can see that we have two spaces between four and books.
And what we want to get is only one space here.
So we're just removing the extra space here.
So let's check it out.
When we select this final line of code of our cleaning process.
Well, let's make sure that this extra space
in the review 841 disappears, this time for good.
All right.
So I'm going to press Command plus enter to execute.
Here we go.
And now let's have a look at the review 841.
So let's press the up arrow here to find it.
Here it is.
We're going to get the new version of the review 841
after we applied this strict whitespace to the corpus.
So let's do it pressing command here.
Well not only the space was removed,
we can see that we don't have any extra space here.
But also we get all the other steps of the cleaning process.
Since indeed we can clearly see that the non relevant words were removed.
So the non relevant words were for example four.
So this was removed.
And then a I, I think that's all.
Yes. As you can see a was removed here and I was removed here.
Okay. So that's the first thing we see very clearly.
And the second thing we see very clearly is the stemming of course,
because we hardly recognize the words here.
Box was replaced by bug.
You know, it's not only about the past tense of verbs,
it's also about the singular and plural of nouns.
So books became bug
and head was not replaced because the root of head is head.
So that's why we kept head here.
And then, really,
really with a why became really with an eye.
So that's how are simply interpreted the root.
So that's okay. It's not a mistake.
And finally expect better.
And food were not replaced because you know these are kind of already
the root of the words.
And we cannot really simplify the words more.
All right.
So that's a good example.
We can clearly see what happened here.
And for what we just did here with the stripped whitespace
we can see that this extra space here was removed.
Well there is still a space here.
But you know before there was the four and two spaces between four and book.
And then after the stopwords step, the four was removed.
So we can still see a space here before bug, but that's actually one space
instead of two.
If we want to get
even more convinced, well, we can see that we have an extra space here
between love and place.
This review is the first review in the previous version
of the corpus, before we applied the strip whitespace here.
And so now we'll see that if we have a look at the first review
of the new version of our corpus, well, this extra space here will be removed,
and we will get only one space between love and place instead of two.
Here. Let's check it out.
Here we go.
We can clearly see that we have only one space here instead of two spaces here.
All right, so good.
Removing the extra spaces worked properly.
And so we are done with the cleaning process.
So great.
That means that now we are ready to build the sparse matrix of features.
We'll do that in the next tutorial.
And until then enjoy machine learning.