Hello and welcome to this art tutorial.
So in the previous tutorial, we imported the data set
which contains 1000 reviews of a restaurant.
And for each of the review,
we have the information whether the review is positive or negative.
So one when the review is positive and zero when the review is negative
and we are trying to build a machine learning model
that will be able to classify each new review
and tell if this new review is positive or negative.
So basically what we are allowed to do is something we already did.
That is classification and part three.
But this time we are working on text and therefore we need to figure out a way
to create a model where we can have some independent variables
to train a machine learning classification model,
to learn some correlations between the independent variables
and the dependent variable, which is of course the liked column.
So now the goal is simply to create some independent variables.
And what could be those independent variables.
Well the IDE and natural language processing
and that's the bag of words model, is to create a model which will basically
be a huge table, a table where the rows are nothing else, and the reviews.
So this table will have 1000 rows because we have 1000 reviews,
one row for each review and the columns will simply be
all the words that we can find in the reviews.
You know, we take the 1000 reviews, we look at all the words
in those 1000 reviews, and we are going to create one column for each word.
And then each cell in the table will correspond to one row that is one
review and one column that is one word in all these words in these 1000 reviews.
And then this cell,
there is going to be the number of times the word appears in the review.
So for example, here is the first word.
Well, well there's going to be a column for this while word here.
And so for the first line that corresponds to the first review.
Well this well column will get a one because the word
well appears once in the first review.
But then for all the other reviews, that is all the other rows,
we don't see any wow here.
So all the cells in this first column for the other rows will get a zero.
And that's what will be our model in the end.
It's going to be this huge table with 1000 rows that are going to be the reviews.
And a lot of columns that are going to be all the words
that we can find in this 1000 reviews here.
So now you might start to figure out why we would need to clean the reviews.
It's because
since we are going to take all the words in these reviews here, well, we are going
to get a lot of columns because we create one column for each word.
But we don't want to get too many columns, because the more we get some columns
and the harder
it will be for our machine learning model to run properly, to execute efficiently.
Not only our machine learning model will have more trouble to execute properly,
but also it will have more trouble understanding the correlations
between the presence of the words in the reviews and the information.
Whether the review is positive or negative.
Because of course, if we keep all the words in these reviews, well,
we will get some irrelevant words, some words that will not help the machine
learning algorithm to predict if a review is positive or negative.
Because you know the words that we can find in the reviews here.
Well, some words
give a much better hint in telling if the review is positive or negative.
Let me give you a simple example.
We have this loved word here in this review.
This is the word that basically tells us that the reviews positive
because there is this love word.
But then if we look at this, this word or even place,
well, these two words don't give the machine learning algorithm a hint
whether the review is positive or negative.
It's of course, this word loved
by which the machine learning algorithm will understand some correlations
between this love word and the fact that the review is positive.
So that's the whole reason why right now we are going to clean the text.
It's not only to reduce the Bigtable we're going to get in the end,
because we want our algorithm to run properly and not be saturated.
And the other reason is that we want to get the most relevant words
to find the best correlations between the presence of the words and the outcome,
whether the review is positive or negative.
All right.
So now we get the point.
So let's start cleaning the reviews.
And an important thing to understand is that what we'll do here
to clean the reviews.
Well it's the same technique to clean any other kind of text.
Well I will give you the main tools.
You will be able to use these tools to clean any text you're working with.
And of course,
if your text is a little more complicated, like for example, an HTML pages
that contains HTML tags, well, you would need to add a little more tools,
but you would still use the tools that we are about to use.
And the good news is that
if you want to use more tools to clean more sophisticated text, well,
you just need to ask me some questions in the Q&A.
And now I'll help you add these tools to your problem and your text.
But what we'll do here, you will definitely do it to perform
natural language processing on your text files for your problems.
All right.
So let's do it. Let's clean the text.
So that's the next step in natural language processing.
We will clean all the text.
And then we will create our bag of words model which is this huge table
which is actually called a sparse matrix because we'll get a lot of zeros
in the sparse matrix.
And then that means that we'll get a model where we have some independent variables
and one dependent variable.
And that's when we'll be able to use our machine
learning classification models that we built in part three to predict
the class of a new review that the model will not have seen yet.
So let's do it.
Let's start with the first step of cleaning the text.
And this first step is going to be about initializing a corpus, because,
you know we will not clean the reviews directly in the data set.
We will instead create a corpus which will contain all the reviews,
and that will be in this corpus that we will clean all the reviews.
So let's start by training this corpus.
And in order to create this corpus we need to import a package.
And you might need to install it
if it's the first time you're doing natural language processing.
So I'm going to type here the command to install this package.
This package is called the TM package.
It's a very famous package in R for natural language processing.
So let's install this package by typing install dot packages.
Here it is.
And so the name of the package has to be input in quotes which is the TM package.
All right.
So if I go to my packages I will find
the TM package.
Here it is.
So I already have it installed so I don't need to install it again.
So I will put that in comment.
Here we go.
And of course if you don't have the team package here, you will need to install it
by executing this line and everything will run properly.
All right.
So of course after we install the package we need to import the package.
Well mine is already imported but we need to automate all this.
So as usual we are going to take the library command.
And in parentheses we input the name of the package.
So TM again not in quotes. All right.
And now we are ready to build the corpus.
And so how are we going to call our corpus.
We're going to call it corpus.
All right. Corpus equals.
And now we're going to use the V corpus function spelled
this way V corpus with capital v and capital C.
And then in parenthesis we need to input vector source.
This way. And again new parentheses.
And it's in these parentheses that we input
the column that contains the text that we want to clean.
In this corpus.
So this column is of course the review column of our data set.
So here we will input this column which can be taken the following way
by typing data set dollar sign.
And then as you can see we have the two variables here that we can pick.
And the one we want to pick is review.
All right.
And that takes all the review column of our data set.
And that's exactly what we want because this column contains the text.
And we want to clean the text in the corpus.
All right.
So that's the first step of cleaning the text.
So let's select this and press
Command or Control plus enter to execute and create the corpus.
Here we go.
As you can see it already says that it's a large corpus.
And we can even see here that our corpus takes 3.7MB of space.
We will simplify this corpus by cleaning step by step other reviews.
And that's what we'll do in the next tutorials.
Until then, enjoy machine learning.