All right, so now that we have are nested list of all the words and all the e-mails stemmed and cleaned.
I want to return to one of our previous topics, namely slicing data frames and creating subsets.
Previously, we looked at the at attribute.
We looked at the Iot attributes and we looked at the eye log attribute.
In this lesson, I want to show you how to use logic to create a subset of the woods.
I want to create a subset from a data frame based on a condition.
Consider this part two of slicing data frames allowed a quick markdown sell him.
That's going to read using logic to slice data frames.
Now, we've already seen how we can use the square bracket notation with a data frame to select a particular
column, say the message column.
But another thing that we can do in these square brackets is to place a condition similar to what you
would find after an if statement.
Say, suppose I want to access a subset in our data frame, say all the spam messages.
Then what I could do in here is to have data dot category.
So selecting a particular column and then saying, well, only give me the rows where the category is
equal to one.
This here will actually create a subset based on this condition being true.
If I change on the shape attribute here, then we can see that the size of this subset is a thousand
eight hundred and ninety six rows and three columns.
This is exactly the number of spam messages that we had.
If I copy paste this code and change that shape at the end to head, we can look at the first few rows
in this subset.
Now, these look very similar to what we've seen before.
But if I change this head here tail, we can see that the last spam message is with document eighty
one thousand eight hundred and ninety five.
Now, I want to pose a challenge to you to put the subsetting into a bit more practice.
I'd like you to create two variables, doc.
Underscore I.D., underscore spam and doc underscore ideas, underscore ham.
And I want you to store the indices for the spam and the non spam emails in these variables.
So pause the video and try and figure this out.
Did you have a go?
Here's the solution.
So I'll say donc I.D. on the school.
Spam is equal to and then it's going to be equal to a subset from my data data frame.
So it's a data square brackets, data.
That category is equal to double equals one.
And then I'll use the index attribute to store the index of the subset inside this variable.
And I'll do something very, very similar for the hand messages, for the non spam messages, that is.
I'll just change the name of the variable to him and then change the category to zero.
And that's it.
Here's what these X values would look like for our non spam messages.
They start at a thousand eight hundred and ninety six and go up all the way to five thousand seven hundred
and ninety five.
And there's a total of three thousand nine hundred of them.
But the thing is, now that we know exactly which indices in the data frame correspond to our spam messages
and which indices correspond to our non spam messages.
I want to show you another trick.
I want to show you how you can create a subset using these indices directly.
So let me quickly add a mark down cell so we can find this again easily.
I'll say subsetting a series with an index.
Now, if you recall, the type of our dog I.D. on underscore ham is in fact, an index.
And the type of our ness, that list is a Pendas series.
When you're working the series or a data frame, you can use the Yellow Sea attribute, right.
Alosi with the square brackets.
And then here you can feed in an index directly.
So if, say, we wanted to access a subset of the nested list that just contained the non spam messages
we could feed in the non spam indices between these square brackets of the Yellow Sea attribute to location
attribute.
I'm actually going to store all of this in a variable.
I'm going to call this variable nest that list ham.
This nested list has.
The following shape, it's got three thousand nine hundred entries and the last few entries in this
series look like this.
Let's do the same thing for our non spam messages some quickly just going to paste this in here.
Change this to spam and change this to spam here.
So I've got a nested list spam variable and I've got my indices for my spam emails.
Now, inside the square brackets of the location attribute for the nested list.
And this will give me a nested list of all the spam messages.
Fantastic.
Now we have two separate lists of all the hash messages and all the spam messages all broken up into
the tokenized stemmed and lowercase words.
At this stage, I want to propose another challenge to you.
This is gonna be a bit of a bigger challenge.
I'd like you to practice some list comprehension.
And then I'd like you to find the total of number of words in our clean data set of spam email bodies.
And I'd also like you to find the total number of words used for the normal e-mails.
And then I'd also like you to tell me what the 10 most common words are in all the spam messages, as
well as the ten most common words in all the normal e-mail messages.
I hope you're up for the challenge.
I'll give you a few seconds to pause the video and figure this out.
Are you ready?
Here's the solution.
Let's tackle the non spam emails first.
So I'll create a flat list from our nested list as follows.
So it's a flat list.
Ham is equal to scrub brackets.
And this is where I'm going to use list comprehension again.
This is a little bit of review from what we've done before.
So I'll say item four, sublist in nested list ham.
For item in sublist.
This is the list comprehension.
Done.
Here we are flattening our nested list of all the words in the non spam messages and putting them into
a single place.
Now, what I'll do is I'll find out the total number of words.
So I'll say maybe normal words is equal to.
PD don't series parentheses.
So create a panda series from our flattened list of ham e-mails and I can then easily find the total
number of words used here.
With the shape attribute.
And it's just at index zero.
So this is the total number of words.
We've got approximately four hundred and forty one thousand words inside our bag of words for our non
spam messages.
But there's one problem here, right?
We've got four hundred and forty one thousand words in our bag of words of normal emails.
But there's gonna be some repetition here.
What if we want to find the unique number of words?
If you wanted to find the number of unique words, then we'd have to use the value counts method.
So check this out.
If I add value on a school counts.
To my code here, then I'll be creating a series using the flattened list of how messages.
But instead of storing this in my variable directly, I want to use the value underscore counts method.
This will then tell me the total number of unique words in the nonce found messages.
In this case, we have approximately 21000 words.
So while we've got a total of approximately four hundred and forty one thousand words, we've only got
about 21000 unique words in our dataset of non spam messages.
To find the most common woods hole we have to do now is take our normal woods and look at the first
10 values.
So square brackets, semicolon 10.
Here we go.
These are the ten most common words in the non spam messages.
And this is the number of times that the word HTP occurs.
This is the number of times the word use occurred in our non spam messages.
So I hope you saw how we use a couple of the techniques that we've covered already.
To answer this question of what the top 10 most common words were when you were able to do this with
very, very few lines of code.
Thanks to the power of pandas, oftentimes of these things, it's just a matter of remembering what
the right method is or Googling for it.
But this is just the hammer, such as this is just a non spam messages.
What about the spam messages?
Let me copy the code here.
Scroll down and then we'll do the spam messages all the same.
So we'll change our variable name here.
We'll change the name of the nested list we're using.
And we'll create these series from the flight list of spam messages.
We'll call this spammy was instead of normal.
Underscore words.
And then we'll print out the shape.
The total number of unique spammy words as well.
So this is the total number of unique words in the spam messages.
Let's see what we get.
In the spam messages, we have a total of thirteen thousand unique words, and the top most common ones
are spammy words, square brackets, semicolon 10, says the top 10 spammy words.
At number one, we've got HTP at number two.
We've got the what e-mail, then the word free.
Then the word click.
Now, one thing that you might be wondering about is, of course.
Well, wait a minute.
So these words, these were all the standard words, right?
These things here aren't real words.
So we can entirely be sure if this word here is distend would form for please or pleasure just by looking
at it.
We might not know what our Porter Stemmer has done behind the scenes.
So that's something to think about for the future lessons in the next lesson.
We're going to be doing some more data exploration in the form of some advanced visualization techniques.
I want to show you how to create something called a word cloud, because it's all very nice and good
printing out the spammy words or the normal words as this list where you see the frequency and the word.
What's a lot nicer usually is to present this graphically, to present this visually, because oftentimes
visuals are just so much more convincing in any meeting or any presentation.
I can't wait to see you again in the next lesson.
Take care.