Hello everyone and welcome to the recommender systems of Python lecture and this lecture we're going
to show you how to create items similarity recommendation system for movies.
Let's go ahead and jump to the Jupiter notebook to get started.
All right.
Here I am at the repository for the machine learning sections and under the recommender systems folder
again you see two notebooks recommender systems of python and then advanced recommender systems of Python
.
We're going to be walking through the recommender systems of Python lecture notebook.
They can see here and this is all the code we're going to be basically walking through and explaining
as we go along.
This is going to be essentially a portfolio project that will walk you through.
And here we have the advanced recommender systems of Python notebook.
This is a code notebook for creating advanced recommendation systems of Python.
But keep in mind that the level of mathematics and use of Pi here is pretty advanced.
So only go through this as an option if you want to actually see through linear algebra how you can
create a more advanced collaborative filtering recommendation system.
And there's pretty good text explanation on every single step.
As you go along as well as just some images to help you understand what's actually going on.
But keep in mind is pretty mathematical.
So if you don't have that background you can go ahead and feel free to skip this.
Let's go ahead and jump right in to our own recommender system which is going to be a little simpler
and it's going to be an item similarity based recommendation system and we're going to go ahead and
create a new notebook here.
I would advise that you do the same.
Again you won't have all those options that I just have because we're going to be using these three
data sets here.
You the item you've got data in this movie underscore ID underscore titles.
All right.
We're going to go ahead and hide the head bar and the toolbars.
Get myself a little more room.
Let's go ahead and get started.
All right.
I'm going to go ahead and start off by just saying important umpires and P and import pandas as PD and
then we're going to go ahead and get the data.
First off I'm going to create a variable called column underscore names and it's just going to be a
list of the column names I already know these beforehand because I've done this before our first column
is going to be a user ID.
Then we're going to have an item id then we'll have a rating and then we'll have a time stamp and we'll
go ahead and do is say ZF is equal to PD.
Read CSFB and we're going to pasan you dot data and it's going to be a tab separated file.
So instead of just leaving it blank unless say Sep is equal to backslash T and then I'm going to set
the names to column names and I just have autocomplete there and then go ahead and check the head of
our data frame.
All right.
There it is.
So here we have user id item id rating and time stamp.
And this is known as the movie lens data.
And if you want to find out more information about the data set which you can actually do is go to the
advance recommendation systems notebook of python and if you scroll all the way down to the bottom where
it says looking for more I have actually set you up with a bunch of links for other data sets where
you can actually build off more recommendation system analysis.
There's other movie recommendation data sets we're going to be using a subset of this movie Lind's recommendation
dataset but you can see if there's other links here.
There's also a music recommendation link for other datasets book recommendation health care and dating
.
Now keep in mind sometimes these sort of links get dropped or go down if people don't want to host those
datasets anymore.
So not all the links may stay up to host the data as of now I've checked and they're all working but
go ahead and post the Q&A forums.
If you find the link that's no longer working and I'll try to help you find that dataset again.
OK let's go ahead and jump back to our original notebook.
We have our data frame and the data frame basically just says a user id the item id in this case it's
a movie.
The rating they gave that movie and a time stamp for at what time they actually gave the movie that
particular rating.
Well we're going to go ahead and do is grab the movie titles.
So I'm going to say movie titles is equal to the read underscore C S V.
And I'm going to pasan movie and I can actually just tab complete this to get movie underscore ID underscore
it titles.
Now if I just check out this movie titles just the head of it I basically have a correlation between
the item id and the actual title and what you want to do is instead of dealing with item ideas here
to replace that with some sort of title early Mer's a data frame so I can have a connection between
item id and the movie title.
That way I'm not just working with kind of meaningless numbers here in order to do this.
I will say DF is equals to PD that merge and I'm going to merge the data frame and the movie titles
and I will do it on item underscore ID and then when I check out the head of my data frame now I have
the relevant movie title for that particular item ID so we can see the first one is the first Star Wars
movie in 1977.
Let's go ahead and explore the data a bit and get a look at some of the best rated movies.
Just so we can kind of get a feel for the data.
We're going to do some imports here import that plotless that pipeline as Piazzi all imports seaborne
as asinus.
I'm going to go ahead and say Asinus set underscore style to white lips and since I'm using a Jupiter
note book I need to say map plot lib in line.
All right well we're going to go ahead and do is create a ratings data frame with average rating and
number of ratings.
Let me go ahead and show you what I mean by this.
I will say DMF thought group by going to group by the title column remember that's the title of the
movie.
And I actually just want to grab the rating call them and grab the average of it.
So if I just do this I have now the average or mean rating for every title by grouping everything by
the title than what I can do is actually just off of this call sort values and then say ascending is
equal to false and just check the head of this.
And now this allows me to see the movies with the best ratings.
Now keep in mind we did do a group buy here so it could be the case that a lot of these movies here
only have one or two people that saw it and happen to give it a five star rating.
Let's go ahead and see the movies the most ratings.
So say DF group by going to group by the title again I just am just interested in the rating columns
.
I'll just grab that and then I'm going to say count and then I'll say sort values all go out and say
ascending is equal to false and just grab the head
and this lets me see which titles of the movies actually have the most ratings.
So it looks like the most famous movies such as Star Wars contact Fargo return of the Jedi in Liar Liar
have the most ratings which kind of makes sense if you've seen at least some American movies you probably
are familiar with at least one of these movie titles vs.
I'm actually not familiar of any of these movie titles so that actually makes sense.
Let's go ahead and try to put this all into a data frame when it created data frame called ratings called
equal to PD data frame and then it is going to pasan group by title
and then say rating mean.
So essentially what I'm doing is grabbing what I did up here.
Except the peer it returned a series after calling values on it.
So I was going to go ahead and pass this all onto a data frame just so it looks a little nicer for me
.
So go ahead and check out the head of this ratings.
Now I have an index based off of the movie title and my rating is going to be a column with the value
of the average rating.
Now something we noted earlier was that the rating kind of depends on how many people rated it because
seeing something is a five star rating doesn't really help if you only have one person that rated it
versus a thousand people that rated it.
So you get an idea of the number of ratings.
Let's go ahead and create a number ratings column right next to this rating column nor to do that.
I'm going to do basically what I did up here the group by title rating dot counts.
I'm going to go ahead and copy this and then say ratings
put a new column in here that says numb of ratings for the number of ratings that will set this equal
to PD that data frame and then just pass in what I had there.
And now if I check out the head of ratings I have the number of ratings and the average rating with
the index being the movie title.
Now let's go ahead and explore this by doing just a couple of histograms.
I'm going to go ahead and just do a histogram off the ratings and you can do this with seaborne or plotless
whatever you prefer.
Or just going to use basic pandas built in visualizations here.
I'll say ratings.
Number of ratings and just do a histogram and census data.
It's kind of large I will say bin's is equal to 70.
All right so now we can see here that most of our number of ratings are actually quite few if we look
at the movies.
It looks like most movies have either 0 or 1 rating which kind of makes sense.
Most people only watch the famous or big hit movies these blockbusters.
So those are going to be the ones a lot of reviews or number of ratings.
Now let's see the actual ratings themselves.
So I mean to say ratings
rating which is going to be a histogram I'll say been 0 to 70.
All right so now we can see visual distribution of the ratings themselves.
You probably notice that there are some peeks at the whole numbers such as 1.000 two three four and
five.
And that kind of makes sense because of the way people are going to rate movies and these movies probably
only have a few people watching them or just one person watched them and happened to give it a whole
number star.
One two three four five and then it looks like most movies are distributed kind of a little bit normally
around three stars kind of three and a half stars and there's actually quite a number of outliers here
of one star movies kind of makes sense because there's probably a bunch of just bad movies where only
a few people watched it just rated it as a kind of a garbage movie of one star and happened to give
it one star there.
And you see there's a little peek here of five stars and that's probably going to be the really popular
movies or the movies that happened to have just one person watching it.
Let's go ahead and do one more visualization to explore the data something we may be interested in is
the distribution or relationship between the actual average rating and the number of ratings.
Well we can do as you Seaborn joint plot to explore this I say s n s joint plot also a sequel to reading
y equal to the number
of rating column
ratings and said data is ratings and then Alpha is a point firelit make sure I have this correct.
Is it number of ratings.
Yep that's right.
Let's go ahead and run this.
OK.
Now looking at this data it looks like as you get more ratings you're more likely to actually have a
higher rating of a movie.
And that kind of makes sense because if your movie is better the better your movie the more people that
are in to watch it and the more people that are going to watch it the more likely you're going to have
a higher number of people reviewing or rating the movie.
So this actual trend of more reviews more ratings makes sense and it makes sense that these Want to
star movies have very few ratings versus these kind of better movies have a lot of viewers.
Now you can see an outlier here at the five stars.
And again that's most likely due to just one person watching one movie putting a five star on it and
that's it.
Because even if you're a blockbuster in a fantastic movie it's really unlikely that everyone is going
to rate you a five star.
OK now that we have a general idea of what that data looks like we can go ahead and move on to creating
a simple recommendation system based off of items similarity.
We'll go ahead and explore that in part two.
Thanks everyone and I'll see you at the next lecture