Hello everyone and welcome to part two of recommender systems of Python.
In part two we're actually going to build our recommendation system for the movie data.
Let's go ahead and get started by jumping back to the Jupiter notebook.
All right here I am back of the Jupiter notebook.
Now we're going to go ahead and do is create a matrix that has the user IDs on one axis and the movie
titles on another axis.
Each cell will then consist of the rating the user gave that movie if you remember back to the previous
lecture Dia was our original data frame with the user id the item id the rating the time stamp and then
the title.
We're not actually concerned the time stamp we're just concerned with the title of the movie the user
id and the rating of the movie.
We're going to go ahead and do is use pivot table to get this into matrix form.
I'm going to go ahead and say movie Max is equal to.
Pivot table and I will set my index equal to user ID cards that my columns equal the title and then
I'll set my values equal to rating.
Let's go ahead and check out movie I'm.
Now just the head of it and we take a look at this matrix.
You'll notice that you have a lot of missing values that actually makes sense because most people have
not seen most of the movies.
Well we can go ahead and do is check out some of the most rated movies again by saying ratings sort's
underscore values numb of ratings and I'm going to go in and say ascending equals false that head of
10.
Now looking back at this or going to go ahead and do ish choose two movies will choose Star Wars this
science fiction movie and liar liar which is it old comedy.
Well we're going to go ahead and do is grab the user ratings for those two movies.
So I'm going to make a variable called Star Wars underscore user underscore ratings and that will be
equal to my movie Matrix star.
We can keep typing here Wars 1977.
News tab to help you autocomplete that.
And I'm also going to do.
Liar liar.
Underscore user underscore ratings as equal to movie mats.
And in this case it's going to be liar liar that can do the complete that as well.
Now just checking one of these out.
Princess Star Wars user ratings the head of it.
I now have basically a series with that user ID and the actual rating they gave that movie or no they
did not rate that movie.
Well we can do is use Corps which has a method to get the correlations between two pandas series.
It's really similar to c o r r the regular correlation method.
But let me go in and show you how we can use or with I can go ahead and say movie mats dots and if I
remember we had a core.
But there's another method or we're in court with will compute the pairwise correlation between rows
or columns of two different objects instead of just the index or columns of a data frame.
Second say movie amat Korwin and I can pasan Star Wars for auto completion their user ratings and if
I just take a look at this I essentially get a bunch of titles of movies and their correlation with
the Star Wars user ratings which remember was just the actual user and then the relation or the rating
they gave that movie.
So I'm going to go ahead and call this similar underscore to underscore Star Wars
because all I'm asking here is for the correlation of every other movie that that specific user behavior
on the Star Wars movie so I can say similar Star Wars is equal to the movie Matrix which remember was
this where the user ID was essentially just a row of all the ratings they gave to all the movies they
watched or no.
They hadn't seen that and then I went ahead and said well how correlated is that.
With essentially this specific Star Wars throw here am I going to do the same thing.
Or liar liar.
So similar to LIAR LIAR sequel to movie Matt or with Liar Liar underscore user underscore ratings and
don't worry if you get this warning it's just a pandas issue.
But everything should still works on an actual air is just a warning.
We're going to go ahead and do is clean this by removing the null values and then using a data frame
instead of a series just so it looks a little nicer.
We'll go in and just start with Star Wars.
So going to say core underscore Star Wars is equal to PD the data frame and I'm going to pass in the
similar to Star Wars and I will set my columns equal to co-relation then I'm going to say car or course
Star Wars drop any one else say in places the true run those two lines and then actually check out the
data frame.
And this will probably be more clear as far as to what we actually just created.
OK so we just created a data frame where the index is the title of the movie and we have this correllation
column in this correllation column tells us how correlated this movies user ratings were to the user
ratings of the Star Wars movie.
So if we sort the data frame by correlation we should be able to do is get the most similar movies.
However you should know that we get some results that don't really make sense when we go in and show
you what I mean by that.
By go in and say see our core underscore Star Wars sort Gollies mindless sort of by correlation with
ascending was ascending to false and I was going to take the first 10 notice that we get a bunch of
titles with perfect correlation.
So according to our model right now although read Beane's of Egypt maned whatever that movie is old
lady who want to see all these kind of movies you probably never heard of before are perfectly correlated
with Star Wars.
Now the reason that doesnt make sense is because most likely these movies happened to have been seen
only by one person who also happened to rate Star Wars five stars.
What we can do is go ahead and fix this by filtering out movies that have less than a certain number
of reviews so we can set a threshold for the number of ratings necessary in order to be put into our
model.
We can go ahead and do it just go back to our histogram on the number of ratings.
Now it looks like we get a serious decline after 100.
So you can go ahead and kind of play around with the number we can do.
But I'm going to go ahead and filter up my movies to have less than 100 reviews but I encourage you
later on when we finish this project to play around with the actual Filcher you're going to use.
I will say core underscore Star Wars is equal to or underscore Star Wars.
And I'm going to join it with ratings of ratings.
So basically what I just did where they just joined the state of France together.
So with the title I can use number of ratings there with the correlation.
Remember if you go back to the merge joining cannonading lecture the reason they used join here and
sort of merge is because I have the title as the index of my data frame which makes it a good fit for
this join method.
All right so we're going to do now is filter out any movies that don't have at least 100 ratings so
I'll say core underscore.
Star Wars where core underscore Star Wars of ratings column is greater than 100 and then Osaze soar
underscore values correllation and basically the same thing I did last time will say ascending the sequel
to false and I go ahead and just check the head of that.
Let's run that.
OK and now we can see some movies that start to make a lot more sense.
We see Star Wars and then another Star Wars movie The Empire Strikes Back and it makes sense that the
Empire Strikes Back is going to be the most correlated movie to Star Wars.
Now we also have Return Of The Jedi which is another Star Wars movie and it's also pretty highly correlated
with Star Wars.
Then you have Raiders of the Lost Ark which if you saw Star Wars you probably enjoyed this Indiana Jones
movie and you'll notice that there's a big drop in correllation when you start to get to Austin Powers
now Austin Powers is a comedy.
So you may or may not like it if you like Star Wars but one of the reasons that it may have relatively
high correlation now 0.37 isn't that high.
But you may be wondering why it showing up here.
The reason is most likely it's a very popular movie and so is Star Wars.
And a lot of times that these recommendation systems really popular movies are really popular items
tend to be recommended to other people who also like another really popular item.
Let's go ahead and explore.
Liar liar.
See if we get some different results.
We'll go ahead and say core liar liar as equal to PD they frame what we're going to go ahead it is kind
of review the steps we then remember we grab the similar to LIAR LIAR.
So the columns equal to correlation.
So that very first step essentially you just created a nice looking data frame for us or we had the
title in the index and then the correlation with the movie.
And in this case our movie is liar liar.
Notice that we have a lot of no values which are kind of useless to us here as far as recommending stuff
.
So go ahead and say drop those all say in place schools.
True.
So we went ahead and dropped those.
And our next step is actually get the number of ratings and we can do that with join.
We'll say core underscore.
Liar Liar schools corr score liar liar.
And I'm going to join it with the ratings numb of ratings column.
And now I check out what I have I have the title on the index the correlation with Liar Liar and the
number of ratings are going to go ahead and do is then filter this out books by a certain number of
ratings and I encourage you to kind of play around with this value.
This case will just kind of arbitrarily choose 100 ratings I'll say I'm a ratings greater than 100 and
I'm going to sort values.
Correlation is correlation.
Ascending equals false.
And I just want the head of that.
OK and now we have the movie suggested to us if we really like Liar liar.
Now obviously lie liar has a perfect correlation.
The next one is Batman Forever which it might be ok of a choice.
The.
That's actually probably a really good choice because liar liar is a commie in the 90s.
That stars Jim Carrey in The Mask is another comedy in the 90s that stars Jim Carrey.
But notice that as we start to drop off in number of ratings our correlations are pretty low to begin
with.
They're all lower than 0.5 where you can go ahead and do is play around with this number of ratings
threshold in order to see if you can get better recommendations.
Or maybe you can filter out some recommendations that don't really make sense.
Ok I hope you enjoyed this project.
Again you can check out the advanced recommendation systems lecture if you want to see more advanced
methods of creating a recommendation system based off of collaborative filtering.
But in case you really want to quickly create a recommendation system and do it kind of easily with
Pancho's by just using and does it capabilities with core or with the core with.
All right thanks everyone and I'll see you at the next lecture