Welcome back.
In the previous video, we learned about importing the data and using the PANDAS library to manipulate
our data frame.
Now let's learn a little bit more about this step, which is to clean the data.
And in this case, let's try to solve an interesting problem.
Let's say that instead of this filtering, what we want to do or what our boss comes and tells us is
that he or she wants to know.
Well.
What players are a good deal?
What players are the best in terms of skill but are also lower salary.
So maybe we can afford them better.
So let's say that the boss wants a few things from you.
If we go back to Kaggle here and look at our data, it will actually have a link to all the columns
that this data has.
And let's say our boss wants to find out the current market value of the player and the wage, so he
or she wants to know which player is undervalued right now.
And maybe we can offer we can grab that player for a really good deal.
So while we want to graph from the data is the value, the wage and obviously the name category of the
player.
Now, if you want, you can pause the video and try to do this yourself.
Otherwise, let's give it a go.
Let's create a new variable called a data frame one.
And this is going to grab.
Using the PANDAS library, the data.
Frame method which will accept our previous data frame that we've grabbed, which is our entire CSV
file.
So let's say data frame and then we're going to select as our second parameter the columns that we want.
In this case, we can give it an array of columns that we want, which will be name.
Wage.
And then finally, a value.
So let's just print DF one to see if this is working.
There you go.
Just like that, we have the name wage and value of the player.
Holy moly.
Messi.
Messi has has quite a high wage.
That's a.
That's a nice wage.
All right, let's keep going.
Now, here's the next part.
We have this table.
But the data doesn't really mean anything to us right now.
Let's say we want to sort the data now.
Because what we want to do is we want to compare each player's wage with the value and have perhaps
a different column called The Difference to see which player has the biggest gap between their wage
and their value.
Well, let's let's try and give this a go.
Now, ideally, we can just create a different column by simply saying DF one, we're going to add difference.
And this difference column is simply going to equal the dataframe that we have built over here.
And it's going to be the.
The value.
Minus the F one.
The wage just like that.
Now.
If I run this, let's do DF one and click Run.
Oh, I get an air.
If I scroll all the way down I see unsupported operant type SW four minus.
So I'm trying to subtract a string and a string.
And we've learned about this before.
Right.
You can see over here that the wage of a player is in euros and they get paid a lot.
So sometimes you have the M, which is 4 million or K for the thousands.
So we need to convert this into an int, right.
So we can do that quite easily again using python.
Now with this part, you can try and solve this yourself and try to convert these values and wages to
an integer and remove the euro sign and the M or the K for the thousands.
Or if you want to just do this quickly, you can even Google around because this is a common problem.
All you do is Google for pandas convert string to int or in my case look of that googling around I found
convert the string 2.90 K to this in pandas dataframe.
So it looks like they are asking the exact same questions that we have.
And again, as a developer, you have to really learn to start Googling and finding answers on your
own because most likely other people have the same questions you have.
Now, if I scroll to the answer here, you see that I get some answers that look overly complicated,
but I really like this one.
So value to float.
So we're changing a value to a float.
We give it some sort of an X and that's it.
So this converts our value to float.
And you see here that it replaces different things for us, such as the K, M and the B.
So that's perfect.
Let's copy this.
And again, I'll link to this resource so you can take a look for yourself.
If I go to our soccer.
Let's enter the function that we have here, make sure that it's formatted properly.
So we now have the value to float function, which is great.
If we go back to the the question here.
Unfortunately, they only remove the last item in the list and not the beginning portion.
So we can just do that ourselves because we want to remove the euro sign as well.
So there's actually a pretty easy way of doing this using regex.
So let me show you.
We can do a wage variable that equals DF one hour dataframe.
We're going to grab the wage column.
Let's make this a little bit bigger so you can see.
And this wage column is going to be replaced.
And the replace method allows us to use regex.
So what we can do is we'll say that we want to remove the euro sign, so let's just copy it from here.
Just like this.
With an empty string.
And then the third parameter is to just say reg x equals to true.
Now if you go to Reg X 101 dotcom.
And we copy what we just did.
So let's just copy this.
If I do anything with a euro sign.
You see that it is highlighted over here.
That's what the regex does again.
This is something that I just experimented with.
Trust me, I don't remember this off the top of my head.
I just looked it up before this video.
So we replace all the your signs with the empty string.
And then finally if we go to our.
StackOverflow answer.
You remember here that in order for us to use the value to float, we just simply do not apply value
to float.
So again, if we go to our soccer.
We'll just say dot.
Apply.
Value to.
Float.
Which is what we have up here.
And we can just do the same thing for the value as well.
So let's say value.
Is going to equal.
Value here and the same thing being applied.
And then finally with these new columns, we can just update them by simply saying DF one wage.
Is going to equal wage and then data frame one.
The value.
Is going to equal value.
So because now these are going to be integers.
I can now run this minus operation.
And hopefully if we didn't make any mistakes, this should work.
Let's have a look.
All right.
Look at that.
We have a new difference column which just subtracts value from wage.
Awesome.
The last thing I want to do is to actually sort this so I can better understand it.
So let's do for the last part, say sort.
Values.
Again, another thing that we get from pandas.
And we'll sort By the difference column.
And we'll say ascending.
Equals to false.
So we get the biggest difference first instead of lowest to highest.
So if I run this now.
Look at that.
I get the players with the biggest difference all the way down to the lowest players.
Very, very cool.
So this was a bit of a longer video, but I wanted to show you the idea of cleaning the data.
That is, we removed any unnecessary data.
We converted to types or clean things up in case we needed anything, and then perhaps even added extra
information that we might need so that we can improve or get insights about what we might want to do
next.
Let's take a break and I'll see you in the next video.