Okay, my friends.
So now let's add a new tool to our data preprocessing toolkit,
which is taking care of missing data.
So indeed, if we have a look again at our data set data dot csv, we notice
that there is a missing salary here for this specific customer from Germany.
Of 40 years old and who purchased the product.
So generally you don't want to have any missing data in your data
set for the simple reason that it can cause some errors
when training your machine learning model, and therefore you must handle them.
There are actually several ways to handle them.
A first way is to just ignore the observation by deleting it.
That's one method, and this actually works if you have a large data set
and you know if you have only 1% missing data, you know, removing 1%
of the observations won't change much the learning quality of your model.
So 1% is fine, but sometimes you can have a lot of missing data
and therefore you must handle them the right way.
So that was the first way to ignore them.
To remove them.
And now a second way.
And this is what we're adding right now in
the toolkit is to actually replace the missing data.
You know, the missing value by the average of all the values in the column
in which the data is missing.
So here we have a missing salary.
What we want to do is to replace this missing salary
by the average of all these salaries.
This is a classic way of handling missing data.
And I'm going to teach it to you right away.
So here we go. Taking care of missing data.
Let's create a new code cell and let's replace that missing salary
by the average of all the salaries here.
All right.
So to do this we're going to use the libraries.
And actually I'm about to introduce you to one of the best data science libraries.
I'm talking about scikit learn.
Scikit learn is an amazing data science libraries containing a lot of tools,
including a lot of data preprocessing tools.
You will see that we will actually use scikit learn a lot in this course.
You know, more than half of the machine learning models we will build
in this course will be built with scikit learn.
So if you don't know scikit learn yet,
I'm telling you you're going to absolutely love it.
And so for the first time here we're going to use scikit learn to handle
missing data.
And to do this the class that we're going to use
from scikit learn is called simple input.
We're actually going to first import that simple input a class.
Then we will create an instance you know an object of the simple input a class.
This object will allow us to exactly replace
this missing salary here by the average of the salaries.
And then we will have an updated data set.
You know, an updated actually matrix of features,
because we will apply this input on the matrix of features only.
So we will have a new matrix of features with no missing data
because the missing salary will have been replaced by the average salary.
All right let's do this.
Perfect.
So first since this class belongs to scikit learn, well
we're going to start here by going from scikit learn which has the name sklearn.
So sklearn.
Then remember in order to access a module we have to add a dot.
Because actually this simple import a class which we want to import
belongs to a certain module of scikit learn called impute.
This one impute.
And from this impute model well we're going to import.
There we go. The simple
import class.
Google collab really exists.
You will simple import a class. Perfect.
Then next step.
As I said the next step is to create an instance of this class
which you can exactly see as the tool itself.
You know, the tool
that you'll use to replace that missing salary by the average of salaries.
So since we're about to
create a new object, well, we have to introduce here a new variable.
And we're going to call this variable imputer okay.
Input it which will be exactly this object of the simple input class.
And therefore since it will be the object of this simple Imputer class, well,
we have naturally to call this class simple input.
So I'm going to copy this and paste that here.
That's how you create an object of the class.
You simply call the class.
Then you add some parenthesis and there you go.
Now you're going to enter the right arguments in order to replace
indeed this missing salary by the average of salaries, because note
that there actually many replacements that you could do.
You could instead of replacing it by the average salary,
you could replace it by the median salary
you know, there is a difference between the average and the median.
You could also replace a missing value by the most frequent value.
Right.
That would be, for example, relevant for categories okay.
So we have many options.
But the most classic one and the one option that I recommend
is the average salary. The mean salary okay.
And so that's exactly what we have to enter here.
First we have to specify which missing values we have to replace.
And so that's why we have to enter here.
First argument called missing values which has to be equal to NP.
You know the numpy library dot none.
And that's just to say that we want to replace
all the missing value in the data set like this one.
This is like an empty value.
This is what this means an empty value.
And then the second argument we have to input here is exactly
the one saying that indeed the missing values here,
you know, the empty values of the data set will be replaced by the mean.
And to do this we have to add the next argument here, which is strategy.
And this argument will be equal to in quotes mean okay.
And that's just to say that we want indeed to replace all the missing values
in the matrix of features by the mean of the feature itself.