Hello and welcome to this art tutorial.
So we just.
Trained our artificial neural network on the training set,
and now it's time to make the predictions on the.
Test set.
So lucky for us, we already have everything ready here,
thanks to our classification. Templates.
That we pasted in the first tutorial.
So actually this. Section predicts.
The test set results.
And this section makes the confusion matrix.
Thanks to which we.
Will obtain the.
Accuracy. On the. Test set.
That is some accuracy. On you. Observations.
On which.
Artificial.
Neural network model wasn't trained.
So first let's take care of this.
Section and.
Let's. See what we need to change.
So first of all this first line gets the predicted probabilities
thanks to this predict function.
But that was the.
Predict function used. For.
Built in our. Packages.
But here since we are using.
The H2O package, that is kind of special. There are.
Actually some things that we need to change here.
But a very few things.
So first, as for.
All the functions we.
Used with this H2O package.
Well, you notice that when we use the function we.
First take the H2O package.
Then a dot and then the name of the function.
Well we need to do the same. Here for the predict.
Function.
So here we just need to add h to zero dot predict.
Okay. So that's the first thing we need to change.
And then let's see. Let's go inside the function.
So the first. Argument. Is classifier.
Let's press here.
F1 to get some information about.
The predict.
Function of the. H2O. Model.
So let's scroll down to have a look at the arguments and let's see.
What they are.
So as we can see we have.
Only two main arguments and then some additional arguments.
But that we. Will not focus on.
Instead, we will focus on the two main arguments here.
Which are the object. And. Mu data.
So the first thing we can see.
Is that there is no type. Argument.
So here simply.
We will remove. This type equals.
Response argument and input.
Because we actually. Don't need it.
All right.
And now we are left. With the.
Two arguments we. Are required. To input.
That is the object which is our classifier here.
That is the A.
And model. Itself that we have just built.
On the training set.
And then the second argument, new data.
And this. New data. Argument is expecting.
Of course, the.
Observations of which it has to make the.
Predictions.
All right. So that's exactly our test set.
And here we remove.
The dependent variable.
Column thanks to this minus three. Here.
But we need to replace this three because.
This number three. Here corresponds.
To the index. Of the.
Dependent.
Variable of the data set that we worked with in part three.
Classification.
And here of course the index of our dependent variable is not three.
But is. 11.
Remember we. Already replaced.
The index three here in this.
Feature scaling part.
So we replaced the four indexes three that were here by 11.
And so here we need to do the same.
We will replace this three index here by.
Index. 11.
All right.
So now this.
Is taking the test set observations as. New data.
That is it will predict the probabilities that the dependent variable
exited equals one.
For the observations in the test set.
And therefore.
It will predict for each customer. In the test.
Set the probability that this customer leaves the.
Bank. And since we.
Have the real results of whether the customers of the test set left or stayed.
In the bank, well, we will compare our predictions to.
These real results, these actual results, and that's how we'll get the accuracy.
By computing the number of correct
predictions divided by the total number of observations in the test set.
That is, two. Thousand.
And then if we get. A good accuracy.
Then maybe we'll get a good and powerful model.
And if that's the. Case, we will give it to.
The bank on the plate and tell the bank, okay, now you can rank.
All your customers, all. The customers in the bank.
By their probability to leave the bank.
That is, for each of your customers.
You can predict with a good accuracy and will be.
Able to tell them precisely. Where this accuracy is.
You'll be able to predict
with a good accuracy the probability that the customer leaves the bank.
And then you can add.
Therefore, I. Can give you a ranking of.
All your customers ranked by their probability to leave the bank.
From.
The highest probability to the lowest probability.
And therefore you can do some customers segmentation and consider, for example,
the top 10%. Probabilities.
That the customers leave the bank.
And in this segment, you can analyze.
Deeper the factors that lead the customers.
To leave the bank.
By. Using some data mining techniques.
Like for example, doing a chi square test or.
Applying the step summary function on your independent variables
to understand which independent variables have the most impact
on the dependent variable.
That is, which.
Independent variable explains the most.
Why customers are leaving?
Well, you know how to do that.
That's exactly what. We did in part two and three.
When we use this summary function to get.
The p.
Values and statistical significance levels.
To see which. Independent variables are the. Most.
Optimistically significant and therefore explain the best, the dependent.
Variable. That is why customers are leaving.
So that's the purpose.
Behind making these predictions on the test. Set.
It's just to get the.
Accuracy on your observations to validate the model. So.
That we can give this model to the bank.
All right.
So now let's make the predictions.
So we are almost done here.
We just need to add. One more thing.
Which is.
Again related.
To the fact that we are using the H2O package.
And as you can see in.
This new data argument well this new data is of course the test set.
But this test set is expected to.
Be an. H2O frame.
Right now it is a standard data frame.
But our H2O.
Predict function is expecting an H2O frame.
So how can we convert. This test a data.
Frame into a needs to frame?
Well, by doing exactly the same as what we did.
To convert.
This training.
Set data frame into this. H2O. Frame.
That is, by applying. On the test.
Set the as dot h to O.
Function.
All right.
So I'm putting. The test. Set in the function.
Like that.
And here we. Go. Now I think.
Everything is ready.
We are ready to make the predictions which so far will be the.
Prediction of the. Probabilities that the class equals one.
That is, the probabilities that the customers leave the bank.
So let's select this.
And get the. Predicted probabilities.
And here we go.
We now. Have the prob pred vector.
Containing all the. Predicted probabilities.
In the form of an environment.
So that's good.
But we. Cannot have.
A look at these predicted probabilities yet.
We will need. To convert. It.
Back into a standard. Vector.
But before we do that.
Convert it into a. Vector.
Well we need to apply this line as well, which will, you know.
Transform the.
Probabilities into the.
Predictions in the form one. Or zero.
That is exactly the.
Predictions. Of the.
Dependent variable. Exited.
And to do this we're using this ifelse function.
And basically what we do is we choose a threshold such that
if the predicted probability is above the threshold, then we predict one.
And if the predicted probability is below the threshold, then we predict zero.
So that's a natural.
Threshold to take when we get our predictions.
In terms. Of probabilities.
No that it is not.
Necessarily always 50. Percent 0.5.
That's the case.
For example, in medicine when we have to predict some sensitive
informations, like for example, predicting if a tumor is malignant.
Well that's. More sensitive.
So in that case with.
Better be sure of. Our predictions.
And therefore we would choose a higher threshold like for example 80%.
But here we are.
Predicting if a customer leaves the bank.
So we are fine with the 50% threshold.
So that's okay.
And by the way there is a more simple way to get these.
Predictions in the form 0 or 1.
Without using.
This if else function. It's by simply.
Removing this one and zero. Here and.
Removing this. If else.
And by using this.
Prop, read. Larger than 0.5.
Because this will return a boolean, which will be true if prop read is.
Larger than 0.5.
And false if prop read is below.
0.5 and wipe read in the form of this boolean, true and false.
Will be accepted.
In this confusion matrix here.
So that's more simple. And now.
Let's get this.
Predictions in the form of booleans.
All right. So I'm going to.
Select this. Line and. Execute it.
All right.
So now we have our white. Print in the form of booleans.
But it is still.
An H2O. Object because it is the result.
In the first.
Place of this H2 dot predict. Function.
So it still needs to object.
And therefore. Now what we. Have to do is.
To convert.
This H2 object back. Into a. Vector.
Because this table.
Function here will only. Accept a vector.
A standard vector.
And of course will never accept this H2 object.
So let's convert it back into a vector.
And that's actually really simple.
It's actually kind of the same as converting a data frame into an H
two frame.
But instead of using H2 here we will use vector.
So here we simply need to type y pred
equals as dot vector.
And in.
Parentheses of course y print.
All right. So let's check it out.
I'm going to select this line and. Execute.
And now as you can.
See y pred.
Became this vector of integers containing 2000 elements.
And that's the standard.
Vector of r we were used to working. With.
So we can actually have a look at the.
Predictions of. The test.
Observations by typing here in the. Console.
Why pred?
Here we go. That's all.
The predictions of the. Tested.
Observations 2000 predictions.
So here we go.
According to the model, the first. Customer stayed.
In the bank. The second customer stayed in the bank.
The third customer left the bank.
The fourth one stayed, the fifth one stayed.
Etc..
So if you want, you can actually compare these predictions with the.
Real results that are in the last column of test set.
This come here.
So for example, 001000 are the real.
Outcomes of the.
First customers.
And if. We compare that with the predictions.
Well we see that the.
Predictions.
Are quite correct because here we get as well zero. 010.
Zero zero.
So the five first. Predictions. Are correct.
So that smells pretty good for our accuracy that.
We were about to compute.
Because when we look at the first observations we can only see.
Correct. Predictions.
So now actually I can't wait. To see the accuracy.
So let's computed right now.
We will start. By making the confusion matrix.
And of course. Here we need to replace.
This index.
Three here by 11.
Because this corresponds to the index of the dependent variable.
And so now. We are ready to.
Make this confusion matrix.
So I'm going to select this line and execute.
Here we go. Confusion matrix. Created.
So now let's have a look.
I'm going.
To. Type cmd here in the console and press enter.
That's our confusion matrix.
We can see. A lot of correct. Predictions.
That's good.
1500. And 36 correct.
Predictions. Of customers who stayed.
In the bank, and 195 correct predictions of.
Customers who left the bank.
And then we have 212. Plus.
57 incorrect predictions.
Of customers who either left or stayed.
In the bank.
So this looks pretty good.
And now let's. Not wait anymore.
Let's compute the accuracy.
So the accuracy is the total. Number of.
Correct. Predictions.
That is 105,030.
Six plus.
190. Five divided.
By the total number of observations in the. Test set.
That is the total number of predictions actually.
Which is 2000.
All right. So let's check it out.
Let's see if we can offer this model to the bank.
Let's see if we'll. Get the bonus.
Let's find out about this.
Accuracy on 332.
One. Go 80. 6.5.
Percent.
That's actually not bad at all 86.5%.
Well, let's say 80.
787% means that on 100 observations.
87. Predictions should. Be correct.
So this is pretty good.
And besides, we. Haven't done any parameter tuning.
And you will see that by doing some parameter
tuning using some techniques like k fold cross-validation.
Well we can get an even better accuracy score.
No worries, we will. Do that in person.
You can actually already. Practice.
To improve. This accuracy score.
And please let me know if you get an awesome one.
And now just one last thing.
Since we were connected to. This.
H2O instance.
It's better to. Disconnect from it now.
And to do. This we.
Only need to.
Apply a last.
Function of.
H2O, which is the H2O dot shut down.
Here it is with no arguments inside.
You just need to select this.
And this will disconnect you from the server.
So let's execute.
Are you sure.
You want to shut down the H2O instance running at this.
Address?
Then you just need to type here capital.
Y and then enter.
And now we are disconnected.
True means yes we did disconnect.
So congratulations.
You have built your first artificial.
Neural network on. R using the H2O package.
I was very happy to build this first deep learning model with you, and we are.
Getting to the end of this section.
Next section will be about Convolutional Neural networks, another branch of machine
learning specialized for computer vision, because it will consider.
Spatial. Structure in the.
Data exactly as it is the case. For images.
Where the. Position of the pixels. Matters.
So we will.
See that in the next section.
And until then, enjoy machine learning.