Okay, my friends.
Are you.
Ready to now implement.
The step three of this upper.
Confidence bound implementation?
We already implemented step two, and we even got.
These values in for each of the ads.
And now we need. To use a trick to.
Find the maximum of them.
All right.
So in the.
Last tutorial we left things at this else you know else.
Meaning if the numbers of selections of.
This. Ad number I we're dealing with right.
Now in the second full loop is equal to zero.
Meaning if the ad was not selected yet, well, in that case,
what we have to do is absolutely select it, right?
We must select it if the ad has not been selected yet, we have to select it.
Why is that?
That's because, as we explained at the end of the previous tutorial.
We need to make sure that the denominator and I n, which is the number of times
the ad. I was selected, is different.
Than. Zero, so that. We can indeed compute.
The average reward.
And then the upper confidence bounds.
So there is a priority.
Now, you know, at the beginning, you know in the first round.
To select all the ads.
And therefore the trick that we have to use is to set
the upper bound of this particular ad we're dealing with.
Right now in the second for.
Loop to a super high value, so that it will be.
Indeed the maximum upper bound, and therefore the one that will be.
Selected, because at the. End we.
Will of course select the ad with the highest.
Upper confidence bound.
So the trick right now.
Is to take that variable again, which so far thanks to this.
If condition is. Equal to this value.
But if we're not in this if condition and if we are in this.
Else, well we would like. That same variable.
To be equal to a super high value.
Like, you know, infinity.
But we can't set it equal to infinity.
However, we can.
Set it equal to a super high value like.
One times ten at the.
Power of 400.
That's a classic.
Trick in Python to, you know, use the infinity.
This is a super high value, which will.
Definitely be the maximum of the upper.
Bound, so that if that ad was not selected yet, well, we will select it because.
Indeed it will have the maximum of the upper confidence bounds.
And now we need to finish.
With a. Final if condition to make sure.
That indeed, we select the ad with the highest.
Upper confidence bound.
And the trick. To do that is to play with.
Max upper bound here, which.
So far is initialized to. Zero.
You know, before we. Start this. Second loop and.
That. Upper bound and the if condition that.
We have to add here is to check if, well, the upper.
Bound. Of the ad.
We're dealing with right now in the second full loop in which we've just computed.
Either through this first if.
Condition or in this else.
We have to check if that upper. Bound.
Is larger.
Than the maximum upper bound.
Right.
Because okay. At the beginning, you know, before.
We start. This full loop, max upper bound is equal to. Zero.
Then we will compute.
This for the first ad we will get either value if this ad had.
Already been. Selected, or we will get this value.
And of course since this or this will be larger.
Than. Zero.
Then this max upper bound will.
Be updated. Right? And this is exactly. Our next step. Here.
If indeed upper bound is larger than this maximum upper bound.
Well we need to update.
That max upper bound.
Value.
There we go to be equal to.
That new upper bound.
Just. Computed. Either through this if.
Condition. Or this else.
All right.
So upper bound okay.
And then in the next step. Of this. Second for.
Loop well.
We will compute a new. Value of the upper.
Bound here. If you know the. Was already selected.
And if this new value of the upper bound.
Is larger.
Than this new max upper bound, which.
Was just updated to the previous upper bound of the previous add.
Well, we will update it again.
You see the you see the trick each time.
You know, at each iteration of this second loop, we compute a new upper bound.
We compare. That upper. Bound with.
The largest upper.
Bound collected so far, you know, with. The previous. Ads.
And if that new upper bound is larger than this maximum upper bound, well. We.
Update the new maximum upper bound.
And of course. For the ads that. Have not been selected.
Yet, this. Upper bound. Will always be.
Larger than the max upper bound.
And therefore this ad will be selected.
And speaking of this, as will be selected, well, that's exactly the final step
we have to do here.
We have to select the add.
In in order. To select it.
Well we need to update that variable here at equal zero to.
Of course I. You know the index.
I love that ad.
We're dealing with right now in the second for loop.
And that's my friend.
That's how.
You implement step. Three.
By making sure at the same time.
That you select the ad.
That was not. Selected yet.
And at some point, you know, after a couple of rounds, will all.
The ads will be selected.
Well, you know, actually, after the ten first rounds, all.
The ads. Will automatically. Be selected.
And then we will only be in this condition.
You know, this else condition will never happen after the ten first round.
Okay. So there you go. Now congratulations.
This step three is implemented.
We selected the eye that.
Has a maximum upper confidence bound.
And now what we simply need to. Do is just.
Finish. This.
Main code here. You know this.
Cell by. Going back to. This first.
For loop.
You know iterating through the rounds, you know,
through the users who connect to the website.
And, well, the.
Way to do this is just to update each of these variables, which we created at the.
Beginning, you know, before starting the for. Loop.
To indeed get, you know, that.
Full list of all the ads selected.
Over the round.
Then of course.
Update this variable to update.
The number of selections for. Each of the add.
Then of course update this one, which gives the accumulated rewards
for each of the ads.
And finally update the total accumulated reward over the rounds.
All right, so I will let you do this.
Please try to do it yourself.
Before the next tutorial and we will implement this solution
together in the next tutorial, which will at the same time finish and complete.
That cell.
Implementing the upper confidence bound algorithm.
All right.
So good luck.
Again and I'll see you in the next tutorial for the solution.
And until then enjoy machine learning.