All right.
So in this lesson we're going to set up our lost function.
We're gonna set up our optimizer and we're going to set up some metrics like the accuracy how add a
markdown cell here loss optimization and metrics.
The very first thing that we're going to tackle is the loss function.
Now the lost function is key for optimizing the neural network.
In this example we'll continue to use the cross entropy loss that we talked about in the last module.
So let's have a look at the tensor flow documentation for cross entropy loss.
If we Google tensor flow cross entropy then the first result I'm getting here is this one if I click
on this result and scroll down a little bit then I can see that this function is deprecated.
So apparently there's already a newer version of this function that's out and this new function pretty
much has the same name except at the end it's got version 2.
So if we click on that then in the description we can see that this function calculates the soft Max
cross entropy which is what we're after between two things.
The lower bits and the labels the low gits are the outputs from our output layer and the labels are
going to be the actual labels that we're going to supply are y values.
So let's use this function in our Jupiter notebook.
Our loss is going to be equal to TAF Dot and Dot soft Max cross entropy with low its version 2.
All we need to supply in the parentheses are the labels which is going to be our place holder tensor
for our y values and our logics which is another name for the outputs that we're gonna get from our
last layer.
So I'm going to set low it's equal to output output here.
Remember is what we're getting out of these soft Max activation function from our last layer.
Now there's only one tiny modification I want to make to this line of code.
And that modification is due to the fact that we're gonna be training our model in batches.
We've got a big training data set and we're gonna split it up when to split it up into smaller pieces
and we're going to train on these smaller pieces one at a time until we chew through the entire dataset.
And that means that while this last calculation works for calculating the loss on the entire dataset
it's not going to give us a good result when we've got individual patches.
So what we need to do when we have individual patches is we need to take the average of the losses and
the good thing is tend to flow has a fantastic function for us called reduce on the score mean that
we'll do just that.
So I'm going to wrap the output from soft Max cross entropy with low gits version to inside the parentheses
for reduce underscore mean now that tensor flow knows what loss function we're going to use we can move
on to the next step which is telling tensor flow which optimizer we want to use.
So we'll store this in a variable called optimizer and that's going to be equal to.
Well let's google for it actually.
So tensor flow optimizer will bring up several optimizer so that we can use like the gradient descent
optimizer or the atom optimizer Adam as we've discussed in the previous module is a state of the art
optimizer.
So I'll tell you what let's use this one again.
We can see from the documentation him that there are default values for all the parameters for the atom
optimizer and we wouldn't actually have to specify anything of our own.
So if we say T F train don't Adam optimizer that's good enough that just sticks with the default values.
I can even hit shift tab on my keyboard to bring up what those are in the quick documentation.
But suppose we want to use our own learning rate instead of this default one then we can simply specify
that learning rate here we can see learning rate equal to learning rate or if you want to use a positional
argument instead of one that is named We can delete that and just leave it like this.
I think that's perfectly fine.
Looking back at the documentation for the optimizer the key thing that it needs to do is minimize our
loss and the optimizer is method that will do that is called minimize.
So in this case we will call this function and we will supply our lost function.
Everything else we will keep the same scrolling down we can actually see that this minimize function
has an output as well and it outputs an operation that updates our variables in tensor flow which variables
while our weights and our biases.
Right.
So let's store the output of minimize in a variable called Train underscore step and that will be equal
to our optimizer dot minimize and then loss.
So in these two lines of code we've told tensor flow which optimizer we want to use we've initialized
our optimizer here.
And we've said what the learning rate should be for the atom optimizer the next thing we've done is
we've nailed down what the operation is that the optimizer will do to minimize the loss.
We've said which loss it should minimize namely this one right here and we're storing this work in something
we're calling train underscore step.
The reason we're doing this is so that we can call this training step again and again and again in a
loop when we're training our model that way we will minimize our loss as we iterate over our data.
Now I know there's quite a bit of setup that we're doing here with tensor flow and this is why karats
exists right.
This is why there's a bridge to make all of this easier.
When you first get started but essentially what we're doing is we're laying out all the calculations
and all the variables ahead of time for tensor flow so that when it comes to running the calculations
it knows what those are another calculation that it will need to run for example is it needs to calculate
the accuracy of the model.
So this is another thing that we're going to have to outline ahead of time.
So what I'll do is add a markdown cell here.
It's gonna be very small and it's just going to read accuracy metric that another markdown sell him
that reads defining optimize them and let another mark down sell him.
That's going to read.
Defining loss function.
There we go to calculate the accuracy.
We need to compare two things our prediction and the true label.
Right.
And we need to check if they're equal.
Once we know if they are equal then we have a correct prediction.
And once we know how many correct predictions we have we can work out the accuracy of our model.
So let's start with that.
The correct on underscore score prediction is going to be equal to the result of a comparison.
The result of a calculation tensor flow has a function called equal which will make this comparison
for us we want to compare two quantities in this case.
One of them is going to be our output.
And the other one is going to be the true label.
So the output was the output from our final layer which we've called output.
Right.
And the true labels will be stored in a tensor called Capital Y.
Now we have to make one more modification to this code because the y values will actually look something
like this.
We're going to have 10 different values most of them going to be zero and one of them is going to be
equal to one.
So what we'll have to do is we'll have to pick out the index of the largest one.
Right.
This one is going to be at index 5 0 1 2 3 4 5.
The index will correspond to the class of the label and we're gonna have to do something very very similar
for the outputs from our final layer.
Right.
Because we're getting 10 outputs each one of them will be a probability between 0 and 1 and we have
to pick the largest probability out of the 10 and then pick out the index where that largest probability
occurs the reason we've got these nice probabilities is because our output is using these soft Max activation
function.
So what's the easiest way to get the index of the largest probability tensor flow has a function called
ARC Max and ARG Max will pull out the index of the maximum value from our output and just has to know
where to look.
And it has to look along a row.
So we'll see.
Axis is equal to 1 and will do the same thing for our y values.
So TFT aren't Max parentheses y comma axis equals 1.
Now we can compare whether this value is equal to this value is our highest probability prediction equal
to the actual label.
And once we've got these predictions we can calculate the accuracy the accuracy is going to be equal
to the average of the accuracy of all the patches where we're doing the calculation.
So once again we're using tensor flow reduced mean and here we're going to supply the correct prediction.
Now I want to make another small modification him because I'd like to be 100 percent sure that I've
got a decimal number here.
So I'm going to just convert this correct prediction into a decimal number with TAF dot cost parentheses
correct underscore.
Fred T of dot float 32.
So now I've converted my correct predictions into a decimal number and I'm averaging them across all
the patches that we're gonna be doing the training on and that way we've calculated the accuracy.
The downside of having to do all the setup ahead of time is we can't press like shift ends around these
cells and see if we've done it all correctly.
We'll only find out when we're actually training our model and we're getting closer to that in every
lesson.
So stay tuned.
It's coming right up.
I'll see you in the next lesson.
Take care.