In this lesson we're finally going to train our model and this is going to involve starting a session
patching our data and then finally running the training loop.
It's gonna be a pretty jam packed lesson coming up.
Let's dive right in.
The first thing I'll do is I'll add a markdown cell here that reads run session.
So the weight tensor flow works is that it's got these session objects and the session objects encapsulate
the environment under which all the operations and all the calculations take place and are executed.
So we've done all this setup previously with worked out these place holders and we've said what calculations
should take place and what the operation is going to be for minimizing the loss of our lost function
and so on.
All of these calculations only take place inside a session.
The way we create one of these is with T F dot session and parentheses at the end.
I'm actually going to store this session that I'm creating here in a variable called cess.
The next thing I have to do is I'm going to have to initialize all the variables.
I'll just add a comment here so I'll say in it.
These are all our variables and they're going to be initialized with T F dot global variables initialized.
So there we go.
And next I want to feed all of these variables to the session.
So says don't run open parentheses in it.
These three lines of code get us all set up.
So let me come up here where I see set up tensor flow graph and go to cell and run all below.
Now that we've create our session initialized our variables and have run our session we can actually
peek inside some of these tenses.
So for example if we wanted to see what weights we've actually got for our first hidden layer then we
can come down here and we can say w 1 will take our tensor and we'll say Val evaluate and then we have
to provide the session as an argument if I had shift into I can actually see our starting weights.
The same is true for our biases right.
B One dot eval and then our session will show us that our biases are all initialized with the values
0 for our first hidden layer.
We've cut a lot of biases but for our output layer b 3 We've only got 10.
So the way the tensor flow works is that with the setup what we're doing is we're building out this
graph these calculations.
It's almost like laying pipes when we actually run a session.
The data starts flowing through these pipes that we've laid out and then we can actually evaluate our
calculations and we can look at what's inside our variables.
We can get some outputs and the calculations are actually executed.
So now that we've covered that we're almost ready for writing our loop for training our model.
But before we do that we're going to write some code that will split up our training data into smaller
components.
And the reason we're doing that is because we want to be able to train our model on batches of 1000
samples at a time.
Chances are that for your own projects you're gonna be working with quite large datasets.
So having the skill of dividing up your data is gonna be essential let me add a quick mark down sell
him and add a variable here.
That's going to read size of batch and set that equal to 1000 and then I'll create three more variables.
The number of examples I'm going to store in num underscore examples.
And that's gonna be equal to the number of examples in our training dataset.
So why on a school train dot shape square brackets zero.
The next thing I'll do is figure out the number of iterations that we need for training.
And here I see an our underscore iterations is equal to the number of examples divided by the size of
the batch.
Now if I wanted to make sure that this was indeed an integer then I could cast it or converted to integer
using I.A. parentheses.
Some examples divided by size of batch.
The last thing I'll do is create an index variable so I'll see index on the score in underscore e Paul
and I'll set that equal to zero.
The index here will help us keep track of where one batch ends and the other batch of samples should
start because what we want to do is we want the first batch to go from zero to nine hundred and ninety
nine and we want the second batch to go from 1000 to one thousand nine and ninety nine and so on until
we've chewed through all fifty thousand examples in our training dataset.
The next thing we're gonna do is we're gonna define a function right.
When I say next underscore batch is gonna encapsulate the logic for going from one batch to the next
batch.
This function of ours is going to take three parameters.
The first will call batch size.
So how big are the batches gonna be.
The second thing is what's the dataset.
And the third thing shall be the labels.
So this function needs our exes and our Y's next I'm going to use the global keyword to get a hold of
a variable that's outside of this function someone a reference number of examples which is outside right
here using this global keyword.
And I'm also going to reference the index in the epoch.
This one here our index variable inside this function.
And the reason I'm doing this is because we need to figure out the starting and the ending point the
starting point is gonna be equal to our next variable.
So it's gonna be equal to zero in the beginning but then our index will update.
Right.
So we'll take our index and we're going to add the batch size to it.
So it starts out at zero and then next time round it's gonna be equal to 1000.
So we'll see index in epoch is equal to index and epoch plus batch on the score size.
Now oftentimes when you're updating a variable like this there is a shorthand notation so you can write
plus equals and then batch size.
So now that we've got the start figured out and we're moving our index along by the batch size we can
think about the end.
Right.
So the end of the batch is gonna be at index in epoch.
What this allows us to do is then return our x values and our y values that are between these two points
between the starting point and the ending point so we can do this with our data square brackets start
colon end.
So what are we getting the very first time round that we call this function well we're gonna get all
the values between 0 and 999.
Why.
Because it won't supply 1000 to the size of the batch.
So the starting value will be equal to zero.
Because that's where index and e pork is starting out then it's gonna be index and epoch is equal to
itself plus 1000 and then the ending value will be equal to 1000.
So if this notation you'll look at all the values between zero and this ending value not inclusive.
Right.
So why underscore train for example square brackets zero colon three.
Well give us all the rows between 0 and 3.
Right.
0 1 2 right.
That last one is at index 2 not 3.
And that means the next time around this method gets called the starting value will be equal to 1000.
Then the next updates to two thousand and then the ending value updates to two thousand.
And we get everything between 1000 and one thousand nine hundred and ninety nine.
But if we're going to return to things we're going to return our features which I've called data and
we're going to return our labels right labels square brackets start colon and the only thing left to
do is to include some logic between these two lines to capture what will happen when we get to the end
of the dataset.
So what will happen if the index is greater than the number of examples.
So in this case we have to do a couple of things in this case.
We should start the next epoch right.
So start should be reset to equal zero and the index should be reset to the batch size.
Brilliant.
So now we can move on to our next step which is actually training the model lot a markdown cell here
and that's going to read training loop.
So what are we going to do him.
What we'll do in our training loop is we'll go from epoch number 0 through to epoch number 5 so we'll
start at zero and we'll go for whatever we've specified as one of our hyper parameters here.
So we'll a for punk in range number of epochs so we're iterating through each epoch and then will iterate
through our data itself.
So we'll see.
For i in range number of iterations number of iterations was equal to the number of examples divided
by the size of the batch.
If you'd like to know what that number is we can actually print it out here.
It's 50 so there'll be 50 iterations.
Paul take us through all 50000 examples 1000 examples at a time.
It's a number of iterations.
And what are we doing in each iteration.
While the first thing is we'll need a batch of 1000 samples for features and 1000 samples for our labels
so we'll see patch underscore X come up batch underscore Y is equal to next underscore batch the method
that we created a minute ago will return to us two things.
Some x values and some y values but it requires three inputs.
So the patch signs are x values and y values right.
So in this case they'll be batch size is equal to size of batch data shall be equal to X on a school
train and labels shall be equal to y on the school train.
That's that how we get to the really fun part.
And by that I mean working with tensor flow.
So what we have to do here is we have to create something called a feed dictionary a feed dictionary
is nothing other than a Python dictionary.
So we'll see feed underscore dictionary is equal to curly braces.
And inside those curly braces will provide two key value pairs.
The first is gonna be our X that capital X is gonna be the key and batch underscore X is gonna be the
value.
The second thing will be no surprise there.
Capital Y as the key and batch underscore y as the value.
The reason that we've created this is so that we can feed it to our session.
Our session is gonna run all our calculations for us.
So we'll take our session as ESFS.
Remember we've created it up here and our session is gonna have this run method and a run method will
do the calculations.
The key calculation in our case is the training step the training step is the operation from our optimizer
that minimizes our loss.
What is it going to minimize the loss on.
Well it's going to minimize it on the data that we're providing through our feed dictionary.
So our feed dictionary is going to have our x's and our Y's are features and our labels and when we
provide this alongside the lost minimization to the session it will run the calculations and update
our weights.
These two lines of code are really the second piece of this tensor flow puzzle.
The first piece is setting up all these place holders and initializing all these variables and setting
up all the calculations ahead of time.
The second piece is actually running the calculations.
So running that training step running our optimizer and running that calculation on our data which in
our case is a batch of 1000 samples.
But even though we're training our model let's not forget our accuracy metric.
We've defined this calculation up here now's our chance to use it once again.
This is where our session is going to come into play using the run method on our session can give us
this accuracy as an output.
So let's say batch underscore accuracy is equal to says Don't run parentheses.
And here we need to specify what the session should fetch for us what output we want to get out of this
session and tensor flow has a really funny word for this.
It's called fetches.
It's almost like a Caris callback.
So what should fetches be equal to it should be equal to our accuracy calculation.
And what's the accuracy going to be calculated on the accuracy is going to be calculated on our batch.
So feed on a score dict is gonna be equal to our feed dictionary once again.
Now since we're fetching this let's print it out but let's not print it out inside this for loop which
will run 50 times.
Let's print it out.
Every epoch has that.
So once as for loop is done well hit a print statement.
We'll see print and I'll put an F string here and the first thing I'll do is I'll print out the epoch
that's running so epoch curly braces lowercase epoch which is gonna be this one right here.
Just the index in our loop and then I'm gonna tab over so I'll have an escape character backslash T
and I'll put a pipe symbol there and I'll see training accuracy equal to curly braces batch underscore
accuracy and when we're all done I'll just say print done training and this is the Mon we've all been
waiting for.
Right.
So let's shift enter on this L in epoch 0.
I've got a training accuracy on my batch of about 64 percent by Epoch 3.
I'm up to 86 percent.
And by epoch number four namely the fifth epoch I'm up to eighty seven point two percent on my training
data set.
Fantastic.
So now that we've been successful in training our neural network we can add some bells and whistles
to our code and we can also investigate in a bit more detail how it works what it does and we can modify
it more to help us better evaluate what's actually going on for all of that.
We're gonna be using tensor board once we have tensor boards set up we can start looking at our performance
a bit more closely especially the performance on the evaluation data set.
So far all of that and more I'll see you in the next lesson.
You know the drill C in a bit.