In this lesson we're gonna be setting up our tensor flow graph and our neural network.
And that means creating our tenses as well as setting up the layers in our neural network the first
thing I'll do is I'll credit a markdown cell here that's gonna read set up tensor flow graph now cutting
a tensor in tensor flow is actually pretty straightforward and tensor flow actually will require us
to create some place holder tenses ahead of time and that so that tensor flow can figure out how did
it setup and it can figure out how data should flow within its graph.
Here's the API documentation for the tensor flow placeholder to create a tensor.
We're essentially going to supply two things a data type and a shape scrolling down.
You can see an example of how to use this code here.
A tensor is created and stored inside this variable here.
This tensor is going to contain a floating point number and it's going to be two dimensional.
It's going to be of shape 1024 by 1000 and 24 so let's do something similar.
Let's set up our X and our y r features and our labels as placeholders.
That's the first step.
So capital X shall be equal to T F dot place holder.
We're also going to be working with floating point numbers so we'll use TAF Dot.
Float 32 and then for the shape we'll say shape is equal to and in our case our shape will be as follows.
I'll be none comma.
Seven hundred and eighty four.
Why is that.
It's because we've got seven hundred and eighty four features.
Now what about this other dimension.
Why did I put none here.
Why am I effectively doing him is leaving this first dimension blank.
And the reason is is that because this dimension here will hold on to how many examples are going to
be used.
How many samples will be contained in the tensor.
And this will actually be determined a little later on because when it comes to training our model we're
going to be splitting up our data set into batches and by leaving this dimension blank at the point
in time when we're creating the place holder we can change the sizes of the batches as we see fit.
We can use 1000 samples per batch we can use two thousand five thousand ten thousand.
It doesn't matter what I'm going to do is I'm actually going to change this 784 to a constant so I'm
going to go up to my constants and I'm going to add a few more constants appear to make this a bit more
explicit.
We've discussed before that the image width is 28 pixels in this dataset.
We've also discussed how the image height is 28 pixels in the dataset.
And finally all the images were grayscale.
So we know that the number of channels is equal to one.
And what I've done in these CSP files that I've provided is I've flattened all the images.
So instead of providing you with an array that was shaped 28 28 and 1 I've flattened the structure to
make the total number of inputs equal to the width times the height times the channels.
Now that we've got this constant appear we can come back down and we can replace this thing here by
the total number of inputs.
Brilliant.
So that's the place holder for our features.
What about the place holder for our labels.
We're gonna create this one with Y is equal to T F dot placeholder and for the data type we're also
gonna use float 32.
And now for the shape it's going to be equal to none because this dimension will be determined by the
size of our batch.
And then they'll be 10 y 10 because this is the total number of classes that we have.
This is the total number of categories that we're looking to predict and what I'm actually gonna do
is I'm going to use a constant here as well number an R underscore classes you can see that I've already
created this constant at the top right here.
Let me hit shift enter on the cell and now we can move on to setting up our neural network.
I'll add a little subheading here that's gonna read neural network architecture the very first thing
that we're gonna determine are the so-called hyper parameters and by hyper parameters I mean parameters
that don't come out of training the model a parameter that comes out of training the model would be
something like the weights right.
But the hyper parameters are basically things that we're gonna determine ahead of time.
So these are things like how long to train our model for number of epochs used for training the learning
rate used in our optimizer the number of layers in our neural network the number of nodes per layer
all of these things are hyper parameters.
So let's add a few here.
The first one I'm gonna add is the number of epochs and I want to set that to maybe five.
To start with the second thing I'm going to add is the learning rate learning on a school right.
And I set that equal to a very small number.
Zero point zero zero zero one.
By the way if you wanted to write this in scientific notation you can as well.
So 1 e minus 4.
We'll give it the very same number so I could write that like so as well.
The next thing that I'm going to specify are the number of neurons and the number of layers.
So I tell you what let's have two hidden layers and the first hidden layer will have 512 neurons.
So and underscore hidden one it's gonna be equal to 512 and that second hidden layer and hidden to is
gonna be equal to sixty four.
We're not gonna make it too complicated for now.
So two hidden layers first hidden layer 512 neurons.
Second hidden layer 64 neurons the output layer.
Remember it's only going to have 10.
So let me hit shift enter on the cell and I'm going to delete this one here and add a few more cells.
So what's next.
We've initialized our hyper parameters.
We've given them some starting values and we've created our place holder tenses for our features and
our labels.
The next thing we're gonna do is we're gonna give some starting values to our weights and our biases
for our neural network all the connection weights in the neural network needs some sort of initial value.
And what we're gonna do is we're going to give a small random value as a starting point and we're gonna
do that for every single weight in the network tensor flow actually has a really nice way of generating
these random values for us and the way it does it is to pick these values from a distribution.
So I'm going to store all of them in a variable called initial on a score w one for initial weights
on the first hidden layer and I'm gonna set that equal to T F dot truncated normal.
So it's gonna be a normal distribution but it's gonna be truncated meaning that there's not gonna be
any extreme values that are gonna be generated on either tail of the distribution to the extreme left
or the extreme right.
It's truncated and now this function has to know how many values it needs to generate What's the shape
of this first layer the shape for this layer is gonna be determined by two things the number of inputs
and the number of neurons in the layer.
Right.
So we said that the number of inputs was seven hundred and eighty four and the number of neurons in
this layer was five hundred and twelve.
And we don't actually have to have these numbers here.
We can just write total inputs and an underscore hidden one in place of these numbers.
So now this function knows how many weights to generate for us in total but it doesn't yet know how
to pick out these weights should they be far apart from one another or should we rather close together
for that it needs to know a little bit about the standard deviation and we can end that with this little
argument here and we can set that equal to a small value like zero point one.
And because everything is randomized and you might want to replicate my results we can set the seed
value here equal to 42.
That way you'll be drawing the same random weights every single time.
So those are our weights.
Let me hit shift enter on the cell.
Now you might think that you can actually just take a look at what values we picked out here what values
we generated and you would be disappointed that you can't actually do that just yet.
And the reason is is that tensor flow kind of has this like two stage approach.
The first stage is all setup and it's only the second stage that these calculations are actually all
done.
So as long as we don't actually tell tensor flow to evaluate any of these tenses and to run all the
calculations we don't actually get to see the values that are generated just yet.
This is one of those things to wrap your head around with tensor flow that it's got this two stage approach
and we're gonna be talking about that a whole lot more in a bit.
So back to the setup we've created our random little values him as the initial values for the weights
but the thing that we should actually do with these values is to create something called a variable
a tensor flow variable and that variable will hold onto all the weights in that first hidden layer.
So when I call it w 1 and I want to set that equal to T F dot variable with a capital V and open parentheses
initial value is gonna be equal to our initial underscore w one in this line of code we're actually
creating the weights the weights remember are more than just the initial values they have to persist
and be updated as all the calculations intense floor are going to be run.
This is why we create them as a variable.
Now let's tackle the biases remember how we said that changing the weights is like a stretching or compressing
the activation function in a neuron.
The bias will work together with the weight and the bias is what shifts the activation function from
left to right.
By adding or subtracting a number.
So what we're gonna do next is we're going to initialize the biases and the way we're going to do this
is we'll get all the initial biases for that first hidden layer with initial underscore B 1 and we're
gonna set that equal to not TFR truncated normal but f dot constant all of our biases are going to start
out with the same value.
What value is that going to be that value is just going to be equal to zero.
The only other argument we have to supply to this function is how many initial values we need.
So in this case we once again need to supply a shape and that's gonna be equal to 512 or n underscore
hidden one to create our biases then we're going to do it very similar to how we did it with the weights
we're going to use tensor flow and we're going to create a tensor flow parable and this variable just
has to know what the initial value is of the biases.
So initial underscore value is equal to initial underscore b 1.
So all of this is still part of the setup just for the first layer and the biases and the weights for
this layer will be updated during the training process.
This is where the network does its learning.
These are the values that will feed into the activation functions of the neurons and represent the strength
of the connections between the different units.
Now what we need to do is think back to that slide where we had that formula.
We need to determine how the weights and the biases work together to determine the inputs into this
hidden layer.
We discussed this in the previous lesson on this slide right here.
So we know that the first step is we have to multiply whatever comes into this Greenlee right here by
the weights.
This is the first step I'm going to store the inputs that come into our hidden layer in a variable called
Layer 1 underscore in and I'm going to set that equal to the result of this multiplication and the way
that we can multiply our tenses together is with this function right here from tensor flow this function
gets used so often that there is even a nice short little alias that we can use to use this function
need to supply two things we'll supply in a and a B and then this function will multiply matrix A by
matrix B producing a times b no surprises that so let's call this function in our notebook TMF Dot.
Matt Mol and now we have to figure out what is it that we're multiplying given that this is our first
hidden layer.
We're going to multiply the place holder for input features.
This is going to be the tensor that we created by our weights.
So w 1 so X in this case represents our raw inputs.
We're still on the very first layer.
So these are our raw inputs of our feature vector.
But remember what actually reaches the neurons is gonna be the inputs multiplied by the weights plus
the bias.
So plus b 1 the result of this calculation is what actually feeds into the activation function and that's
the next step.
Now let's complete the whole puzzle by working out the output from our hidden layer.
And I'm going to call this layer 1 out and that will be equal to the output of the activation function
from all the neurons in the previous module.
We've used these really activation function with Caris so let's stick with reload.
This time around as well looking at the documentation it really only requires one input namely the features
what are the features.
Well it'll be the inputs to the layer.
Right.
So TAF Dot and Dot really do layer 1 underscore in will now calculate the output from our first hidden
layer so we've done a lot of work just now.
We've initialized the weights for the first time in lab.
We finish lines the biases for the first in lamb.
We figured out what the features were that were going into the first hidden layer.
And this was gonna be the weighted average of the inputs plus the bias and the output of this layer
is going to be the result of the calculations that happen inside the activation function.
Now I've got a challenge for you.
I'd like you to complete the code to set up the second hidden layer.
Remember this layer has 64 neurons and that layer will depend on the output of the first hidden layer
and I'd also like you to setup the output layer.
And here the trick is that the activation function will be the soft max function this challenge is gonna
be a little bit more tricky.
You have to get a couple of things right to solve it but pause the video and give this a go.
Ready.
Here's the solution.
The first thing that we need to do for that second hidden layer is to initialize the weights and the
biases so copy this line and I'll copy this line and I'll just pasted in the cell below.
And what I wanna do is I want to change my variable names so these are not gonna be the same weights
or biases.
As for the first hidden layer and I won't have to do this everywhere right.
So I've got my initial weights here.
I've got my initial weights here.
I've got a separate variable W2 the values for my initial bias is here and the variable for my initial
bias is here.
Now the first trick is actually setting up the weights and the biases for the second layer correctly
because the thing that's different between the first layer and the second layer is the shape.
Right.
So the number of inputs that these second hidden layer gets is not equal to seven hundred and eighty
four.
It's actually equal to the number of neurons in the first hidden layer.
Right.
Instead of 780 for him we'll have an underscore hidden one.
The total number of neurons in the second hidden layer is also not an underscore hidden one but an underscore
hidden to if this is the shape that you've provided for those initial weight values.
Well done.
Now what about the biases in this case.
We need 64 neurons whether we've got a bias so the shape here should be an underscore hidden to the
inputs for the second layer.
Well then use these values.
Right.
So layer two in will be equal to the multiplication between x and w one No because that second hidden
layer follows the first hidden layer.
It'll be the output of that first hidden layer.
Right.
It'll be Layer 1 out and then it won't be w 1 but it'll be w 2.
And of course also be the biases for that second hidden layer.
So in this case it'll be the output of the first hidden layer multiplied by the weights of the second
hidden layer plus the biases of the second hidden layer that will form the inputs into that second hidden
layer.
Now what about the output.
Well in this case I'll be quite similar right.
We'll have Layer 2 out and there'll be also redo we're gonna stick with the same activation function
and it'll just be Layer 2 on a score.
So now we've got the output for that second hidden layer.
What about that final layer though.
What about the output layer.
Well in this case we have to update everything for the output layer starting at the top.
We've got to change the weights right.
So I to go with W3 here and W3 here and here.
The shape in this case will be 64 and 10.
Right.
We've got 64 neurons in that second hidden layer so an underscore hidden 2 and we've got 10 neurons
in that output layer or the number of classes for the weights we're also have a shape determined by
the number of outputs.
Our number of categories that we want to predict I'll update the names of course as well.
Initial underscore B three and initial on his work b 3.
Ok so now we've got the weights and the biases for that output layer.
So what's coming into the output layer.
Well it's kind of B layer three underscore in is equal to the multiplication of.
Well in this case it's gonna be the output of layer number two right.
Number two is gonna feed into Layer number three and we're gonna multiply it by the weights that connect
those two layers and of course we're gonna add the biases that are specific to a layer number three
the output layer finally our output as a whole is kind of equal to whatever comes out of the activation
function in our output layer.
But in this case it's not gonna be really you it's gonna be soft Max.
So that way we get a nice probability that is associated with each of the outputs.
If you weren't sure how to find the soft max function from tensor flow then if you google for it you
actually get directed to this page right here which is the API reference for tensor flow and there you
see that the soft max function requires also only really a single input which here have a name called
Low gits.
But effectively what this function needs to calculate the probabilities is of course the weighted inputs
for the output layer.
So I hope that didn't throw you off.
This was definitely one of the hotter in video challenges that I've asked you to complete because it's
really built on understanding the code that we've written a minute ago plus understanding the concepts
that we've kind of discussed over the last two modules but don't worry if you didn't quite get this
right there's still plenty of opportunities throughout this module to solidify your understanding and
see how this works it'll become a lot more clear.
Once we actually run our code trainer model and you get to see these things in action in the next lesson
we're gonna continue with our setup.
We're going to hammer out three important points.
They need the loss function that we want to use the optimizations that we want tensor flow to do and
the metrics like the accuracy that we want tensor flow to calculate along the way.
And only after we've done all that we can actually train our model.
I'll see you in the next lesson.
Take care.