In this lesson we're going to pre process our data so that it's easier to feed it into our neural network.
Let's add a markdown cell in our notebook.
It's a pre process data.
So one of the things that we're going to do is we're going to change the kind of numbers that are being
fed into our neural network because at the moment if we look at X underscore train underscore all and
say we look at the very first entry then we'll get an array like this.
If we drill down a little further right we go one level deeper then we see that an array is nested inside
another array.
But if we drill a little deeper.
So we go to a particular pixel.
We can see the three values that are stored for this pixel.
The red green and blue values.
Now if I just want to look at this number 59 in isolation some four levels down right and I look at
the type that I can see that this is an integer right.
But it is a you into it.
And what does this mean.
This means it's an 8 bit unsigned integer now an unsigned integer is simply a fancy name for a positive
number.
So 1984 would be an unsigned integer but negative 10 would not be.
Why.
Because there's a sign in front of the ten the negative sign.
So unsigned just means a positive number if I take my training dataset.
Excellent as quatrain trained on the scroll all and I divide this whole thing by two hundred and fifty
five point zero then I would accomplish two things.
First off I would make all of these numbers a lot smaller.
Right.
I know that 255 is the largest value that I'm going to have because on the RG B scale this is what I
see right.
It goes only up to two hundred and fifty five for any of these particular sliders.
So dividing by 255 means that all the values inside my ex on this quatrain underscore all will be between
zero and one right.
The next thing that this will accomplish is there will be some conversion taking place.
We're going to be converting from an integer to a float.
We're going to be converting to a decimal number so a float is simply a number that has a decimal point
and then some numbers after the decimal point floating point numbers is what you're going to encounter
when you drink some sort of scientific calculation.
Now the reason I'm dividing by two hundred and fifty five and bringing those values down to a range
between 0 and one is because of our learning rate.
Now if you watch the module on gradient descent you'll see that the learning rate is typically quite
small.
And by scaling our numbers down to values between 0 and 1 it's going to be much easier.
Calculating the loss and adjusting the weights given a very typical learning rate.
So this is why we're bringing that range down.
Now I'm going to do this for both our training dataset and our testing dataset.
So when I write X and the score train underscore all comma X underscore test is equal to x on the school
train and a score all divided by two hundred and fifty five point zero comma X underscored test divided
by two hundred and fifty five point zero.
If I hit shift enter on this and say copy this cell paste it again and put it below this line.
After this line has evaluated I can re-evaluate this self and I can see that the type has changed to
a 64 bit floating point number.
Brilliant.
And if I want to see what that number is it'll be fifty nine divided by two hundred and fifty five which
is around zero point 2 3 1 3 7 and so on right now.
The next thing I'm going to do as part of our pre processing step is I'm going to flatten out our data
set so you know having four dimensions is is fine and sometimes we'll work with that but I think I'll
make it a lot easier conceptually if we put all of these values into a single row if you will.
Right.
Going to have a single vector a single row of numbers to represent one image.
So this means that these three dimensions will be flattened and to do this I'm going to use num PIs
reshape method to check it out how overwrite our X underscore train underscore all again by setting
it equal to X on this quatrain on the square all dot reshape and then I have to supply basically two
inputs the first input is the length.
So I'm going to say X on a squad train and a squirrel dot shape the square brackets zero and that value
is equal to 50000.
Now I could have also said alien X and the squat train underscore all get the same answer.
And the second value I'm going to supply to reshape is equal to what I want to collapse.
These three dimensions.
Right.
What I want to do is I want to multiply the number of pixels in the width of the image.
The number of pixels in the height of the image and the color channels.
So this would read 32 times 32 times three but I don't really like these magic numbers in my code that
much.
So when I come back appear to our constants and I'm just going to make this very explicit it's going
to see image on a square width is equal to 32 image on a score.
Height is equal to 32 and then I'll add another one called image on a school pixels and that's gonna
be equal to image width times image height.
All right.
And then my color channels.
I'll stick with the American spelling here.
It's gonna be equal to three.
So the total number of inputs I'm going to have total underscore inputs should be equal to.
Well it's gonna be the number of pixels times the color channels.
Agreed.
Great.
This means that I can come down here where I'm pre processing my data and I'll replace this just with
total on the school inputs.
So let me hit shift enter on the cell and after a number I has completed its work.
I'll say X and the SWAT train and all that shape and we'll take a quick look at what we got.
We are now dealing with an umpire a that is 50000 by three thousand and seventy two.
Now of course we can't just do that to the training data is that what we do to the training data set.
We should also due to our testing data set.
So I'll do exactly the same thing and a lot of print statement on here with an F string shape of X on
the score test is curly braces.
Excellent score test that shape that shift enter and let's see what we get 10000 images in our testing
dataset and they share the same dimension as our training dataset.
Brilliant.
The next thing that we're going to do is we create something called a validation dataset.
So imagine we've got our entire data set.
So both our training and our testing data and what we're gonna do is we're gonna split our training
data into two.
We can have a part of the training dataset that's part of our validation dataset.
So in total we're going to have three parts.
How why would we do this.
Well it has to do with our workflow because if you think about it there's many little knobs and little
variations that we can make to our model.
Many little tweaks and the validation data set will allow us to then select our best model because the
validation data set is because of where we're gonna be evaluating all these little tweaks.
In other words we're going to be doing two things.
We're gonna be training our model but we're also gonna be tuning in and making slight variations to
it.
And the validation data set will provide us an unbiased evaluation as to how our model is doing.
While we're tuning it and this has the big advantage of saving our test data set for later the test
data set will be untouched and it'll be therefore our final evaluation only our best model will get
to see our test data set because the job of our test data set is to give us a realistic impression of
horror model would do in the real world if we didn't create this validation dataset and we only had
a training data set and a test data set and we're tuning our model and we're always showing at the test
data set.
Then we'd actually be in danger of tuning our model such that it does well under test data set and as
a consequence we end up with unrealistic results so before we head back into Jupiter notebook and split
up our number higher rate one question you might be asking at this point as well.
How large should your validation dataset be.
And I think this really depends on the size of your data set as a whole.
For smaller data sets.
The general rule of thumb is 60 percent training 20 percent validation 20 percent.
Testing on the other hand if you have an absolutely enormous amount of data then what people might actually
do is only reserve about 1 percent for validation and 1 percent for testing.
But the kind of data that I'm usually working with the 60 20 20 rule has served me very very well.
So this is not a bad rule of thumb to stick by back in Jupiter notebook.
I'm going to insert a subsection here that's going to read great validation data sets and I want to
call this dataset X on the score.
Val set that equal to x underscore train a score all square brackets and I'm going to take the first
say ten thousand values.
So semicolon and then the number 10000.
Now if we want to get rid of this magic number here we can once again add up to our constants and put
in validation on a score size as a constant and set that equal to ten thousand and then before we come
back down we have to hit shift enter of course scroll down and replace our 10000 number with validation
underscore size now of course we need to do this to the y values as well.
Right.
So we're going to create another variable called Y on a square.
And just to check what it is we're doing.
Quickly print out the shape here.
So X and a score for hold up shape and it shift enter on the cell.
So I've got 10000 values with the same dimensions as above.
Now what about our training dataset.
Right.
We've created our validation bit but given that the first 10000 values are now part of our validation
dataset we should take maybe the last 40000 values and store those in our training is that our new training
data set that we're actually going to use for our model.
That way we'll have 40000 images for training 10000 images for validation and 10000 images for testing.
So I'll actually leave that up to you as a challenge.
So what I'd like you to do is create two num higher res X on a squat train and why on a squat train
and extend the school train has to have the shape 40000 by three thousand seventy two.
And why on this go train needs to have the shape 40000 by one since we've used up the first 10000 values
from our X underscore train those are all data set for our validation data set.
What I'd like you to do is to store the last 40000 values so the ones we haven't used inside this X
on this quatrain and y on a squat train no higher rate I'll give you a few seconds to pause the video
and give this a go.
I'll see it a bit
all right.
So here is the solution X and this good train is equal to x.
I was quatrain underscore all square brackets and then since we're not taking the first 10000 values
but we're taking all the values from ten thousand onwards we can see validation underscore size semicolon.
And that will give us the last 40000 values from X underscore train underscore all that we're looking
for and we can do the very same thing for our labels of course.
So why is train is equal to why on a score train underscore all square brackets validation size semicolon
and that's it X and the square train that shape is 40000 by three thousand and seventy two so that's
really it.
But I tell you what training our models can actually be a little bit intensive.
Even with only 40000 images.
So what I think would be nice would be to have an even smaller dataset to work with in the beginning
so that we can iterate and not slow down our computers too much.
And then once we're happy move on to the larger dataset.
So this would almost simulate training on a smaller data set that you have at first but then going out
and gathering more data and then training on a larger data set later on.
This is something you'd often encounter in the real world.
So I'm going to add another little markdown cell here and says create a small data set.
And this is mostly for illustration
so I'll just create two more umpire raise X on the school train and score access for extra small.
And I'm just going to take the first thousand values from X underscore train that we created earlier.
And if I do the same thing for our y values then I get something like this that a little constant at
the top that just reads small underscore train underscore size.
And that's gonna be 1000.
That way I know what's the and what it does.
Small on a score train underscore size.
It's gonna take the place of our magic number in the cell.
Now one of the really good things about using these practice data sets is that they tend to be pretty
clean.
It doesn't tend to be much wrong with them and that saves us a lot of time when it comes to data cleaning.
We're actually done with the pre processing we're done with the data exploration side of things.
So now we get to move on to our next steps setting up our neural networks and training our algorithms.
And this is what we're all here for right.
So I'll see you in the next lesson.
Take it.