So we're getting up to the end of this module and the last bit that we're gonna do is we're going to
make a prediction using our tensor flow model for how it's going to evaluate its performance on the
test dataset.
First let's tackle how to make a single prediction.
So when I had to mark down cells so we've got a subsections ready that first markdown cell is going
to read make a prediction.
And that second markdown cell is going to read testing and evaluation.
Notice I'm inserting all of this before closing our session and closing our writers and resetting the
graph in your downloads.
When you downloaded the zip file it with the amnesty data I've included a single test image and that's
this one right here.
This test underscore image top PMG file that test image should be in your M.A. folder which is which
is in the same directory as your Jupiter notebook.
Scrolling up to our imports we can write something like From pile import Image.
So this is the pillow module which is really great for manipulating and displaying images that shift
enter on the cell and I'll come down to where I want to make my prediction.
And now I can see image onto open and then I can supply the relative path.
So that's amnesty forward slash test on the square image don't P PMG.
And if I had shift enter then I see it as an output here.
But if I have a pin file How do I converted so that I can feed it into tensor flow and get a prediction.
How does that work.
Well the first thing I want to do is I might actually sort of send a variable called image and if I
still want to display it I can write it like so.
Well right now I have a g image but at the other end I've got a tensor that will expect seven hundred
and eighty four inputs.
So I've got to get the grayscale image for this thing and I have to flatten it.
The other thing that I have to do with this image is actually to invert this image because at the moment
it's black ink on a white background.
But I've trained my neural network on a black background with a white digit right.
This is a little different from what I've got at the moment.
So I have to do quite a few things already.
The first thing I'm going to do is actually convert this thing to grayscale and I can do that using
pillow because pillow has a convert function and with convert I can select a mode and there is a mode
here that is called l and this is when translating a color image to black and white.
So at the moment even though it looks like it's black and white I think I've actually got a color image
here.
So when I convert this to black and white with mode L so colors BW is equal to image dot convert and
then l what's next.
Now I've got a black and white image I can use num pi to invert this thing believe it or not.
So I'm going to say n p dot invert open parentheses BW which is my black and white image.
And this will be my image and a score array so far so good.
Now that I've done that I can take a look at the shape of my image right.
And this is 28 by 28 by.
Well nothing here.
Right.
Because it's a black and white image with one color channel.
So that third dimension has already been dropped but it's still 28 by 28 and not flattened to seven
hundred and eighty four.
So we can do that with another name pie method and that's called Ravel.
I'll store the result of this in a variable called Test underscore image and so that equal to image
and a square array.
Dot Ravel parentheses at the end.
Hitting shift enter on my keyboard.
We'll show you what the description is for this method.
Reading the docs string we see that Ravel will return a flattened array.
We don't want a 28 by 28 array but we actually want is a flattened array and we can see this when we
pull up test to underscore image dot shape.
Now we're ready to feed this into tensor flow.
To do that we're gonna have to get a hold of our session our session object was called Sex and the way
we get our session to actually do some calculations for us.
We're going to have to see says Dot run the run method we'll do some calculations for us and the way
we get this run method do some calculations for us is to provide two things.
The first thing was the kind of output that we want from it.
In this case we want a prediction the outputs are called fetches.
So if we supply the fetches we can specify what kind of output we want.
The other thing that we have to supply is the feed dictionary it's the feed dictionary are the inputs
that's the data.
Our session is going to use to run its calculations the feed dictionary is always a python dictionary.
So we have to supply an X and this is what's going to go into our placeholder tensor.
In this case it's gonna be our test underscore image right.
Our test image will go into the place holder tensor.
I'm not going to supply a Y because I don't really care what the accuracy or the losses so I'm just
going to supply the X..
Now I have to specify in the fetches what kind of output I want what I want to know is from our output
layer from all the outputs all 10 of them.
Which one hand the highest probability so it's gonna give us 10 outputs and I don't want to know which
one had the highest number.
And I want to use T F arg.
Max open parentheses output which is the output from our last layer and then I supply the Axis axis
is equal to 1 meaning what's the largest number in that row.
That's how we get tensor flow to make us a prediction based on some data.
But we're actually going to get a return value for this code.
So I'm going to save this under prediction.
Equals since we're fetching a number from our session.
I to store that in a variable called prediction before I run all my code though I want to print out
my prediction.
Someone has enough string him.
I'll see prediction for test.
Image is curly braces and right prediction.
Now I can go back up him to where I'm setting up my tensor flow graph and I'm just going to run all
the cells below.
And here we go our model got it right.
Hurray.
I've got a challenge for you.
I'd like you to calculate the accuracy of our model over the test dataset our test dataset has 10000
images and we've got all the features stored in X underscore test and we've got all the labels stored
in y underscore test.
What I'd like you to do is use your knowhow of how to use the session to calculate the accuracy over
the test data set.
Once you've calculated the accuracy I'd like you to print it out and display the accuracy down to two
decimal numbers.
I'll give you a few seconds to pause the video and give this a go.
Far show you the solution.
Here it goes.
So test underscore score accuracy is equal to says Dot run open parentheses fetches is equal to accuracy.
Why can I use this because I've defined what the accuracy is up here.
TENSOR flow already knows how to calculate the accuracy because we've nailed it down in this cell right
here.
That's why we can reuse this calculation here in the fetches the fetches are what has to be output ID
by intercession when it runs.
The only thing it needs to know is what to give to the place holders what data to feed in and that happens
in the feed dictionary the feed dictionary is going to have an X this time for the features and those
are gonna be equal to while x on the score test.
That's where our test data set has the features and the Y is going to be mapped to y underscore test
our place holder for the labels it's gonna get the testing labels with y on a score test to print the
test accuracy out to two decimal places when using f string and we'll see accuracy on tests it is curly
braces test on the score accuracy and then I'm gonna use the colon to format the thing and say zero
point two percent.
I can't actually just hit shift enter to execute the cell because I've already closed my session.
I'm going to have to come back.
I'm just gonna rerun all my cells below.
So what kind of accuracy would be a good result.
Well I would expect our model to give us a accuracy on our test data set that is somewhat close maybe
slightly lower than what I see on the validation data set.
So I'm hoping that I'm getting something on the 98 97 percent mark maybe 96 if I'm not lucky.
If you take a look around online and take a look at what the baseline results are for accuracy on the
end list database then you'll find that a lot of models actually are able to achieve a prediction era
of less than 1 percent.
And if you're talking really state of the art prediction errors then we're talking around zero point
two percent and those kind of prediction errors are achieved by very large convolution all neural networks
in our case.
We've actually got a very very simple precept Tron and we're training it on what really his consumer
hardware which is not really all that expensive and not all that powerful.
So getting a 97 98 percent test accuracy is actually really really cool.
One of the reasons why this dataset and this problem is actually so famous is because recognizing handwritten
digits is actually a very very practical problem and it's something that's actually used by the postal
services.
So for example the Postal Service in the United States processes around 500 million pieces of mail per
day and recognizing handwritten addresses handwritten digits means that postal workers don't individually
have to handle and examine millions and millions of mail and parcels every day corrected recognizing
handwritten addresses is definitely a step up from processing individual digits but it's not too far
off and we're still learning to walk before we can jog.
If you're curious how other models have done with the atmos database I recommend you to go to this Web
site and here you can see all the classifiers that people have trained on amnesty.
So these researchers have collated a wonderful Web site and they are reporting the testing error in
this table here.
So for example boosted stumps here by these researchers in 2009 had about seven point seven percent
test there.
So this will give you a little bit of an idea of the lay of the land.
And it's very interesting to see how the best non convolution of neural net fit on amnesty.
So I've actually had a look around who built the most accurate multilayer perception that's out there
and what I found was that in 2010 these four researchers had a perception that was massive and it achieved
like a really really low error rate something like zero point three five percent.
And the way they did this was to have a massive neural network that had something like six layers and
that first hidden layer had two thousand five hundred neurons.
The second hidden layer had two thousand neurons.
The third one one thousand five hundred and so on all the way down to 1000 five hundred.
And then finally ten and the other thing they did was they started messing around with the training
data.
He started stretching the images flipping the images and kind of like deforming the images and transforming
them so that they were actually generating more training data than the original sixty thousand.
So through this process of having basically an enormous percent drawn and also generating more data
from the existing data they were able to get the error rate down to zero point three five percent without
using a different architecture from what we were using.
So again congratulations for making it all the way to the end.
This was a very very difficult module.
There was a lot of things that we covered him.
It was heavy on the tensor flow API and it meant we really had to get stuck in with how tensor flow
works under the hood.
So hats off again to the team at Karas who makes tensor flow so easy to learn and so accessible because
a lot of things are definitely not that straightforward.
When you first learn them but when you finally do you unlock a lot of power more power than what carries
is actually able to give you.
And combined with tensor board you're able to really track how your training is going how to compare
like the different models.
If your model is learning the way you expect it to or if there are some kind of problems do you have
to fix tensor board is an incredible tool for working with complex models and giving you insights into
how to improve them and how to compare them and to check how your training is going.
So I hope you enjoyed this module.
That's all for me.
Well done again on all the hard work.
Take care.