Hello and welcome back to the course on Deep Learning.
Today we're talking about ReLU which is the rectified linear units.
And this is an additional step on top of our convolution step.
So it's not a separate big step. It's a small step.
It's step one be basically.
And what is going on here.
Well we have our input image.
We have our convolution layer which we've discussed.
And then on top of that we're going to apply wait for it.
Our favorite rectifier function.
And you're already familiar with the rectifier function from
the previous section on artificial neural networks.
And in our.
So sometimes authors or instructors,
separate the convolution and the rectifier as two separate steps.
In our examples.
We're just going to, consider them the,
just one big step, for the convolution than the rectifier.
And the reason why we're applying the rectifier
is because we want to increase non linearity in our image
or in our network in our convolutional neural network.
And rectifier acts as that,
filter
or acts as that function which breaks up linearity.
And the reason why we want to increase non-linearity in our network
is because images themselves
are highly nonlinear, especially if you're recognizing different objects,
next to each other, just on backgrounds and stuff like that.
Like the image is going to have lots of nonlinear elements, and the transition
between pixels, adjacent pixels is often going to be non-linear.
That's, you know, because as borders is different colors is different.
there's different elements in your images.
And but at the same time, when we're applying
mathematical operations such as convolution,
you know, and,
running this feature detection to create our feature maps,
we risk that we might, create something linear
and therefore we need to break up the linearity.
so let's have a look at an example.
here is a image, an original image.
Now when we apply a, feature detection,
a detector to this image, we get something like this.
So you can see here that black is negative.
White is positive values.
Well, when you apply, a feature detector to a, like a proper image,
which has not just zeros and ones but has lots of different values.
And you, apply,
as we saw previously, feature text is can have negative values in themselves.
Sometimes you'll get negative values.
And here their black ones are negative, white ones are positive.
And what a rectified linear unit.
function does is it removes all the black.
Right. Anything below zero turns into zero.
And so from this it turns into this. Right.
And so it's it's pretty hard to see what exactly is the benefit
in terms of for in terms of breaking up linearity.
I'll try to explain.
I'll try to like show an example on this image.
But at the end of the day, it's this is a very mathematical concept,
and we would have to go into a lot of math to really explain what is going on.
But let's let's try let's have a look.
So, for instance, let's look at this, this building here.
Right. So this is a building on its own.
and then you can see this shadow, this black part, this shadow over here.
Well, you can see that it's white.
The the reflection of the light, and then it's a gray,
and then it gets darker and then it gets darker again.
Right.
So and when we take it out, we take out that black part.
So think of it in terms of linearity. Right.
So it looks like when you go from white to gray
the next step would be black right. The next step would be black.
It's it's a linear progression from bright to dark.
And therefore this is kind of like a linear situation.
When you take out the black you break up the linearity.
let's try another one.
Let's have a look here.
And at the same time it's still that same building, right?
It's not it's not like you are, you're like it's not like you're
blending two buildings into each other, but that is secondary.
The main point is breaking up the linearity.
So let's have a look here. Same thing.
So you see white gray black gray white.
And when you break it up you don't have that anymore right.
You don't have that progression the gradual progression that you just have
like an abrupt change.
And that helps introduce non-linearity into your image.
So it's a very rough explanation, very, kind of like on the, on the fingers
explanation, rather than, technical, but hopefully
it kind of helps you understand a bit better what we're talking about here.
So here again, you can see white gray is a better example.
Even still bright darker darker darker darker darker darker.
So this part looks like it's linear.
Then you break it up like that.
again.
So this is a very rough explanation is not absolutely perfect,
but at least it gives you some idea of what's going on.
but if you'd like to learn more, there's a good paper.
as always, there's always a paper.
This one is by CCJ. Cool.
From the University of California.
And it's called Understanding
Convolutional Neural Networks with a mathematical model.
And basically there he answers to questions and
you need to just look at the first one.
And the question is why in a nonlinear activation function,
he's essentially at the filter output of all intermediate layers.
So that kind of explains it in a bit more detail,
both in terms of intuition and mostly in terms of mathematics.
So that's an interesting paper where you can get some more additional
information on this topic.
And if you really want to dig in and explore,
some, some cool stuff here,
then there's another paper that you might be interested in.
It's called, delving deep into rectifiers, surpassing
human level level performance on image net classification.
And here, the authors,
coming here and others from Microsoft Research,
they propose a,
different type of, rectified linear unit.
function.
They proposed,
parametric rectified linear unit function, which you see here on the right.
And they argue that it delivers better results without sacrificing performance.
So interesting read if you'd like to get a bit more into this topic.
And, that's all for today.
The ReLU layer is pretty simple, pretty straightforward.
Just just applying the rectifier function.
And I look forward to seeing you next time.
Until then, enjoy deep learning.