Hello and welcome back to the course on Deep Learning.
Today we're talking about the neuron,
which is the basic building block of artificial neural networks.
So let's get started.
Previously we saw an image which looked like this.
And these are actual real life neurons,
which have been smeared onto a glass and, colored a little bit.
And they are observed. Through a. Microscope.
So this is what they look like, as you can see, quite an interesting structure.
body and then lots of different,
tails, kind of branches coming out of them.
And, this is very interesting,
but the question is, how can we recreate that in a machine?
Because we really need to recreate that machine.
Since the whole purpose of deep learning is to mimic how the human brain works,
in the hope that by doing so, we're going to create something
amazing, we're going to create an amazing infrastructure for machines
to be able to learn.
And why do we hope for that?
Well, because the human brain is,
well, just happens to be one of the most powerful learning,
learning tools on the planet or like learning mechanisms on the planet.
And we just hope that if we recreate that, we'll have something.
As awesome as that.
So our challenge right now,
our very first step to creating artificial neural networks is to.
Recreate a neuron.
So how do we do that?
Well, first of all, let's have a closer look.
At what it. Actually is.
This image was first created.
By, Spanish neuroscientist Santiago Ramon Cajal, in.
1899.
And what he did was he died.
Neurons in actual brain tissue.
And look at them under a microscope.
And while he was looking at them, he actually drew what he saw.
And this is what he saw.
He saw two.
Neurons or two large neurons over there at the top.
which had all of these, branches coming out of them towards their top part.
And then the each one of them had this rod.
or like thread
coming out towards the bottom, very long one.
And, yeah. So that's what he saw.
And now, you know, technology has advanced quite a lot.
And we, have seen neurons much closer in more detail.
And now we can actually draw, what it looks like.
Diagrammatically. So let's have a look at that.
Here's a neuron. This is what it looks like.
Very similar to. What? Santiago.
Ramon, drew over here
and here in this neuron, what we can see is that it's got a body.
Well, that's the main part of the neuron.
And then it's got some branches at the top which are called dendrites.
And it's also got an axon which is at the long tail of the neuron.
And so what are these.
Dendrites went for and what's the axon for? Well.
the key point to understand here
is that neurons by themselves are, pretty much useless.
It's like it's like an ant, right?
An ant on its own can't do much like five ants together.
Maybe they can. Pick something up.
But again, they they don't.
They can't build an anthill or the colony. Establish a colony.
They can't.
work together as a as a huge organism.
But at the same time, when you have lots and lots of ads,
like you have a million ants, they can build a whole colony.
They can build an anthill.
Same thing with neurons by itself. It's not that strong.
But when you have lots of neurons together.
They work together to do magic.
And, How do they work together? That's the question.
Well, that's what the. Dendrites and the axon offer.
So the dendrites are kind of like the receivers of the signal for the neuron.
And the axon is the.
Transmitter of the signal for the neuron.
And here's an image of.
How it all works. Conceptually.
So at the top you've got a neuron.
And you can see that its dendrites are connected.
To axons of other neurons that are like even further away above it.
And then the signal from this neuron travels down its axon
and connects or passes on to the dendrites of the next neuron.
And that's how they're connected.
And, in that small image over there, you can see that,
the axon doesn't actually touch the dendrite.
A lot of, a lot of machine learning or like a few machine learning scientists.
are very adamant about the fact that it doesn't touch, it's.
Like the it doesn't touch it.
Has been proven that there is no.
Physical connection there.
But the point that we're interested.
In is that that connection between them that, the whole concept of.
The signal being passed, that's. Called the sign ups.
You can see over there in that little image, that's,
Finger bracket is a sign ups, and that's the term we're going to be using.
So instead of calling our artificial.
Neurons
the lines that we're going to have or the connectors for artificial neurons,
we're not going to be calling them axons or dendrites.
because then the question is whose connection is this?
Is it that neurons or is this neurons we. Just call and.
We're just going to call them synapsis.
And that's kind of just, answers all questions right away.
That's just basically where the signal is passed.
Doesn't matter who.
That element belongs to.
That's just a representation of the signal to passed.
And we'll see that just now.
So basically that's how a neuron works.
And yeah. So let's move on to how are.
We going to represent neurons or how we're going to create neurons in, Machines.
So we're moving away now.
We're moving away from neuroscience and moving into.
technology.
And here we go.
So here's our neuron also sometimes called the node.
the neuron gets some input signals and it.
Has an output signal.
So dendrites and axons remember.
But again we're going to call these sign ups and.
Then.
The these input signals, we're going to represent them with other neurons as well.
So in this specific case you can see that this neuron,
this green neuron is getting signals from yellow neurons.
And in this course we're going to try.
And stick to, a.
Certain color coding regime where yellow means an input layer.
So basically all of the neurons.
That are on the outer layer or in the first front of,
where the signal signals coming in, and.
By signal it might be like a bit of an over overkill to call this a signal.
It's just basically input values.
So so you know how even like in a simple linear regression
you have input values and then you have a a predicted value.
Same thing here.
So you have input values.
And there they are the yellow ones.
And then on the right we will see just now it'll be red.
It'll be the output value.
the, the thing that I wanted to point out here
is that in this specific example, we're looking at a neuron
which is getting its signals from the input layer neurons.
So they're also neurons.
But they're they're input layer neurons.
sometimes you'll have neurons which
get their signal from other hidden layer neurons.
So from other. Neurons.
And the concept is going to be exactly the same in just in this case.
We for simplicity's sake, we're portraying this example.
And in terms of the input layer, the way to think about it is,
in, in the analogy of the human brain,
the input layer is your senses, right?
So whatever you can see here feel.
touch or smell.
And of course, it's like there's, there's a lot of things you can see.
There's a lot of information coming in.
But those are your that's what your brain is.
Limited to is pretty much a, like.
It's pretty much lives in a box made out of bones.
And it's only just, it's it's a mind blowing concept to think about that
your brain is just locked in a black box, and the only thing like it can't see,
you can't hear.
The only thing it's getting is electrical impulses
coming from these, organs that you have, which are called your ears.
And those eyes, you know, your sense of touch and.
Whatever, and you and your and your taste.
Right?
So it's just getting signals, but it basically lives in this dark
black box, and it's making, making sense of the world through your senses.
It's, it's it's phenomenal.
and so, yeah.
So you have these, inputs that are coming in in terms of human brain.
Those are your five senses.
And, in terms of machine learning.
Or deep learning, that is basically your.
input values.
So your independent variables and we'll get that in a second.
So, your input values, they the signal is passed through synapsis.
To your neuron, and then your neuron.
Has an output value that it passes it further on down the chain.
In this specific case in terms of color coding.
Again yellow means input layer.
So we kind of simplifying everything here.
We're saying we're only going to have like the input layer.
Then we're going to have one.
Hidden. Layer with the green which is a hidden layer.
And then we're going to have the output layer right away.
So just so that we can get used to these colors for now.
So there we go.
That's the basic structure.
So now let's look in a bit more.
Detail at these different elements that we have.
So we've got the input layer.
And what do we have here.
Well we have These inputs which are in fact independent
variable soon depend variable one independent variable two.
Independent variable m.
The important thing to remember here is that these
independent variables are all for one single observation.
So think of it as just one row in your database.
One observation.
You just take all of the independent variables.
you know, maybe it's, the age of the person there,
the amount of money in the bank accounts, and then how how do they drive
or walk to work?
What method of transportation do they use?
So but that's all descriptors of one specific.
Person that you are.
Either you're training. Your model on.
or you're performing some prediction on, and.
the other thing you need to.
Know about these variables is that you need to standardize them.
So you need to either standardize them, which means,
make sure that they have a mean of zero and variance one.
Or you. Can also. Sometimes.
And Hedlund will point out these situations in a bit more.
detail. Perhaps in the practical tutorials.
You might come across these.
Sometimes you might want to
not standardize, you might want to normalize them, meaning that instead of,
making sure the mean and I mean the zero variance is one you just take,
you know, to subtract the minimum value, and then you divide by the maximum
minus minimum.
So by the range of your values and the for you get values between 0 and 1.
And it depends on the scenario.
You might want to do one or the other.
But basically you want all of these variables to be quite similar in the, in
about. The same.
range of values.
And why, why is that? Well.
all of.
These values are going.
To go into a neural network where as we'll see just now,
they'll be added up and multiplied by weights added up and so on.
And just going to be this is going to be easier for.
The neural network to process them, if they're all about the same.
And and in fact, you know, that's, that's
just how it is going to be able to work properly.
And if you want to read.
More about, standardization, normalization and other.
Things that. You can do if you know.
What variables, a good additional reading.
paper is called Efficient Backprop by Yann LeCun, 1998.
the links over there.
So Yann LeCun, we're actually going.
To talk more about, this, phenomenal person in the space of deep
learning in, the part of the course where we're talking about convolutional.
Neural. Networks.
And you'll, you'll see that this is definitely a person
who knows what he's talking about.
He's a close friend of, Geoffrey Hinton, who we've already seen,
who've already mentioned. So, in.
This paper, you'll learn more about.
Standardization and. Normalization, but also you can.
Pick up lots of other different tips and tricks, and it'll be a good
a good source for additional reading as you go through this course.
So definitely check it out.
if you're interested in
so yeah, check it out if you're interested in some additional reading.
There we go. So that's.
What we need to do with the. Variables.
And here we've got. The output value.
So what can our output value be.
Well we got a couple options.
well we've got a couple of options.
Output value can be it can be continuous.
Like for instance price.
It can be binary.
For instance a person will exit or. Will stay or it can.
Be a categorical variable.
And If it's a kind of categorical variable.
The important thing to remember here is that in that case, your output value.
Won't be just one. It'll be.
Several output values.
Because these will be your dummy variables.
Which will be representing your categories.
And that just this how it works.
And it's just important to remember that in that case, that's how you're going
to be getting your categories out of the, artificial neural network.
but let's go back to a simple case of one output value.
And now let's, one more point or kind of like an,
the point that we've already made.
I just wanted. To reiterate this point.
on the left, you've got a single observation.
So one of your from your data set.
And on the right you have a single observation as well.
And that is the same observation.
So important to remember that
like whatever inputs you're putting in that's for one row.
And then the output you get that is for that same exact row.
Or if you're training your,
neural network, then, you know, you're putting the inputs in for that one
row, you're putting the output in for that one row.
So like if you want to simplify the complexity, think of it as a.
like a simple linear regression or a multivariate or linear regression.
So you're putting in your. values.
You have your output.
There's, there's like there's no question about it when we're talking about,
things like. Regression because we're so used to it.
Same thing here. It's it's nothing too complex.
We're just putting in values. We're getting output.
But just remember.
That every time it's one row you're dealing with so you.
Don't get confused.
And start putting in like, thinking that these are different, different rows
that you're putting into your, artificial neural network or something.
This is all just values in that one row.
So different observation, different characteristics of or attributes
relating to that one observation every single time.
Okay.
So next thing that we want to talk about here is.
or the sign ups is, is a sign of sense.
here we've got sentences and they all actually get assigned weights.
weights.
We're going to talk more about weights for the down.
But in short, weights are crucial to artificial.
Neural networks functioning because.
Weights are how neural networks learn.
By adjusting the weights, the neural network
decides in every single case what single signal is important.
What signal is not. Important to a certain neuron, what.
Signal gets passed along and what signal.
Doesn't get passed along, or to what strength?
To what extent signals get passed along.
So weights are crucial.
They are, and they are the things that get adjusted.
Through the process of learning. Like when.
When you're training on artificial.
Neural network, you're basically. Adjusting all of the weights.
In all of the sign ups across this whole neural network.
And and that's where gradient descent and,
backpropagation come into play.
And those are concepts that we'll also discuss.
so. Basically those are the weights.
That's all we. Need to know for now.
And here we've got the neuron.
So signals go into the neuron and what happens in the neuron.
So this is the interesting part.
Like we're talking about the neuron today.
What happens inside the neuron.
So a few things happen.
First thing and the first step is that all of these values that it's getting.
Gets added up.
So it Takes the added so the weighted sum of all of.
The input values that it's getting very simple.
Right.
It's very, very straightforward.
Just add up multiply by the way. Add them up.
and then it applies an activation function.
Now we're going to talk more about activation functions further down.
But it's basically a function
that is assigned to this neuron or to this whole layer.
And it.
Is applied to this weighted. Sum.
And then from that the neuron understands, if it needs to pass.
On a signal like.
The, that's the signal that it passes on that, the function
applied to the weighted sum.
But basically depending on the function,
the neuron will either pass on a signal it or it won't pass this signal on.
And that's exactly what happened here in step three.
the neuron passes on that signal to the next neuron down the line.
and that's.
What we're going to talk about in the next tutorial, because it is.
Quite an important topic we want to delve deeper into, the activation function.
But hopefully for now everything is it should be pretty clear how, you know,
you've got input values, you've got.
Weights, you've got the synopses, you've got something that you know, happens in
the neuron, you've got the weighted sum and then the activation function applied.
And then that is paused down the line.
And that is just repeated
throughout the whole neural network on and on and on and on.
you know, thousands, hundreds of thousands of times
depending on how big, how many neurons you have,
how many sign ups as you have in your neural network.
So there we go.
Hope you enjoyed today's tutorial.
Can't wait to see you next time.
And until then, enjoy deep learning!