Hello and welcome back to the course on Machine Learning.
Today we're talking about Support Vector Machines or SVMs for short.
So SVMs were initially developed in the 1960s.
Then they were refined again in the 1990s.
And only now they're becoming a very popular in machine learning
because they are demonstrating that they can be very, very powerful
because they are somewhat different to other machine learning algorithms.
And we'll find out how they're special towards the end of this tutorial.
But for now, let's understand how support vector machines actually work.
All right.
So here we've got as usual points on a two dimensional space.
For simplicity's sake we've got just two columns x1 and x2.
And we've got some observations.
Some are red, some are green. So we've already classified them.
But now how do we derive a line that's going to separate them.
So how do we actually separate these points.
Because that's a separation.
Or in other words that decision boundary is going to be very important
for us going forward when we start adding new points.
So that's that's a point about classification.
That's the purpose of our classification.
We want to create a boundary between these two
so that when we in the future
add new points that we want to classify that haven't been classified yet,
we will know where they will fall
either in the green green area or in the red area.
So how can we separate these, points?
We see here?
Well, one way is to draw a line like that in our two dimensional space.
And then,
so anything to the right will be green, anything to the left will be red.
And if a new point falls somewhere on this space, we will know right away
if it's red or green, because we'll know where it falls.
However, there's another way we can draw a horizontal line like that,
or we can draw a diagonal line like that.
We can actually draw another diagonal line, or we can draw another diagonal.
So there's lots of different lines that we can create that will achieve
the same result will separate our points to two classes.
But at the same time they all in the future will have different
consequences.
So when we add new points,
depending on where that point will fall, it'll either be classed
as part of the green zone or part of the red zone.
So we want to find the optimal line.
And that's what, SVMs are all about.
They're about finding the best line or the best,
decision boundary, which will help us separate our space into classes.
So let's find out how the SVM actually searches for this line.
Well, the line is searched through the maximum margin.
So here you can see a line.
And this is the line an SVM will draw.
And so basically it's the line that separates these two classes of points.
And at the same time it has the maximum margin which means this distance.
So this line is drawn equidistant from this point and this point.
And we'll find out exactly why these points in a second.
And then the distance between the line and each one of these points.
So that's equidistant and that's margin.
So the sum of these two distances has to be maximized
in order for this line to be the result of the SVM.
And these two points are actually called the support vectors.
Why they're called vectors. We'll also find out in a second.
But so basically they these two points are supporting this whole algorithm.
So even if you get rid of all the rest of the points,
nothing will change the algorithm will be exactly the same.
So these other points, they don't, contribute to the result of the algorithm.
Only these two points are contributing.
And therefore they called the supporting vectors.
You can you can call them supporting points, but in reality they're vectors.
And this is why, because in a multidimensional space,
when you have more than just two variables, you can have three, five, ten,
100 variables each.
point is actually no longer point because you can't visualize it on a
two dimensional plane or even a three dimensional space.
And therefore, each of those points that we see here is
considered is actually a vector in a multi-dimensional space.
So the more general term for points that we see here are vectors.
And this is something that is studied in, mathematics in a university
or high school mathematics.
And basically so generally speaking they're all vectors.
Just in this particular example,
when we have two dimensions, then we can call them points.
But in reality their vectors.
And that's why they're called support vectors.
So hence these two specific vectors are the ones
supporting kind of supporting this decision boundary.
Or this way we're building this algorithm. That's why they're important.
And that's why this whole algorithm is called the support vector machine.
So now what else do we have here.
Well we've got the line in the middle which is called the maximum margin
hyperplane or the maximum margin classifier.
So in a two dimensional space, it's it's just like a classifier is just a line,
but actually in a multi-dimensional space, it's a hyperplane.
And I know it's a very a bit of a confusing term,
but that's what it's called a maximum margin hyperplane.
So those all other ones that we saw were also hyperplanes.
But there weren't the maximum margin hyperplanes.
And you can check that just also
you can draw a different hyperplane here and just check what the margin will be.
It'll always be less because this is the one with the maximum margin.
And then you've got the green and the red dotted lines.
So the green one is called the positive hyperplane.
And the red one is called the negative hyperplane.
It doesn't really matter in which order you name them.
Just the point is that one of them is positive, the other one is negative.
I basically anything to the right of the positive is classified
as, the green category or the positive category.
Anything to the left
is classified as a negative category or the red category in our case.
So that's, how the, support vector machine algorithm works.
Of course, there's
some complicated mathematics behind it, but the essence, the intuitive part of it
is exactly this that we're working with a linearly separable,
data set where we can actually it's given to us by default
that we can put a line through our chart which will separate the two categories.
And then we're just searching for the one with the maximum margin.
So conceptually, when you think about it, it's actually a pretty,
simple algorithm when you think about it this way.
If I going into the mathematics and the question is what's so special
about SVMs?
Why are they so popular and why are they different?
to other machine learning algorithms?
And that's exactly what we're going to talk about right now.
So imagine you're trying to teach a machine
how to distinguish between apples and oranges,
how to classify, a fruit into either an apple, an orange.
So you're telling the machine that.
All right, I'm going to give you some test data.
So here's have a look at all of these apples.
These are apples to oranges.
Analyze them. Look at them.
see what, parameters they have.
And then next I'm either going to give you I'm
going to give you a fruit which will be either an apple, an orange.
And you're going to need to classify it and tell me
whether it's an apple, an orange. Right.
So that's kind of a standard machine learning problem.
Now in our case here you can see let's say on the right we have oranges.
On the left we have apples.
So what predominantly machine algorithms would do is they would
look at the most apple the apples and the most orange oranges.
So they would look at the most stock standard common type of apples
and the most stock standard common type of oranges.
And now case it would be some apple, some over there in the
in the very heart of the apple, class, far away from the oranges.
And for the oranges would be somewhere over there.
So also in the very heart of the orange class, far away from the apple.
So they would try a machine, would try to learn from the apples that are very,
like apples. So it would know what an apple is.
And it also try to learn from oranges, so it would know what an orange actually is.
And that's how most of the machine learning algorithms work.
And then based on that, it would be able to come up with some predictions
and classifying for new data elements and new variables that you would give it.
In the case of Support Vector Machine, it's a bit different.
Instead of looking at the most stock standard apples and stock
standard oranges, what does Support Vector Machines do
is they actually look at the apples that are very much like an orange.
So here you can see an apple which is not your standard apples, orange in color.
Right. So it's very easy to confuse this apple with an orange.
And they would look at oranges which are not stock, standard oranges
which are more like apples than anything else.
So ignore the lemon here.
So those of us in the image, just out of the oranges,
the SVM would pick the one that is that looks the most like an apple.
In this case, we have a green orange.
It's, not normal to have a green orange.
When you think of orange, you think of an orange orange.
And so what that is, is those are the support support vectors.
So the support vectors, you can see that
they're actually very close to the boundary.
So they're very close to to the apple or the red one would be very close
to the green ones.
And the orange of the green mark
here would be very close to the red ones and therefore the support vector machine.
In that sense, you can think of it as like a more extreme type of algorithm,
a very rebellious type of algorithm, or very risky type of algorithm,
because it looks at the very extreme case, which is very close to the boundary,
and it uses that to construct its analysis.
And that in itself makes the support vector machine algorithms very special,
very different to, most of the other machine learning algorithms.
And that's why at times they perform much better
than, non support vector machine algorithms.
So there you go.
I hope is explanation and intuition of support vector machines was useful.
And now not only you know how they work, but also why they are different
to other algorithms out there that are used in machine learning.
And on that note, we're going to end today's tutorial.
I look forward to seeing you next time.
And till then, enjoy machine learning.