Hello everyone and welcome to the introduction to tree methods lecture in this lecture will discuss
decision trees and ran them for us as machine learning models.
If you want the mathematics behind these topics go ahead and read Chapter 8 of an introduction to systematical
learning.
Let's start off with a thought experiment to give a little bit of motivation behind why we would use
a decision tree method.
Imagine that I play tennis every Saturday and I always invite a friend to come play with me.
Sometimes a friend shows up sometimes not for him.
It depends on a variety of factors such as weather temperature humidity wind etc..
I start keeping track of these features and whether or not he showed up to play with me in my data to
look something like this.
Each of these columns represents a feature and each of the rows represents a certain day.
I go ahead and for each row or day take no other features.
And finally at the final column I have a label.
Whether or not my friend came to play with me so you can see on the first day I have temperature mild
a sunny outlook.
Humidity of 80.
Windy it was not and my friend came out to play.
However in Day 2 it was a hot temperature sunny outlook 75 humidity.
It was windy and my friend did not comply with me.
I can build a decision tree off of this process.
I want to use this data to predict whether or not my friend is going to show up to play in an intuitive
way to do this is through a decision tree.
And this is what a decision tree may look like or the data I just showed in this tree.
We have notes and notes split for the value of a certain attribute we can he can see here we have Outlook
nodes humidity nodes and windy nodes.
So each of these attributes are nodes and then we have edges edges are the outcome of a split to the
next node in this tree.
We also have a root node.
That's the No that performs the first split.
So in this particular example the root node is the outlook feature where I performed the first split
and I split into the possible categories of sunny overcast and rain and then those will lead to other
nodes such as humidity and windy.
The coloured notes which are yesses and knows are leaves and those with terminal knows that predict
the outcome.
Let's go ahead and discuss intuition behind splitting on these nodes.
Imagine now that I have this data with three features X Y and Z and I'm trying to predict two possible
classes.
Class A or Class B splitting on y gives us a clear separation between classes.
So I asked the Weichel one I get perfect separation of the two class instances of class A and the two
instances of class B.
Again if I go back.
Check out this imaginary data.
Note how the Y.
Feature.
Split the classes perfectly.
And also note how the x and z features don't split the classes perfectly meaning I could have also tried
splitting on other features.
First if I split on X first or Z first.
I don't get proper separation of those classes.
Entropy and information gain are the mathematical methods of choosing the best split.
Go ahead to refer to the reading assignment if you're interested in the mathematics behind the intuition
but the actual intuition is just trying to choose the features that best split your data.
This is called trying to maximize your information gain off of this split.
Again refer to chapter 8 if you want more information on the mathematics behind these topics of entropy
in information gain.
Now let's talk about random force as a way to improve performance off single decision trees.
The primary weakness of decision trees is that they don't tend to have the best predictive accuracy
.
This is partially due to the high variance meaning that different splits in the training data can lead
to very different trees.
Bagging is a general purpose procedure for reducing the variance of machine learning method that's discussed
in Chapter 8 of the reading assignment.
We can build off the idea of bagging by utilizing random force random force.
It's just a slight variation of these bag to trees that has even better performance.
Which is what we're going to be discussing and coding out in our what we do of a random forest as we
create an ensemble of decision trees using bootstrap samples of the training set bootstrap samples of
the training set just means sampling from the training set with replacement.
However we're building each tree each time a split is considered a random sample of M features is chosen
as a split candidate from the full set of p features.
The split is only allowed to use one of those m features a new random sample of features is chosen for
every single tree every single split for classification.
This M. random sample of M features is typically chosen to be the square root of P where P is the full
set of features.
You may be wondering what's the point of using random force.
Well suppose there is one very strong feature in the data set a feature that's really strong at predicting
a certain class when using bagged trees.
That's that bootstrap sampling.
Most of the trees will use that feature as the top split that's going to result in ensemble of really
similar trees that are highly correlated which is something you want to avoid averaging highly correlated
quantities does not significantly reduce variance.
I randomly leaving out candidate features from each split random force are going to Dijk correlate to
trees.
Meaning making the trees in the pendant of each other such that the averaging process can reduce the
variance of the resulting model.
So we won't be affected by features that really strongly predict declasse data.
Now that we've taken a look at the theoretical background behind this isn't trees in random force.
Let's go ahead and move on to seeing some examples with Python.
We'll start off by taking a look at a small example dataset of Infosys which has a medical spine condition
and will look at patients and try to see or predict whether or not a corrective spine surgery was successful
.
Then for your portfolio exercise project will go ahead and use lone data from lending club to predictive
default rates on some real world data.
Thanks everyone.
I'll see at the next lecture