We've got our data now in X train y train X validation y validation but in a previous video we evaluated
our model using the score function.
What does the score function use that uses the coefficient of determination.
But if we remember right back at the top will go right back out that are just to check it out because
we wrote this down.
We know that the evaluation metric for this competition.
So for this project that we're working on is R M S L E or in other words root means squared log error.
That's not the same as coefficient of determination.
And if you're wondering where we got this evaluation function from it's from the code competition overview
dot evaluation which we saw right at the start but we'll just check that out again.
Evaluation.
So the evaluation metric for this competition is the R M S L we can Google that to see what it actually
is.
Let's go here.
Root means squared log error
what is the difference between how do you interpret root mean squared log root error.
Okay so we got a fair few options.
So if you wanted to read further these are something you can look up but we're going to work through
this in code.
Let's go down.
You might be wondering we haven't seen or really mentioned root means squared log error.
That's a bit of a mouthful in itself.
So what we might do we're gonna have to create our own evaluation function because if we come back to
our keynote so classification and regression metrics we're working on regression problem here.
This is one of the slides from the SBA loan section.
We got to mean absolute error.
Okay means squared error MSE.
Okay.
Root means squared error.
But this is missing the log component.
Let's see if we could get that in through SBA loan.
So we'll go back here what we might do is build our own.
So building and an evaluation function now.
This is something you may have to do depending on the project you're working on right.
So if we're working on this cable competition and they say that for some reason we have to use this
metric.
Who knows the people who posted this challenge may require this they may have decided this is their
best evaluation metric depending on what project you're working on.
You may have to find an evaluation function which suits your use case the best.
And so what we're going to do is create a evaluation function because we're going to be running multiple
experiments we want to be feeding multiple different machine learning models and evaluating them.
That's why we're going to build it into a function so we can use this functionality multiple times without
having to repeat ourselves because we're kind of lazy when it comes to writing or this code.
You know I just want to experiment more and more and create a valuation function.
The competition uses root mean square low error M S L A.
So you go from S K lined up metrics that can tell you.
So I get learn you could go to the SBA loan valuation metrics but we can go through here we can have
a look at some evaluation metrics main squared log era.
Okay.
So we might use that mean squared log era but we're going to have to add our own little piece on it
because this is not root mean square logout.
So what we might do is it might take this one we can import that let's do that.
So from S.K. learned up metrics import mean squared log error and for the hell of it or do mean absolute
error you make predictions on all the examples and then you minus your predictions from your actual
values and take the average that's the mean absolute error.
So how much on average is your prediction from the actual sale price.
But Main squared log error is more to do with the ratio and most often times with regression problems
right.
If you're like 10 dollars off something like So if we were 10 dollars off this sale price prediction
wouldn't really matter.
But if we were 10 per cent off that'd be pretty bad right.
We wouldn't want to be more than 10 percent off whatever value insert something it could be 20 percent
could be 30 percent.
But that's kind of the difference between these two right.
Anytime you see absolute think of it as being 10 dollars off and anytime you see like squared or some
sort of log or think of it like that the ratio so being like 10 percent off that's when are these two
you can really compare them to each other.
But we've got a little bit more of that in the slide.
You might want to take a screenshot of that that is a pretty good one.
But this will all be in the slide.
You can check out later.
We want to create a valuation function we've got main squared log era.
So how would we turn that into root mean square logo.
So let's try it out so maybe we go here.
I don't want to type out root I mean that's too much R S M S Ali root main squared logger.
Beautiful and that can just take Y test and Y prance gonna take the same inputs as my psychic learn
function so see how this takes true why red it could be y true could be why test doesn't really matter.
Why test.
And then we go here we'll leave a little doctoring for ourselves calculates
root main squared log era between predictions and true true labels.
Wonderful.
And then we might go return.
Nice and simple.
We can use shoes empty square root of main squared log era it will have to pass this to the test matches
and the predictions to mean squared log error.
That's actually pretty easy.
What we're doing here we're taking the square root so right sort of root component of the means squared.
Log era the squared log error between predictions and true labels.
Wonderful.
So we've created our own evaluation function that lines up with Cargill's requirements.
The root means squared log error.
Okay.
And we probably want to check a few more things like we want and create an evaluation function to check
a few things at the same time so we don't have to keep calling something like this.
So let's create function to evaluate model on a few different levels.
So again evaluation is just as important as building the actual model.
So we'll go here show scores models so we'll pass it our model we'll make some predictions on the training
data some model not predict.
So we'll evaluate how our model did on the training data X train and then we'll evaluate how our model
did on the validation data model dot predict we'll pass it our X validation here if our model is performing
better on the validation data set that's hinting at us that our model is over fitting.
So between training and validation generally because our validation set only has 12000 examples and
we're not training our model on it you'd want to see slightly worse metrics on the validation set.
Investing the training set.
So we'll go scores equal maybe we'll return a dictionary of some sort.
Yeah.
So training and they a.k.a. mean absolute error.
We're gonna go.
Can I tab auto complete mean absolute error.
Yes I can.
Why train.
So remember we're just comparing the Why train.
So this is the training mean absolute error on the training labels versus the training predictions.
And beautiful.
We're gonna go valid M.A..
So again you'd expect this valid M.A. To be slightly worse and the training is going to be why valid
then this is gonna be valid reds and then we're gonna go training and excel for root means squared error
and we can do this using our fancy little function that we've just created up here.
This is exciting but writing our own evaluation functions.
Maybe that's a pull request potential cyclone.
Wonder why they haven't got root mean squared log error.
Maybe it's not that popular.
Or maybe you can just easily create it yourself like that.
Is that correct.
Why train.
Train prints.
I'm talking too much.
Don't talk in code yet.
No one can.
You can do whatever you want.
Here we go here and there Sally.
Except don't under evaluate your machine learning models.
That's the only thing you're not allowed to do.
Everyone loves a good model but everyone loves and a well evaluated machine learning model even more.
And finally for good measure since we're here let's do the default score and we just import that.
That's probably a bit better to try to score
what is this.
Why is it not working tab.
I can play is given me some issues.
So we've got to ask.
Just on a R2 score.
Yeah.
Trust your instincts Daniel come on.
R to score.
So that's all we're changing there rather than NA Rican puting something will are just imported now.
This is fairly decent function not going to lie.
Takes care of a few things.
Train friends.
Probably some redundancy we could fix but that's alright.
We're trying to experiment as fast as possible you might be thinking about how experiments are getting
slowed down but with your own practice without having to listen to me talk and go through these sort
of things you'll be whizzing through experiments.
No that's wrong isn't it.
This needs to be veiled threats prints are beautiful and we're going to return the scores.
Okay.
So what have we done here.
So it built an evaluation function inline with what's required for this particular project and in our
case its root mean square logger.
I'm gonna go hoarse saying that if I keep doing it and then we've created another little helper function
that's gonna do a whole bunch of other metrics is gonna make some predictions using a model we pass
it to.
And then it's gonna compare those predictions on a whole bunch of different playing fields on the mean
absolute error on the root mean square log error and on the coefficient of determination.
So that's enough for this one video.
Building our own custom evaluation functions now again is something you might want to do if your own
projects because we are running as many experiments as possible.
It is helpful to have evaluation functions like this.
You can quickly do a whole bunch of different things rather than having to retype out this every time
we try a new model but lets our press shift an intermezzo our functions are instantiated and we're going
to try and test them out the next video.