All right so in this lesson what we're gonna do is we're gonna run this gradient descent without 1
SymPy. 2
We're going to look at making our Jupyter notebook a little bit more performant. 3
This gives us a chance to solve this problem the old fashioned way, the way we've done it for the previous 4
examples 5
and one reason to do this is that you'll notice this performance difference in your Jupyter notebook 6
because it turns out running it without SymPy is actually much faster. 7
But first, let's add the markdown for our partial derivatives in LaTeX notation. 8
So I'm going to go back to this section heading here and we're going to write down the partial derivatives 9
in mathematical notation because it will give us a good opportunity to see a few more of these LaTeX 10
tags in action. In our print statement, on the cell below, 11
we've got the partial derivative with respect to x and it looks like this. 12
So the question is how to translate this very little bit ugly and unreadable equation into LaTeX 13
notation? 14
The first thing I'm going to do is use the opening double dollar sign tags and I'm going to have a little 15
fraction here, 16
I'm going to say "\frac{f}{x}" 17
and then two more dollar signs. Let's see what this looks like. 18
So we can see that our fraction tag and the dollar signs together center our equation like this. 19
So let's take a look at this, to get this a bit bigger, 20
I'm going to add the two pound symbols before this thing and press Shift+Enter again so we can see it 21
a bit larger. 22
Now I'm going to add the mathematical notation for the partial derivatives, so I can go back into my 23
fraction and use the backslash again and before the f, I'm going to use a tag called "partial" and have a space 24
and you can see that the markdown actually highlights this tag differently, 25
It highlights it in green just like it does with the fraction tag. 26
So that way I know just from the syntax highlighting that this is a legitimate LaTeX tag. 27
I'm going to also add the partial tag to the bottom. 28
So I'm going to "\partial x" and then after the curly braces I'm going to put an equal sign 29
and I'm going to press Shift+Enter and see what it looks like. And here we go. Here we can see the partial 30
tag translates into these little delta symbols for the equation. 31
Okay so now we can have the top part of the fraction on the right hand side. 32
And that was "2x" had log, I'm going to use the natural log here, 33
So "\ln(3)*3^{-x 34
^2 - y^2}" 35
Let's see what this looks like. Now, 36
now one thing I don't like is this times symbol that we've got here. It would be much nicer to have a dot 37
and there is a LaTeX tag for that as well 38
and it's "\cdot". Pressing Shift+Enter, 39
we can see that it now looks like this which is a lot nicer. 40
So we've got our top part of a fraction now. 41
Now, all I have to do is actually, 42
well now all I have to do is actually put it in a fraction. 43
So I'm going to say "\frac", open curly braces and at the end closing curly braces. 44
And I'm going to open another pair of curly braces at the en. This is going to be for the bottom part of our 45
fraction. Now at the bottom we had 3 to the power of, 46
and then it was "{-x^2-y^2}", 47
+ 1. 48
Let's press Shift+Enter and see what this looks like. 49
So now our equation looks like that - we've got a fraction, 50
we've got the top part and we've got the bottom part. But this isn't quite correct, 51
we have to add a little bit more notation since the bottom is actually squared. 52
So how do we do that? 53
One of the things I can do is I can take a parentheses here and a parentheses at the end and then have 54
that whole thing to the power of two and pressing Shift+Enter, 55
it would then look like this. 56
I have the parentheses and then the two, 57
like so. 58
And you know the thing is this isn't even all that terrible, 59
but I can use LaTeX notation again to style these parentheses a little differently. 60
So instead of having just a normal parentheses here, I can actually have a backslash and say "\left", 61
so this is one of the tags, 62
and then at the end here instead of having the closing parentheses I can actually also put 63
"\right" and then have the closing parentheses like so. 64
So this code here, this markdown, now refers to the right or closing parenthesis. Then you press Shift+ 65
Enter to show what this looks like in comparison. 66
You can seen now with the "left" and "right" parentheses using the LaTeX tags, it actually looks 67
a lot better. 68
And just like that we're done. We have gut our partial derivative with respect to x formatted very 69
beautifully in LaTeX notation. Let's add the partial derivative with respect to y now, so this is the 70
really easy part because all we have to do is copy this, paste it and change this x to a y, and change 71
this x to a y, and press Shift+Enter. Now we've got both our partial derivatives displayed here. 72
Now if you're suspicious and you don't believe me that this is indeed the partial derivative for y, then 73
you can go down here and you can refresh this cell here, just by changing this a here where we're calling 74
the differentiation function from SymPy to b, hitting Shift+Enter and then you should be able to 75
confirm that this indeed is the partial derivative with respect to y. 76
So now that we've figured out the functional form for both of these partial derivatives, can you as a 77
challenge, can you write these partial derivative functions as Python functions? 78
I'll also give you a hint - when you're writing the Python functions, 79
there's actually one additional requirement that you have to consider. Also for the function names use 80
the names fpx and fpy for the names of the partial derivative functions. 81
I'll give you a few seconds to pause the video and give this a shot. 82
All right, you ready? 83
Here's the solution. For the solution, 84
I'm going to add a little Python comment, 85
Now let's just say "Partial Derivative Functions 86
example 4". 87
And here they are. 88
It's gonna be "def fpx", which needs two inputs, an x and a y, colon and to write the partial derivative 89
for this, I'm going to again use a little simplification and define a variable called r and that's gonna 90
be equal to part of my expression, 91
I'm going to have 3**(-x**2 - y**2), 92
so then my derivative is going to be 2*x*log(3*r). 93
This is the top part and the bottom part of that fraction is gonna be (r+1)**2 and that's it. 94
That's my partial derivative with respect to x and as we've already discussed this is very, very similar 95
to the partial derivative 96
with respect to y so I'm just going to copy this and change this to y and change this x in my return 97
statement to y as well. 98
Now I said there was one requirement that you have to take into account and this is the log function 99
here. 100
This comes from the math library, 101
the math module and we have to import this functionality in our Python notebook for it to work. 102
Otherwise, when it comes to running fpx or fpy to evaluate one of these functions you're actually gonna 103
get an error. 104
So if I had fpx(1.8, 1.0) and I hit Shift+Enter on this cell, I'm actually 105
going to get an error. 106
And the reason is is that log is not defined. 107
I can't just use the log functionality like I would up here without importing the module first. 108
So let me go back up to the very, very top and then down here, 109
I'm going to say "from math import log", 110
then I'm going to hit Shift+Enter on this and I can scroll back down again I can try rerunning this cell 111
here and I can see that my slope, my partial derivative with respect to x, is equal to 112
0.037 113
which is exactly what I am getting here. 114
In this case I'm using SymPy to evaluate my partial derivative 115
and in this case I've already got my partial derivative here as a function and I can just plug in the 116
values. 117
So one question you might ask is: well why did you use log, why is it not ln? And the answer is is that 118
if I check the documentation on this, then you see here, if I hit that little plus sign that if the base 119
is not specified, 120
so if there's no second argument being passed to this function it returns the natural logarithm, 121
so ln, base e, of x. 122
So in fact this log(3) is the natural logarithm or ln which is what we've got up here for our partial 123
derivative function in the LaTeX notation. 124
So now we've got that out of the way we can do a bit of a horse race between these two methodologies - 125
we can namely take this cell here and we're gonna copy it, we're going to copy this cell and what we're gonna 126
do is we're gonna paste that cell here. 127
I don't need this one, I'm going to delete that. 128
So I'm going to go to "Edit" > "Delete Cells" and then I'm going to modify this cell. In particular, I'm going to modify 129
these two parts 130
to use my fpx and my fpy functions instead. 131
You know what? 132
Maybe you should give this a quick shot, see if you can figure out what code should go here in order 133
to use fpx and fpy, the partial derivative functions instead of what we had before. 134
I'll let you pause the video and I'll show you how to do it in a second. All good? 135
Here's the solution. 136
You simply call the two functions and supply the x value and the y values. 137
So the x value was under "params 138
[0]", and the y value was under "params[1]". 139
And that's it. 140
That's the gradient or partial derivative with respect to x. And for our gradient in the y direction, 141
we do exactly the same thing with one difference - 142
we're gonna use fpy, 143
the partial derivative with respect to y 144
and we're gonna supply the same two inputs, the first value in our array and the second value in our 145
array. 146
Now I'm going to run the cell with 500 iterations and see how well it performs. 147
Ready, steady, go! And it's done. Going back up here to where we're using SymPy, I can change the 148
maximum number of iterations to 500 and rerun the cell. Thinking, thinking, thinking, thinking, thinking, 149
there we go. 150
So you can see that Python is actually doing a lot of extra work in this case. Every single time this 151
loop runs it has to differentiate our cost function and then evaluate that cost function. 152
And these instructions actually are a little bit more resource intensive than if it already knew from 153
the get go what the partial derivative was. 154
And this is one of the drawbacks of why you might not want to use SymPy in your loop if you're 155
running an optimization algorithm many, many, many times. 156
So I hope you enjoyed that. 157
So I really think this is an interesting exercise because it really shows some of the pros and cons 158
of the different tools that are at our disposal. 159
On one hand SymPy can make our life a lot easier when it comes to calculating derivatives and doing 160
symbolic mathematics and much, much more. 161
But on the other hand, it does show that you have to be clever when it comes to running your optimization 162
algorithm. 163
You do have to think about a little bit about the resources that you're using and running your optimization 164
and this kind of goes back to thinking like an engineer, 165
thinking about the resources that are at your disposal and how you can write your code and how you can 166
choose your algorithm to make the most use of the resources that you have. OK, 167
so what's left to do? 168
Well, only one thing right? 169
Plotting the gradient descent on our wonderful 3D chart and this is what we're going to do next lesson. 170
So I still got that coffee pumping through my veins so I hope I'll see you there.