In this lesson we're going to play around with our gradient descent algorithm a little bit and we're 1
gonna see how it is affected by our initial guess, by our starting value. Also so we're gonna be building 2
on our Python programming skills by covering some of the more advanced features of functions in Python. 3
The first thing we're gonna do is we're going to generate a new cost function and kind of go into a 4
second example. 5
So I'm going to change the cell at the bottom here again to Markdown so that we have a nice clean section 6
heading and we're going to call it "Example 2 7
- Multiple Minima vs. Initial Guess & Advanced Functions". 8
The cost function that we're gonna be working with is going to look like this. 9
It's gonna be two dollar signs g(x) = x^4 - 4x^2 + 10
5, 11
And then two dollar signs at the end. 12
So again we're gonna be using LaTeX markdown to write our function in mathematical notation. 13
So as we've talked about before remember that LaTeX uses tags to mark text for special formatting and 14
there's two tags here. 15
Right. 16
There's an opening tag of two dollar signs and there is a closing tag of two dollar signs. 17
Now let's get stuck into the Python code. 18
So the first thing we're gonna do is we're going to make some data and we're going to to create a variable called 19
x_2 since this is gonna be our second example. 20
And again we're gonna be using numpy's linspace to generate our values. 21
So I'm gonna have value starting at -2, going to 2 and I'm going to space them out with about 1000 22
values. 23
Now, as a challenge can you write the g(x) function and the derivative - the dg(x) function in Python? 24
So remember you gonna be writing two functions with the 25
def keyword and applying that power rule that we covered in the previous lesson. 26
I'll give you a few seconds to pause the video and write these two functions 27
Ready? Here's the solution. 28
It s "def" key word, "g(x)", colon, "return x**4 - 4*x**2+ 29
+5" 30
So that's our first function, but the derivative of this function applying the power rule is going to be 31
be "def dg(x):" and then it's "return 4*x**3 - 8*x" - so four 32
gets multiplied by two, becomes eight and the constant drops out. 33
And that's it. 34
I'm going to hit Shift+Enter and if you get this error that I've gotten you've got to scroll all the way up to 35
where you're importing numpy and just hit Shift+Enter on the cell again because np was not recognized 36
by Python because I haven't come back to this notebook in a while. Now I can click back into the cell 37
and hit Shift+Enter and it will run just fine. 38
Now it's time to plot the cost function for this example. 39
I'm going to scroll back up for a second here and I'm going to grab this cell here and I'm going to 40
copy it because we're gonna be reusing some of this code. So I'm going to copy this cell and I'm going to come here 41
and go to "Edit" > "Paste Cell Above", but I'm going to have to make a couple of changes here. 42
So first off I'm going to change the x axis to go from -2 to 2 and the y axis to go from 43
0.5 to 5.5 and then I'm going to change the label here on the y axis as well to g(x) 44
and, of course, on my plot I'm going to take x_2 and g(x_2) and 45
similarly for my derivative the y label is gonna be dg(x) and for my x axis I'm going to go from 46
-2 to 2 as well and for my y axis I'm going to go from -6 to 8. And when it comes to plotting, 47
I'm going to plot x_2 and dg(x_2). I'm going to hit Shift+Enter, 48
see what I get. Voila! So these two plots 49
help us visualize our second example's cost function. 50
So what can we observe here? 51
What can we see? 52
Well there's a couple of things that are of note right. 53
So if we look at the chart on the left we can see that there are two minima. 54
There are two places where the cost is very low. 55
One here and the other one here. Also looking at the right hand chart we see that the derivative intersects 56
the x axis on three points, right? 57
One here one here at x equals zero and one here, so there are three points when the slope is equal to 58
zero. 59
And those three points correspond to this minimum, this minimum, but also this maximum here. 60
Now it's gonna be very interesting how this affects our gradient descent algorithm but before we start 61
playing around with the starting values, we're going to make some modifications to our Python code because 62
this is a really, really good time to talk about some of the advanced features of functions in Python 63
programming. 64
Now, we've written our code already for the gradient descent algorithm but we're gonna do is we're going to 65
scroll up 66
and we're gonna take this cell and copy it. 67
And then we're gonna insert that cell above. Now we're in a good position to start modifying this little 68
bit of code. 69
I really can't wait to show you some more advanced Python programming features when it comes to functions, 70
because Python functions are actually incredibly powerful and versatile things. 71
And in this lesson we're going to cover how to pass a function as an argument, how to make an argument 72
optional by specifying a default value and also how to have a function return multiple values. 73
I'm going to show you all these things by turning our gradient descent algorithm into our very own function. 74
In fact let's add some markdown to show this in our Python notebook. 75
So I'm going to to move this cell, this one here at the bottom, I'm going to press that up arrow here, I'm going to move 76
this cell up and then I'm going to convert this cell to markdown and then I want to use two hashtags 77
and just put down "Gradient Descent as a Python Function". 78
There we go. 79
OK let's get started. 80
The first thing we're gonna do is write our function header as always. 81
So it's gonna be "def", and then we're going to give our function a name, let's call it "gradient_descent". 82
Yeah. 83
Followed by two parentheses and a colon. 84
Now our gradient descent function is gonna be taking four arguments - the derivative function itself, a 85
value for an initial guess, the learning rate and the precision. 86
So let's put these in as parameters between these two parentheses. 87
So the first we said was the derivative function, I'm going to call this "derivative_func", then 88
the initial guess, 89
comma, then a multiplier or learning rate and then the precision. Now if you're looking at this then you're 90
going to be maybe wondering about my intentions here. 91
What do I mean by putting this derivative function as a parameter here? 92
See, the thing about Python is that a function is actually a full blown object. 93
Yeah. 94
So functions are stored in a piece of the computer's memory all on their own. 95
Just like other objects are. 96
And this means that you can stick a function in a variable and pass functions around our program. 97
So in our gradient descent function this derivative_func parameter will be our place holder 98
for the actual derivative function and you'll see this when we call this function. 99
So now let's fill in our function body. In order to make all of these lines part of our function body, 100
we have to indent them because the indentation is how Python knows that these lines should be part of 101
our function. 102
Now to indent a whole group of lines what you can do is you can select them and then use a keyboard 103
shortcut. 104
So on Windows you can press Control and then and then this key here to indent the whole group. 105
And then on Mac instead of Control you simply press Command and the square bracket key. 106
Let me show you. So I'm going to select all of these up to the break statement here and then I'm going to a press 107
control and then the square bracket key and they all move over by 1. 108
So now they're all part of our function body. 109
That's pretty neat. 110
Right? Now I'm going to modify these lines, so our new_x will be equal to to the initial guess that we're 111
making here. 112
Yeah. 113
So the initial guess is our place holder. 114
And when our function gets called we're gonna supply an initial guess and we're going to set new_x equal 115
to that initial guess. 116
I don't need this line anymore. 117
And I also don't need this line anymore, and I don't need this line anymore as well because these variables 118
will get their value when the function is being called. And I'm going to keep these around but I'm going to 119
have to make a modification here - our derivative function here isn't gonna be called df anymore, right? 120
It's gonna have the name of our place holder. 121
Right. 122
It's gonna be called "derivative_func" and the same, 123
yeah, is gonna be the case down here. 124
This is the other occurrence where we're referring to our derivative function. 125
So this is gonna be also called derivative_func. I'm going to delete this comment here, 126
don't need this anymore. 127
And then when it comes to graphing there's another reference here to our previous example. 128
But now I'm going to replace this again with derivative_func. 129
I'm also going to take away this print statement. 130
Don't need this anymore as well. 131
And then I'm also gonna delete these print statements here. 132
We don't need these anymore as well. 133
But there's one additional thing that I do want to add. 134
I'm just gonna delete a couple of these lines here to tidy it up a little bit better so we can actually 135
tell what's going on and now I can add that last thing to our gradient descent function that I wanted 136
to add. 137
I'm going to add my return statement. 138
So this is the keyword return followed by whatever the function spits out. 139
Now, what do we want this function to return? 140
What are the important things that we want out of our gradient descent function? 141
We want three separate things, three separate values that we're interested in. 142
So we want the new_x value, we want the list of x values and we want our list of slopes that we calculated 143
because our list of x values and our list of slope values is what we're going to be using for graphing, 144
and obviously our minimum is going to be that new x value that we spit out. 145
So one of the easiest ways of having a function return more than one value is simply by having our return 146
values separated by a comma. 147
So new_x, x_list, slope_list will return three values. So let's press Shift+Enter 148
now and see if we get any errors. 149
Okay, so far so good. 150
Now it's time to call this function. 151
This is where the rubber meets the road as they say right. 152
Now, since we have multiple return values we can store these return values in three separate variables. 153
So I'm going to call these variables "local_min, list_x, deriv_ 154
list". 155
Yeah. 156
So my function is gonna be returning three values and these are gonna be stored in this order in three 157
variables. 158
Okay, so time to call this function gradient_descent, 159
open parentheses, 160
now it's time to supply those four arguments. 161
So the question is what are those gonna be? 162
So let's supply these arguments by their position. 163
The first thing that will pass into our function is going to be, well, another function, it's gonna be 164
another function, we're going to pass in our derivative function which is kind of crazy, right? 165
Like we're passing in a function to another function like it's no big deal. 166
So I already mentioned that functions are just objects in Python, just like pretty much anything else. 167
So our derivative function is an object and it's got the name dg. That's it. 168
So if we want to get technical what's in fact happening here is that we're giving our gradient descent 169
function a pointer to our dg function that we've defined in the cell above. 170
Now if you've got a programming background then you might be interested to know that what we're doing 171
here is we're not copying our dg object, we're simply pointing to our dg object. Now let's supply other 172
three variables, let's have our starting position be 0.5, let's have our learning rate be 173
0.02 and let's have our precision to be 0.001 and let's add some print 174
statements for good measure. 175
Yeah, "print('Local min occurs at: ', local_min)". Let's print out that first value and let's also 176
print out the number of steps, so the number of steps is gonna be, 177
well it's gonna be the length of our list, so I can have "len", the length function of our list, 178
list_x, so this will include the initial guess plus the number of times that the loop ran - that's 179
gonna be the number of values that are gonna be stored in this list right here. Now as a challenge, 180
can you figure out what the problem is if I tried to run this as it is? And also what I would have to 181
fix in order for our function to run properly? Because there's something that I've missed in our gradient 182
descent function that I have not taken into account yet. And here's the solution - so this variable here 183
step_multiplier exists locally within our function, so it only exists within the function itself but 184
the problem is is that it hasn't been defined anywhere. 185
This means that Python actually does not know what this name refers to - in short, we've got to be consistent 186
with our naming here. 187
So multiplier is the name of the parameter which we actually want to use, we cannot use the name 188
step_multiplier which we've defined earlier and that's the fix. 189
So let me press Shift+Enter to rerun the cell. 190
Now I can press Shift+Enter again to run the cell below, and here's our answer - our local minimum occurs 191
at 1.4 and the number of steps it took to get there was 23. 192
So our function is working. Now looking at how we're calling this function here, 193
gradient_descent(dg, 0.5, 0.02, 0.001) is not 194
very readable. 195
Yeah this is something I really, really dislike when writing code because these numbers just appear like 196
magic numbers. 197
Looking at this we don't know what they mean. 198
Right. 199
We'd have to actually know what the function definition is. 200
It's much nicer to add keywords to these arguments and we can do that by modifying our function call. 201
So I'm going to take this code here, 202
Copy it, paste it down here for reference and then I can fill in the names of these arguments. 203
So this was our derivative_func, it's going to be equal to dg; our initial_guess is gonna 204
be equal to 205
I'm going to say, uh, 0.5 and then I can, uh, I can actually hit enter here and go to a new 206
line, 207
this won't affect the function call at all, but at least that's gonna make our code a lot more readable. 208
So this was our multiplier, 209
and this was our precision. let's see what happens when our initial guess has a starting value of 210
-0.5. 211
Let's run it now. So we can already see a difference here. 212
So the first time the local minimum occurs at 1.4 and the second time round the local minimum 213
occurs at -1.4. 214
But, uh, before we investigate this let's talk a little bit more about arguments. 215
We already know that arguments are how a function gets its inputs. 216
Arguments are how objects are sent to a function as an input. 217
And I promise to show you how we can give our arguments a default value and also make some of these 218
arguments optional. And to do that we're gonna have to modify that header in our gradient descent function, 219
because this is where we can specify our default values. So let's specify a default value for this multiplier 220
here. 221
We can do that by setting it equal to 0.02, and to specify a default value for the precision, 222
we also can just simply set it equal to 0.001. 223
And now our gradient descent function has two required arguments - 224
the derivative function and the initial guess, and two optional arguments - they're optional because they 225
have default values. 226
So let's call this function again, this time we're only gonna specify the required arguments. 227
Now copy this code here, paste it here and I'm going to delete 228
this part of my function call, so I'm only going to specify the derivative function and the initial guess. 229
Now I'm going to change this guess to -0.1 and let's see where we end up. 230
If you're getting this error. 231
Yeah. 232
If you're getting "gradient_descent() is missing 2 required positional arguments" 233
despite having added this code, it's because you haven't rerun the cell. 234
So remember to press Shift+Enter on the cell to rerun this code and make sure that our Jupyter notebook 235
is updated and then come down here and run this one and you should see that we end up at the same minimum 236
as before but this time it takes us 34 steps instead of 23. 237
Now one thing that you might try is you might want to rerun this cell here. 238
Question is: will this still work? 239
And the answer is: yes, it will. 240
Even though we don't have to specify a value for the multiplier and the precision, we can do. 241
So I can add an extra zero here to the precision and make our step size even smaller by changing it 242
from .02 to .01 and I can overwrite the default values that this function usually 243
has and we can see that having made the step size smaller and our cut off point even more precise, now 244
the number of steps increases to 56. 245
So this is really, really cool right. 246
We've got really powerful capabilities with functions. 247
We've got a very descriptive way we can call them. 248
We've got multiple outputs that we can store in variables just separated by commas and we can have optional 249
arguments, so we can give default values to some of our parameters in the function when we create it. 250
But all this stuff with arguments and optional arguments it's kind of hidden from view. 251
Right. 252
I mean how would you know what the arguments are for a function that you've never seen before? 253
Wouldn't it be nice to be able to pull up this information really quickly and really easily inside Jupyter 254
notebook without having to guess? 255
Well, I got you covered. 256
Let me show you a neat little trick in Jupyter notebook. 257
So if I put my cursor over my function here gradient_descent and then I hit Shift and Tab on my keyboard, 258
so I'm holding down the Shift key and I'm pressing Tab, then Jupyter notebook will pull up a little bit 259
of documentation, a little bit of information on this function, so I can see here, 260
I need four arguments, I can see what the arguments are called - derivative_func, initial_guess, multiplier, 261
precision; and I can even see what the default values are for my other two arguments. 262
Isn't this really, really cool? Jupyter notebook is actually smart enough to give us some information 263
on our function right there and then. 264
And it doesn't just work with our own function. 265
So, scroll up and pick out matplotlib scatter function in this notebook and give this a try, Shift 266
and then Tab on your keyboard. 267
Go on. 268
I'm going to wait for you right here. 269
Did you try it in a couple of places? You might have noticed something. 270
Sometimes it's really, really informative. 271
And other times it's not. 272
So, let me show you what I mean. 273
So I've got my Python code here from our previous example. 274
If I go to scatter and press Shift+Tab, then I get a wonderful description here with very, very detailed 275
information on the Signature. 276
I can click this little plus sign to even take a look at all this information here and all this documentation 277
for our scatter function. 278
And similarly, when I go up here to plt.figure, and I press Shift+Tab again I get a wonderful description 279
here, a wonderful signature, 280
these are all the things in the header and click the little plus sign and I get really, really descriptive 281
information, right? "Facecolor: the background color. If not provided, 282
defaults to our rc figure.facecolor." 283
Fair enough, right? 284
Cool. 285
Let's go to this plot functionality here. 286
Right. 287
If I press Shift+Tab on this all I get is "plt. 288
plot", and then args and kwargs, so this isn't really informative and I do have to kind of go to the 289
documentation here and really, really figure out what it is. But it's not all that readable, right? 290
So, I mean you can be scrolling around here for a long time trying to make sense of all this. 291
So at this point you're probably much better off going to the website of the official documentation 292
where you can actually read up on this in a much nicer format and you can search on this page, etc.. 293
In other words - if you want to know how something like plant or a subplot works, you're probably still 294
better off in pulling up the documentation for these things in your browser. 295
But speaking of plots, it's time to chart our gradient descent. 296
So let me copy the cell that generates these charts here. 297
"Edit" > "Copy Cells", scroll all the way down and then paste the cells above. 298
This is our chance to play around with the starting values and our algorithm. 299
So I'm going to add a comment here: 300
"Calling gradient descent function"; and I'm going to edit this comment here - 301
"Plotting function and derivative and scatter plot 302
side by side"; then I'm going to take this one here and copy it and I'm going to paste it here. 303
I am going to change our initial guess from -0.1 to 0.1. 304
Now, 305
let's add a couple of lines of code to have that scatter plot on here as well. 306
This is gonna be "plt. 307
scatter" 308
and here we had our list of x values and then our cost function 309
"g(list_x)" but we can't leave it like that because we're doing some calculations, 310
remember? And the power function doesn't play nice with a list type. 311
So I'm going to convert it to np.array and then I'm going to have it inside our g function. 312
So this is going to be the input - I'm nesting two functions here, 313
instead of splitting it up. And then we're going to add a color, I'm going to say color is red, the size of the dots will 314
be 100 as before, and we gonna give it some transparency as well, 315
alpha = 0.6, and then a closing parentheses for the scatter plot. 316
Now, transparency looks pretty good on the other one as well. 317
So, I'm going to add alpha=0.8 on our cost function chart as well. 318
And let's do something similar for our derivative chart below. 319
So it's also gonna be a scatter plot, "plt.scatter" and it's gonna be a list of x values again 320
for the x axis, 321
and then what was previously our slope list, we've stored in a variable called the deriv_list. 322
We're gonna go again with the red color for the chart. 323
We're gonna go with size 100 and alpha 0.5 324
and this sky blue line is also gonna get some transparency with alpha equals 0.6. 325
Now let's run the cell. 326
See what happens. And voila! 327
This is what we get. 328
We get our initial starting value up here at 0.1 and we're descending to the minimum here. 329
And on our derivative it looks like this. 330
We're gonna go down down down down down down down until our slope is equal to zero. 331
Okay so let's try out a couple of different starting values for our gradient descent function in this 332
example. 333
So here we've started at 0.1 and we've converged to this minimum here, 334
this right hand minimum here. 335
Let's see what happens when we start out with the value 2. 336
So I'm going to say my initial guess up here is gonna be the value 2 and I'm going to hit Shift+Enter. 337
In this case, our gradient descent algorithm goes down here and we end up at the very, very same minimum. 338
And if we start out somewhere else, say like -1.8, 339
then we end up at this minimum 340
instead. We end up in the left hand minimum and we'll end up in this left hand minimum as well 341
if we start out at -0.1. So if we start at -0.1, we end up here as well, 342
we end up in the left hand minimum. 343
So what we can learn from this, what we can see from this example is that our algorithm isn't perfect, 344
it's got some weaknesses because conceptually it's a little bit disturbing, 345
right? 346
That we end up at completely different minima when we start out at 0.1 and 347
-0.1. 348
So if we're unlucky in our choice of initial starting position, we can actually end up in very, very different 349
places. 350
So we can see in this example that the path of the descent can be very, very much influenced by that 351
initial guess in certain situations. 352
Now would you like to venture a guess what happens when we have an initial starting value of 0? So have a think about, 353
what would happen with our gradient descent if we feed in the value zero into our initial 354
guess before you run the algorithm. Let's try it out. 355
So I'm gonna say instead of 0.1, I'm going to start at 0 and press Shift+Enter. What ends up 356
happening is that we don't descend to any of the two minima here. 357
Instead we end up sitting right here on the maximum. 358
And that's because the slope at this very, very point is also equal to 0. 359
And we can see this here on the right hand function here, on the slope of the cost function. 360
Remember, a gradient descent algorithm stops running once the slope is equal to 0. 361
So this problem is also related to this sensitivity to the starting position. 362
Right? 363
The sensitivity of the path of the gradient descent algorithm to that initial guess. 364
Now, in this case, both of the two minima have the same cost. 365
Right? The cost is equal at both of these two minima but we can also imagine a very different situation. 366
If our cost function looked something like this, then we would have two minima, one of which has a much lower 367
cost than the other one. 368
One of these is a global minimum. 369
And the other one is a local minimum. 370
So the local minimum actually has a higher cost than the global minimum, 371
but our gradient descent would not discover the global minimum if the initial starting point was on 372
the right hand side of that local maximum, of that small hump right there in the middle of the chart. 373
Okay. 374
So we've outlined the problem. 375
What's the solution? 376
Right? How do we get around this weakness? 377
Well, the easiest thing to do would be to simply try out multiple different starting values and see if 378
they end up in the same place. 379
Think of this as like injecting a little bit of randomness into our gradient descent, so we could do 380
this by choosing a whole host of like random starting values and then running our gradient descent over 381
and over again to see where it ends up. 382
And this might be an approach you could take if you actually don't know what the cost function looks 383
like, right? 384
If you don't know where the minimum is, 385
this might be something that you want to try. 386
Another thing we could do is to try a completely different algorithm to find the minimum. 387
Right? 388
Because, let's face it, this particular version of gradient descent isn't our only option. 389
Similar to how we can try a bunch of different random starting points, other algorithms have randomness 390
baked into them. 391
One of the versions of gradient descent that has more randomness baked in is called Stochastic Gradient 392
Descent. 393
And this is in contrast what we're currently doing which is called Batch Gradient Descent. But the thing 394
to note about any of these approaches is that none of them are perfect, right. 395
You'll find that no matter which approach you choose, it has certain strengths and it has certain weaknesses 396
and it's important to understand what the pros and cons are of of each approach. 397
So on that note, we're not done yet examining this particular algorithm. 398
Our Batch Gradient Descent algorithm might actually face another problem and this is what we're gonna 399
be looking at next. I'll see you in the next lesson. 400
Take care.