All right. 1
So welcome back. 2
In this lesson, we're going to start taking a look at another quirk of the gradient descent algorithm. 3
We're going to be taking a look at an example where our gradient descent algorithm actually diverges 4
and spirals more and more out of control. 5
And also we're gonna be working through some more hardcore Python programming concepts. So I'm going to click 6
in my last cell and then I'm going to go change the cell from Code to Markdown here. I'm going to add a hashtag 7
and say 8
"Example 3 - 9
Divergence, Overflow and Python Tuples". The function that we're gonna be looking at in this example is 10
gonna be this one - it's gonna be h(x) = x^5 - 2x^4 +2 11
Forgot my closing tags in the end. 12
There we go. 13
Maybe add another cell below actually, one, two, three, so that we're not at the bottom of the screen. 14
And then I'm going to generate some data, as always that's gonna be my first thing. 15
So I'm going to make data and since this is example 3, I am when I use x_3 as my variable 16
and I'm going to use np.linspace starting at -2.5 going to 2.5 and with the num argument set to 17
1000. 18
Okay. 19
So now it's time to write that equation above in Python code. 20
It's gonna be "def f(x):", "return x**5 - 2*x**4+2" 21
And then our derivative of this function is gonna be "def dh(x):", "return" 22
and then you've probably already worked this out by applying the power rule. 23
It's gonna be 5*x**4 - 8*x**3 24
Okay. 25
So let's plot this function. 26
I'm gonna scroll up and I'm gonna take this cell here. 27
I'm going to say "Copy Cell" and then I'm going to paste it above. 28
Now I'm going to have to make some changes to this code. Over here 29
for the gradient descent, 30
we're gonna be calling dh. 31
And I'm going to say my initial guess should be equal to 0.2. 32
I'm going to leave a lot of the other stuff as it is but I'm going to change my x axis and y axis on my graph. 33
So this graph is gonna go from -1.2 to 2.5 and the y axis is gonna go from 34
-1 to 4. 35
So that's our cost function, I'm gonna change the y label to read h(x). 36
I'm going to plot x_3 on h(x_3) and on the scatter plot 37
I'm also going to be plotting h(x) not g(x). Similarly, for my derivative I'm going to change the label 38
and on the axes, we're gonna go from -1 to 2 and then from -4 to 5. 39
And then on the plotting, I'm going to change this to x_3, 40
dh(x_3) 41
and then I'm gonna hit Shift+Enter. Voila! So we can see here on our graph, we start at positive 42
0.2 and then our gradient descent slowly, slowly, slowly makes its way down into this local minimum 43
right here. 44
Let's modify the cell here to print out what the values actually are. 45
So I'm going to say "print('Local min occurs at ', local_min)" and then I'm going to print 46
"The cost at the minimum is", and then I'm going to say h(local_min), right. 47
So local_min remember is the last value calculated by our gradient descent. 48
We're going to work out what the y value is on our chart here at this particular point. 49
And finally, let's print out the number of steps that our algorithm has taken including the initial guess. 50
So that's gonna be the length - len of our variable called list_x. When I run this 51
now, I can see the output here at the bottom 52
below our charts. The local minimum occurs at around 1.6, the cost of this minimum is about 53
-0.62 and the number of steps is 117. 54
So our gradient descent algorithm has run over 100 times to get to this point. Okay, 55
so we're converging to this minimum here 56
and so far nothing new. 57
We've seen all of this before. But let's see what happens if instead of at 0.2 we start at 58
-0.2. 59
Let's see what happens if we start a little bit on the other side of this chart. 60
So I'm going to scroll back up and here where my initial guess is 0.2 I want to change this to 61
-0.2. 62
And I'm going to rerun the cell and see what happens. Scrolling down we see an error. 63
In fact we see an overflow error, where it says the result is too large. 64
What does that mean? 65
I'm going to change the initial guess back to 0.2, hit Shift+Enter, reload the graph so we can 66
think about this. If our initial guess was -0.2 67
then we would be on the left hand side of this hump and this means that the algorithm would be starting 68
to move down and down and down and down this line here. Now we continue going all the way down to 69
like negative infinity. 70
Now I know what you're thinking. 71
Negative infinity. 72
That makes h(x) a very a very unrealistic cost function but it does illustrate several things. 73
First, we can see how our gradient descent algorithm behaves when the algorithm diverges. 74
And second, it shows us how our Python program behaves when this happens. 75
So conceptually we've explained the problem, but I do think that seeing this overflow error gives us 76
an opportunity for understanding something a little deeper about our Python code. 77
So let's run our code in slow motion if you will and examine what's actually happening. And to do that, 78
I'm going to modify our gradient descent function. So I'm going to scroll back up where we've defined our gradient 79
descent and in our function header, I'm going to tack on another parameter. This parameter is gonna be 80
called max_iter for max iterations and give it a default value of say 300 and instead 81
of this hardcoded 500 value here in our range, I'm going to substitute our argument I'm going to substitute 82
in max_iter from max iterations. So this way we can specify the maximum number of times our loop 83
will run when we are calling our function. 84
So I'm going to press Shift+Enter to update the Python code now. 85
So I'm going to update the cell and then down here where we're generating this graph with our third example, 86
I'm going to change our function call as follows - 87
for our initial guess I am going to use -0.2, but then I'm going to give a max iteration 88
value of 10 and see where this leaves us. So I'm going to hit Shift+Enter, 89
take a look at the graphs. OK, 90
so in 10 iterations we're still pretty much on this hump. If instead the max iterations of our loop are 91
set to 40, then we can see we start moving a little further down. 92
And if I change it to say 60 I can see we're moving down even more. 93
Now, let me update this to 70 and rerun this function. When we examine the chart, 94
now, what do we notice? 95
Well, yes, it's moving down to the left but very interesting here is that the step size gets bigger and 96
bigger with each step as this slope starts getting steeper and steeper and steeper. 97
Our steps start getting larger and larger and larger. 98
So what's the last x value that the algorithm calculates? 99
We're printing out the last x value with this print statement. So we can see below our graphs are the 100
printouts from our three print statements. 101
We can see that the last x value that's printed out is negative 2 million. 102
That's the first print statement here. 103
And this is definitely not a local minimum in this case, but when we feed our negative 2 million back 104
into our function then we can see that the cost, 105
yeah, 106
what's on the y axis at this point 107
is equal to -3.8*10^31. 108
So in our print statement here, Python is giving us this number in scientific notation but it's actually 109
an enormous number. 110
It's around 3 8 0 0 0 0 0 0 0 0 0, 111
it actually continues going, right. 112
I'd have to copy this, paste it three times. 113
This is how large this number is 114
that's being printed here in scientific notation - It's 3.8 with a lot of zeros after it. 115
Yeah I mean you're going to be looking at this number and you're like "Well it's a computer, right? 116
So what, we sent a man to the moon over like 40 years ago - surely my computer can handle calculating large 117
numbers, right? 118
What's the big deal?" 119
And you're not wrong. 120
We can and we should be able to do math with very large numbers. 121
But the thing is your computer and Python doesn't do this straight out of the box. If you want to work 122
with numbers of this sort of magnitude, 123
if you're are, I don't know, calculating the number of atoms in the universe or what have you, 124
then you have to employ a couple of tricks. 125
The thing is, you can actually see what the maximum is that you can reach on your particular machine 126
at home right now in Python 127
straight out of the box and that's without importing any libraries or any modules and using Python as 128
it is that you've got installed right now. 129
So if you're curious and you wanted to pull up this sort of system specific information you can actually 130
do so with a module called "sys". 131
So "import sys". 132
This is the module where the system's specific information resides and there you can pull up a number 133
of different things. To see the kind of things that I'm talking about, 134
you can write something like "help", and then put sys in there and then you'll get some documentation on 135
the system module. So you can read this. 136
This is by the way very, very similar to what you can pull up by pressing Shift and then Tab and then 137
hitting that little plus sign. 138
You'll see that this also pulls up the same documentation. 139
But let me show you two things that might be quite useful. 140
I'm going to comment out the the help here. 141
Don't need this. 142
So for example one thing that you might be interested in looking up is what version of Anaconda you're 143
using or what version of Python. 144
And you can pull this up by writing "sys.version". 145
So version is an attribute of this system module. 146
So right now you can see that I'm using Python 3 and I've got a 46 bit system and you can also see that 147
I'm running this on a Mac. 148
Let me comment this out again and let's look at something else. 149
Let's pull up what the largest floating point number is that I can calculate in my Python program now. 150
Now you might ask: "Why am I interested in floating point numbers? Why do I say floating point numbers?" 151
Well if you have that type of the thing that we're looking up, right, the type of thing that we're calculating 152
h of, 153
take a look, 154
in this case it's h(local_min). 155
So this is the thing that gave us the problem. 156
So this is a float and this is what we're looking up. The largest float that we can use is "sys.float_ 157
info.max" and here's our answer. 158
Right. 159
And this number that you see printed out here is specific to my machine. 160
If you're using a different type of machine with a different architecture, say 32 bit, then you may see 161
something else printed below the cell right now, but this is my maximum floating point number that I 162
can use on on my architecture. 163
Yeah, it's 1.79 times 10^308. 164
Yeah. 165
This is huge, but it's still well shy of the 10^31 that we had just a moment ago. 166
Right. 167
It's many, many orders of magnitude larger. 168
So you might ask why are we running into this problem? 169
Well, looking at our chart we can see that our step size increases dramatically with each step. 170
And if I go up here and I change this from max iterations, the number of times I run my loop, from 70 171
to say 71 and I look at my cost, then I get -2.1* 172
10^121. 173
So this is the crux of the problem - I'm going to blow through my limit at the very next iteration, on 174
iteration 73. 175
This is when I get the overflow error. 176
Now on your machine at home, if the number that you're seeing printed here, the number that spat out by 177
sys.float_info.max is smaller than this then you might actually get that 178
overflow error much sooner than I do, right. 179
You might not get it at iteration 73, 180
you might actually be getting that error far earlier, that overflow error. 181
Now one thing that you might like to know about Python lingo is that errors like this are also referred 182
to as exceptions, but no matter if you call it an exception or an error, we still crash and burn. 183
So yeah, I hope you enjoyed that little detour into the the low level of representation of numbers 184
inside your Python computer program. But I think that while we're on the topic of Python programming, 185
we should revisit a piece of code that we've written in a previous lesson, namely the code up here, the 186
code for our gradient descent algorithm, because I have to confess something - I've been a little cheeky 187
in having our Python gradient descent function return multiple values without actually explaining how 188
this works. 189
And this is a good point to cover the Python code 190
before we go back to actually analyzing our algorithm. So let's add a new section heading at the bottom. 191
I'm going to click this little plus sign here to insert some cells below. 192
And this cell here I'm going to convert from Code to Markdown and the section heading I'm going to give 193
this is "Python tuples". 194
So, what's a tuple? A tuple is a data structure that's very, very similar to a list - a tuple is just a sequence 195
of values that are separated by a comma. 196
And this is what we've used in our gradient descent function. 197
Let me show you how you can create a tuple. I'm going to click 198
Plus again here and let's do this. Let's, 199
let's insert a quick comment here 200
"Creating a tuple". My first tuple is gonna be called "breakfast" and it's going to contain three values 201
bacon, eggs and avocado. 202
This, by the way, is a fantastic way to start your day. 203
It also illustrates the general format for tuples. 204
You have a sequence of values that are separated by a comma. I'm going to create another tuple here call it 205
unlucky_ 206
numbers. I'm going to give it 13, 4 for China, 9 for Japan, 26 for India and 17 for Italy. 207
So it's the same pattern as above. 208
And this way of creating tuples actually has a name. 209
This is called tuple packing, because we're packing multiple values into a single tuple. 210
So now that we've got our tuples, how do we access them? 211
Well, I'm going to add some print statements here like "I love", comma breakfast 212
[0]. 213
I'm going to hit Shift+Enter. My lack of spelling ability has foiled me once again, I'm going to take out the superfluous 214
e here and then hit Shift +Enter again and then we can see that the syntax here with the square brackets 215
for working with tuples is actually very, very similar to working with a list. 216
So you've got a tuple that has a name, 217
in this case breakfast and you're accessing the values inside the tuple through the index. 218
So zero is the first item in the tuple. And to show you a second example, 219
I'm going to print out the string 220
"My hotel has no ", and then I'm gonna have two plus signs, another string at the end "th floor". 221
Now in between here, I could put unlucky_numbers, and then square brackets and say provide 222
the index 1, 223
and if I try to run this right now, I'll get an error because the string concatenation with the pluses 224
does not convert the integers here to strings, so I have to actually wrap this in a function called str, 225
and only now can I press Shift+Enter and run this. If I try to do this without wrapping it then we'll 226
get an error like this - must be string not int. 227
And that's because my tuple here contains ints and those are not converted to strings by the plus operator. 228
So I'm going to wrap this in a string function and press Enter. 229
So we've covered how to how to access a value in a tuple. Brilliant! And how we can try something else, 230
because I'm sure you're looking at this and you're saying "Well how are tuples different from lists? 231
How how are tuples used? 232
Why do we have something that's so similar and yet different?" 233
Well, in contrast to lists, tuples are often used when the data they contain is heterogeneous. 234
Now, what do I mean by that? Tuples often contain a mix of data in contrast to lists. 235
So lists often contain the same kind of data, like all strings, all integers, but tuple like say, "not_ 236
my_address" equals 1, comma and then the string "Infinite Loop", and then a comma and then 237
another string "Cupertino", 238
and then comma, "95014" for our postcode. 239
And we've just created a tuple with a mix of data, a mix of different data types if you will and this 240
is something that you don't usually see in practice with lists. 241
Lists are usually homogeneous, meaning people don't tend to mix and match the different types of data. 242
Now, another difference with lists is that tuples are immutable. 243
What does that mean? 244
It means that we can't change the tuple after we've made it. 245
So for example, if I had, say breakfast, and I wanted to change bacon which is at index 0, and set that equal 246
to a, say, sausage and just you know innocently swap out the value then Python will actually yell at us, 247
it's gonna give us a type error it's gonna say the "tuple object does not support item assignment" and 248
this basically means that once we've created a tuple like this, we cannot change the values here and 249
we also can't append a new value say we can't stick this at say index 3 right. 250
We get the same error in other words the immutability of tuples means that once you've created a tuple 251
you can't mess around with it. 252
You can't change it up. 253
This is quite different from a list right. 254
Because if you remember in our gradient descent function we were running our loop and we were appending 255
items to our lists. Every time the loop ran, our list grew in length because we're appending new items 256
and this is something we couldn't do with tuples. Now one more thing I want to show you on the topic 257
of tuples is a little gotcha. 258
Say we want to create a tuple with a single value. 259
So just one value inside our tuple. 260
I know it's strange but, for the sake of argument, have a think about how you would create a tuple with just 261
one value. 262
What would the Python syntax look like to store a single value inside this tuple? 263
Here's the solution. 264
So if I put a single value in here, say 42, then I would have to put a trailing comma after it. 265
Now I've got a tuple with a single value, so if I print it out, print(tuple_with_single_value), 266
then I can see it looks like this. 267
It's got a single value and a comma. 268
And if I substitute the print for type and check, then I can see that indeed tuple with single value 269
is indeed a tuple. 270
Now the very first time I saw this I found this syntax like super weird and confusing - trailing comma 271
right? 272
My goodness. 273
So here it is. Now, 274
you too have shared this experience and the weird syntax. 275
You're welcome. 276
Now it's time to come full circle. 277
We've packed a bunch of values into a tuple, but we can also do the very opposite. 278
So we can unpack these values as well. 279
So if I take my tuple - breakfast, and I want to grab the values that are stored inside this tuple and 280
put them into some separate variables, I can do that by writing, say, "main, side, greens" is equal 281
to breakfast. 282
And this is called sequence unpacking. 283
Yeah. 284
So if I print out "Main course is ", and then comma, "main", then I get "Main course is bacon". 285
So here's the reason I mentioned this and why I say we've come full circle. 286
If I scroll back up to where we had our gradient descent function, we can see here in our return statement 287
what we're doing is we're returning three separate values, but in fact we're packing all these values 288
into a single tuple. 289
Yeah. 290
And when we're calling our gradient descent function, say here, then we are unpacking this sequence and 291
storing the results in three separate variables - local_min, 292
list_x, deriv_list. 293
So we've actually been using tuples, but we've never had to access any of the values from the tuple by 294
index. 295
But we can do that. 296
Let me show you how it would work. 297
So if I created a variable called data_tuple and set that equal to gradient_descent 298
and for my derivative function I supply dh and for my initial guess I supply 299
0.2, then I can print out the local_min at data_tuple[0] because that's 300
the very, very first thing that is stored inside our tuple. I can print out the cost at the last x value 301
which would be equal to, 302
well in this case it would be h(data_tuple[0]) and then I could also 303
print out the number of steps, which in this case would be the length of, so it would be "data_ 304
tuple[1]" and we run this. Then you can see that it works exactly the same way 305
as before, but instead of unpacking the sequence we're using the tuple now explicitly. 306
Okay, so we've paused a little bit on analysing our algorithm and we've talked a little more about Python 307
and Python programming, so now it's time to change tracks and go back to our gradient descent algorithm. 308
Now a reasonable question to ask is why did I show you this h(x) example function? I mean this function, 309
by my own admission, seems a little bit contrived and I already confessed that a non convex cost function 310
is not very realistic, but truth be told, is that we can get this divergence and the very, very same overflow 311
error in another way too and we can see this divergence and see the same error even if we are working 312
with a very nice clean cost function where we know for a fact that we should be able to reach a minimum. 313
And this is what we're gonna be examining in the next lesson. In the next lesson we're gonna be looking 314
at the elephant in the room - the gradient descent learning rate. I'll see you there. 315
Take care.