1 00:00:00,268 --> 00:00:01,950 All right, it's time to talk about 2 00:00:01,950 --> 00:00:05,870 debugging Spark jobs using the Spark UI. 3 00:00:05,870 --> 00:00:07,470 In this lesson, we're going to jump in, 4 00:00:07,470 --> 00:00:09,580 and I'm going to talk to you just very briefly 5 00:00:09,580 --> 00:00:12,870 about how to track an application in the Spark UI, 6 00:00:12,870 --> 00:00:14,970 talk a little bit about what can be tracked, 7 00:00:14,970 --> 00:00:16,850 and we'll talk a little bit about stages 8 00:00:16,850 --> 00:00:19,290 and directed acyclic graphs. 9 00:00:19,290 --> 00:00:20,780 And I know we've touched on that a little, 10 00:00:20,780 --> 00:00:23,900 so this will be a very brief overview. 11 00:00:23,900 --> 00:00:27,210 Then we'll take a look at how to access that in the portal. 12 00:00:27,210 --> 00:00:28,290 So let's get started 13 00:00:28,290 --> 00:00:32,170 by talking about tracking applications in the Spark UI. 14 00:00:32,170 --> 00:00:34,550 First, we can track jobs. 15 00:00:34,550 --> 00:00:36,945 So we're going to pull detailed information 16 00:00:36,945 --> 00:00:40,590 on submitted jobs. We can also track executors. 17 00:00:40,590 --> 00:00:41,640 And when we do that, 18 00:00:41,640 --> 00:00:44,370 we're going to break those executors down by ID. 19 00:00:44,370 --> 00:00:46,170 And I'll actually show you the executor part here 20 00:00:46,170 --> 00:00:47,343 in just a second. 21 00:00:48,320 --> 00:00:50,020 So included in that executor, 22 00:00:50,020 --> 00:00:53,820 we can also track the task information of that executor. 23 00:00:53,820 --> 00:00:56,090 And then we can also take a look at the memory 24 00:00:56,090 --> 00:01:00,720 and the shuffle usage for each individual executor. 25 00:01:00,720 --> 00:01:03,960 Then you can also track stages in the Spark UI. 26 00:01:03,960 --> 00:01:06,720 So we can take a look at shuffle read/write. 27 00:01:06,720 --> 00:01:10,140 We can take a look at the duration and the input/output. 28 00:01:10,140 --> 00:01:13,670 And then we can also see a directed acyclic graph, 29 00:01:13,670 --> 00:01:16,700 or a DAG, visualization of each stage. 30 00:01:16,700 --> 00:01:18,840 And you can see that over there on the right. 31 00:01:18,840 --> 00:01:22,840 So you'll note here that we have our individual stage. 32 00:01:22,840 --> 00:01:24,500 This is stage 42. 33 00:01:24,500 --> 00:01:28,130 You can see our shuffle read for that individual stage. 34 00:01:28,130 --> 00:01:32,320 You can see our graph with each action broken down 35 00:01:32,320 --> 00:01:35,130 in that directed acyclic graph. 36 00:01:35,130 --> 00:01:36,560 So we get quite a bit of information 37 00:01:36,560 --> 00:01:39,180 that we can dive into for each stage of a job 38 00:01:39,180 --> 00:01:40,433 to see what's happening. 39 00:01:41,940 --> 00:01:44,110 All right, so with that, let's jump of into the portal 40 00:01:44,110 --> 00:01:48,140 and just take a super brief look at how we access Spark UI. 41 00:01:48,140 --> 00:01:51,630 So we find ourselves here in an HDInsight cluster. 42 00:01:51,630 --> 00:01:52,750 Over here on the right, 43 00:01:52,750 --> 00:01:55,900 we can just click on Spark History Server, 44 00:01:55,900 --> 00:01:58,730 and that's going to spin up my Spark History Server. 45 00:01:58,730 --> 00:02:02,530 We can also access it by going to my 46 00:02:02,530 --> 00:02:07,530 hdinsightname.azurehdinsight.net/sparkhistory. 47 00:02:07,582 --> 00:02:09,600 So we can do that as well. 48 00:02:09,600 --> 00:02:14,210 So once we get in there, we can choose our application ID, 49 00:02:14,210 --> 00:02:16,530 and then we can see an event timeline 50 00:02:16,530 --> 00:02:18,790 as to what's happened with Spark jobs 51 00:02:18,790 --> 00:02:20,540 that we've been running. 52 00:02:20,540 --> 00:02:23,830 So then we can also see our stages and our environment. 53 00:02:23,830 --> 00:02:25,530 And since we talked about stages already, 54 00:02:25,530 --> 00:02:28,270 let's jump in and talk about executors. 55 00:02:28,270 --> 00:02:30,740 So you can see here, under Executors, 56 00:02:30,740 --> 00:02:34,710 we have a summary of the executors that we have. 57 00:02:34,710 --> 00:02:37,160 And it shows you all the information about 58 00:02:37,160 --> 00:02:40,050 what's being used, cores that are active, 59 00:02:40,050 --> 00:02:42,300 failed tasks, complete tasks. 60 00:02:42,300 --> 00:02:45,110 So we can really get a good idea of what's going on, 61 00:02:45,110 --> 00:02:46,601 and then we can see full logs 62 00:02:46,601 --> 00:02:48,359 if we want to see that as well. 63 00:02:49,480 --> 00:02:51,960 And then finally, under SQL, we can go here 64 00:02:51,960 --> 00:02:55,090 and we can see a list of any completed queries that we have. 65 00:02:55,090 --> 00:02:56,360 And if I clicked on one, 66 00:02:56,360 --> 00:02:58,520 it's actually going to drill down and give me details 67 00:02:58,520 --> 00:03:00,630 about the query that was run. 68 00:03:00,630 --> 00:03:02,950 So quite a bit of information here. 69 00:03:02,950 --> 00:03:05,320 Again, you just access it through the Azure portal 70 00:03:05,320 --> 00:03:08,960 by going to Spark History or by going to this 71 00:03:08,960 --> 00:03:13,278 clustername.azurehdinsight.net/sparkhistory. 72 00:03:14,590 --> 00:03:16,500 All right, that's it for this lesson. 73 00:03:16,500 --> 00:03:18,370 A couple of key points to talk through. 74 00:03:18,370 --> 00:03:21,380 One, this is blended concepts. 75 00:03:21,380 --> 00:03:25,680 So, we use the Spark UI to understand shuffle optimization. 76 00:03:25,680 --> 00:03:27,040 Here's where the blending comes in. 77 00:03:27,040 --> 00:03:30,120 We've talked a lot about shuffle optimization, 78 00:03:30,120 --> 00:03:32,470 and we've also talked about Spark. 79 00:03:32,470 --> 00:03:34,600 We talked about what an executor was 80 00:03:34,600 --> 00:03:38,400 and we talked about what a directed acyclic graph is. 81 00:03:38,400 --> 00:03:40,400 So at this point in the course, 82 00:03:40,400 --> 00:03:43,650 you should understand a lot of those core concepts. 83 00:03:43,650 --> 00:03:46,570 And when I blended them together in this lesson, 84 00:03:46,570 --> 00:03:48,320 a lot of that should make sense 85 00:03:48,320 --> 00:03:50,603 just from the prior stuff that we've learned. 86 00:03:52,020 --> 00:03:56,020 Also, remember what you can access with the Spark UI. 87 00:03:56,020 --> 00:03:58,230 This is helpful for the DP-203. 88 00:03:58,230 --> 00:04:02,000 You can access jobs, executors, stages, and queries 89 00:04:02,000 --> 00:04:03,383 with that Spark UI. 90 00:04:04,700 --> 00:04:07,763 That's it for this lesson. I'll see you in the next.