All right, it's time to talk about
debugging Spark jobs using the Spark UI.
In this lesson, we're going to jump in,
and I'm going to talk to you just very briefly
about how to track an application in the Spark UI,
talk a little bit about what can be tracked,
and we'll talk a little bit about stages
and directed acyclic graphs.
And I know we've touched on that a little,
so this will be a very brief overview.
Then we'll take a look at how to access that in the portal.
So let's get started
by talking about tracking applications in the Spark UI.
First, we can track jobs.
So we're going to pull detailed information
on submitted jobs. We can also track executors.
And when we do that,
we're going to break those executors down by ID.
And I'll actually show you the executor part here
in just a second.
So included in that executor,
we can also track the task information of that executor.
And then we can also take a look at the memory
and the shuffle usage for each individual executor.
Then you can also track stages in the Spark UI.
So we can take a look at shuffle read/write.
We can take a look at the duration and the input/output.
And then we can also see a directed acyclic graph,
or a DAG, visualization of each stage.
And you can see that over there on the right.
So you'll note here that we have our individual stage.
This is stage 42.
You can see our shuffle read for that individual stage.
You can see our graph with each action broken down
in that directed acyclic graph.
So we get quite a bit of information
that we can dive into for each stage of a job
to see what's happening.
All right, so with that, let's jump of into the portal
and just take a super brief look at how we access Spark UI.
So we find ourselves here in an HDInsight cluster.
Over here on the right,
we can just click on Spark History Server,
and that's going to spin up my Spark History Server.
We can also access it by going to my
hdinsightname.azurehdinsight.net/sparkhistory.
So we can do that as well.
So once we get in there, we can choose our application ID,
and then we can see an event timeline
as to what's happened with Spark jobs
that we've been running.
So then we can also see our stages and our environment.
And since we talked about stages already,
let's jump in and talk about executors.
So you can see here, under Executors,
we have a summary of the executors that we have.
And it shows you all the information about
what's being used, cores that are active,
failed tasks, complete tasks.
So we can really get a good idea of what's going on,
and then we can see full logs
if we want to see that as well.
And then finally, under SQL, we can go here
and we can see a list of any completed queries that we have.
And if I clicked on one,
it's actually going to drill down and give me details
about the query that was run.
So quite a bit of information here.
Again, you just access it through the Azure portal
by going to Spark History or by going to this
clustername.azurehdinsight.net/sparkhistory.
All right, that's it for this lesson.
A couple of key points to talk through.
One, this is blended concepts.
So, we use the Spark UI to understand shuffle optimization.
Here's where the blending comes in.
We've talked a lot about shuffle optimization,
and we've also talked about Spark.
We talked about what an executor was
and we talked about what a directed acyclic graph is.
So at this point in the course,
you should understand a lot of those core concepts.
And when I blended them together in this lesson,
a lot of that should make sense
just from the prior stuff that we've learned.
Also, remember what you can access with the Spark UI.
This is helpful for the DP-203.
You can access jobs, executors, stages, and queries
with that Spark UI.
That's it for this lesson. I'll see you in the next.