In this lesson, we are going to talk about
how to troubleshoot a failed Spark job.
And we're going to start with some basic
troubleshooting logic.
We're going to learn about thinking it through.
Then we're going to jump back into Ambari UI,
look at configuration settings,
look at cluster health, stack and version.
Follow that up by talking about log files,
configure settings, and then finally,
how to reproduce an error.
So let's get started by gathering data about the issue.
At this point, you need to employ
basic troubleshooting skills.
Now, I know what you're thinking,
"HDInsight is not a basic service".
However, we still need to think
about those basic troubleshooting skills.
You know, taking a look at: is it plugged in?
Is it turned on?
Now I know this is the cloud and yes,
it's plugged in and turned on.
However, think about those basic troubleshooting pieces
and see if there's something super simple
before you jump into much more complicated fixes,
because oftentimes, it's something very simple.
"Oh, I forgot the comma.
"That's 8 hours I'll never get back out of my life."
We've all been there, if you spend any time in coding.
Just think it through and start with the basics.
And then we want to ask ourselves some basic questions.
When did the problem start?
Is this a repetitive issue?
Not only is it repetitive, can I force the issue to repeat?
And what was I expecting to happen?
By understanding some of these basic questions,
we'll have a much better idea
as we jump into some more complicated fixes.
So with this, let's actually stop now
and jump into Ambari and talk about configuration settings,
cluster health, and stack and version.
So, here we find ourselves in Ambari,
and we always want to start by looking at our dashboard
and seeing if everything looks like it should.
Is it green?
Do we have red, angry faces?
Is there something wrong with disk usage, or CPU,
or nodes, or something else?
If we've gone through these basic settings,
the next thing that we want to do
is jump into the service that we're trying to configure.
And we want to make sure that the components are set up
like they're supposed to be.
And we want to take a look at our config files as well.
Now it may be that we don't need to do any configuration.
However, for your individual service,
you might want to jump in and look.
And I'm not really going to spend any time here,
because as you can see,
there are hundreds of different metrics
that we can edit and play with.
So that's not really going to be helpful for the DP-203,
other than understanding
that you do need to look at your configs,
if appropriate for your solution
and your job that you're trying to run.
The next thing that we want to look at
is under Cluster Admin.
We want to take a look at our stack and versions.
So this is going to give us our service stack,
and you can see all of the services that we have,
and you can see if they're installed.
And then you can also see, under Version,
a really nice chart that shows you all of the versions
and how current they are.
So this will help you as you're looking at
a starting place for if there's any obvious errors.
That's kind of what we're looking at.
If you've done that, then we want to jump in
and we want talk about log files.
So we want to check our 2 different types of log files,
and we want to check our Hadoop step logs.
So make sure that you've looked at those.
If you've done that
and we still haven't encountered anything that's obvious,
now we need to start going into configuration settings.
If you have not optimized your configuration settings,
some places to start are your cluster settings,
hardware configuration, and nodes.
Next, we want to see if we can reproduce the error.
If everything fails, Microsoft recommends
that we try again on a brand new cluster
to see if we're getting the same type of error.
All right, so
we ran through those points rather quick.
A few key points to remember. One,
remember that this is a process that you need to complete,
even if it's boring.
Next, you want to make small changes.
Step by step, you want to make a change
and see what's actually impacting in your environment.
If you make too many changes,
you might skip some steps and fall down the stairs.
So don't do that.
Small changes, and then look and see
what's actually impacting or changing
within your environment.
Also, exploring should be done in dev, not prod.
Should be a fairly simple concept,
but make sure that you keep out of prod
when you're exploring with your services.
Finally, for the DP-203, what is it that I need to know?
Well, I would recommend that you know the basic steps.
So you need to look at Ambari UI,
you need examine your log files,
examine your configuration settings,
try and reproduce the error.
If you remember those basic concept pieces,
that should be what you need to know for the DP-203.
I don't think it's going to go much deeper than that,
but just understand those basic points.
So with that, we're done with this lesson.
I'll see you in the next.