So now you have data,
but you need some know‑how. I'm going to give you a little secret.
A little birdie told me that there are some bad things going on in
this network that we're monitoring.
And yes, if you're asking,
I am indeed the master of spies for my own fictional
company that I'm using for this scenario.
But before we get to that, let me first quench your curiosity.
You saw those services that popped up at the end of install,
and I haven't shown you how any of them work or what they do.
So let's walk through some of those services that were in the
print out. From the bottom, KSQL server.
There's nothing to hunt here, but it's showing that the service is good to go.
It should be accessible, but again,
this is part of the back end that ties everything together.
And this portal shows that the Spark Master server is up and
running, but this is also boring because you actually have to
interact through the Jupyter Notebook,
which is here. The first time that you log in, you're going
to want to use the token that you got as a password in the
printout after the install.
Included are a demo for different tactics, tutorials, and a ton of Sigma rules.
But we're going to come back to that. Now that we have the
data, let's walk through these dashboards and see how Kibana
can help us hunt the baddies.
Looking back through some of these dashboards,
we can see that they're actually now populated.
The Global Dashboard is just what it sounds like, and as we look through,
we can see the data populated with a focus on the command usage
and hashes, but nothing jumping out here.
The Global Dashboard is going to give you a roll up,
but not necessarily point to anything specific that's concerning.
Remember, this is meant to be an investigation tool,
and that is exactly how these other dashboards are structured. For the Host
Investigation Dashboard, we get process execution by user,
which, as a timeline dashboard, is pretty helpful on its own.
Network connections not showing up here are excluded to make some of these
internal connections that are made all the time less noisy.
I don't have any particular techniques that require me
to have network connections working,
but it's good to note that this should be coming from event ID 3
within Sysmon and can be used for cross correlation.
Notice also that not much else pops up until I enter the system name
as it asks me to. And we will do that in a moment, but first we need
to know what we want to investigate.
Probably something to do with this weird spike, but let's leave it for now.
I don't want it to be just an artifact of the fact that I only used a few
boxes and not much user emulation for this quick course.
Now the same idea applies for the processes and the user investigation
dashboards, and those are going to be super helpful in a bit, but I want to
follow the good hunt workflow that this was really built for.
The Sysmon Dashboard is,
however, a general roll up just like you would expect. And expanding the
time a bit, we start to get some pretty cool information.
The Logstash filters take time to be built,
and index patterns are not always going to map perfectly.
I always take the open source goodness for what it is and use the
data to develop what I need for my purposes.
If I want something to change or if I want to get something
fixed, I am very welcome to fix it myself.
That's the whole nature of open source.
Interesting note as we look through this is that the
count for the Windows event utility, for example,
is kind of interesting.
And we should take note of that and see if it pops back up anywhere. But again,
not a ton of stuff popping out at you here.
So without knowing what to look for,
let's instead move over to the main hunt utility,
the Jupyter Notebook.
Now here in the Jupyter Notebook,
you're going to want to take some time to get familiar
with what you have available to you.
Remember,
this is the one and only interface with Apache Spark and
GraphFrames, providing the machine learning statistical analysis
capability that really make this platform stand out.
To get used to that, there are tutorials and demos built out,
some using demo material and some that are actually then using your data.
The other demos use sample data to walk you through how to set up and
run through Elasticsearch data ingested with Apache Spark. Notice, to
use this yourself you need to mimic the same setup in pyspark.sql,
create the session, and verify the Spark variable and then initiate the
Elasticsearch DataFrameReader.
This is kind of the beginning setup to be able to use this on your
own data anytime that you want to make a notebook.
Once that's done,
you can load the indexes into variables and then view
and manipulate them accordingly.
Now,
where even more of the power comes in is instead of
reading one dataset at a time, we can join the various
data feeds together in a custom way.
This one initiates the Spark session and connects to Elasticsearch,
then imports the winevent security events that are imported through KSQL,
then runs an SQL query against the data to look for specific conditions of
log ons. And with that done and loaded into the security_4624_3 variable,
you can then load up the Sysmon events as well and perform a query of
process events created during the same time period as the log ons. And this
is interesting because now you can really hone in on what kind of processes
were created in a specific time period at which we also had log ons over the
network, which is what that type 3 code stands for.
And then you can use the join command to join the two
queries together from separate tables and filter for just
the columns that you're interested in.
This is a clear extension of the native capability in
Elasticsearch and is just the tip of the iceberg and the
additional functionality that's gained here.
Now, there's no results in this live demo,
of course, because that data is not in the demo time periods.
But I told you I was taking you hunting, and I'm not going to let you down.
There is a ton of information in the tutorials that's
really worth the experience itself,
just simply on ways that you can manipulate information and log sources.
But hunting is not for the light of heart because what you're doing is you're
looking for things that our existing detections don't find.
So you have to really get at the information and the data at its source.
You've got to take it and you've got to throw it in a
barrel and filter it and toss it this way and that and mix
it together and get new insights.
Let's look at PySpark.
Run through the same setup that you saw before except now it's time to
add in GraphFrames. If we want to get to the bottom of process creation,
then relationships is the name of the game,
and we can do a little test here to look at the
degree of separation of the edges.
If you want a dissertation on graph theory, that is a different course.
But I will show you how HELK uses it here to map process
spawning relationships to catch bad guys.
In case you didn't know, getting to the bottom of process creation in
your environment is doing something that's kind of hard.
It's linking processes with their parents and spawned processes.
This data is a bit messy to work through normally,
but can hold telltale signs for adversary activity. Import the
Sysmon index and change the process_guid column to the id. This is
so that GraphFrames can use it properly.
GraphFrames is looking for the ID name in the column as
the unique identifier for each row.
Also,
remember that process_guid was one of the atomic indicators that
we can use for the investigation dashboards.
Next, you add the specific columns as vertices in the motif,
and then you go ahead and show what you have imported so far as the
vertices. Now we can look at the relationships across the
environment for process parents by indicating the parent as the
source and the process as the destination.
And with that, we can see which process spawned which other process.
So not amazing hunt information yet,
but we're on the trail and it's starting to heat up.
With the vertices and the edges defined, you can build the graph.
And no, this is not a visual graph,
though there are projects for drawing them based on the
output that we're about to receive.
After that,
you define the relationships motif that you want to show. In
this case, A is spawning B and B is spawning C so that we
can follow the process chain.
Okay, so this is cool.
Now let's look at this and try to make a little bit of sense from our
extensive defender knowledge base. Process parent userinit.exe means
that we have a human logged in running cmd​.exe live on this device,
and that is nearly confirmed by the process name of rubeus popping up
here as well.
You can see a ton of wevtutil.exe processes spawned in the same way. Windows
event log accessed by an actor who just used rubeus. Well,
I'm going to write that down as suspicious, and I now have reason to
believe that some Kerberoasting activity was taking place, and then
potentially the logs were modified or cleared.
There is a lot of hunt potential here,
but simply grouping the data by least to most prevalent
process parent name reveals that cmd​.exe as a parent process
that could be suspicious as well.
This is another clue to help you identify what's happening to go hunt.
Okay, so that's cool, right?
But what I want is more information.
All right, now right here, this is fancy and I like it,
but I need more information about the user and the host that this happened on.
What about the GUID? Well, all I have to do is ask.
Let's talk about Rubeus. We know that it's bad, and for reference, if
you don't know why it's bad, it's part of GhostPack,
and that's a common set of post‑compromised tools, and there's
a Red Team Tools course on that as well.
What we see here is that that is in the C process column in the GraphFrame.
So I want the C user, the C host, and the C GUID. So here we go.
C.user_name was the column, then c.hostname, then I remembered
changing guid to id for compatibility, so this is actually going to
be c.id. Now throw that into the vat of acid,
release the bones, and come full circle to right where we want to be
that was initiated by ultron on hr‑01 with this GUID.