In this episode, we're going to talk about one of my favorite subjects, thread hunting.
How do we go in and actually find those different pesky little threats that are invading our network,
and how have they corresponded to the mitigation factors that we've already talked about?
Threat hunting really is one of those things that you love it or you hate it.
But as a CSA, as far as the exam is concerned, we're going to cover the high levels.
Don't worry.
I'm going to input my own little spin on everything we talk about.
Threat hunting really starts when you get an alarm, or you get a problem within your Siem system that
points to a direction of malicious activity.
This could be anything from a virus going off or a piece of machinery within your system suddenly crashing
for no apparent reason.
Threat hunting comes into play where I've got a problem within my network or within my systems, and
I need to go through and find out the root cause of that problem.
We start off with a simple hypothesis, meaning I need to take into account what the alert tells me,
and that may be the minimal information that I get and come up with a hypothesis as to why that happened.
For instance, let's say that I've got a piece of malware that suddenly went off and hit my system saying,
hey, there's malware on the system.
My hypothesis would be that I've got a Trojan horse or other virus that has infected this machine.
That is a very simple hypothesis.
We're going to go through and we're going to start to investigate the problem.
We're going to dig in and find out what did that virus or that piece of malware do to our system.
We're going to look for different patterns.
Did it infect just this system or multiple systems?
Is it just on the core operating system or has it expanded across the entire network?
Those patterns really point us in the right direction and formulate what we need to do for threat hunting.
Then finally we need to respond.
That could mean eradicating the virus, removing the the piece of equipment from our network.
We could isolate it.
We could do all those different things with our incident response plan.
Threat hunting really is at the core of what you do as a cybersecurity analyst, because it comes into
play with saying, I've got this problem that my Siem pointed me to and I need to dive in and find out
what happened.
That's really all threat hunting is when you talk about threat hunting.
We talked about the overall process.
But why do I threat hunt?
Our first question we need to ask ourselves before we start threat hunting for any legitimate problem
on our network is what is the purpose of my hunt?
The purpose could be that I have a bug in the machinery, and so I need to start researching that to
find out.
Hey, where did the bug start from?
Was it from a piece of malware?
Was it two pieces of software that didn't like each other?
Maybe there was a hardware fault.
Any of those could be the purpose of the hunt, but I need to find out why I'm actually going after
that piece of problem.
Sometimes it doesn't make sense to go after a hunt, meaning I may have a virus that went off on our
antivirus program and it scrubbed it and it was one machine.
We're not seeing patterns.
We're not seeing any other problems.
It may just be time to stop.
Where will it be conducted?
Is it across the entire network?
Maybe it's just one Vlan, maybe it's one subnet, maybe it's just one machine.
It doesn't matter.
But we need to ask ourselves to provide a scope of what we're trying to do within our threads.
What resources do I need to conduct the hunt?
Do I just need my own machine?
Do I need access to the antivirus program that's on there?
Do I need access to the IPS?
Is it a network holds a hold?
Do I need access to the server?
All of these come into play with again our scope.
We need to identify our key stakeholders.
And this is important because if I start going after a virus on a certain machine, and yet that machine
is supposed to be used for something else that could cause some major problems.
Usually one machine isn't going to stop us from moving on with our threat hunt, but let's say that
it's a server.
If I had to take an entire server offline, we may have some problems within our business scope.
What if our servers are all at 80% capacity and I only have three of them?
By taking one server off the line, our company could be losing money on a drastic rate, and those
stakeholders may not like that too much.
And then finally, what is the desired outcome of our hunt?
Is it just to find the problem?
Usually?
Not usually.
The overall outcome is supposed to be eradication or isolation, meaning I'm removing that threat from
our actual systems.
If we don't have a desired outcome, then we don't have an endpoint.
And if we don't have an endpoint, we could go down rabbit holes.
We don't want to go down.
Remember, your job as an analyst is to find the problem, find out how the problem occurred, why it
occurred, where it took place, and route into the system.
Did it expand?
Identify the stakeholders that are associated with that problem, and then let properly communicate
to them with the desired outcome that we've already formulated, i.e. isolation, eradication, threat
hunting really comes into play with those basic questions that you need to be asking yourself before
we jump into the fray.
The Threat Hunting Maturity Model, or HRM, is a maturity model that we use not as a person but as
an organization.
This means that we as an organization, need to understand where we come into play as a threat.
Hunting infrastructure, meaning how well does our organization actually handle those different threats?
If we're very at the point of emphasis in our threat hunting maturity model, it means that we don't
usually have a lot of data collected.
This means that our network usually doesn't have a Siem.
It doesn't normally have a lot of logs that were corresponding.
Maybe it's a brand new organization that's being stirred up for the first time.
And all we really have on there are some machines with a basic network, so we really don't have a lot
of information that's coming into play.
This would be an Hmm zero infrastructure, meaning that we don't have the resources or the personnel
to probably start searching for threat huts.
Then there's MX one.
Now this is where we have some data collected, maybe some minimal information is gathered, maybe we
have a scene, but it just got kicked off yesterday.
And so we're getting alerts, but we really don't have those alerts properly tracked or maintained.
We're not really sure where everything came from.
But we can develop a new hypothesis and start very, very low level threat hunting activities.
This usually comes into play with organizations that are brand new to the scope.
They just started their Siem.
They just have a cyber security program, but they're very, very low maturity level when it comes into
play.
With MX two, we have high collection of some data types, meaning maybe we're collecting data from
our routers or our Or firewalls or IPS, but maybe we're missing it from the servers.
Or maybe we have it from the servers, but we're not collecting logs from our from our firewalls or
our routers.
This is where we have a lot of good information from some items and no information from others.
Not the best practice comes into it, but we're developing our threat hunting or our maturity model.
In this case, we can generate general Intel reports, we can use basic tools, and we can automate
some of those threat handling procedures that we would in a normal SOC environment.
But again, still standing up our systems and a layer three or excuse me, a level three.
Hmm.
We have a high collection of many data types, meaning maybe we're getting data from different servers.
We're getting it from switches, we're getting from routers, we're getting it from firewalls, IPS,
IDs, a whole gambit of different information.
We may be missing a log file here and there, but overall we have a very high collection methodology.
We also have described Intel reports meaning outsider intelligence reports are coming into us and we're
getting those different feeds.
We talked about that in an earlier episode, but if you remember, we saw different feeds from all types
of blogs.
We saw them from different websites, from the government, even from different other organizations.
Those Intel reports provide us different trends and availability of information that we can visualize
and better prepare ourselves for threat detection and mitigation.
Finally, in level four, we have continuous mapping of possible threats to our organization.
This is the highest level of maturity model available here.
And we can not only search, but visualize and analyze different data sets to those corresponding information
that we're getting.
Meaning we're taking those Intel reports that we've received.
We're also taking our data from our infrastructure.
We're kind of combining them two together, and we're providing our own analysis of the different facts
associated with it.
That's usually reserved for high level organizations that can do both threat hunting and have a very
robust security network with lots of manpower.
Think about those high intrinsic SoCs that utilize various different departments, usually with something
associated with maybe with AT&T or Verizon that have those very, very advanced socks.
There's different techniques that you as a threat analyst can utilize in order to find different aspects
of your different networks.
That means that if I come into play, I can look at the different alerts and corresponding intelligence
reports that I'm getting and set that data in a way that I can actually read it and go through the process.
Volume is probably the easiest.
This is pure simple, just the aspect of different alerts that are coming into play.
If I get over 1200 different alerts for the same machine, that's volume.
I'm getting a bunch of information from one specific machine or one type of alert that I need to be
aware of.
Pure and simple.
Volume is just numbers.
Then there's frequency.
How frequent is the specific alarm or specific problem coming into play?
Now, this is honestly sometimes confused with volume where we're looking at the full number, where
frequency is more of the pattern that comes aligned with it, which means I could get a lot of volume
for the same alert.
But how often is it occurring?
Let's say that I have one alarm coming into one system every day.
It doesn't sound like much because it's only happening one day, but then it happens again, and then
again and then again.
And for seven straight days, we've got the same alarm at the same time.
That only provides us a pattern, but it also provides this frequency, and that's where frequency analysis
comes into play.
Finally we have clustering.
Now clustering is finding similarities in data by comparing the different characteristics.
This is often confused with grouping.
Clustering comes into the fact that I'm not actually looking for similarities, but I kind of am.
I take this giant glob of data and I start sifting through it, and I notice different similarities
as I'm going through.
This is where costing comes into play.
Let's say that I've got a bunch of data, and I start sifting through the data, and it all points out
to the color blue.
So I start to see blue pop up a lot more often.
That's finding a similarity in the data I found blue.
This is different from grouping.
Grouping is where I have a bunch of data, and I specifically look for the color blue.
That's where grouping is comes into difference with clustering.
Clustering I'm not looking for the data or specific attributes grouping.
I am looking for specific attributes.
And then there's stacking.
Stacking is sorting data and finding the extremes.
Let's say that I have an average of five.
I've got 500 different numbers and they're all different all over the place.
Right.
I take the median or the average of those numbers.
And I say, you know what?
My average is between 200 to 300.
That's the average of those numbers from 1 to 500.
That would be my main grouping.
However, stacking takes the extremes.
Maybe I have a number at 102.
And on the flip end, maybe I have a number at 498.
Those are the extremes.
And that's where stacking comes into play.
We're not looking at the core foundation or the medium or the average.
We're looking at the extremes, the two ends where the data just doesn't support what's going on within
our system.
We talked about analysis.
We talked about hypothesis, the process, the procedures and how you found everything.
But what do we do after that?
That's where reporting comes into play.
Reporting is taking everything that we did and putting it into a well-documented aspect.
Now you heard me pound on documenting.
And we beat that dead horse into a million different pieces.
But I'm going to say it again.
You got a document.
We need to document what our hypothesis came from.
If I'm going threat hunting, I need to start my documentation right at the get go.
Why am I threatening?
What was the original alert or problem that I saw?
And I need to document that along with my original hypothesis.
Now, our hypothesis can change as we're going forward, and we can be proven wrong on a hypothesis.
That's perfectly okay.
It doesn't matter.
We just need to document the differences from what we went.
If we go from one hypothesis, i.e. I believe the sky is orange and now I believe it's green, then
I need to document that hypothesis.
If I'm suddenly wrong and the sky isn't green, it's now blue.
Well, that's a finding.
That's the overall finding that we we identified.
That's okay.
Our hypothesis is wrong.
There's nothing wrong with being wrong.
We just need to document why we found the color blue for the sky.
We need to identify our different processes.
How did we go through the actions of going in and finding what we found was a process to simply go through
the antivirus program and figure out, hey, yeah, it stopped the virus and it isolated it.
And so I'm done.
Or was it something more complex where we went through the different procedures in order to actually
identify that a piece of malware came in from this IP address because someone posted on a phishing link.
We need to identify that process and how we went about that.
We need to identify the different procedures that we utilize.
Did we utilize a tool such as Pfsense for our firewall?
Did we use a different router and trace the logs through there?
Did we use our Siem system?
All those tools need to be identified within our process and procedures and annotated correctly on our
documentation.
Finally, we need to identify our findings.
If I finally discovered that the sky is blue, that's fine.
I found out the sky is blue.
Let's mark down our findings.
We've identified how we got there, and then finally we need to identify our recommended actions.
Well, the sky is blue.
My recommended action.
Nothing.
However, what if it was a piece of malware that was on our systems and we discovered not only did we
alert, but that malware has expanded into our servers and even some of our clients.
Well, maybe our recommended action would be during the maintenance window to scrub those systems and
get rid of and isolate those networks so that we can scrub those systems.
When we go through and we go through the reporting process, we need to be very clear, very concise
and have a proper workflow.
Usually a company will have a template for you to follow, and usually just following that template
just comes into play and we just go A, B, C, D and E and boom, we're done.
But we want to annotate everything.
We want to go through that documentation to make sure that it's very easy to read, and that anybody
can follow our line of thought as we're going forward.
Who knows, there may be another action two weeks from now that a brand new analyst is trying to identify,
and your report will help them get through that process.
We discussed today different threat hunting methodologies.
We discussed the threat hunting maturity model.
We went through the different process reporting of documentation and annotation.
When you go through the Cisa exam as an analyst, you need to understand again that this is a very high
level overview.
You're not expected to be able to go through and say, oh, I found a piece of malware.
And from that malware, I need to use this tool.
And from that tool, depending on what it's not like that it's very high and very high level.
Scenario based questions, much like we've discussed in the past, you should expect to see questions
such as what's the threat maturity model would you use?
Or what phase of that threat maturity model would you be in for your organization if you had access
to some reports, but not all, some inaccuracies within your intelligence gathering and very low,
limited tools at your discretion, then you would be able to answer that question between 0 and 4 of
what maturity model level you were at.
You should also expect to see different questions on different threat hunting, aspects of where the
process from hypothesis all the way down to reporting.
You would lie within this specific, uh, scenario.
As you're going through these different questions, realize they're all scenario based.
You shouldn't have any problems with it.
Just understand and memorize the threat hunting maturity model as well as threat hunting graph that
we propose.
Remember, everything starts with a hypothesis.