We will attempt to implement the beacon detection
technique that we have described previously.
In an initial phase, we'll group the HTTP logs by the connection UID value.
Then we will iterate through the connections log,
filter http connections based on the service field,
and save the results in an array.
The end goal is to correlate the aggregated HTTP request data and
the filter connection data based on the UID.
This will allow us to identify beaconing based on the connections with
a long duration and a high number of HTTP requests.
In the first part of this module,
we'll attempt to aggregate the HTTP requests based on the UID field,
essentially establishing how many HTTP requests were made during a session.
The name of this function can be something like
getHttpByUid and the path to the log file.
I will use a Counter dictionary and import it from the collections module.
Next,
I will open the log file using the path as the argument
and iterate through it line‑by‑line.
I'm using a try/except block to catch exceptions in case there
are lines which cannot be parsed correctly.
Then for each log entry in the log file,
I will parse it and put the results in the log data variable and
update the counter using the log data UID field,
and I will return the counter object.
To get a better feeling of how this works,
let's open a terminal and start Python.
I will import this function first and then call it using the HTTP log file,
and we get the result, but not in a beautiful format.
Because the function returns a counter object,
this has a specific method called most_common.
This returns an array sorted descending based on the values,
and we can select the top five connection UIDs based on
the highest number of the HTTP requests.
Getting the number of requests is the first part.
Now it's time for the second.
The next function will call detect beacons as it will correlate
the information between the two log files,
the HTTP log and the connection log.
Even though we extract different information,
the structure and methods will be similar.
So first, I will extract the number of requests by UID into a variable named req.
Then I proceed to open the log file.
This will be done for the connections log.
For each log entry in the connections log,
I will parse the entry and filter it to ensure the service field is HTTP.
Now let's pause right here and understand this next part.
Essentially,
we need to correlate the UID from the HTTP log file which is in the
req variable with the UID from the connection log.
The way Zeek works,
there will always be a connection log entry that will correspond to
one or multiple entries in the HTTP log file.
With this enriched information,
we'll add the log_data object to the beacons array.
Next, we will sort this array using the sort function.
Since we are sorting an array of dictionaries,
we need a way to specify the key or field by which it should be sorted,
and this can be done using the itemgetter function.
To sort descending, I will use reverse=true.
Because this function implements a detection mechanism,
I want to print the results in a much nicer form.
So I will use format strings and only extract the important fields.
To test this,
I will go back to the terminal and call this
function using the log files provided.
Wow.
We can immediately spot the outlier as we have a connection which has a very
strong duration and a very high number of HTTP connections.
And here is the interesting part,
the information that we extracted now is very similar to an existing tool
called Real Intelligence and Threat Analytics developed by Black Hills
Information Security which extracts similar information and then applies
machine learning algorithms to identify beaconing.