Log enrichment refers to the ability to add additional information
or context to log data to provide more insight.
This can be done by performing computations on existing log data or by
correlating with data from another log source or service.
So you might be wondering, what does log enrichment actually means in practice?
Well, let's consider that you identify suspicious file by the hash value.
You can certainly verify this hash value against the list of potentially known
malicious files or associate it with an exploit or intrusion.
In the case of a domain,
we may want to check if it's a similar or part of a list of domains associated
with malicious activity. And in the case of an IP address,
we can check if it's on a suspicious IP list,
or as we'll see later in the demo,
actually obtain location information based on the public IP data.
Within the scope of information security,
geolocation data can be a very powerful insight.
Leveraging the GeoIP Python module,
we are able to obtain the country code based on the public IP address.
Please do note that this will not work for private IP addresses.
To highlight this in practice, let's consider the following demo.
We'll start off by installing the python‑geoip package,
and we will run a few tests to see how the location
data can be extracted based on it.
Then,
we'll update the log parsing function and add the country
code based on the public IP address.
Each service will log different information and in the different format.
A good example of this is the auth.log file that can be found on Linux systems.
As you can observe, multiple processes send logs to this file like SSHD,
as well as Cron, and apart from the timestamp,
the host, and the process, the log message varies quite a lot,
so writing a parser for this type of log can be quite challenging.
I have already written one that focuses on failed and accepted passwords.
This is done in the same style as the SMB log parser.
I have also separated each group on separate lines and
added some comments based on a log entry so it's easier to
understand what it should detect.
One unique aspect of this pattern is how it detects if the user is invalid.
This is done with a named group called invalid.
This group is marked as optional using the question mark.
If the group is detected, it will contain the string invalid user,
and if not, it will be none.
And similar to the other parsers,
after extracting the timestamp in a string format,
I parse this and convert it to a date and time object instead.
So let's use this in an actual example and see how it works, so I will open a
terminal. And the quick tip here, the keyboard shortcut for the terminal in
VS Code is Ctrl+J or Cmd+J if you're on a Mac.
I will run Python, import the samples, and display the auth log entry.
Next,
I will import the parseAuth function from the module and execute it
using the sample auth log entry as a parameter.
And we can see that the important information is extracted.
Since this log is from a server which is open to the internet,
we can expect a lot of activity.
So looking at this log,
is there any information that we can add to enrich this log?
For all public services,
sometimes it's useful to have the location by country
that the clients are connecting from,
and this is where I introduce the next important package named python‑geoip,
which based on the public IP address,
will produce some information regarding the location.
And as we can see, we can install this using pip.
So I will open a terminal and install the two components.
Python‑geoip‑geolite2 is the publicly available database.
To test this out, I will open up Python and import the library.
The lookup function can be used to retrieve the location for an IP address.
I will use the Google public DNS IP which is 8.8.8.8.
We see that the function returns more information like the time zone,
as well as the coordinates.
To get the specific field, I will add country to get the two‑letter code.
Now it's time to implement this in our code starting with
adding python‑geoip to the import section.
In the auth parser right before the end,
I will add another key to the dictionary with the country and call the
geoip 2 lookup using the extracted IP as the parameter.
Additionally, I will also update the year which is missing from the log file,
and I will set this to the current year.
Looking at the script that we have developed so far,
we notice that the code quantity has increased a lot,
and in time, it might be difficult to maintain it.
So the approach here is to separate all the different parsers
to another file called parsers and paste in all of the parser
functions including the imports.
And in the main file,
we import everything from the parsers module. And now we can see that
the code is a lot more clean and easy to work with.