Based on the previous modules,
you have learned how to leverage Python to gain insight into log files.
We have parsed log data,
after which we have identified and extracted indicators of compromise,
as well as aggregated data to build and visualize trends.
But Python offers much more opportunities to automate manual analysis,
as the work does not stop right after we parse the log files.
Most times, there are other steps in the process,
so you can focus on finding opportunities where they
can be automated with Python as well.
Also Python does not limit us to only log files,
you can consume data from existing repositories to analyze or feed
back enriched data. And some great investigation workflows have
already been covered in the previous modules.
For example,
we started from the DNS logs and then implemented some logic in
Python to identify the least frequently‑visited domains and
perform similarity analysis on them,
from which we extracted some high‑fidelity indicators,
including the IP to the attacker infrastructure.
In another workflow, two log sources were used.
The connection and HTTP logs were correlated to identify beaconing based on
the long duration and high number of requests from which we also obtained the
same IP address to the attacker infrastructure.
So you can see that you can take things forward and
build an additional layer of logic,
as well as coupling these to some machine learning or AI system.
When it comes to building log analysis workflows in practical terms,
we can push generated data results into centralized repositories.
We can correlate with data from other services,
such as IP or domain‑reputation services.
And finally,
building standalone solutions for log parsing and
analysis that others can leverage as well.
From a technical perspective,
let's look at the mechanisms that will allow us to extend the usage
of Python within the wider scope of log analysis.
For starters, if you want to store data for use at a later time,
you can just put them in a file.
But this kind of takes us back to where we first started,
so let's look at some nicer options.
Web services and APIs are the way servers communicate to each other,
so these can be leveraged to either pull in additional information
or feed information upstream to another platform.
Databases are also a good option for persistent storage,
especially for scenarios where log analysis is done distributing and
you want to gather all results in a single place.
Interacting with web services requires us to
serialize the log data leveraging JSON, for example.
And to interact with SQL databases, it may be that you need to use an ORM.
On the other hand, NoSQL databases, such as Mongo,
come with a native client for interacting with the database.
Next, we'll try to implement two investigation workflows.
We'll start from the SMB logs as the source.
We then move to parse the log data, like we already did in a previous module.
As for the problem that we want to solve,
the goal is to detect known ransomware extensions.
And to preserve these alerts, we can store them in a database.
The second investigation workflow starts with the auth logs.
Then, we parse the data.
We then enrich the logs with GeoIP data,
like we've done in a similar workflow,
and index this data into Elastic so that others can use the data as well.
The main reason I have put them side by side is to showcase
the fact that they have common stages,
which means we can reuse components that we've written.