For log analysis in Python, I propose the following modular pipeline.
In the first stage,
we'll have a set of components responsible for reading
through the log entries one by one.
The next layer is composed of the parsing logic.
Since log files are mainly logged as large strings,
we want to extract the individual fields so we can run comparisons and
calculations depending on their specific data type,
like date and time.
And the last layer is for analytics.
This is where we implement our aggregation and detection logic
based on structured data to gain some insights.
This processing pipeline is modular because each layer
focuses on different functionality.
Now that we know the plan,
it's time to focus on the toolset that we'll use to build it.
Python 3 is supported natively on the majority of Linux distributions.
If you want to go through the demos on a Window system,
you have the option of either downloading and installing Python 3 or use
it through the Windows Subsystem for Linux feature.
You can use your favorite text editor or integrated development environment.
I will use VS Code in all of the demos.
In order to make this a constructive learning experience,
I have some log files extracted from some simulated attacks.
There is an SMB log from a file share where multiple machines host
files and contains traces of a ransomware infection.
Next,
the auth log from a Linux server that was internet facing and had received a
lot of SSH‑connection attempts, including some brute forcing,
which resulted in someone or something gaining access.
There is also a more complex infection scenario that is tracked from two angles.
From the network side, there are Zeek connection,
DNS,
and HTTP logs that contain some C2 becoming traffic, and
also, there is a Windows log fall,
which contains details on the infection with a backdoor.
It will be quite interesting, not only to detect these attacks,
but automate them in such a manner that it takes no effort
from us apart from writing the script.