Hello, and welcome to the Log File Analysis with Python course.
My name is Cristian Pascariu, and in this module,
we'll tackle how to move away from manual analysis and
start processing log files with Python.
We'll start off by going over some of the important aspects of log files.
Log files have been used for a very long time by IT
professionals to troubleshoot services and applications that
are the backbone of IT in all organizations.
For security people,
log files represent a detailed and high fidelity source of insight that
enable defenders to gain visibility and enable them to reconstruct the
chain of events in case of an intrusion.
Logs are considered of such high value that attackers will
try to purge log files or suspend the logging services in
order to cover their tracks.
With log analysis, we have a wide spectrum of activities.
At one end, we have manual log analysis,
and on the other,
we have the state of the art SIEM or SOAR solutions that
extract meaningful details from log data.
Each come with their own set of opportunities and challenges.
Manual log analysis is perfect for small investigations where
one or a couple of hosts are compromised.
They contain high fidelity events.
Onto the challenges side, manual analysis can become time consuming.
Also manual analysis does not scale well and it depends a lot on the
size and the skills of the investigation team.
On the other hand,
SIEM and centralized monitoring solutions provide at scale
monitoring and correlation of events.
However, all of these nice features come with a cost.
Also analysis capabilities are limited by the product features.
Essentially,
the benefit lies in the features that the solution already comes with.
Additional features may not exist or may come at an additional price.
Automating log analysis with Python comes somewhere in the middle.
It can enable a high level of automation.
It can be used to refine and enrich log data by the
means of correlation with other sources.
While not necessarily a big challenge,
but developing tools and scripts will require
maintaining them in the future as well.
I hope that these opportunities have captured your attention.
In this module,
we'll start off with an overview of the opportunities and some
of the areas that you should avoid when starting to automate log
analysis with Python and the value that you can add to your team
and your organization, as well as the time you'll save.
We will primarily focus on parsing log files.
First,
we'll parse log files based on the standard delimiter to extract
individual fields. And then, we'll leverage regular expressions in
situations when log files don't come in a structured form.
Windows logs are also a critical point in any investigation,
so we'll cover how to parse Windows log files based on the EVTX file format.
As we start off this module,
please consider all the scripts more as building blocks that you can
modify and build upon rather than one finished tool.
Also, in order to focus on teaching you the core concepts,
I have omitted some of the verification that would have made
the code way too complex to understand.