Now that we're able to retrieve log entries from log files,
it's time to get to the next level.
We need to parse each log entry and extract individual fields.
One of the benefits of doing this is that it will enable us to apply
filters based on the value of the individual field.
We will go beyond this to perform operations based on the data type.
For example, we can apply math functions and do calculations on numerical data,
and perform timeline analysis to establish the chain of
events that led to a system being compromised.
There are situations where the log entries are very well structured,
and each field is separated by a known delimiter.
This can be in the form of the pipe character, comma, or tab.
By splitting the string based on this delimiter,
we'll be able to extract the individual fields.
Onto the technical part,
we can leverage the re module to split the log entry based
on the value of the known delimiter.
Rstrip is used in this case to remove any trailing white space or
new line characters at the end of the line.
Then, we can create a dictionary, where the key is the name of the field,
like uid or timestamp,
and the value is the extracted data corresponding to that field.
To get a better understanding, let's consider the following demo.
The main goal is to parse a Zeek log file and extract the
individual fields based on the tab separator.
In order to get more insight into the log‑file structure,
let's get acquainted with the log fields.
This information is available on the Zeek documentation page.
The way it's described in the documentation is in the JSON format,
but the log files are actually tab separated.
Now let's return back to VS Code and see how a log entry actually looks like.
I will open up Python and import the samples from the log_samples module.
The samples object contains a class variable called zeek_conn.
For this log record, we notice it's in raw format because we can see the \t,
which encodes the top character.
With this information, we can start writing our parser.
First, we'll import the re module, as this contains the split function.
Next, we'll define the parseZeekConn function that takes one parameter,
which is a single line from the log file.
This variable will be called log_entry.
The log_data variable will become an array as the
result of the re.split function,
which splits the log file based on the tab separator.
Log_entry is passed as a parameter,
but with rstrip used to remove any white space at the end of the line.
The results will be a dictionary with a field name as
the key and the log_data as the value.
So I'll create an empty dictionary called r.
The first field in r will be the timestamp.
Noted ts will be assigned the first element from the log_data array.
This is an important aspect that we need to keep the same
order of the fields that there is in the file.
And I will continue to fill in a couple of other important parameters.
The last instruction in this function will be return r,
to return the dictionary object which contains all the data.
Time to see this in action.
In the terminal, I will import this function from the log_analyzer module,
and I will call this function and use the sample.zeek log line as a parameter.
And it works.
We can see the object returned.
One important thing to note is that this is not the full list of fields,
so I have to return to this function and fill in all of the
fields based on the Zeek log_file structure,
which I showed you from the Zeek documentation.
Dates are really important when it comes to log analysis,
but Zeek actually captured this in epoch format.
In order to make this usable,
we have to convert it to a proper date and time format,
which will support date and time operations.
To achieve this, I will import the datetime module as dt,
so it's easier to type.
To make things simple, in the samples module,
I have the timestamp that can be used for conversion.
This is called samples.zeek_ts.
The datetime module has a specific function called fromtimestamp,
but this actually requires a number and not a string as a parameter,
so I have to use float to convert the string to a float,
and we can notice the datetime object being created.
Now let's implement this in the code.
First, I will import the datetime module,
similar to the example in the terminal.
And now, in the parser function,
instead of passing the timestamp string directly,
I will use this as a parameter to the datetime function.
So instead of the epoch number, we'll have an actual datetime object.
Moving forward in time, I have added all of the fields,
as well as created two more parsers for the HTTP and DNS log files from Zeek.
I will minimize this to keep the code tidy.
Now we're in a position where we can actually start analyzing log files.