Log sources come in different formats, and there may be situations,
as we'll see in the next part, where log file types include unstructured data.
Essentially,
there isn't a single common delimiter that we can leverage for parsing.
This is where we need to take advantage of regular
expressions for pattern identification to carve out each
field based on their type and order.
Then,
the challenge lies in identifying and crafting patterns that are inclusive
of the important data and don't skip over log entries.
While this course is not meant to provide a
detailed view on regular expressions,
we'll be making use of them in the next section,
so it's better to get familiar with some of the basic
elements for pattern recognition.
Specific to Python,
the RE module provides access to the regular expression functions.
Search and match are two important functions that can be
used to identify patterns in strings.
A really important topic is that Python supports the concept of named groups.
A named group can be added for each field in a pattern and then
extract these individual fields as a dictionary.
To provide a more practical example,
let's consider the following log entry and the regular expression below it.
Let's walk through it so that I can explain how it was
created and how you can craft your own.
The first named group identifies the timestamp using three groups of
two numerical characters separated by a colon.
Outside the group, we also need to match the colon surrounded by whitespace.
The hostname can contain lowercase and uppercase characters,
as well as digits and the dash symbol, which is escaped by backslash.
The plus at the end specifies that the hostname should have at
least one character from the previous set.
Outside of the groups, all pipe characters are matched using
backslash to escape them and avoid them being interpreted as
an or by the regex engine.
The client_ip, similar to the timestamp,
identifies four groups of one to three numerical
characters separated by the dot.
The share and operation fields are quite similar.
The goal is to detect lowercase and uppercase characters.
And finally, the path.
I use the any operator together with a star to match a
variable string length until the end of the line.
This is because the path can contain more details than just the document path,
name, and extension.
Once the pattern is prepared,
in order to parse the log entry and extract the individual fields,
the re.search function is used where the first parameter is the
pattern and the second one is the log entry.
Rstrip is applied only to remove any trailing whitespace
or new line characters at the end.