Now it's time to get started with the first stage, reading the log file.
For the majority of services,
log files are essentially text files where every line
in the file represents a log entry.
Each log entry contains multiple fields.
Depending on the service,
log files can become quite large so we'll have to analyze them line‑by‑line
as opposed to loading all their contents into memory.
If speed is important,
consider using filters when loading these events into memory for processing.
We'll also leverage exception handling for parsing errors
as logs may contain additional entries,
either for metadata or in a different format.
This function takes one argument which is the path to the log file.
The with statement is used to handle opening and closing of the file.
Information will be accessible through the log file object,
and then in a for loop, It iterates through it line‑by‑line.
For simple scripts,
we can add the parsing and the processing within the for loop,
but this will not scale, so let's consider the next example.
By leveraging a generator rather than calling the function once
to iterate through all of the log entries,
with a generator, this will be exposed as an object that can be iterated through.
Then we can separate the detection logic in another function.
Generators are a Python topic which are a bit more advanced,
but you'll get more familiar with them in the next demo.
In this demo,
we'll start off with an overview of the demo files
including their name and location.
Then, we'll check the Python version and ensure we're on version 3.
Then, we'll create the virtual environment to manage dependencies.
Please consider that for Windows this will be a different process,
but altogether, it's still optional.
And finally,
we'll write and test our function to open log files
and retrieve information from them.
So let's start by exploring the log files that we'll be working with.
I'll open up the terminal and list the directory
structure using the tree command.
The logs folder contains the logs that will be used
throughout the rest of this course.
The extra folder contains some utilities including some log samples
that we can use while developing our log parsers.
Next, it's time to double‑check the Python version,
and this can be done by using python3 followed by ‑V.
On this Linux distribution, the Python version is 3.8.10.
All of the demos will be compatible with Python version 3.
You might need to look out that if you're using just Python without the 3,
you might actually be running Python 2 depending on the
version of Linux or Windows that you're using.
I will create a virtual environment to manage all dependencies,
and this can be done with the following command.
To note, this is not mandatory.
It's just a good practice.
And I got the first issue.
It seems that I need to install this package in
order to use virtual environments, so I'll do just that.
Now let's try to create the virtual environment again and it works.
To activate it,
I will use the source command followed by the path to the
activation utility which is venv/bin/activate.
Do note that once the virtual environment has been activated,
this is displayed in the terminal at the beginning of the command line.
Within the virtual environment,
I can also just use Python without the 3 and this will return the same version.
This is because I used Python 3 to create the virtual
environment in the first place.
To deactivate it, just type deactivate in the terminal.
Now it's time to write some code.
For this, I'm using Visual Studio Code,
but any text editor or integrated development environment can do it as well.
This can be done from the command line just by typing code
followed by the folder that should be opened.
VS Code is great because it has also built‑in terminal which
I will use heavily throughout the demos.
Now I'll create a new file called log_analyzer.pi,
and we can see it appear in the File Explorer.
Now we can write our first function that will allow us to open files.
I will leave a couple of lines free from the top where all the imports will be.
Def is used to define a function followed by the name,
which in this case,
will be open log file and then use the path as the parameter followed by colon.
I'm using the with statement to handle the opening and
closing of the file, then iterate with a for loop through
every single line in the file.
Typically, with any function, I should have used the return statement,
but instead,
I will use a yield statement instead to leverage this function as a generator.
So now you may wonder how does a generator work?
Well, let's find out.
In the terminal,
the first thing that I will do is to activate the virtual environment.
Next,
open up Python and use a form statement to import the open log
file function from the log_analyzer module.
I will be using a variable called log_file which I assigned
the result of the open file function for which I use one of
the log files as a parameter.
If I write log_file, we can see that it returns a generator object,
but we were expecting lines of log entries from the log_file.
To get this, I will use the next function with the log_file as parameter,
and we notice an actual log entry.
At this point,
we have everything in place for the next stage which
will be to actually parse the log data.