Persistence is an important topic.
We can leverage a database exactly for this purpose.
For example, we can store newly processed data. At the same
time, we can also extract information from the database to
further enrich the current log analysis.
For example,
identifying patterns based on previous events. To interact with the
relational databases, an object relationship mapper can be used, ORM for
short. A very popular one for SQL databases is SQLAlchemy.
Meanwhile, for NoSQL databases such as Mongo,
there is a Python native client called by PyMongo.
Since we'll attempt to do a demo on this topic later on,
it's useful to familiarize yourself with the MongoDB
hierarchy. The top‑level is the MongoDB instance.
There might be multiple databases on one instance.
A database can contain multiple collections. You can think
of these as tables in the SQL world. And the collection can
contain multiple documents.
You can think of this as actual rows or records in the SQL world.
So let's assume we want to leverage Mongo database to host
alerts that we generate based on our log data. We can consider a
dedicated database for logs or alerts.
Then, at the collection level, we can have one collection for each type
of log, and then individual log entries or alerts.
Working with the PyMongo is quite easy.
After establishing the connection,
you can select the database and collection, just like you would access
regular values from a dictionary. And then you get into the specific
functions for inserting and finding documents. To better visualize this
concept, let's consider the following demo.
Initially, I will set up a Mongo database using Docker, and test
connectivity from Python using the PyMongo module.
Then, I will implement some detection logic that after parsing the SMB logs,
the script would look for files encrypted by ransomware, based
on a couple of known ransomware extensions.
In order to save the results of our log analysis efforts,
we first need a database. To provisions one for this demo,
I'm using Docker. Going over the command,
I'm using docker run, then ‑d so the container runs in detached mode,
then ‑b to map the port on localhost port to that of the container.
I'll give the container the name mongodb, and the image
that I will use is mongo:5.0.11‑focal.
It takes a while to download the image and build a container.
To test if the container is running,
I'll use docker ps. And the container is up and running.
Moving forward, it's time to install the MongoDB client called by PyMongo.
I will install this using pip.
Now it's time to start up Python. Import the MongoClient,
establish the connection with the database, and to test that it's
successful, actually list the available databases.
Perfect.
Now it's time to import PyMongo into our script.
Next, I want to implement a function that will save the alerts generated when
files with known ransomware extensions are discovered when analyzing the SMB
logs. The function name will be ransomwareAlert, and it takes one argument the
path to the SMB log file. To save time, I will declare the location of the SMB
log as the default one.
Next, I will instantiate the MongoDB client, create a new database for
alerts, and add the new collection called SMB. These are created
automatically once the first document is inserted. To detect
ransomware‑infected files, I will create a pattern.
The pattern contains common ransomware encryption names
such as encrypted, locked, or wncry.
Next, I will open the log file and iterate to each of the log
entries in a for loop. For each log entry, I will parse the log
data and then via regular expression, check if there are any
ransomware extensions in the path field.
If so, it means that the ransomware infected file has been discovered.
I will also add a new field called timestamp added, ts added for
short, and the sign is the current date and time.
This is because we might analyze old log data and we want
also the timestamp of the analysis.
Then, the collection object has a function called insert_one.
I will call this with the parsed log entry. To test this, let's run
this in the terminal. I will import the ransomwareAlert function and
also the MongoClient from the PyMongo module.
Next, I will call the ransomwareAlert function with no argument,
since the default is the location of the SMB log file. To check
if this has been done successfully, I will connect to the
database from the command line.
I will instantiate the MongoClient and list the
database names. And as we can see, there is a new database called alerts.
Next, I will also list the collection names for the alerts DB. And notice
the SMB collection. To see an actual document, I will use the function
find_one against the SMB collection. And it works.
We can also use the function count with an empty filter to get
the number of alerts that we have inserted.
Working with databases offers a whole new area of possibilities.