Getting into some more complex solutions,
the Elastic Stack is a very popular framework.
Initially, Elasticsearch was designed for fast and distributed search.
Since distributed document indexing and search is a
foundational challenge for many topics,
it has been successfully adopted for log analytics.
There is a Python client available to interact with the
cluster directly from the script,
and the name of the Python module is Elasticsearch.
A small preview of the important functions that are used to
interact with the Elasticsearch index,
the connector can be used to specify the connection
parameters as well as credentials.
In this demo,
I will spin up a single‑node Elasticsearch instance and Kibana using Docker.
Then from Python,
using the Elasticsearch module, we'll ship off logs that
have been enriched with your location data.
This will allow us to build a map visualization in Kibana using the
geolocation data to build a heat map of the events,
but first, I will need an elastic instance to send data to.
I will make use of Docker Compose to set up a single‑node
instance of Elasticsearch and Kibana.
I will use the command docker‑compose up and ‑d for detached mode.
We have to wait a few seconds as the images are downloaded and built.
Now that this is done,
let's open up a browser and see if we can access Kibana on port 5601.
Let's press Try again, maybe Kibana it's still starting and it works.
I'll click Configure elastic and continue to Kibana.
We don't need any integrations for this upcoming demo.
Now let's return back to VS Code and install the Elasticsearch client.
This can be done using pip.
Next, I'll run Python, import the Elasticsearch module,
establish a connection, and run es.info,
and we notice that everything works perfectly.
Now let's implement this in the script.
I will import the Elasticsearch module as well,
then create a new function called exportSshActivity.
This will take one argument, which is the path to the auth logs.
I will establish the connection, then open the log file,
and iterate through all of the entries.
After parsing the log entry and extracting the individual fields,
I will then index a document into Elasticsearch.
I will use the index function.
I have to specify the index, this will be called auth,
and for document, specify log‑data and that's it.
Let's test this in the terminal and see if it works.
I will import the exportSshActivity function and execute it.
Now to test if it was successful,
I will switch to Kibana and go to the Discover section.
Create a data view, type in the auth index,
and specify the timestamp field as being TS.
The reason we don't see any records is because the
default search is for the last 15 minutes.
We need to change this.
Let's zoom in a bit, and we can see our data.
Now it's time for something interesting.
In a previous module,
we have enriched these logs with the country code based on the originating IP.
How about we plot this on a map?
We'll go to the Maps section, add the layer,
choose choropleth, select world countries as boundaries,
select the auth index for the data view,
and the country.keyword field, and Add layer.
Give this a name and save it.
Wow, this is really interesting, and this is fully interactive as well.
We can run queries against the index and see how it reflects on the map.
Let's have a look at only the accepted connections,
and we can see that only two connections were successful coming from the US.
The possibilities with Python are limitless at this point.