Before we get into a deeper discussion about those components,
you should learn about the formats of the Zeek logs
and how we can manipulate those.
When we're talking about Zeek logs,
there are two main formats to think of: the Zeek TSV format,
and the JSON format.
So let's talk about these two formats and break them down.
The Zeek TSV format is known as the tab‑separated format,
and is what we saw in the previous module with all of the jumbled,
difficult‑to‑read information from the output of the cat command.
This format is the default format for the Zeek logs.
We have to essentially remember which column and field
name correspond to which actual value.
This makes it difficult to analyze the logs and
extract the needed values from them.
The other type, JSON,
is great for readability and support with SIMs and extracting specific data.
JSON uses key value pairs for the data to make this possible.
This format may cost us a little bit extra in storage,
but it's typically the preferred method for analysis.
To address the readability issue with the TSV logs,
we have some tools that are available to help us in order to
interact with and filter out the data.
We can use a tool like awk to give us a cleaner view of
just the information that we want.
This is great if we're trying to inspect individual logs that
we know the information we want to see for,
or if we're not expecting big changes in the platform.
But using this tool and method will require us to modify the setup,
if we modify the log output,
because we need to specify the fields that we want to see by a location.
More on that in the coming demo.
The other tool that's more preferred to use with Zeek TSV logs is zeek‑cut.
Looking at some of the popular uses of the zeek‑cut tool,
we can do many things like extract the specific fields that we want,
and normalize the timestamps.
This tool is easy and gives us just the items that we want.
The final tool that I want to discuss is a common JSON
parser written by Stephen Dolan.
It's called jq.
If you're familiar with this tool,
then you know how easy it is to use to display JSON logs in
its pretty‑print format for readability.
We can use it to filter out and parse logs,
too,
which is beneficial if we're trying to focus on specific
information or sending the logs to other applications and
need to customize the format.
Let's hop into the lab again and look at these
three tools and see them in action.
Alright, and here we are in the lab.
The first thing that I'm going to do is just show you what directory we're in.
So we're in the local site directory for Zeek,
and so let's just go over to our log directory so we can see
the current logs that are being generated.
And as you can see, we have quite a few logs here.
Let's just take one of these for example, so the DNS log, let's say.
As you can see,
we have the logs that are very difficult to read;
they're in the Zeek TSV format here.
So we want to try to clean it up a little bit first.
So I'm going to change the command and use more so we can
actually see the columns at the top of this.
And as you can see here, we have things like the timestamp,
we have the originating host, the originating port,
responding host, and responding port protocol,
so on and so forth.
And so each one of these is a separate number when we're
referring to the columns using awk.
Before we go ahead and actually look at this log through awk,
let's look at the man page real quick.
So we have a couple commands that we can easily use for
this to help us clean up the logs, or at least clean up the print view for it.
And so we can designate the program file,
we can designate showing the characters as bytes,
and we have a whole bunch of options here.
For our purposes today though, we're just going to use awk in its simple format,
and we're going to use a little bit of Regex with it.
So let's go ahead and do the awk command,
and we're going to go ahead and type in some Regex strings
to help us filter on some of the data.
Then we're going to have to designate which columns that we want to print.
And so this is designated by the dollar sign and the column number.
So for example, if I wanted to print just the first column here,
I would just do a 1 right here,
and then I would close it out and then designate which log.
And as you can see, since timestamp was the first column here,
Zeek only printed that timestamp column for us,
and so we could repeat this command and do the same for our other columns.
Now we have the source and destination IP addresses or the
originating and responding host IP addresses,
and the destination port numbers.
So this is a really cool tool to be able to filter out the logs.
Now, another tool that we have that's very similar is going to be zeek‑cut,
and so we do a very similar thing when we're talking about using zeek‑cut,
and we identify the columns that we need,
but really what we're doing is instead of
identifying the columns in the TSV format,
we're identifying the fields that we want to see instead,
so we don't have to map field names to numbers anymore.
We could just designate it by the actual field names at the top.
So for example, if we did zeek‑cut id.orig_h and then the dns.log,
it would just give us that IP address.
And so we can add additional ones to this as well.
So we could put the timestamp,
we could put the originating host or the originating port,
and all of these different fields that we want to see
are separated by just a simple space.
And if you take a look at that,
we have our timestamp with the originating host with the originating ports,
so that ephemeral port,
the responding host, and the destination port or the responding port.
Alright, one more that we have to look at real quick.
I'm going to jump back over to our previous working
directory, so the local site directory,
and we're going to go ahead and we're going to cat the con.log,
which is just the connection log, right?
And so these are all in TSV format, right?
What we want to do now is edit our local.zeek file,
so that local Zeek script that we'll talk about later on,
and we want to load a specific tuning script for ourselves.
That way, we have the Zeek logging framework logged to JSON format only.
And now that that's loaded, all we have to do is re‑deploy Zeek,
so we're going to use zeekctl to deploy it.
And now if we go up and look at the connection log,
we should start getting this log coming through in JSON format.
We don't have anything here yet.
Also, since we have our signature framework loaded,
we should be able to detect a Telnet packet as well.
And let's just double check that, using the nano command again,
and make sure we have our actual Telnet signature loaded,
and we do not, so let's do that again.
Let's load it.
Alright, now with all of that said,
we can go ahead and try to Telnet to somewhere,
so it can trigger log generation into the notice framework.
Alright, and as you can see, we have a notice.log.
So let's cat that notice.log, and look at that; the
format is a little bit different.
We can also do the same thing for the connection log as well,
and see how it looks different.
And so now that we have our logs coming in in JSON format,
we can use the jq tool to pretty print this.
So taking the connection log, for example,
we can just use the command jq and then period,
and then the log path, and there you go.
We have the field value pairs aligned, and it's in an easier format to read,
as opposed to that previous one, and as opposed to the Zeek TSV logs.