Okay.
So, now we're going to practice VPC flow logs
and to do so,
we'll go into our demo VPC under flow logs,
and we can create a flow log.
So, we have a few types of flow logs.
So, I'll call this one DemoS3 flow log,
and we have a filter.
Do we want the accept type,
reject type or all kinds of traffic?
So, if you were trying to debug
why some traffic is not going through,
maybe reject is more relevant.
Otherwise, all or accept are good as well.
The maximum aggregation interval,
so, how long you need to wait to aggregate.
And you can look at the info
so you can optionally specify a one minute aggregation,
but if you do so then there's going to be
more records being created because it's possible
that you have lots of records getting created over time.
And obviously if you wanted it written to S3
or CloudWatch logs can be very expensive.
So let's just do one minute just for the sake of the demo
and to go quicker.
But 10 minutes is usually a better option
if you're looking at it,
especially after a little bit of time.
So two options,
we can send it to CloudWatch logs or to Amazon S3.
So for Amazon S3,
we need to specify our bucket ARN and for CloudWatch logs,
we need to specify a log group.
So, let's start with Amazon S3.
So, to do so I'm going to go into the S3 service,
in here,
and it will create a bucket.
And I'll call it demo Stephane VPC flow logs V2, okay.
In the same region where my VPC is, obviously,
and then I will create these buckets.
Now view the details of the bucket,
and now we need to get the buckets ARN.
So, to do so here it is,
I can just copy it from the properties.
So, let's play this in.
And as we can see,
a resource based policy will be created for you
and attached to the target buckets.
So, a bucket policy will be created for us automatically so
that the VPC service can send data into our S3 buckets.
Now, the format looks like this.
This is the default format of AWS,
and next I will click on create a flow log.
Okay? So, this first log has been created.
Perfect.
And I will go ahead and create a second flow log now.
So, I'll click on create flow log.
And this one is going to be demo flow log CloudWatch logs,
all kinds of traffic,
one minute interval,
to CloudWatch logs,
and you need to create a log group and an IAM role.
Okay. So, let's do both of these things.
So, to do so we need to click on setup permissions,
and this is going to create for us a flow logs role, okay.
We're going to allow this
and now this role is created.
So, if I refresh this and look for flow log roles,
here we go.
It is created, and now we need a log group.
So, let's go into the CloudWatch logs console.
So, just click on it here.
And under logs, log groups,
I'm going to create a log group,
I'll call it VPC flow logs,
and I will set the retention to one day
and click on create.
Click on this log group here. Now refresh this.
And the VPC flow logs is appearing.
So, we're good to go.
Now let's create this flow log,
and yet again,
so, now we have two flow logs.
One flowing into Amazon S3
and one flowing into CloudWatch logs.
So, into Amazon S3 and they're my objects.
I can refresh this.
And as we can see some AWS logs have already been created,
I can click on it.
Look at the VPC flow logs for EU central one.
And then we have a timestamp
and we have here the VPC flow logs available to us.
So, now let's go into the CloudWatch logs
and refresh this.
And I have two log streams.
And these log streams corresponds
to the ENI's within my accounts.
So if we have a look at the ENI of my Bastion host,
this one,
and we have a look at the ENI number.
So, we go under networking.
And what I'm going to do is pull this up a little bit
and look for the ENI ID.
So, the ENI ID is 0835.
So, let's go into here and look for a 0835,
This one, perfect.
So, this is showing us the kind of traffic
that is happening on to my EC2 instance.
And if you look at it,
well, it seems that some people
are trying to access my EC2 instance,
but it's getting rejected.
So, this is the version.
This is like all these fields, okay.
The ENI ID and this IP address,
which is a public IP address,
is trying to access my EC2 instance.
And this is really like this kind of attackers
were like attacking your instance
and just scanning the web for holes and so on.
So, you're going to have a lot of this kind of traffic.
And, you know, if I want to get some defense,
I could, for example,
if this IP address was annoying me too much,
I could block it at the Necole level, for example,
and not have this traffic,
but as we can see,
a lot of traffic is appearing here
and it's all rejected.
But if I were to do some activity on my students instance,
connect to Google, for example,
we would see some traffic with accept as well.
And the same kind of data is going to appear on Amazon S3.
So, if I go here, you will have a lot of data, okay?
So, if you use CloudWatch events,
I mean CloudWatch logs, sorry.
If you want to have like some kind of metrics filter
to do some like real-time analysis
in case you're being attacked,
a lot of traffic is being rejected and so on.
So, this is very helpful.
And you use Amazon S3.
If you wanted to do some more bigger analysis,
for example, using Athena.
So, let's go ahead and practice using Athena
to query this data into my S3 buckets.
So, in Athena,
what I'm going to do is to first set up
a query result location in Amazon S3.
So, I'm going to click on view settings,
manage,
and then I need to specify an S3 bucket
where I'm going to store my results.
So, back into Amazon S3,
when I'm going to do is just quickly create an S3 bucket
because this wasn't set up for this account.
So, I'll call this one demo Athena Stefan V2
and this should work.
Now, let's click on create these buckets,
view details.
And I can just use this as the bucket's properties,
I will copy the ARN,
back into Athena and then click on save.
So, I need to just do slash and then Athena,
and then something like, S3 colon slash slash,
here we go.
Okay, this is good.
So, let's click on save.
And now we have this query location being saved,
and I need to go ahead and start creating a database
and then create some data.
So, what I need to do is to go in here
and I will type AWS VPC flow logs,
Athena.
And it's going to take me into a tutorial on how to do it.
So, to do so we need to create a table.
And this whole statement right here shows you how
so, let's go ahead and paste this in my Athena UI.
So, here we go.
And we need to specify where the data is.
So, the log bucket,
the prefixed logs,
and an account ID VPC flow logs and region code.
So, let's find it.
So, let's go back in here.
This is my Athena bucket.
This is my VPC flow logs bucket.
Here we go.
So, let's go to the top of it just to go back.
So, logs, account ID, VPC flow logs, region.
Okay. We got it.
Now we're going to copy this S3 URI
directly back into Athena,
and I'm going to paste this in.
Okay, perfect.
And let's run it, the statements.
So, this is completed,
and now we have a VPC flow log table in here.
Okay, and it is partitioned.
So, we can see we have all the information in here.
Now, the second thing we need to do
is to run another statement.
So, let's go here
and find this alter table, okay.
To create partitions, to read this data.
So, let's copy this,
paste this in, and again
we need to specify a date as well as
the entire replacement for month, year and day.
Okay. So, let's go ahead and do this.
So, back into Amazon S3,
into my objects,
I will click, click and click,
go back to properties and paste this,
copy and paste this in.
So, we are good to go.
And then we need to add a partition.
So 2021,
10, 06,
and this is quite manual,
but a glue can help with automating this,
not as friendly statements
and the statement was successfully completed.
So, now we have added one partition into our table,
and then finally,
we can query the flow logs just by doing a select.
So, let's do this
to find all the reject traffic
and then click on run.
And this has worked and now we have 46 results
and we can have a look so,
day, dates, interface ID, source address,
action, rejects, protocol, six.
So, we can get a lot of information from Athena
and we can start doing some complex queries
to try to group, for example, by IP addresses.
See who is attacking us the most,
where we are attacked the most, and so on.
So, this could be quite helpful
for doing some batch analysis, okay.
But that's it, we've seen how to set up VPC flow logs.
We sent you CloudWatch logs into Amazon S3.
We've used Athena to query these logs in AmazonS3.
So, very complete demo.
And what I'm going to do now is just disable these logs.
I'm going to delete them,
just not to run any ongoing cost.
Okay. So, that's it.
I hope you liked it.
And I will see you in the next lecture.