1
00:00:00,470 --> 00:00:01,303
Okay.

2
00:00:01,303 --> 00:00:04,140
So, now we're going to practice VPC flow logs

3
00:00:04,140 --> 00:00:04,973
and to do so,

4
00:00:04,973 --> 00:00:08,080
we'll go into our demo VPC under flow logs,

5
00:00:08,080 --> 00:00:09,480
and we can create a flow log.

6
00:00:09,480 --> 00:00:11,250
So, we have a few types of flow logs.

7
00:00:11,250 --> 00:00:14,080
So, I'll call this one DemoS3 flow log,

8
00:00:14,080 --> 00:00:15,470
and we have a filter.

9
00:00:15,470 --> 00:00:17,080
Do we want the accept type,

10
00:00:17,080 --> 00:00:19,427
reject type or all kinds of traffic?

11
00:00:19,427 --> 00:00:21,050
So, if you were trying to debug

12
00:00:21,050 --> 00:00:22,387
why some traffic is not going through,

13
00:00:22,387 --> 00:00:23,867
maybe reject is more relevant.

14
00:00:23,867 --> 00:00:26,357
Otherwise, all or accept are good as well.

15
00:00:26,357 --> 00:00:28,660
The maximum aggregation interval,

16
00:00:28,660 --> 00:00:31,020
so, how long you need to wait to aggregate.

17
00:00:31,020 --> 00:00:32,106
And you can look at the info

18
00:00:32,106 --> 00:00:35,880
so you can optionally specify a one minute aggregation,

19
00:00:35,880 --> 00:00:37,280
but if you do so then there's going to be

20
00:00:37,280 --> 00:00:39,670
more records being created because it's possible

21
00:00:39,670 --> 00:00:43,020
that you have lots of records getting created over time.

22
00:00:43,020 --> 00:00:44,799
And obviously if you wanted it written to S3

23
00:00:44,799 --> 00:00:47,950
or CloudWatch logs can be very expensive.

24
00:00:47,950 --> 00:00:50,380
So let's just do one minute just for the sake of the demo

25
00:00:50,380 --> 00:00:51,270
and to go quicker.

26
00:00:51,270 --> 00:00:52,890
But 10 minutes is usually a better option

27
00:00:52,890 --> 00:00:53,723
if you're looking at it,

28
00:00:53,723 --> 00:00:55,758
especially after a little bit of time.

29
00:00:55,758 --> 00:00:56,680
So two options,

30
00:00:56,680 --> 00:00:58,849
we can send it to CloudWatch logs or to Amazon S3.

31
00:00:58,849 --> 00:01:00,017
So for Amazon S3,

32
00:01:00,017 --> 00:01:02,400
we need to specify our bucket ARN and for CloudWatch logs,

33
00:01:02,400 --> 00:01:04,230
we need to specify a log group.

34
00:01:04,230 --> 00:01:06,430
So, let's start with Amazon S3.

35
00:01:06,430 --> 00:01:10,210
So, to do so I'm going to go into the S3 service,

36
00:01:10,210 --> 00:01:11,285
in here,

37
00:01:11,285 --> 00:01:13,560
and it will create a bucket.

38
00:01:13,560 --> 00:01:18,230
And I'll call it demo Stephane VPC flow logs V2, okay.

39
00:01:18,230 --> 00:01:21,137
In the same region where my VPC is, obviously,

40
00:01:21,137 --> 00:01:23,453
and then I will create these buckets.

41
00:01:24,470 --> 00:01:26,320
Now view the details of the bucket,

42
00:01:26,320 --> 00:01:28,260
and now we need to get the buckets ARN.

43
00:01:28,260 --> 00:01:29,760
So, to do so here it is,

44
00:01:29,760 --> 00:01:31,373
I can just copy it from the properties.

45
00:01:31,373 --> 00:01:32,653
So, let's play this in.

46
00:01:32,653 --> 00:01:34,160
And as we can see,

47
00:01:34,160 --> 00:01:36,230
a resource based policy will be created for you

48
00:01:36,230 --> 00:01:37,400
and attached to the target buckets.

49
00:01:37,400 --> 00:01:39,425
So, a bucket policy will be created for us automatically so

50
00:01:39,425 --> 00:01:43,094
that the VPC service can send data into our S3 buckets.

51
00:01:43,094 --> 00:01:45,100
Now, the format looks like this.

52
00:01:45,100 --> 00:01:46,886
This is the default format of AWS,

53
00:01:46,886 --> 00:01:50,782
and next I will click on create a flow log.

54
00:01:50,782 --> 00:01:53,920
Okay? So, this first log has been created.

55
00:01:53,920 --> 00:01:54,930
Perfect.

56
00:01:54,930 --> 00:01:57,140
And I will go ahead and create a second flow log now.

57
00:01:57,140 --> 00:01:59,210
So, I'll click on create flow log.

58
00:01:59,210 --> 00:02:04,210
And this one is going to be demo flow log CloudWatch logs,

59
00:02:05,750 --> 00:02:06,680
all kinds of traffic,

60
00:02:06,680 --> 00:02:07,513
one minute interval,

61
00:02:07,513 --> 00:02:08,910
to CloudWatch logs,

62
00:02:08,910 --> 00:02:12,260
and you need to create a log group and an IAM role.

63
00:02:12,260 --> 00:02:15,190
Okay. So, let's do both of these things.

64
00:02:15,190 --> 00:02:18,963
So, to do so we need to click on setup permissions,

65
00:02:20,704 --> 00:02:24,040
and this is going to create for us a flow logs role, okay.

66
00:02:24,040 --> 00:02:25,380
We're going to allow this

67
00:02:28,551 --> 00:02:30,220
and now this role is created.

68
00:02:30,220 --> 00:02:33,157
So, if I refresh this and look for flow log roles,

69
00:02:33,157 --> 00:02:34,270
here we go.

70
00:02:34,270 --> 00:02:36,360
It is created, and now we need a log group.

71
00:02:36,360 --> 00:02:41,230
So, let's go into the CloudWatch logs console.

72
00:02:41,230 --> 00:02:43,623
So, just click on it here.

73
00:02:46,560 --> 00:02:48,500
And under logs, log groups,

74
00:02:48,500 --> 00:02:50,278
I'm going to create a log group,

75
00:02:50,278 --> 00:02:53,086
I'll call it VPC flow logs,

76
00:02:53,086 --> 00:02:56,294
and I will set the retention to one day

77
00:02:56,294 --> 00:02:57,823
and click on create.

78
00:02:59,370 --> 00:03:02,320
Click on this log group here. Now refresh this.

79
00:03:02,320 --> 00:03:04,130
And the VPC flow logs is appearing.

80
00:03:04,130 --> 00:03:04,963
So, we're good to go.

81
00:03:04,963 --> 00:03:07,540
Now let's create this flow log,

82
00:03:07,540 --> 00:03:08,373
and yet again,

83
00:03:08,373 --> 00:03:09,420
so, now we have two flow logs.

84
00:03:09,420 --> 00:03:10,710
One flowing into Amazon S3

85
00:03:10,710 --> 00:03:13,147
and one flowing into CloudWatch logs.

86
00:03:13,147 --> 00:03:15,510
So, into Amazon S3 and they're my objects.

87
00:03:15,510 --> 00:03:17,190
I can refresh this.

88
00:03:17,190 --> 00:03:19,987
And as we can see some AWS logs have already been created,

89
00:03:19,987 --> 00:03:21,067
I can click on it.

90
00:03:21,067 --> 00:03:23,787
Look at the VPC flow logs for EU central one.

91
00:03:23,787 --> 00:03:25,724
And then we have a timestamp

92
00:03:25,724 --> 00:03:29,846
and we have here the VPC flow logs available to us.

93
00:03:29,846 --> 00:03:32,160
So, now let's go into the CloudWatch logs

94
00:03:32,160 --> 00:03:33,430
and refresh this.

95
00:03:33,430 --> 00:03:34,806
And I have two log streams.

96
00:03:34,806 --> 00:03:36,500
And these log streams corresponds

97
00:03:36,500 --> 00:03:38,600
to the ENI's within my accounts.

98
00:03:38,600 --> 00:03:41,846
So if we have a look at the ENI of my Bastion host,

99
00:03:41,846 --> 00:03:42,915
this one,

100
00:03:42,915 --> 00:03:45,830
and we have a look at the ENI number.

101
00:03:45,830 --> 00:03:47,537
So, we go under networking.

102
00:03:47,537 --> 00:03:50,267
And what I'm going to do is pull this up a little bit

103
00:03:50,267 --> 00:03:52,140
and look for the ENI ID.

104
00:03:52,140 --> 00:03:55,700
So, the ENI ID is 0835.

105
00:03:55,700 --> 00:03:59,570
So, let's go into here and look for a 0835,

106
00:03:59,570 --> 00:04:00,640
This one, perfect.

107
00:04:00,640 --> 00:04:02,380
So, this is showing us the kind of traffic

108
00:04:02,380 --> 00:04:05,100
that is happening on to my EC2 instance.

109
00:04:05,100 --> 00:04:05,933
And if you look at it,

110
00:04:05,933 --> 00:04:07,540
well, it seems that some people

111
00:04:07,540 --> 00:04:09,812
are trying to access my EC2 instance,

112
00:04:09,812 --> 00:04:11,800
but it's getting rejected.

113
00:04:11,800 --> 00:04:13,060
So, this is the version.

114
00:04:13,060 --> 00:04:14,410
This is like all these fields, okay.

115
00:04:14,410 --> 00:04:16,420
The ENI ID and this IP address,

116
00:04:16,420 --> 00:04:17,470
which is a public IP address,

117
00:04:17,470 --> 00:04:19,485
is trying to access my EC2 instance.

118
00:04:19,485 --> 00:04:21,730
And this is really like this kind of attackers

119
00:04:21,730 --> 00:04:22,880
were like attacking your instance

120
00:04:22,880 --> 00:04:25,470
and just scanning the web for holes and so on.

121
00:04:25,470 --> 00:04:27,652
So, you're going to have a lot of this kind of traffic.

122
00:04:27,652 --> 00:04:29,754
And, you know, if I want to get some defense,

123
00:04:29,754 --> 00:04:30,852
I could, for example,

124
00:04:30,852 --> 00:04:33,954
if this IP address was annoying me too much,

125
00:04:33,954 --> 00:04:37,092
I could block it at the Necole level, for example,

126
00:04:37,092 --> 00:04:38,690
and not have this traffic,

127
00:04:38,690 --> 00:04:39,650
but as we can see,

128
00:04:39,650 --> 00:04:41,664
a lot of traffic is appearing here

129
00:04:41,664 --> 00:04:43,150
and it's all rejected.

130
00:04:43,150 --> 00:04:45,850
But if I were to do some activity on my students instance,

131
00:04:45,850 --> 00:04:46,830
connect to Google, for example,

132
00:04:46,830 --> 00:04:49,125
we would see some traffic with accept as well.

133
00:04:49,125 --> 00:04:54,125
And the same kind of data is going to appear on Amazon S3.

134
00:04:54,150 --> 00:04:56,960
So, if I go here, you will have a lot of data, okay?

135
00:04:56,960 --> 00:04:59,030
So, if you use CloudWatch events,

136
00:04:59,030 --> 00:05:00,100
I mean CloudWatch logs, sorry.

137
00:05:00,100 --> 00:05:01,781
If you want to have like some kind of metrics filter

138
00:05:01,781 --> 00:05:03,770
to do some like real-time analysis

139
00:05:03,770 --> 00:05:04,770
in case you're being attacked,

140
00:05:04,770 --> 00:05:06,370
a lot of traffic is being rejected and so on.

141
00:05:06,370 --> 00:05:07,762
So, this is very helpful.

142
00:05:07,762 --> 00:05:09,213
And you use Amazon S3.

143
00:05:09,213 --> 00:05:11,952
If you wanted to do some more bigger analysis,

144
00:05:11,952 --> 00:05:14,162
for example, using Athena.

145
00:05:14,162 --> 00:05:16,802
So, let's go ahead and practice using Athena

146
00:05:16,802 --> 00:05:19,240
to query this data into my S3 buckets.

147
00:05:19,240 --> 00:05:20,453
So, in Athena,

148
00:05:20,453 --> 00:05:23,210
what I'm going to do is to first set up

149
00:05:23,210 --> 00:05:25,413
a query result location in Amazon S3.

150
00:05:25,413 --> 00:05:27,642
So, I'm going to click on view settings,

151
00:05:27,642 --> 00:05:28,602
manage,

152
00:05:28,602 --> 00:05:31,450
and then I need to specify an S3 bucket

153
00:05:31,450 --> 00:05:33,450
where I'm going to store my results.

154
00:05:33,450 --> 00:05:35,029
So, back into Amazon S3,

155
00:05:35,029 --> 00:05:37,790
when I'm going to do is just quickly create an S3 bucket

156
00:05:37,790 --> 00:05:39,140
because this wasn't set up for this account.

157
00:05:39,140 --> 00:05:43,838
So, I'll call this one demo Athena Stefan V2

158
00:05:43,838 --> 00:05:45,693
and this should work.

159
00:05:45,693 --> 00:05:47,743
Now, let's click on create these buckets,

160
00:05:48,977 --> 00:05:50,900
view details.

161
00:05:50,900 --> 00:05:53,110
And I can just use this as the bucket's properties,

162
00:05:53,110 --> 00:05:54,563
I will copy the ARN,

163
00:05:55,930 --> 00:05:58,740
back into Athena and then click on save.

164
00:05:58,740 --> 00:06:02,580
So, I need to just do slash and then Athena,

165
00:06:02,580 --> 00:06:07,070
and then something like, S3 colon slash slash,

166
00:06:07,070 --> 00:06:08,180
here we go.

167
00:06:08,180 --> 00:06:09,013
Okay, this is good.

168
00:06:09,013 --> 00:06:11,180
So, let's click on save.

169
00:06:11,180 --> 00:06:13,682
And now we have this query location being saved,

170
00:06:13,682 --> 00:06:16,990
and I need to go ahead and start creating a database

171
00:06:16,990 --> 00:06:18,120
and then create some data.

172
00:06:18,120 --> 00:06:20,690
So, what I need to do is to go in here

173
00:06:20,690 --> 00:06:24,613
and I will type AWS VPC flow logs,

174
00:06:24,613 --> 00:06:26,490
Athena.

175
00:06:26,490 --> 00:06:29,772
And it's going to take me into a tutorial on how to do it.

176
00:06:29,772 --> 00:06:32,515
So, to do so we need to create a table.

177
00:06:32,515 --> 00:06:35,580
And this whole statement right here shows you how

178
00:06:35,580 --> 00:06:40,490
so, let's go ahead and paste this in my Athena UI.

179
00:06:40,490 --> 00:06:41,327
So, here we go.

180
00:06:41,327 --> 00:06:45,030
And we need to specify where the data is.

181
00:06:45,030 --> 00:06:46,184
So, the log bucket,

182
00:06:46,184 --> 00:06:47,670
the prefixed logs,

183
00:06:47,670 --> 00:06:49,784
and an account ID VPC flow logs and region code.

184
00:06:49,784 --> 00:06:51,300
So, let's find it.

185
00:06:51,300 --> 00:06:52,875
So, let's go back in here.

186
00:06:52,875 --> 00:06:54,403
This is my Athena bucket.

187
00:06:55,386 --> 00:06:57,480
This is my VPC flow logs bucket.

188
00:06:57,480 --> 00:06:58,313
Here we go.

189
00:06:58,313 --> 00:06:59,780
So, let's go to the top of it just to go back.

190
00:06:59,780 --> 00:07:03,517
So, logs, account ID, VPC flow logs, region.

191
00:07:03,517 --> 00:07:04,845
Okay. We got it.

192
00:07:04,845 --> 00:07:08,080
Now we're going to copy this S3 URI

193
00:07:08,080 --> 00:07:10,575
directly back into Athena,

194
00:07:10,575 --> 00:07:13,560
and I'm going to paste this in.

195
00:07:13,560 --> 00:07:14,455
Okay, perfect.

196
00:07:14,455 --> 00:07:17,173
And let's run it, the statements.

197
00:07:18,920 --> 00:07:20,030
So, this is completed,

198
00:07:20,030 --> 00:07:23,670
and now we have a VPC flow log table in here.

199
00:07:23,670 --> 00:07:25,480
Okay, and it is partitioned.

200
00:07:25,480 --> 00:07:28,600
So, we can see we have all the information in here.

201
00:07:28,600 --> 00:07:30,040
Now, the second thing we need to do

202
00:07:30,040 --> 00:07:31,220
is to run another statement.

203
00:07:31,220 --> 00:07:36,010
So, let's go here

204
00:07:36,010 --> 00:07:38,017
and find this alter table, okay.

205
00:07:38,017 --> 00:07:40,860
To create partitions, to read this data.

206
00:07:40,860 --> 00:07:42,763
So, let's copy this,

207
00:07:44,640 --> 00:07:45,950
paste this in, and again

208
00:07:45,950 --> 00:07:49,890
we need to specify a date as well as

209
00:07:49,890 --> 00:07:54,278
the entire replacement for month, year and day.

210
00:07:54,278 --> 00:07:56,100
Okay. So, let's go ahead and do this.

211
00:07:56,100 --> 00:07:59,203
So, back into Amazon S3,

212
00:08:00,540 --> 00:08:01,373
into my objects,

213
00:08:01,373 --> 00:08:03,787
I will click, click and click,

214
00:08:03,787 --> 00:08:06,310
go back to properties and paste this,

215
00:08:06,310 --> 00:08:09,690
copy and paste this in.

216
00:08:09,690 --> 00:08:10,930
So, we are good to go.

217
00:08:10,930 --> 00:08:12,320
And then we need to add a partition.

218
00:08:12,320 --> 00:08:14,310
So 2021,

219
00:08:14,310 --> 00:08:15,597
10, 06,

220
00:08:15,597 --> 00:08:16,990
and this is quite manual,

221
00:08:16,990 --> 00:08:19,920
but a glue can help with automating this,

222
00:08:19,920 --> 00:08:22,050
not as friendly statements

223
00:08:22,050 --> 00:08:23,590
and the statement was successfully completed.

224
00:08:23,590 --> 00:08:25,841
So, now we have added one partition into our table,

225
00:08:25,841 --> 00:08:26,910
and then finally,

226
00:08:26,910 --> 00:08:30,170
we can query the flow logs just by doing a select.

227
00:08:30,170 --> 00:08:31,810
So, let's do this

228
00:08:32,996 --> 00:08:35,320
to find all the reject traffic

229
00:08:36,299 --> 00:08:37,982
and then click on run.

230
00:08:40,299 --> 00:08:43,409
And this has worked and now we have 46 results

231
00:08:43,409 --> 00:08:44,800
and we can have a look so,

232
00:08:44,800 --> 00:08:47,958
day, dates, interface ID, source address,

233
00:08:47,958 --> 00:08:50,810
action, rejects, protocol, six.

234
00:08:50,810 --> 00:08:52,307
So, we can get a lot of information from Athena

235
00:08:52,307 --> 00:08:53,990
and we can start doing some complex queries

236
00:08:53,990 --> 00:08:56,101
to try to group, for example, by IP addresses.

237
00:08:56,101 --> 00:08:57,851
See who is attacking us the most,

238
00:08:57,851 --> 00:08:59,720
where we are attacked the most, and so on.

239
00:08:59,720 --> 00:09:00,590
So, this could be quite helpful

240
00:09:00,590 --> 00:09:03,020
for doing some batch analysis, okay.

241
00:09:03,020 --> 00:09:04,990
But that's it, we've seen how to set up VPC flow logs.

242
00:09:04,990 --> 00:09:07,360
We sent you CloudWatch logs into Amazon S3.

243
00:09:07,360 --> 00:09:09,650
We've used Athena to query these logs in AmazonS3.

244
00:09:09,650 --> 00:09:10,711
So, very complete demo.

245
00:09:10,711 --> 00:09:13,320
And what I'm going to do now is just disable these logs.

246
00:09:13,320 --> 00:09:15,330
I'm going to delete them,

247
00:09:15,330 --> 00:09:17,310
just not to run any ongoing cost.

248
00:09:17,310 --> 00:09:18,660
Okay. So, that's it.

249
00:09:18,660 --> 00:09:19,500
I hope you liked it.

250
00:09:19,500 --> 00:09:21,450
And I will see you in the next lecture.