1 00:00:00,160 --> 00:00:02,290 ‫Okay, so now let's talk about another concepts 2 00:00:02,290 --> 00:00:04,650 ‫of CloudWatch Logs which is the Metric Filter. 3 00:00:04,650 --> 00:00:06,770 ‫So CloudWatch Logs can use filter expressions, 4 00:00:06,770 --> 00:00:10,840 ‫for example, to search for a specific IP inside of a log, 5 00:00:10,840 --> 00:00:12,940 ‫or we can use a Metric Filter, 6 00:00:12,940 --> 00:00:14,130 ‫to do something a bit more advance, 7 00:00:14,130 --> 00:00:16,380 ‫such as counting the number of occurrences 8 00:00:16,380 --> 00:00:18,980 ‫of the word "ERROR" in your logs. 9 00:00:18,980 --> 00:00:20,740 ‫And so these metric filters, we can define, 10 00:00:20,740 --> 00:00:22,820 ‫can be used to trigger alarms. 11 00:00:22,820 --> 00:00:23,793 ‫So, this is the whole point of metric filters, 12 00:00:23,793 --> 00:00:25,520 ‫they're a bit more advanced, 13 00:00:25,520 --> 00:00:27,500 ‫they will look for specific occurrences 14 00:00:27,500 --> 00:00:29,210 ‫or counts stuff in your logs, 15 00:00:29,210 --> 00:00:31,280 ‫and if you reach a certain threshold, 16 00:00:31,280 --> 00:00:32,370 ‫then they'll trigger an alarm 17 00:00:32,370 --> 00:00:34,030 ‫and you can react to this. 18 00:00:34,030 --> 00:00:36,130 ‫So, these filters, once you create them, 19 00:00:36,130 --> 00:00:38,660 ‫they do not retroactively filter data. 20 00:00:38,660 --> 00:00:41,060 ‫The filters only publish the metric data for events 21 00:00:41,060 --> 00:00:44,290 ‫that happen after the filter was created. 22 00:00:44,290 --> 00:00:45,300 ‫So let's have a look, 23 00:00:45,300 --> 00:00:48,240 ‫we have the CloudWatch Logs Agents on EC2 Instance 24 00:00:48,240 --> 00:00:50,180 ‫streaming into CloudWatch Logs, 25 00:00:50,180 --> 00:00:53,010 ‫then the CloudWatch Logs will create a Metric Filter, 26 00:00:53,010 --> 00:00:54,750 ‫and the Metric Filter can be defined, for example, 27 00:00:54,750 --> 00:00:57,200 ‫to look at the number of errors in your logs. 28 00:00:57,200 --> 00:01:00,090 ‫Then, if we reach a threshold, we have a CloudWatch alarm 29 00:01:00,090 --> 00:01:02,030 ‫that gets triggered and the CloudWatch alarms, 30 00:01:02,030 --> 00:01:05,350 ‫for example, can send your data into a SNS topic, 31 00:01:05,350 --> 00:01:07,340 ‫from which we can do a lot of animations. 32 00:01:07,340 --> 00:01:10,070 ‫So let's go in the hands-on, to see how this works. 33 00:01:10,070 --> 00:01:12,858 ‫So, I'm in my CloudWatch logs and I want to take 34 00:01:12,858 --> 00:01:14,770 ‫the nginx access logs, 35 00:01:14,770 --> 00:01:18,360 ‫and I want to create a metric filter on these log streams. 36 00:01:18,360 --> 00:01:21,270 ‫So what I'm going to be looking for is to see 37 00:01:21,270 --> 00:01:24,010 ‫if somehow there's error code 400. 38 00:01:24,010 --> 00:01:27,070 ‫So I will type 400, as in here, 39 00:01:27,070 --> 00:01:30,920 ‫and we can see there's a lot of HTTP/1.1" 400 error codes, 40 00:01:30,920 --> 00:01:34,030 ‫so I want to create a metric filter on those and be alerted. 41 00:01:34,030 --> 00:01:36,220 ‫This is just a dummy use case. 42 00:01:36,220 --> 00:01:38,110 ‫So, I'm going to create a metric filter, 43 00:01:38,110 --> 00:01:40,430 ‫I can create from here, or I can go back, 44 00:01:40,430 --> 00:01:43,570 ‫and also go to metric filters in here and create one. 45 00:01:43,570 --> 00:01:45,310 ‫So, I'm going to create a metric filter, 46 00:01:45,310 --> 00:01:46,660 ‫and then you've to enter a pattern. 47 00:01:46,660 --> 00:01:48,410 ‫Now, patterns can be quite complicated, 48 00:01:48,410 --> 00:01:51,270 ‫there's a whole documentation on the Filter and Syntax 49 00:01:51,270 --> 00:01:52,320 ‫for the pattern, 50 00:01:52,320 --> 00:01:54,780 ‫but right now, I'm just going to look for 400 51 00:01:54,780 --> 00:01:56,530 ‫and make it extremely simple. 52 00:01:56,530 --> 00:02:00,000 ‫And then we can send custom log data to test, 53 00:02:00,000 --> 00:02:03,570 ‫or we can just get stuff directly from my logs, 54 00:02:03,570 --> 00:02:05,160 ‫and then test the pattern, 55 00:02:05,160 --> 00:02:05,993 ‫and the result is that, 56 00:02:05,993 --> 00:02:09,670 ‫it found 14 matches out of 50 events in the sample logs. 57 00:02:09,670 --> 00:02:11,240 ‫So that means that, my pattern, 58 00:02:11,240 --> 00:02:13,630 ‫very, very simple, is working fine. 59 00:02:13,630 --> 00:02:15,960 ‫Then, I will scroll down and click on next, 60 00:02:15,960 --> 00:02:19,320 ‫and then I have to give a name to this metric filter, 61 00:02:19,320 --> 00:02:22,433 ‫so, I will say, MetricFilter400Code, okay. 62 00:02:25,630 --> 00:02:28,080 ‫Then we need to give a metric namespace, 63 00:02:28,080 --> 00:02:29,980 ‫so I'll call it MetricFilters, 64 00:02:31,000 --> 00:02:33,620 ‫and then a metric name, MyDemoFilter, 65 00:02:35,510 --> 00:02:37,190 ‫and then the metric value, 66 00:02:37,190 --> 00:02:40,670 ‫so whenever a match occurs, so we can say, for example, 67 00:02:40,670 --> 00:02:44,100 ‫published value number one, okay. 68 00:02:44,100 --> 00:02:46,980 ‫And then, the default value, if no value is published, 69 00:02:46,980 --> 00:02:48,690 ‫is going to be zero. 70 00:02:48,690 --> 00:02:52,250 ‫I click on next, and I create this metric filter. 71 00:02:52,250 --> 00:02:54,730 ‫So now, this Metric Filter has been created. 72 00:02:54,730 --> 00:02:58,070 ‫And so, if I go into my metrics, in here, 73 00:02:58,070 --> 00:03:01,840 ‫I'm able to see, so currently, nothing has been published, 74 00:03:01,840 --> 00:03:05,490 ‫because as I said, the metric filter is not retroactive. 75 00:03:05,490 --> 00:03:07,720 ‫So we need to make this Metric Filter work, 76 00:03:07,720 --> 00:03:09,440 ‫and for this, very simply, 77 00:03:09,440 --> 00:03:12,710 ‫I'm gonna go into MyFirstBeanstalk-environment, 78 00:03:12,710 --> 00:03:15,130 ‫and then I'm going to do an environment action, 79 00:03:15,130 --> 00:03:17,700 ‫and I went to restart the app servers, 80 00:03:17,700 --> 00:03:19,440 ‫and this should trigger a lot more logs 81 00:03:19,440 --> 00:03:22,220 ‫to be written out into CloudWatch logs. 82 00:03:22,220 --> 00:03:24,430 ‫So when I wait, is just go back in here, 83 00:03:24,430 --> 00:03:26,660 ‫and I wait about five minutes for my environment 84 00:03:26,660 --> 00:03:28,190 ‫to be rebuilt, and hopefully, 85 00:03:28,190 --> 00:03:29,700 ‫the metric filter will start showing up 86 00:03:29,700 --> 00:03:31,470 ‫in CloudWatch metrics. 87 00:03:31,470 --> 00:03:33,360 ‫So my environment has now been restarted, 88 00:03:33,360 --> 00:03:35,880 ‫and I'm going to go and open it up as well, 89 00:03:35,880 --> 00:03:37,690 ‫and I'm going to do /test, 90 00:03:37,690 --> 00:03:40,590 ‫just to trigger something, and we're good to go. 91 00:03:40,590 --> 00:03:42,610 ‫Okay, so now let's go back into CloudWatch, 92 00:03:42,610 --> 00:03:44,350 ‫and I'm going to refresh this, 93 00:03:44,350 --> 00:03:45,550 ‫and hopefully, very, very soon, 94 00:03:45,550 --> 00:03:48,120 ‫we should start seeing some metrics. 95 00:03:48,120 --> 00:03:51,630 ‫Okay, so I have now refreshed my CloudWatch metrics page, 96 00:03:51,630 --> 00:03:53,270 ‫and thankfully, what we start seeing, 97 00:03:53,270 --> 00:03:55,940 ‫is a Custom Namespace, called metric filters, 98 00:03:55,940 --> 00:03:57,520 ‫that was the one we created, 99 00:03:57,520 --> 00:03:58,830 ‫and then we create this metric, 100 00:03:58,830 --> 00:04:00,530 ‫and this is MyDemoFilter. 101 00:04:00,530 --> 00:04:02,270 ‫Now, it's not very interesting as a metric, 102 00:04:02,270 --> 00:04:03,800 ‫because the value is zero right now, 103 00:04:03,800 --> 00:04:06,450 ‫so that means we haven't detected any 400 events, 104 00:04:06,450 --> 00:04:07,430 ‫but what I wanna show you is that, 105 00:04:07,430 --> 00:04:10,620 ‫it didn't backfill the data for previous events, 106 00:04:10,620 --> 00:04:14,170 ‫so metric filters only added data as soon as I created. 107 00:04:14,170 --> 00:04:16,070 ‫So, it's not very interesting in this graph, 108 00:04:16,070 --> 00:04:17,030 ‫but that's okay. 109 00:04:17,030 --> 00:04:19,100 ‫Another thing we can do with this metric filter 110 00:04:19,100 --> 00:04:22,660 ‫is to click on it, and then create an alarm, 111 00:04:22,660 --> 00:04:24,310 ‫and so by creating CloudWatch alarm, 112 00:04:24,310 --> 00:04:25,530 ‫we can do some automation, 113 00:04:25,530 --> 00:04:27,930 ‫so I'm just going to create a dummy CloudWatch alarm. 114 00:04:27,930 --> 00:04:31,410 ‫So, I will use MyDemoFilter, currently, there's nothing, 115 00:04:31,410 --> 00:04:34,810 ‫but I can say, okay, if as a static thresholds, 116 00:04:34,810 --> 00:04:38,410 ‫you are greater than, I would say 50, 117 00:04:38,410 --> 00:04:40,000 ‫then something is really, really wrong 118 00:04:40,000 --> 00:04:41,310 ‫with my web application, 119 00:04:41,310 --> 00:04:42,990 ‫and therefore, I'm going to click on next, 120 00:04:42,990 --> 00:04:46,190 ‫and we could say, okay, the alarm should be In alarm, 121 00:04:46,190 --> 00:04:50,340 ‫and I'm going to send my alarm to an existing SMS topic, 122 00:04:50,340 --> 00:04:52,200 ‫maybe this one, maybe another one, 123 00:04:52,200 --> 00:04:54,510 ‫and then, I can say next, 124 00:04:54,510 --> 00:04:57,320 ‫and say, DemoMetricFilterAlarm, 125 00:04:59,840 --> 00:05:00,790 ‫and that's it. 126 00:05:00,790 --> 00:05:02,370 ‫So, now we've created a CloudWatch alarm 127 00:05:02,370 --> 00:05:04,910 ‫on top of our MetricFilter coming from CloudWatch logs, 128 00:05:04,910 --> 00:05:06,500 ‫so you can see, there's a lot of different 129 00:05:06,500 --> 00:05:09,960 ‫CloudWatch services coming together, in this example, 130 00:05:09,960 --> 00:05:11,180 ‫and create the alarm. 131 00:05:11,180 --> 00:05:14,070 ‫And now, I have the basis for my notifications. 132 00:05:14,070 --> 00:05:15,470 ‫Obviously, this won't happen right now, 133 00:05:15,470 --> 00:05:16,910 ‫I won't get any notification, 134 00:05:16,910 --> 00:05:18,480 ‫but you get the general idea, 135 00:05:18,480 --> 00:05:19,650 ‫and this is how you would go ahead 136 00:05:19,650 --> 00:05:21,390 ‫and create your own Metric filters. 137 00:05:21,390 --> 00:05:24,120 ‫And so, if I refreshed this now, this page, 138 00:05:24,120 --> 00:05:26,850 ‫what I should be seeing is on the bottom, 139 00:05:26,850 --> 00:05:30,150 ‫that's yes, this metric filter is linked to an alarm 140 00:05:30,150 --> 00:05:32,490 ‫called DemoMetricFilterAlarm. 141 00:05:32,490 --> 00:05:33,380 ‫So, this is great. 142 00:05:33,380 --> 00:05:34,240 ‫I hope you liked it, 143 00:05:34,240 --> 00:05:36,190 ‫and I will see you in the next lecture.