1 00:00:00,500 --> 00:00:03,330 ‫Okay, so now let's practice event mappers 2 00:00:03,330 --> 00:00:05,950 ‫and for this we'll just author a new function from scratch 3 00:00:05,950 --> 00:00:08,890 ‫and the invent mapper we'll be using is SQS 4 00:00:08,890 --> 00:00:12,740 ‫so I will name my function lambda-SQS, python 3.8 5 00:00:13,590 --> 00:00:16,150 ‫and I will go ahead and create my function. 6 00:00:16,150 --> 00:00:18,270 ‫Also, we need to create an SQS queue for this 7 00:00:18,270 --> 00:00:21,070 ‫for my lambda function to be able to retrieve 8 00:00:21,070 --> 00:00:23,390 ‫the messages from our SQS queue. 9 00:00:23,390 --> 00:00:25,640 ‫So I'm just gonna to create a queue 10 00:00:25,640 --> 00:00:30,640 ‫and I will name my queue lambda-demo-SQS. 11 00:00:30,880 --> 00:00:32,990 ‫Okay. So this is a standard queue 12 00:00:32,990 --> 00:00:35,650 ‫and I will just scroll all the way down 13 00:00:35,650 --> 00:00:38,040 ‫and create this queue. 14 00:00:38,040 --> 00:00:40,160 ‫Okay. So my queue is created 15 00:00:40,160 --> 00:00:41,710 ‫and my lambda function is created as well 16 00:00:41,710 --> 00:00:43,610 ‫so now for my lambda function to 17 00:00:43,610 --> 00:00:45,780 ‫retrieve messages from our SQS queue, 18 00:00:45,780 --> 00:00:48,940 ‫what I need to do is to click on add trigger. 19 00:00:48,940 --> 00:00:50,293 ‫So add a trigger and here we have 20 00:00:50,293 --> 00:00:53,020 ‫all the options of triggers within, 21 00:00:53,020 --> 00:00:54,800 ‫for our lambda function so as you can see, 22 00:00:54,800 --> 00:00:57,690 ‫there's a lot of AWS type of integrations, 23 00:00:57,690 --> 00:01:00,190 ‫but also some partner events sources 24 00:01:00,190 --> 00:01:02,970 ‫that you can get directly into your lambda function, 25 00:01:02,970 --> 00:01:04,740 ‫but so for this example, 26 00:01:04,740 --> 00:01:08,150 ‫what I'm going to do is go into SQS, 27 00:01:08,150 --> 00:01:10,510 ‫which happens to be right here. 28 00:01:10,510 --> 00:01:12,740 ‫Okay. So we need to choose an SQS queue 29 00:01:12,740 --> 00:01:16,580 ‫so I will choose the lambda-demo-SQS 30 00:01:16,580 --> 00:01:18,670 ‫and next we have to specify the batch size. 31 00:01:18,670 --> 00:01:19,860 ‫So how many messages we want to 32 00:01:19,860 --> 00:01:22,360 ‫receive as part of a single batch. 33 00:01:22,360 --> 00:01:26,670 ‫And you can go anywhere from one message, 34 00:01:26,670 --> 00:01:28,920 ‫all the way up to maybe a bigger batch size. 35 00:01:28,920 --> 00:01:31,160 ‫Okay? And you need to look into the documentation 36 00:01:31,160 --> 00:01:33,590 ‫to see what is the biggest batch size. 37 00:01:33,590 --> 00:01:34,580 ‫I thought it used to be 10, 38 00:01:34,580 --> 00:01:36,600 ‫but now I guess it is longer. 39 00:01:36,600 --> 00:01:38,520 ‫So then the batch windows so how long to 40 00:01:38,520 --> 00:01:40,520 ‫gather records before invoking the functions 41 00:01:40,520 --> 00:01:42,950 ‫in seconds so this is to be more efficient 42 00:01:42,950 --> 00:01:45,580 ‫and invoke the function less while having 43 00:01:45,580 --> 00:01:48,880 ‫a bigger batch for your lambda function to process. 44 00:01:48,880 --> 00:01:51,270 ‫And finally, you need to make sure to enable the trigger. 45 00:01:51,270 --> 00:01:54,910 ‫So I will enable this trigger and click on add. 46 00:01:54,910 --> 00:01:56,820 ‫Now we're going to get an error saying that 47 00:01:56,820 --> 00:01:59,350 ‫the execution role does not have the permissions 48 00:01:59,350 --> 00:02:01,450 ‫to call ReceiveMessage on SQS. 49 00:02:01,450 --> 00:02:03,530 ‫So yes, because our lambda function 50 00:02:03,530 --> 00:02:05,310 ‫is reading from the SQS queue 51 00:02:05,310 --> 00:02:08,140 ‫and there is a IAM role attached to the lambda function, 52 00:02:08,140 --> 00:02:10,600 ‫then we need to make sure that it is properly configured 53 00:02:10,600 --> 00:02:14,060 ‫and has the permission to read it from SQS. 54 00:02:14,060 --> 00:02:17,440 ‫So fairly easy. Let's go back to our lambda SQS function. 55 00:02:17,440 --> 00:02:19,760 ‫We're going to configuration, permissions 56 00:02:19,760 --> 00:02:21,570 ‫and here is the role name. 57 00:02:21,570 --> 00:02:23,450 ‫So we're going to click on this role, 58 00:02:23,450 --> 00:02:26,240 ‫which is going to take us into the IAM console 59 00:02:26,240 --> 00:02:28,530 ‫and I can attach a policy to my role 60 00:02:28,530 --> 00:02:32,040 ‫and I will just type Lambda SQS 61 00:02:32,040 --> 00:02:34,150 ‫and there should be a role for that. 62 00:02:34,150 --> 00:02:36,750 ‫So let's just type SQS then 63 00:02:38,020 --> 00:02:41,840 ‫and we have an AWS Lambda SQS queue execution role. 64 00:02:41,840 --> 00:02:43,460 ‫And this role is enough for it 65 00:02:43,460 --> 00:02:45,140 ‫to read it from the SQS queue. 66 00:02:45,140 --> 00:02:46,870 ‫So attached to this policy, 67 00:02:46,870 --> 00:02:49,380 ‫this Lambda SQS queue execution role 68 00:02:50,280 --> 00:02:55,280 ‫and if I go back into my Lambda trigger and add again, 69 00:02:55,780 --> 00:02:57,830 ‫this time it works because we have the permission. 70 00:02:57,830 --> 00:02:59,870 ‫So as we can see now, SQS of the Lambda 71 00:02:59,870 --> 00:03:03,590 ‫function is linked to my SQS queue. 72 00:03:03,590 --> 00:03:05,300 ‫Okay so now in terms of what the code of 73 00:03:05,300 --> 00:03:07,020 ‫my Lambda function will be doing, 74 00:03:07,020 --> 00:03:08,610 ‫it's going to learn the function 75 00:03:08,610 --> 00:03:11,330 ‫and we're just going to print the events 76 00:03:11,330 --> 00:03:13,940 ‫yet again to get some information. 77 00:03:13,940 --> 00:03:17,483 ‫And instead of returning this, let's just return success. 78 00:03:18,700 --> 00:03:21,670 ‫Okay, let's deploy our changes. 79 00:03:21,670 --> 00:03:23,170 ‫Perfect. This is good. 80 00:03:23,170 --> 00:03:24,950 ‫And now let's test our Lambda function 81 00:03:24,950 --> 00:03:28,450 ‫by simply going into SQS and sending a message. 82 00:03:28,450 --> 00:03:32,050 ‫So in my SQS queue I'm going to send a message. 83 00:03:32,050 --> 00:03:34,950 ‫The message body is going to be hello world 84 00:03:34,950 --> 00:03:36,730 ‫and then we can add some message attributes 85 00:03:36,730 --> 00:03:41,730 ‫for example, the foo is going to be of value bar. 86 00:03:42,250 --> 00:03:43,960 ‫Just to try it out. 87 00:03:43,960 --> 00:03:45,500 ‫Then we're going to send it this message 88 00:03:45,500 --> 00:03:48,490 ‫so now the message is sent and is ready to be received. 89 00:03:48,490 --> 00:03:50,290 ‫And so, because my Lambda function is 90 00:03:50,290 --> 00:03:52,300 ‫continuously pulling from SQS, 91 00:03:52,300 --> 00:03:54,580 ‫then it should process this message. 92 00:03:54,580 --> 00:03:55,413 ‫And how do we make sure? 93 00:03:55,413 --> 00:03:57,710 ‫Well we are logging the events right here 94 00:03:57,710 --> 00:04:00,100 ‫by going into the console logs. 95 00:04:00,100 --> 00:04:02,520 ‫So if we go into monitoring and again 96 00:04:02,520 --> 00:04:04,253 ‫we open the logs in CloudWatch, 97 00:04:07,770 --> 00:04:09,010 ‫now we see one log stream 98 00:04:09,010 --> 00:04:11,250 ‫so this is perfect let's click on it 99 00:04:11,250 --> 00:04:13,720 ‫and we do have a request in here indeed. 100 00:04:13,720 --> 00:04:15,590 ‫So we have a look at everything. 101 00:04:15,590 --> 00:04:18,390 ‫and as we can see, the body is hello world 102 00:04:18,390 --> 00:04:20,465 ‫as well as the message attributes contains 103 00:04:20,465 --> 00:04:23,270 ‫foo bar in terms of attributes. 104 00:04:23,270 --> 00:04:25,350 ‫We get some information around when it was sent, 105 00:04:25,350 --> 00:04:27,110 ‫the event source being SQS and so on. 106 00:04:27,110 --> 00:04:29,920 ‫So that means that the invocation was working. 107 00:04:29,920 --> 00:04:33,990 ‫And also, well, if we go back to SQS into our queue 108 00:04:33,990 --> 00:04:37,730 ‫and go into send and receive messages, 109 00:04:37,730 --> 00:04:39,550 ‫as we can see there are zero 110 00:04:39,550 --> 00:04:41,430 ‫messages available in our queue, 111 00:04:41,430 --> 00:04:42,950 ‫well that's because the messages has been 112 00:04:42,950 --> 00:04:45,940 ‫processed by our lambda function. 113 00:04:45,940 --> 00:04:48,400 ‫So finally you need to make sure to 114 00:04:48,400 --> 00:04:50,170 ‫disable this event mapper. 115 00:04:50,170 --> 00:04:51,610 ‫So as we can see right now, 116 00:04:51,610 --> 00:04:53,970 ‫it should be in the enabled state, 117 00:04:53,970 --> 00:04:55,480 ‫but you need to disable it 118 00:04:55,480 --> 00:04:57,290 ‫because if you don't disable it, 119 00:04:57,290 --> 00:04:59,280 ‫then lambda is gonna be continuously 120 00:04:59,280 --> 00:05:00,870 ‫pulling from your SQS queue, 121 00:05:00,870 --> 00:05:02,940 ‫which can incur some costs in the long run 122 00:05:02,940 --> 00:05:05,880 ‫or count against your frequent out for SQS pulse. 123 00:05:05,880 --> 00:05:08,300 ‫So let's just disabled SQS queue, 124 00:05:08,300 --> 00:05:10,400 ‫but let's go ahead and add another trigger. 125 00:05:10,400 --> 00:05:12,890 ‫I just wanna show you the Kinesis trigger 126 00:05:12,890 --> 00:05:15,220 ‫because it is another event mapper 127 00:05:15,220 --> 00:05:16,770 ‫and I want to show you the different options 128 00:05:16,770 --> 00:05:19,610 ‫that are available when you use Kinesis to get lambda. 129 00:05:19,610 --> 00:05:22,020 ‫So Kinesis stream, you need to choose obviously 130 00:05:22,020 --> 00:05:25,080 ‫Kinesis stream to listen for updates on. 131 00:05:25,080 --> 00:05:29,170 ‫Then consumer, so this is if you have a enhance final 132 00:05:29,170 --> 00:05:31,810 ‫consumer then you can create a consumer application 133 00:05:31,810 --> 00:05:34,690 ‫and enable the enhance on that consumer mode right here. 134 00:05:34,690 --> 00:05:36,910 ‫But right now we have no so we'll be falling back 135 00:05:36,910 --> 00:05:40,460 ‫to the standard way of consuming from Kinesis. 136 00:05:40,460 --> 00:05:42,700 ‫Batch size, which is how many records 137 00:05:42,700 --> 00:05:44,520 ‫should read at once so 100. 138 00:05:44,520 --> 00:05:47,010 ‫Batch window again, if we want to have a bit 139 00:05:47,010 --> 00:05:49,170 ‫of wait time to create a bigger batch 140 00:05:49,170 --> 00:05:51,310 ‫before invoking our lender function. 141 00:05:51,310 --> 00:05:54,610 ‫Starting position so do we want to read the latest data? 142 00:05:54,610 --> 00:05:56,720 ‫Do we want to get the earliest data? 143 00:05:56,720 --> 00:05:58,910 ‫Or do we want to get a specific timestamp 144 00:05:58,910 --> 00:06:01,390 ‫to read from for our lambda function? 145 00:06:01,390 --> 00:06:02,790 ‫And then you can see some additional 146 00:06:02,790 --> 00:06:05,030 ‫settings such as on-failure destination 147 00:06:05,030 --> 00:06:08,040 ‫to discard some data in case it cannot be read. 148 00:06:08,040 --> 00:06:10,330 ‫How many retries, the maximum age of record, 149 00:06:10,330 --> 00:06:12,060 ‫if you wanna split batch on error, 150 00:06:12,060 --> 00:06:13,500 ‫how many concurrent batches per shard? 151 00:06:13,500 --> 00:06:16,360 ‫So if you want to process the same shard concurrently 152 00:06:16,360 --> 00:06:18,740 ‫and if you do so then you have more record processors, 153 00:06:18,740 --> 00:06:21,670 ‫but Lambda will be ensuring the fact that there 154 00:06:21,670 --> 00:06:25,963 ‫are still read in order at the partition key level. 155 00:06:26,890 --> 00:06:28,260 ‫Tumbling window duration in case 156 00:06:28,260 --> 00:06:30,340 ‫you want to do an aggregation 157 00:06:30,340 --> 00:06:31,820 ‫and report batch item failures. 158 00:06:31,820 --> 00:06:33,300 ‫So lots of options for Kinesis, 159 00:06:33,300 --> 00:06:34,830 ‫you don't need to know them all for the exam, 160 00:06:34,830 --> 00:06:36,730 ‫just need to know that Kinesis is an event 161 00:06:36,730 --> 00:06:39,050 ‫source member for your lambda function 162 00:06:39,050 --> 00:06:41,053 ‫and if you want to have more details into all these options, 163 00:06:41,053 --> 00:06:44,570 ‫then the documentation is going to be your best friend. 164 00:06:44,570 --> 00:06:45,950 ‫And then enable trigger if you're good, 165 00:06:45,950 --> 00:06:47,290 ‫but let's just cancel this cause 166 00:06:47,290 --> 00:06:49,000 ‫we just wanted to see the options. 167 00:06:49,000 --> 00:06:50,550 ‫So that's it for this lecture. 168 00:06:50,550 --> 00:06:52,200 ‫We've seen how event mappers work 169 00:06:52,200 --> 00:06:54,440 ‫and we've seen how to add the missing 170 00:06:54,440 --> 00:06:55,990 ‫item permissions for lambda functions to 171 00:06:55,990 --> 00:06:58,550 ‫be able to read from our event mapper. 172 00:06:58,550 --> 00:06:59,910 ‫So that's it, I hope you liked it 173 00:06:59,910 --> 00:07:01,860 ‫and I will see you in the next lecture.