1 00:00:00,113 --> 00:00:01,460 ‫Now let's talk about 2 00:00:01,460 --> 00:00:03,420 ‫some more concepts you need to know for SQS, 3 00:00:03,420 --> 00:00:05,600 ‫but more at the developer level. 4 00:00:05,600 --> 00:00:07,960 ‫So, the first one is called Long Polling. 5 00:00:07,960 --> 00:00:10,870 ‫So when a consumer requests a message from SQS, 6 00:00:10,870 --> 00:00:14,790 ‫it has the option to "wait" for messages to arrive 7 00:00:14,790 --> 00:00:16,860 ‫if somehow the queue is empty. 8 00:00:16,860 --> 00:00:18,200 ‫And this is called Long Polling. 9 00:00:18,200 --> 00:00:20,010 ‫So let's take an example. 10 00:00:20,010 --> 00:00:22,460 ‫We have an SQS queue and it is empty 11 00:00:22,460 --> 00:00:26,830 ‫and the consumer does a poll request into the SQS Queue. 12 00:00:26,830 --> 00:00:28,500 ‫Now we have the option to "wait" 13 00:00:28,500 --> 00:00:29,780 ‫and it's fine to wait, right? 14 00:00:29,780 --> 00:00:31,150 ‫Because there are no messages. 15 00:00:31,150 --> 00:00:33,565 ‫So that means that if during the waiting period 16 00:00:33,565 --> 00:00:36,570 ‫a message goes into the SQS Queue, 17 00:00:36,570 --> 00:00:39,570 ‫then the message will be received at the consumer then. 18 00:00:39,570 --> 00:00:41,060 ‫So why do we do Long Polling? 19 00:00:41,060 --> 00:00:43,100 ‫Well, we do Long Polling because we are doing 20 00:00:43,100 --> 00:00:46,290 ‫less API calls into the SQS Queue. 21 00:00:46,290 --> 00:00:47,730 ‫And on top of it, we know that 22 00:00:47,730 --> 00:00:51,070 ‫as soon as the message will arrive in the SQS Queue, 23 00:00:51,070 --> 00:00:54,450 ‫then the SQS Queue will send it back to the consumer. 24 00:00:54,450 --> 00:00:56,500 ‫So we are increasing the efficiency 25 00:00:56,500 --> 00:00:58,360 ‫because we do less API calls, 26 00:00:58,360 --> 00:01:00,111 ‫so less CPU cycles are used. 27 00:01:00,111 --> 00:01:03,360 ‫And we are also decreasing the latency 28 00:01:03,360 --> 00:01:05,440 ‫because as soon as a message will be received 29 00:01:05,440 --> 00:01:06,650 ‫by your SQS Queue, 30 00:01:06,650 --> 00:01:08,170 ‫it will be received by your consumers 31 00:01:08,170 --> 00:01:10,343 ‫so Long Polling is a no-brainer. 32 00:01:11,240 --> 00:01:14,960 ‫Long Polling you can set between one second to 20 seconds, 33 00:01:14,960 --> 00:01:17,830 ‫but 20 seconds is preferable, why not? 34 00:01:17,830 --> 00:01:21,680 ‫And overall it is a recommendation in your applications, 35 00:01:21,680 --> 00:01:24,040 ‫to use Long Polling to Short Polling. 36 00:01:24,040 --> 00:01:25,860 ‫So you will see in the exam some questions 37 00:01:25,860 --> 00:01:27,205 ‫maybe telling you that the consumer 38 00:01:27,205 --> 00:01:29,570 ‫is doing too many calls into the SQS Queue, 39 00:01:29,570 --> 00:01:32,400 ‫and that is costing you money and CPU cycles, 40 00:01:32,400 --> 00:01:34,187 ‫and maybe increasing the latency, 41 00:01:34,187 --> 00:01:37,390 ‫then Long Polling is going to be the option. 42 00:01:37,390 --> 00:01:40,230 ‫Now long Polling can be either enabled at the Queue level, 43 00:01:40,230 --> 00:01:43,460 ‫so Queue level setting, or at the API call level 44 00:01:43,460 --> 00:01:45,390 ‫so whenever your consumer does an API call 45 00:01:45,390 --> 00:01:47,310 ‫into SQS Queue for polling, 46 00:01:47,310 --> 00:01:49,973 ‫using the ReceiveMessageWaitTimeSeconds parameter. 47 00:01:50,890 --> 00:01:52,360 ‫The second thing we need to look at, 48 00:01:52,360 --> 00:01:54,150 ‫is the SQS Extended Client. 49 00:01:54,150 --> 00:01:57,940 ‫So, as we know the max message limit is 256 kilobytes, 50 00:01:57,940 --> 00:02:00,640 ‫so how do we go ahead by sending large messages 51 00:02:00,640 --> 00:02:04,130 ‫into your SQS Queue, for example one gigabytes messages? 52 00:02:04,130 --> 00:02:06,390 ‫For this, we can use a Java Library 53 00:02:06,390 --> 00:02:08,670 ‫called the SQS Extended Client 54 00:02:08,670 --> 00:02:09,980 ‫which does something very simple 55 00:02:09,980 --> 00:02:12,730 ‫that you could implement in any other language. 56 00:02:12,730 --> 00:02:15,850 ‫But the idea is that it will use an Amazon S3 bucket, 57 00:02:15,850 --> 00:02:19,380 ‫as a repository for the large data. 58 00:02:19,380 --> 00:02:20,230 ‫So lets take an example, 59 00:02:20,230 --> 00:02:23,760 ‫your producer wants to send a large message into SQS 60 00:02:23,760 --> 00:02:25,186 ‫but first what's going to happen is that 61 00:02:25,186 --> 00:02:29,710 ‫the actual large message, will end up in Amazon S3. 62 00:02:29,710 --> 00:02:31,688 ‫And what will be sent into your SQS Queue 63 00:02:31,688 --> 00:02:34,390 ‫will be small metadata message 64 00:02:34,390 --> 00:02:37,370 ‫that has a pointer to the larger message 65 00:02:37,370 --> 00:02:39,020 ‫in your Amazon S3 bucket. 66 00:02:39,020 --> 00:02:41,700 ‫So the SQS Queue will contain small messages 67 00:02:41,700 --> 00:02:45,210 ‫and your Amazon S3 bucket will contain large objects. 68 00:02:45,210 --> 00:02:48,340 ‫And your consumer, when he reads from the SQS Queue, 69 00:02:48,340 --> 00:02:51,280 ‫using this library, the SQS Extended Client, 70 00:02:51,280 --> 00:02:54,360 ‫then he will consume at this moment a data message, 71 00:02:54,360 --> 00:02:55,563 ‫which will say to the consumer, 72 00:02:55,563 --> 00:02:59,100 ‫" Hey, glory that bigger message out of Amazon is free." 73 00:02:59,100 --> 00:03:00,588 ‫And the consumer will be able to read 74 00:03:00,588 --> 00:03:03,420 ‫and retrieve large messages from S3. 75 00:03:03,420 --> 00:03:05,060 ‫So a typical use case for this 76 00:03:05,060 --> 00:03:07,200 ‫is if you're processing video files, 77 00:03:07,200 --> 00:03:10,653 ‫you don't send the entire video file into your SQS Queue No, 78 00:03:10,653 --> 00:03:14,090 ‫you upload that video file into your Amazon S3 bucket, 79 00:03:14,090 --> 00:03:16,780 ‫and you send a small message with a pointer 80 00:03:16,780 --> 00:03:19,690 ‫to that video file into your SQS Queue. 81 00:03:19,690 --> 00:03:21,600 ‫And that allows you to accommodate 82 00:03:21,600 --> 00:03:24,590 ‫any message size really through this pattern. 83 00:03:24,590 --> 00:03:28,690 ‫Finally, you will maybe see some API calls 84 00:03:28,690 --> 00:03:31,440 ‫are given through by the exam and so, 85 00:03:31,440 --> 00:03:32,970 ‫it's just normal API calls 86 00:03:32,970 --> 00:03:34,140 ‫that you should understand by now, 87 00:03:34,140 --> 00:03:35,550 ‫but let's go over them. 88 00:03:35,550 --> 00:03:38,220 ‫So CreateQueue is used to create a Queue 89 00:03:38,220 --> 00:03:41,490 ‫and you can use the argument MessageRetentionPeriod 90 00:03:41,490 --> 00:03:44,250 ‫to set how long a message should be kept in a queue 91 00:03:44,250 --> 00:03:45,780 ‫before being discarded. 92 00:03:45,780 --> 00:03:47,464 ‫And DeleteQueue is used to delete a queue 93 00:03:47,464 --> 00:03:50,900 ‫and delete all the messages in the queue at the same time. 94 00:03:50,900 --> 00:03:53,770 ‫PurgeQueue is an API call that is used to 95 00:03:53,770 --> 00:03:55,990 ‫delete all the messages in the queue as well. 96 00:03:55,990 --> 00:03:58,020 ‫Now, when we are sending messages, 97 00:03:58,020 --> 00:03:59,880 ‫we use the SendMessage API. 98 00:03:59,880 --> 00:04:01,830 ‫And if we want to send messages with delay, 99 00:04:01,830 --> 00:04:04,490 ‫we can use the DelaySeconds parameter. 100 00:04:04,490 --> 00:04:06,280 ‫ReceiveMessage is to do polling, 101 00:04:06,280 --> 00:04:08,410 ‫and DeleteMessage is to delete a message 102 00:04:08,410 --> 00:04:11,430 ‫once it has been processed by a consumer. 103 00:04:11,430 --> 00:04:14,130 ‫So when you receive a message by default, 104 00:04:14,130 --> 00:04:18,340 ‫the parameter MaxNumberOfMessages is set to one 105 00:04:18,340 --> 00:04:20,890 ‫that means that you receive one message at a time, 106 00:04:20,890 --> 00:04:24,680 ‫but you can receive up to 10 messages at a time in SQS 107 00:04:24,680 --> 00:04:27,950 ‫so you can set the MaxNumberOfMessages parameter, 108 00:04:27,950 --> 00:04:30,260 ‫for the ReceiveMessage API to 10 109 00:04:30,260 --> 00:04:32,423 ‫to receive a batch of messages from SQS. 110 00:04:33,350 --> 00:04:36,000 ‫The ReceiveMessageWaitTimeSeconds 111 00:04:36,000 --> 00:04:37,833 ‫is telling your consumer how long to wait 112 00:04:37,833 --> 00:04:40,890 ‫before getting a response from the queue, 113 00:04:40,890 --> 00:04:44,790 ‫and this is equivalent of enabling long Polling 114 00:04:44,790 --> 00:04:46,861 ‫and the ChangeMessageVisibility, 115 00:04:46,861 --> 00:04:48,740 ‫is used to change the message timeout 116 00:04:48,740 --> 00:04:51,620 ‫in case you need more time to process a message. 117 00:04:51,620 --> 00:04:53,760 ‫Now, if you want to use Batch API calls 118 00:04:53,760 --> 00:04:56,610 ‫you can do so for SendMessage, DeleteMessage 119 00:04:56,610 --> 00:04:58,800 ‫as well as ChangeMessageVisibility, 120 00:04:58,800 --> 00:05:00,500 ‫and this help decrease the number of API calls 121 00:05:00,500 --> 00:05:01,680 ‫you're doing into API, 122 00:05:01,680 --> 00:05:03,730 ‫and therefore help decrease your cost. 123 00:05:03,730 --> 00:05:04,563 ‫So that's it. 124 00:05:04,563 --> 00:05:07,193 ‫Now let's have a look at how Long Polling works in AWS. 125 00:05:08,980 --> 00:05:10,500 ‫So let's go into our DemoQueue 126 00:05:10,500 --> 00:05:13,163 ‫and we'll edit the settings of our DemoQueue. 127 00:05:14,000 --> 00:05:15,170 ‫So as we can see here, 128 00:05:15,170 --> 00:05:18,320 ‫the Receive message wait time is currently zero 129 00:05:18,320 --> 00:05:20,070 ‫which is called a Short Polling, 130 00:05:20,070 --> 00:05:23,260 ‫but we can set it anywhere between zero and 20 seconds. 131 00:05:23,260 --> 00:05:25,690 ‫So as soon as I set it to one you enable Long Polling, 132 00:05:25,690 --> 00:05:27,340 ‫but we'll set it to 20, 133 00:05:27,340 --> 00:05:28,229 ‫and this is saying that 134 00:05:28,229 --> 00:05:32,060 ‫you should wait up to 20 seconds to receive a message 135 00:05:32,060 --> 00:05:33,720 ‫if the queue is empty. 136 00:05:33,720 --> 00:05:35,940 ‫So I will just go and apply the settings. 137 00:05:35,940 --> 00:05:38,330 ‫I'll save, make your configuration, 138 00:05:38,330 --> 00:05:41,740 ‫and then I will go into my Send and receive messages, 139 00:05:41,740 --> 00:05:46,100 ‫and in here, I'm gonna go ahead and start a consumer. 140 00:05:46,100 --> 00:05:47,800 ‫Now this consumer is doing Long Polling 141 00:05:47,800 --> 00:05:49,770 ‫because he was set at the Queue level, 142 00:05:49,770 --> 00:05:52,780 ‫and so that means that only one API call is happening, 143 00:05:52,780 --> 00:05:55,490 ‫and it's waiting for a message coming from the SQS Queue, 144 00:05:55,490 --> 00:05:56,470 ‫because right now there is none 145 00:05:56,470 --> 00:06:00,790 ‫but if I say, "hello world," and just press sent, 146 00:06:00,790 --> 00:06:01,830 ‫as soon as I press sent, 147 00:06:01,830 --> 00:06:03,230 ‫the message was received by my consumer 148 00:06:03,230 --> 00:06:04,970 ‫this was extremely low latency 149 00:06:04,970 --> 00:06:07,070 ‫because my consumer was in Long Polling mode 150 00:06:07,070 --> 00:06:09,440 ‫and he was waiting for a message from SQS. 151 00:06:09,440 --> 00:06:13,070 ‫Thanks to the wait message time setting 152 00:06:13,070 --> 00:06:13,970 ‫that we set from before. 153 00:06:13,970 --> 00:06:15,710 ‫So that's, it's very simple demo 154 00:06:15,710 --> 00:06:17,120 ‫but hopefully that makes sense. 155 00:06:17,120 --> 00:06:18,870 ‫I will see you in the next lecture.