1 00:00:00,660 --> 00:00:02,740 ‫So let's about SQS, 2 00:00:02,740 --> 00:00:05,020 ‫and at the core of SQS is a queue 3 00:00:05,020 --> 00:00:07,330 ‫because SQS is a simple queuing service. 4 00:00:07,330 --> 00:00:08,970 ‫So we have an SQS queue, 5 00:00:08,970 --> 00:00:11,150 ‫and it is going to contain messages. 6 00:00:11,150 --> 00:00:12,610 ‫And to contain messages, 7 00:00:12,610 --> 00:00:16,070 ‫well, something needs to send messages into our SQS queue 8 00:00:16,070 --> 00:00:19,140 ‫and whatever sends a message into our SQS queue 9 00:00:19,140 --> 00:00:21,110 ‫is called a producer. 10 00:00:21,110 --> 00:00:23,490 ‫So it's possible for us to have one producer, 11 00:00:23,490 --> 00:00:24,580 ‫but also have more. 12 00:00:24,580 --> 00:00:26,380 ‫You can have multiple producers 13 00:00:26,380 --> 00:00:29,370 ‫sending many messages into an SQS queue. 14 00:00:29,370 --> 00:00:31,120 ‫And the message could be whatever he wants. 15 00:00:31,120 --> 00:00:33,670 ‫For example, it could be process this order, 16 00:00:33,670 --> 00:00:35,180 ‫or process this video. 17 00:00:35,180 --> 00:00:38,350 ‫Whatever message you create goes into the queue. 18 00:00:38,350 --> 00:00:41,680 ‫Then something needs to process the messages from the queue 19 00:00:41,680 --> 00:00:44,610 ‫and receive them, and it's called consumers. 20 00:00:44,610 --> 00:00:47,420 ‫So consumers will poll the messages from the queue, 21 00:00:47,420 --> 00:00:49,620 ‫so that means that they will ask the queue, 22 00:00:49,620 --> 00:00:50,970 ‫do you have any message for me? 23 00:00:50,970 --> 00:00:52,880 ‫And the queue says, yes, here it is. 24 00:00:52,880 --> 00:00:54,600 ‫And the consumer will poll these messages 25 00:00:54,600 --> 00:00:56,210 ‫and get some information. 26 00:00:56,210 --> 00:00:57,540 ‫And then with that message, 27 00:00:57,540 --> 00:01:00,640 ‫it will process it and then delete it back from the queue. 28 00:01:00,640 --> 00:01:02,240 ‫And you may have multiple consumers 29 00:01:02,240 --> 00:01:04,670 ‫consuming messages from an SQS queue. 30 00:01:04,670 --> 00:01:07,770 ‫So a queuing service is here to be a buffer 31 00:01:07,770 --> 00:01:11,980 ‫to decouple between your producers and your consumers. 32 00:01:11,980 --> 00:01:14,190 ‫Now, SQS is a complicated service 33 00:01:14,190 --> 00:01:15,320 ‫and we'll see that in-depth, 34 00:01:15,320 --> 00:01:18,670 ‫but the very first offering is called Amazon SQS 35 00:01:18,670 --> 00:01:20,430 ‫for standard queues. 36 00:01:20,430 --> 00:01:24,590 ‫And SQS, historically on AWS, it is the oldest offering. 37 00:01:24,590 --> 00:01:27,470 ‫It was one of the first services on AWS. 38 00:01:27,470 --> 00:01:29,540 ‫It's over 10 years old so it's quite, 39 00:01:29,540 --> 00:01:31,670 ‫quite set in stone about how it works. 40 00:01:31,670 --> 00:01:33,020 ‫It's a fully managed service, 41 00:01:33,020 --> 00:01:35,510 ‫and it will be used to decouple applications. 42 00:01:35,510 --> 00:01:39,310 ‫So anytime you see application decoupling in your exam, 43 00:01:39,310 --> 00:01:41,850 ‫think about Amazon SQS. 44 00:01:41,850 --> 00:01:45,250 ‫Now, why is SQS so special? 45 00:01:45,250 --> 00:01:47,380 ‫Well, we get an unlimited throughputs. 46 00:01:47,380 --> 00:01:48,213 ‫That means you can send 47 00:01:48,213 --> 00:01:50,620 ‫as many messages per second as you want, 48 00:01:50,620 --> 00:01:53,670 ‫and the queue can have as many messages as you want as well. 49 00:01:53,670 --> 00:01:55,390 ‫So there's no limit on throughputs 50 00:01:55,390 --> 00:01:58,710 ‫and no limits in the number of messages in the queue. 51 00:01:58,710 --> 00:02:00,920 ‫Now, each message is short-lived. 52 00:02:00,920 --> 00:02:01,770 ‫What does that mean? 53 00:02:01,770 --> 00:02:03,310 ‫That means that by default, 54 00:02:03,310 --> 00:02:06,860 ‫the message will stay in the queue for four days 55 00:02:06,860 --> 00:02:08,530 ‫and the maximum amount of time 56 00:02:08,530 --> 00:02:11,860 ‫that a message can be in a queue is 14 days. 57 00:02:11,860 --> 00:02:14,380 ‫That means that as soon as you send a message to a queue, 58 00:02:14,380 --> 00:02:18,250 ‫it has to be read by consumer and deleted from the queue 59 00:02:18,250 --> 00:02:21,410 ‫after being processed within that retention period, 60 00:02:21,410 --> 00:02:23,340 ‫otherwise, it will be lost. 61 00:02:23,340 --> 00:02:24,990 ‫Then we have low-latency. 62 00:02:24,990 --> 00:02:27,560 ‫So SQS means that, for them, 63 00:02:27,560 --> 00:02:28,930 ‫is that whenever you send a message 64 00:02:28,930 --> 00:02:30,330 ‫or read a message from SQS, 65 00:02:30,330 --> 00:02:32,010 ‫you will get a response very quickly, 66 00:02:32,010 --> 00:02:34,900 ‫less than 10 milliseconds on publish and receive. 67 00:02:34,900 --> 00:02:37,330 ‫And the messages in SQS have to be small. 68 00:02:37,330 --> 00:02:41,880 ‫They have to be less than 256 kilobytes per messages sent. 69 00:02:41,880 --> 00:02:43,030 ‫Now SQS is a queuing service, 70 00:02:43,030 --> 00:02:45,820 ‫so you can see high throughput, high volume, and so on, 71 00:02:45,820 --> 00:02:49,450 ‫and so, it is possible to have duplicates messages. 72 00:02:49,450 --> 00:02:50,760 ‫That means that, for example, 73 00:02:50,760 --> 00:02:53,250 ‫sometimes a message will be delivered twice, 74 00:02:53,250 --> 00:02:56,070 ‫and so this is why it's called at least once delivery. 75 00:02:56,070 --> 00:02:58,860 ‫And if you go ahead with writing an application, 76 00:02:58,860 --> 00:03:00,870 ‫you need to take that into account. 77 00:03:00,870 --> 00:03:03,690 ‫It can also have out of the order messages, 78 00:03:03,690 --> 00:03:06,450 ‫which means it's best effort ordering, 79 00:03:06,450 --> 00:03:09,640 ‫and we'll see there is another type of offering from SQS 80 00:03:09,640 --> 00:03:11,450 ‫that can deal with that limitation, 81 00:03:11,450 --> 00:03:13,740 ‫but we'll see this later on in this section. 82 00:03:13,740 --> 00:03:16,810 ‫So, let's go back to our message producers. 83 00:03:16,810 --> 00:03:20,410 ‫So, messages that are up to 256 kilobytes 84 00:03:20,410 --> 00:03:23,900 ‫are sent into SQS by producers. 85 00:03:23,900 --> 00:03:25,950 ‫And how does it happen? 86 00:03:25,950 --> 00:03:28,290 ‫Well, the producers will send the messages to SQS 87 00:03:28,290 --> 00:03:31,310 ‫using an SDK, software development kits. 88 00:03:31,310 --> 00:03:35,710 ‫And the API to send a message to SQS is called SendMessage. 89 00:03:35,710 --> 00:03:36,840 ‫Very simple. 90 00:03:36,840 --> 00:03:38,790 ‫Now the message, it will be written, 91 00:03:38,790 --> 00:03:41,270 ‫it will be persisted into the SQS queue 92 00:03:41,270 --> 00:03:44,920 ‫until a consumer reads that message and deletes it, 93 00:03:44,920 --> 00:03:47,940 ‫which signifies that the message has been processed. 94 00:03:47,940 --> 00:03:49,110 ‫We know the retention. 95 00:03:49,110 --> 00:03:51,220 ‫And so, what's a use case for producing messages? 96 00:03:51,220 --> 00:03:53,760 ‫For example, you want to process an order, 97 00:03:53,760 --> 00:03:56,920 ‫for example, packets, and then ship it to the center. 98 00:03:56,920 --> 00:03:58,780 ‫And so you wanna do this in your own time 99 00:03:58,780 --> 00:04:01,530 ‫so you will send a message into the SQS queue 100 00:04:01,530 --> 00:04:02,890 ‫with maybe some information, 101 00:04:02,890 --> 00:04:05,870 ‫such as the order ID, the customer ID, 102 00:04:05,870 --> 00:04:07,180 ‫and any attributes you may want. 103 00:04:07,180 --> 00:04:08,840 ‫For example, the address, and so on. 104 00:04:08,840 --> 00:04:11,230 ‫And then your consumer, that is in application rights, 105 00:04:11,230 --> 00:04:14,260 ‫will have to deal with that message itself. 106 00:04:14,260 --> 00:04:15,490 ‫So, again, to confirm, 107 00:04:15,490 --> 00:04:19,330 ‫SQS standard has unlimited throughputs. 108 00:04:19,330 --> 00:04:21,610 ‫So we've seen about producers. Very simple. 109 00:04:21,610 --> 00:04:23,750 ‫Now let's look about consumers. 110 00:04:23,750 --> 00:04:25,663 ‫So consumers, they are applications 111 00:04:25,663 --> 00:04:28,150 ‫that we have to write with some code 112 00:04:28,150 --> 00:04:31,910 ‫and these applications can be running on EC2 instances, 113 00:04:31,910 --> 00:04:34,550 ‫so your virtual servers on AWS, 114 00:04:34,550 --> 00:04:37,930 ‫but also they can be running on your own on-premises servers 115 00:04:37,930 --> 00:04:39,350 ‫if you wanted to, 116 00:04:39,350 --> 00:04:41,000 ‫or, we haven't seen it yet, 117 00:04:41,000 --> 00:04:42,930 ‫but they can also be running 118 00:04:42,930 --> 00:04:45,620 ‫on Lambda functions on AWS Lambda. 119 00:04:45,620 --> 00:04:46,453 ‫We'll see in this course, 120 00:04:46,453 --> 00:04:49,220 ‫it is a serverless compute type of service. 121 00:04:49,220 --> 00:04:51,550 ‫What this means is that you can just read messages 122 00:04:51,550 --> 00:04:52,660 ‫directly from them as well. 123 00:04:52,660 --> 00:04:55,040 ‫We'll see this later on this course, do not worry. 124 00:04:55,040 --> 00:04:59,700 ‫So, going back to our simple use case about EC2 instances, 125 00:04:59,700 --> 00:05:02,200 ‫our queue has a consumer 126 00:05:02,200 --> 00:05:05,767 ‫and the consumer polls for SQS for messages. 127 00:05:05,767 --> 00:05:08,280 ‫And that means that the consumer will ask the queue, 128 00:05:08,280 --> 00:05:10,360 ‫do you have messages for me? 129 00:05:10,360 --> 00:05:13,630 ‫And the consumer may receive up to 10 messages at a time. 130 00:05:13,630 --> 00:05:16,420 ‫So if there are messages in the SQS queue, 131 00:05:16,420 --> 00:05:17,940 ‫it will receive a valid response saying 132 00:05:17,940 --> 00:05:21,140 ‫here are the messages that are waiting for you. 133 00:05:21,140 --> 00:05:23,140 ‫Then the consumer, it's your code, 134 00:05:23,140 --> 00:05:25,620 ‫has a responsibility to process these messages. 135 00:05:25,620 --> 00:05:29,110 ‫For example, insert some orders into an RDS database. 136 00:05:29,110 --> 00:05:31,670 ‫So you will go ahead and for each order, 137 00:05:31,670 --> 00:05:33,910 ‫you will insert it into your RDS database, 138 00:05:33,910 --> 00:05:36,350 ‫something that you have to write, obviously with your code, 139 00:05:36,350 --> 00:05:39,210 ‫and then because these messages have been processed 140 00:05:39,210 --> 00:05:40,500 ‫because they have been received 141 00:05:40,500 --> 00:05:43,320 ‫and inserted into an Amazon RDS database, 142 00:05:43,320 --> 00:05:44,660 ‫your consumer will go ahead 143 00:05:44,660 --> 00:05:47,030 ‫and delete these messages from the queue 144 00:05:47,030 --> 00:05:49,330 ‫using the DeleteMessage API. 145 00:05:49,330 --> 00:05:51,990 ‫And this will guarantee that no other consumer 146 00:05:51,990 --> 00:05:53,590 ‫will be able to see these messages 147 00:05:53,590 --> 00:05:56,973 ‫and therefore, the message processing is complete. 148 00:05:57,870 --> 00:05:59,830 ‫So now we can scale this up. 149 00:05:59,830 --> 00:06:02,000 ‫We can have multiple consumers at a time. 150 00:06:02,000 --> 00:06:04,293 ‫So our SQS queue can have multiple consumers 151 00:06:04,293 --> 00:06:08,270 ‫that will receive and process these messages in parallel. 152 00:06:08,270 --> 00:06:10,660 ‫So here we have three on EC2 instances, 153 00:06:10,660 --> 00:06:12,920 ‫and so each consumer will receive 154 00:06:12,920 --> 00:06:16,560 ‫a different set of messages by calling the poll function. 155 00:06:16,560 --> 00:06:18,560 ‫And so, if somehow a message 156 00:06:18,560 --> 00:06:20,260 ‫is not processed fast enough by a consumer, 157 00:06:20,260 --> 00:06:22,660 ‫it will be received by other consumers, 158 00:06:22,660 --> 00:06:26,080 ‫and so this is why we have at least once delivery. 159 00:06:26,080 --> 00:06:29,520 ‫And this is also why we have best-effort message ordering. 160 00:06:29,520 --> 00:06:31,030 ‫Now, as I said, when the consumers 161 00:06:31,030 --> 00:06:33,040 ‫are done with the messages, they will have to delete them, 162 00:06:33,040 --> 00:06:36,090 ‫otherwise, other consumers will see these messages. 163 00:06:36,090 --> 00:06:38,300 ‫And so what this means is that with SQS queues, 164 00:06:38,300 --> 00:06:41,480 ‫if we need to increase the throughputs 165 00:06:41,480 --> 00:06:43,000 ‫because we have more messages, 166 00:06:43,000 --> 00:06:46,500 ‫then we can add consumers and do horizontal scaling 167 00:06:46,500 --> 00:06:49,410 ‫to improve the throughput of processing. 168 00:06:49,410 --> 00:06:51,820 ‫And so, if you remember what we said, 169 00:06:51,820 --> 00:06:54,530 ‫this is a perfect use case for using SQS 170 00:06:54,530 --> 00:06:56,860 ‫with your Auto Scaling groups, or ASG. 171 00:06:56,860 --> 00:06:58,170 ‫So, what does that mean? 172 00:06:58,170 --> 00:07:00,310 ‫Well, that means that your consumers 173 00:07:00,310 --> 00:07:02,190 ‫will be running on EC2 instances 174 00:07:02,190 --> 00:07:04,170 ‫inside of an Auto Scaling group 175 00:07:04,170 --> 00:07:07,530 ‫and they will be polling for messages from the SQS queue. 176 00:07:07,530 --> 00:07:10,110 ‫But now your Auto Scaling group has to be scaling 177 00:07:10,110 --> 00:07:11,850 ‫on some kind of metric, 178 00:07:11,850 --> 00:07:15,240 ‫and a metric that is available to us is the Queue Length. 179 00:07:15,240 --> 00:07:17,850 ‫It's called ApproximateNumberOfMessages. 180 00:07:17,850 --> 00:07:21,650 ‫It is a CloudWatch Metric that's available in any SQS queue. 181 00:07:21,650 --> 00:07:23,190 ‫And we could set up an alarm, 182 00:07:23,190 --> 00:07:28,110 ‫such as whenever the queue length go over a certain level, 183 00:07:28,110 --> 00:07:30,310 ‫then please set up a CloudWatch Alarm, 184 00:07:30,310 --> 00:07:33,670 ‫and this alarm should increase the capacity 185 00:07:33,670 --> 00:07:36,750 ‫of my Auto Scaling group by X amount. 186 00:07:36,750 --> 00:07:38,330 ‫And this will guarantee that 187 00:07:38,330 --> 00:07:40,220 ‫the more messages you have in your SQS queue, 188 00:07:40,220 --> 00:07:43,260 ‫maybe because there's a surge of orders on your websites, 189 00:07:43,260 --> 00:07:45,670 ‫the more EC2 instances will be provided 190 00:07:45,670 --> 00:07:47,140 ‫by your Auto Scaling group, 191 00:07:47,140 --> 00:07:49,520 ‫and you will accordingly process these messages 192 00:07:49,520 --> 00:07:51,160 ‫at a higher throughputs. 193 00:07:51,160 --> 00:07:52,880 ‫So this is a very common integration 194 00:07:52,880 --> 00:07:55,190 ‫that you will see in the exam. 195 00:07:55,190 --> 00:07:57,240 ‫Now SQS, again, the use case 196 00:07:57,240 --> 00:08:02,040 ‫is to decouple between applications, so application tiers. 197 00:08:02,040 --> 00:08:04,660 ‫So, for example, let's take an example of an application 198 00:08:04,660 --> 00:08:06,520 ‫that processes videos. 199 00:08:06,520 --> 00:08:09,560 ‫We could have just one big application 200 00:08:09,560 --> 00:08:12,220 ‫that's called a front-end that will take the request 201 00:08:12,220 --> 00:08:14,370 ‫and whenever a video needs to be processed, 202 00:08:14,370 --> 00:08:15,370 ‫it will do the processing 203 00:08:15,370 --> 00:08:17,850 ‫and then insert that into an S3 bucket. 204 00:08:17,850 --> 00:08:19,150 ‫But the problem is that processing 205 00:08:19,150 --> 00:08:21,210 ‫may be very, very long to do 206 00:08:21,210 --> 00:08:23,680 ‫and it may just slow down your websites 207 00:08:23,680 --> 00:08:26,330 ‫if you do this in the front-end here. 208 00:08:26,330 --> 00:08:29,210 ‫So instead, you can decouple your application here and say, 209 00:08:29,210 --> 00:08:32,310 ‫wait a minute, the request of processing a file 210 00:08:32,310 --> 00:08:34,120 ‫and the actual processing of a file 211 00:08:34,120 --> 00:08:36,230 ‫can happen in two different applications. 212 00:08:36,230 --> 00:08:38,200 ‫And therefore, whenever you take a request 213 00:08:38,200 --> 00:08:39,420 ‫to process a file, 214 00:08:39,420 --> 00:08:42,660 ‫you will send a message into an SQS queue. 215 00:08:42,660 --> 00:08:45,210 ‫Now, when you do the request to process, 216 00:08:45,210 --> 00:08:46,950 ‫that file will be in the SQS queue 217 00:08:46,950 --> 00:08:50,100 ‫and you can create a second processing tier 218 00:08:50,100 --> 00:08:52,160 ‫called the back-end processing application 219 00:08:52,160 --> 00:08:54,260 ‫that will be in its own Auto-Scaling group 220 00:08:54,260 --> 00:08:57,180 ‫to receive these messages, process these videos, 221 00:08:57,180 --> 00:08:59,770 ‫and insert them into an S3 bucket. 222 00:08:59,770 --> 00:09:01,990 ‫So as we can see here with this architecture, 223 00:09:01,990 --> 00:09:04,590 ‫we can scale the front-end accordingly, 224 00:09:04,590 --> 00:09:07,010 ‫and we can scale the back-end accordingly as well, 225 00:09:07,010 --> 00:09:08,410 ‫but independently. 226 00:09:08,410 --> 00:09:11,430 ‫And because the SQS queue has unlimited throughputs 227 00:09:11,430 --> 00:09:13,490 ‫and it has unlimited number of messages 228 00:09:13,490 --> 00:09:16,370 ‫in terms of the queue, then you are really safe, 229 00:09:16,370 --> 00:09:20,310 ‫and this is a robust and scalable type of architecture. 230 00:09:20,310 --> 00:09:22,030 ‫And also, for your front-end, 231 00:09:22,030 --> 00:09:25,030 ‫you can use the optimal type of EC2 instances 232 00:09:25,030 --> 00:09:26,520 ‫or architecture for your front-end. 233 00:09:26,520 --> 00:09:27,530 ‫And for the back-end, 234 00:09:27,530 --> 00:09:29,120 ‫maybe if you're doing some video processing, 235 00:09:29,120 --> 00:09:31,150 ‫you can use some EC2 instances 236 00:09:31,150 --> 00:09:34,350 ‫that have a GPU, a graphical processing unit, 237 00:09:34,350 --> 00:09:36,170 ‫because you know that these type of instances 238 00:09:36,170 --> 00:09:38,490 ‫will be optimal for doing this kind of workload. 239 00:09:38,490 --> 00:09:40,680 ‫So this is the kind of architecture 240 00:09:40,680 --> 00:09:41,830 ‫that will come up in the exam 241 00:09:41,830 --> 00:09:42,920 ‫and that you're expected to know, 242 00:09:42,920 --> 00:09:45,170 ‫and this is an amazing and tremendous use case 243 00:09:45,170 --> 00:09:47,110 ‫for SQS queues. 244 00:09:47,110 --> 00:09:48,980 ‫Finally, SQS security. 245 00:09:48,980 --> 00:09:50,910 ‫So we have encryption in-flight 246 00:09:50,910 --> 00:09:54,320 ‫by sending and producing messages using the HTTPS API, 247 00:09:54,320 --> 00:09:56,920 ‫we get at-rest encryption using KMS keys, 248 00:09:56,920 --> 00:09:59,440 ‫and if we wanted to, we can do client-side encryption, 249 00:09:59,440 --> 00:10:01,160 ‫but that means that the client has to perform 250 00:10:01,160 --> 00:10:02,950 ‫the encryption and the decryption itself. 251 00:10:02,950 --> 00:10:06,270 ‫It's not something that's supported by SQS out of the box. 252 00:10:06,270 --> 00:10:09,420 ‫For access controls, IAM policies are going to be able to 253 00:10:09,420 --> 00:10:11,860 ‫regulate access to the SQS API, 254 00:10:11,860 --> 00:10:14,800 ‫but we also have SQS access policies, 255 00:10:14,800 --> 00:10:17,160 ‫which are similar to S3 bucket policies, 256 00:10:17,160 --> 00:10:19,150 ‫and they're very helpful when you want to do 257 00:10:19,150 --> 00:10:21,290 ‫cross-account access to SQS queue, 258 00:10:21,290 --> 00:10:23,150 ‫or when you want you to allow other services, 259 00:10:23,150 --> 00:10:26,240 ‫such as SNS that we'll see very soon, or Amazon S3, 260 00:10:26,240 --> 00:10:27,870 ‫to write to an SQS queue, 261 00:10:27,870 --> 00:10:29,980 ‫for example, with S3 events. 262 00:10:29,980 --> 00:10:31,840 ‫So that's it for SQS for an overview. 263 00:10:31,840 --> 00:10:32,673 ‫I hope you liked it, 264 00:10:32,673 --> 00:10:35,480 ‫and I will see you in the next lecture for some practice.