So let's talk about event processing in AWS
and the different possibilities we have
and the kind of constraints that we have alongside them.
So the first one is to use SQS and Lambda.
And if you use SQS and Lambda with our SQS queue
in the Lambda function
and so the events are going to be inserted
into our SQS queue.
And the Lambda service is going to pull the SQS queue.
And in case there are issues,
it's going to put back the messages into the SQS queue
and try and retry to pull them, okay?
And this can go into some sort of infinite loop.
So in case there is a big problem with one message,
what we can do is set up SQS to send the message
to a dead letter Q after say, five tries, okay?
This will be our way out of it.
Now we can also use SQS FIFO and Lambda.
In this case, FIFO means first in, first out.
And that means that the messages
are going to be processed in order.
So now our Lambda function is going to try
and retry to get the messages from the queue.
But because it's just to process them in order,
in case one message doesn't go through,
it's going to be a blocking, never ending process
and the whole queue processing will be blocked.
In which case, yet again, we can set up a dead letter queue
to send these messages off to SQS queue
and allow our functions to keep on processing it.
Okay.
Next, we have another option, SNS and Lambda.
In this case, SNS is a service
and the message goes through with it
and then the message is sent asynchronously to Lambda.
In which case, the Lambda function
is a different retry behavior.
In case it cannot process that message,
it's going to retry, but internally.
And that retry will be three times.
And if the message is not processed successfully,
it will be discarded, or we can set up a DLQ.
But this time, at the Lambda service level
to send that message into, for example, the SQS queue
for letter processing.
So as we can see, this is a different kind of architecture.
With SQS, the DLQ is set up on the SQS side.
And for Lambda, the DLQ is set up on the Lambda side.
So different architectures with different needs.
Now let's talk about the Fan Out pattern,
which is, how do you deliver a data to multiple SQS queues?
So, option one, we have our application
and it has the AWS SDK installed on it.
And we have say three SQS SQS
we'd like to deliver a message on to.
So what we could do, very easily,
is to write our application,
to first send a message to the first queue
and then send a message to the second queue.
And then send again the same message to the third queue.
And that would work,
but that would be somewhat not too reliable, right?
Because for example, if our application crashes
after we send a message to the second queue,
then the third queue will never receive the last message.
And the content of each queue will be different.
So while this works, this is not very reliable
and this is not pretty.
Instead, we can use a fan out pattern,
which is to combine your SQS queue
with an SNS topic in the middle.
And in this case, what happens?
Well, our SQS queues are going to be subscribers
of our SNS topic in the middle?
And what will happen is that anytime we send a message
to the SNS topic, it will be sent by the SNS service
into all of our SQS queues, which is a higher guarantee.
So from our application perspective now,
we just do a put into our SNS topic.
And automatically, the SNS service
will fan out that message into your other SQS queue.
And this works really well.
And that's a very common design pattern on AWS.
Regarding S3 event notifications,
it's possible for you to react
to specific events on your Amazon S3 buckets
such as when the objects is created, removed, restored,
when replication happens,
and you can filter by name as well.
So a use case would be to generate thumbnails
of images uploaded to Amazon S3.
So from Amazon S3 events, you can send to SNS,
SQS or a Lambda function
and you can create as many S3 events as desired.
And these event notifications
typically get delivered within seconds,
but sometimes it can take a minute or longer.
So you have to remember these three integrations.
And there's one last, which is to use Amazon event bridge.
So in this case
the events will happen in your Amazon S3 buckets.
And all the events are actually sent
to now Amazon event bridge.
And using rules, we can send them
to over 18 AWS services as destination.
So why would you use EventBridge?
Well, because you get filtering options
with adjacent rules.
So you can filter on metadata, on object-size,
on name and so on.
You can also send it to multiple destinations at once,
such as step functions, Kinesis Streams or Data Firehose.
And then, you can also the EventBridge capabilities,
such as archiving, replaying,
and reliable delivery of events,
which is something that may be desirable.
Talking about EventBridge.
Remember, you can intercept any API call
with Amazon EventBridge
by using the integration with CloudTrail.
So for example, if you wanna react to the events
of a user deleting a table from DynamoDB
so using the delete table API call.
Then this API call is going to be logged in CloudTrail.
Actually all of them will be logged in CloudTrail.
These will trigger an event in Amazon EventBridge.
And from this, you can create alerts to alert Amazon SNS.
And finally, you can have external events onto AWS.
For example, using the API gateway.
So they say clients will send the request
into an API gateway.
The API gateway will send messages
into Kinesis Data Stream.
The records will end up at Nemesis data Firehose
and then the data for example, can end up in Amazon S3.
So you've seen a lot of options to integrate events
in AWS and to build some cool automations.
I hope you liked it and I will see you in the next lecture.