‫Another analytics service you will see
‫is Amazon Managed Streaming for Apache Kafka,
‫also called Amazon MSK.
‫And what is Kafka?
‫Well, Kafka is an alternative to Amazon Kinesis.
‫Kafka and Kinesis both allow you to stream data.
‫So, MSK is the ability to get
‫a fully-managed Kafka cluster on AWS.
‫And it allows you to create, update
‫and delete clusters on the fly.
‫And MSK is going to create and manage Kafka broker nodes
‫and Zookeeper broker nodes in your cluster for you
‫and you deploy the cluster in your VPC, across multiple AZ,
‫up to three for high availability.
‫You also have automatic recovery from common Kafka failures
‫and the data is stored on EBS volumes
‫for as long as you want.
‫So, from personal experience,
‫I know it's very difficult to set up Apache Kafka
‫and the fact you can just do one click
‫and then deploy Kafka on AWS is great
‫and this is the Amazon MSK service.
‫So on top of it, you have the option to use MSK Serverless.
‫And this is that you run Apache Kafka on MSK,
‫but this time you don't provision servers,
‫you don't manage capacity,
‫automatically MSK will provision resources
‫and scale, compute and storage for you.
‫So what is Apache Kafka then?
‫Apache Kafka is a way for you to stream data
‫and a Kafka cluster is made of multiple brokers
‫and then you will have producers that will produce data
‫and so they will have to ingest data from places,
‫such as Kinesis, IoT RDS, et cetera, et cetera,
‫and they will send the data directly
‫into a Kafka topic that is going to be fully replicated
‫into other brokers.
‫Now, this Kafka topic is having real-time streaming
‫of data and consumers will pull from the topic
‫to consume the data itself
‫and then your consumer can do whatever he wants,
‫process it or send it to various destinations,
‫such as EMR, S3, SageMaker, Kinesis and RDS.
‫So the idea is that Kafka is quite similar to Kinesis,
‫but there are differences to look out for.
‫So what are the differences between Kinesis Data Streams
‫and Amazon MSK?
‫Well, in Kinesis Data Streams,
‫you have one megabyte message limit,
‫which is the default in Amazon MSK,
‫but you can configure it for a higher message retention.
‫For example, 10 megabytes.
‫You can have Data Streams with Shards
‫in Kinesis Data Streams or in MSK,
‫it's called Kafka Topics with Partitions,
‫but the concept are sort of similar.
‫To scale Kinesis Data Stream,
‫you need to do Shard Splitting and to scale it down Merging.
‫But in Amazon MSK to scale a topic,
‫you can only add partitions.
‫You cannot remove partitions.
‫You have in-flight encryption for Kinesis data streams
‫and then you have either plain text
‫or TLS in-flight encryption for MSK.
‫You get at-risk encryption for both of these clusters
‫and, in the exam level, this is enough.
‫Just so you know, a few differences.
‫And also for Amazon MSK, you can keep data
‫for as long as you want, you can go over one year,
‫as long as you pay for the underlying EBS storage,
‫you're good to go.
‫So to produce to MSK you need to create a Kafka Producer
‫and then to consume from MSK, you have multiple options.
‫The first one is to use Kinesis Data Analytics
‫for Apache Flink.
‫So you want a Flink Application
‫and you make it read it directly from the MSK cluster.
‫You can use Glue as well to do streaming ETL jobs
‫and they're powered by, the time, Apache Spark Streaming.
‫You can use Lambda functions to directly have Amazon MSK
‫as an event source or you can write your own Kafka consumer
‫and you can make it run on whatever platform you want,
‫for example, your Amazon EC2 instances,
‫or an ECS cluster or an EKS cluster.
‫And once you know this,
‫you know pretty much everything there is to know
‫for Amazon MSK at the exam.
‫So I hope you liked it
‫and I will see you in the next lecture.