2 min

Tags in this article

, ,

The new service will make life a whole lot easier for Apache Kafka users.

Amazon Web Services this week announced the general availability of Amazon MSK Serverless. This is a new serverless option for Amazon Managed Streaming for Apache Kafka.

Amazon MSK is designed to scale resources instantly to meet an application’s demand, the company says. The service greatly simplifies real-time data ingestion and streaming. It automatically provisions and manages the resources necessary to provide on-demand streaming capacity and storage for applications.

Kafka is an open-source system for transporting real-time data that enjoys widespread adoption in enterprises. Originally developed by LinkedIn to process user activity metrics from its website, the platform is used for a wide range of applications ranging from infrastructure performance monitoring to cybersecurity.

Marcia Villalba, senior developer advocate for AWS Serverless, described the new release in a blog post. “Amazon MSK reduces the work needed to set up, scale, and manage Apache Kafka in production,” she states.

“With Amazon MSK, you can create a cluster in minutes and start sending data. Apache Kafka runs as a cluster on one or more brokers,” she continues.

Providing on-demand streaming capacity

“When creating a new Amazon MSK cluster, you need to decide the number of brokers, the size of the instances, and the storage that each broker has available,” Villalba explains. “The performance of an MSK cluster depends on these parameters. These settings can be easy to provide if you already know the workload. But how will you configure an Amazon MSK cluster for a new workload? Or for an application that has variable or unpredictable data traffic?” she asked.

Amazon MSK Serverless automatically provisions and manages the required resources to provide on-demand streaming capacity and storage for your applications. It is the perfect solution to get started with a new Apache Kafka workload, she explains. This is especially true “where you don’t know how much capacity you will need or if your applications produce unpredictable or highly variable throughput and you don’t want to pay for idle capacity.,” she added.

“Also, it is great if you want to avoid provisioning, scaling, and managing resource utilization of your clusters.”

Also read: Serveless Framework 3.0 released with “stage parameters”.