The distributed event streaming platform keeps getting better.
The Apache Kafka team is showing their commitment to following through on larger projects. Examples of this commitment have beeb the replacement of cluster management tool ZooKeeper and the introduction of topic identifiers.
Now they have released version 3.1, which provides some useful security and usability additions. The new Kafka also boasts new metrics FencedBrokerCount and ActiveBrokerCount. These help users to find brokers that for some reason are alive but can’t establish a connection.
With these, KRaft and ZooKeeper controllers are able to share the number of active and fenced brokers. They can do this in a cluster they have registered or otherwise are aware of.
Kafka 3.1 also delivers production ready support for OIDC. This means users should be able to connect easily to external identity providers. such as Auth0 or Octa, and should work with Java 17. The project’s attempts to introduce topic identifiers to reduce confusion around topic names progressed as well. FetchRequests now know how to handle such IDs.
The release also includes new Streams
As usual, the Kafka update includes some enhancements for client library Kafka Streams as well, one of the more helpful being the option to raise range queries for ReadOnlyKeyValueStore, ReadOnlySessionStore and ReadOnlyWindowStore without the need to specify both start and end key. Starting with the release, uncaught exceptions are wrapped as StreamsExceptions with a TaskId field to help identify the source of an exception.
Streams now also offers the option to realise foreign key joins on tables using custom instead of the default partitioners. The custom partitioners will be passed into a new TableJoined object that extends the NamedOperation. Join methods accepting the latter will be replaced with alternatives working with TableJoined, and deprecated. The join implementation for stream-stream joins also saw some reworking and should no longer emit eager left/outer join results. Instead the function will now wait until the end of the window grace period before putting anything out.
The team is planning to remove the eager rebalance protocol. This is why users should slowly but surely upgrade to the cooperative protocol instead.
Unfortunately, KRaft is still not production ready. While getting ready to lose its predecessor ZooKeeper was a big topic in the 3.0 release, and a KRaft mode is already available for testing, important security features are still in development and might need another couple of months until they are where they need to be. Details on the current state of the efforts are available in the project’s repository.
9 hours ago
