Kafka in the Cloud: Why it’s 10x better with Confluent | Find out more

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | November 2016

Written By

Last month the Apache Kafka community released version 0.10.1.0, the announcement blog contains a good description of new features and major improvements. In other exciting news, the PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a committer and we are pleased to announce that he has accepted.

As usual, there are a large number of improvement proposals being discussed and voted on:

  • KIP-72: Limiting the amount of memory used by incoming requests is in discussion. Currently admins can configure the number of requests in the request queue, but arrival of very large requests can cause brokers to run out of memory. This proposal will allow admins to avoid that by explicitly configuring the amount of memory available for incoming requests and not accept additional requests from clients until memory is available.
  • KIP-81: Bound Fetch memory usage in the consumer – Another memory improvement suggestion, this time to prevent consumers from running out of memory while fetching data. It is interesting to see how, even though KIP-72 and KIP-81 solve different problems in different components – we converged on very similar solutions, which is to limit the bytes the consumer / broker read from socket when it exhausts available memory.
  • KIP-84: Support SASL/SCRAM mechanisms is in discussion. This KIP adds another authentication mechanism – SCRAM supports convenient username and password authentication, but it addresses many of the security concerns involved with SASL/PLAIN authentication.
  • KIP-85: Dynamic JAAS configuration for Kafka clients was approved. Kafka clients currently rely on configuration file to configure SASL security. With this improvement, developers will be able to pass JAAS configuration as a normal configuration parameter – super useful for container deployments where clients don’t always have a persistent file system to use.
  • KIP-87: Add Compaction Tombstone Flag is in discussion. Right now, the way to remove keys from a compacted topic is by sending a record with the key and null value. This prevents both legitimate usage of nulls and is challenging in cases where the value has “magic bytes” or versions. This proposal will replace the null value with a message flag that will indicate a delete operation.

And few recommended presentations and blog posts relevant to Apache Kafka and streaming platforms:

  • Gwen Shapira is a Software Enginner at Confluent. She has 15 years of experience working with code and customers to build scalable data architectures, integrating relational and big data technologies. She currently specialises in building real-time reliable data processing pipelines using Apache Kafka. Gwen is an Oracle Ace Director, an author of books including “Kafka, the Definitive Guide”, and a frequent presenter at data related conferences. Gwen is also a committer on the Apache Kafka and Apache Sqoop projects.

Did you like this blog post? Share it now