Apache Kafka 0.8.0 released

Kafka is a Message Queue developed by LinkedIn that persists messages to disk in a very performant manner. It provides the functionality of a messaging system,  with a very unique design.

The 0.8.0 bring new features such as:

  • [KAFKA-50] – kafka intra-cluster replication support
  • [KAFKA-188] – Support multiple data directories
  • [KAFKA-202] – Make the request processing in kafka asynchonous
  • [KAFKA-203] – Improve Kafka internal metrics
  • [KAFKA-235] – Add a ‘log.file.age’ configuration parameter to force rotation of log files after they’ve reached a certain age
  • [KAFKA-429] – Expose JMX operation to set logger level dynamically
  • [KAFKA-475] – Time based log segment rollout
  • [KAFKA-545] – Add a Performance Suite for the Log subsystem
  • [KAFKA-546] – Fix commit() in zk consumer for compressed messages

Downloads version 0.8.0 here: http://kafka.apache.org/downloads.html

LinkedIn open sourced Kafka

LinkedIn bring a great contribution to open source and NoSQL community with Voldemort.

They also open sourced a couple of other exciting projects, and now the open sourced another project named Kafka.

Its a distributed publish-subscribe messaging system. It is designed to support the following

  • Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.
  • High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.
  • Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
  • Support for parallel data load into Hadoop.

Kafka is aimed at providing a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by “logging” and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to an offline analysis system like Hadoop, but is very limiting for building real-time processing. Kafka aims to unify offline and online processing by providing a mechanism for parallel load into Hadoop as well as the ability to partition real-time consumption over a cluster of machines.

Find out more details on Kafka website