Beyond Traditional Kafka: Building a Stateless, Pluggable Broker

Operating Kafka at scale is challenging due to resource planning, partition management, high availability, and storage needs. Kafka's design does not align with cloud-native principles of simplicity, statelessness, and elasticity. To address this, an experimental project called stateless-kafka-broker was started, which removes internal state from the broker and delegates everything to external storage backends. The architecture is designed to be simple and minimal, separating metadata, logs, and index management into pluggable external stores. This design brings benefits such as easy horizontal scaling, flexibility in choosing storage backends, and no need to maintain broker-local metadata or logs. The broker design is split into three distinct stores: Meta Store, Log Store, and Index Store, each implemented as a Rust trait, enabling fully pluggable implementations. While this approach has trade-offs in performance, it offers simplicity, flexibility, and operational ease. The project is fully open source and invites contributions to implement new backends, improve protocol support, optimize performance, and write documentation. The goal is to build a truly stateless, cloud-native Kafka broker that is easy to operate and scalable. The project is still in its early stages and welcomes contributors from around the world to join and build together.