Building a Next-Generation Key-Value Store at Airbnb
Airbnb rearchitected its core key-value store, Mussel, from V1 to V2 to meet evolving real-time data demands. Mussel V2 addresses V1's operational complexity, capacity limitations, and consistency issues with a cloud-native NewSQL backend. Key improvements include automated Kubernetes deployment, dynamic range sharding for better capacity management, and flexible consistency options. The new architecture features a stateless Dispatcher service for translating API calls and an event-driven write path using Kafka for durability. Bulk loading capabilities are preserved, enhanced with stateless controllers and stateful worker fleets for high throughput. Automated Time-To-Live (TTL) expiration is now topology-aware and more efficient at scale. The migration from V1 to V2 employed a complex blue/green strategy with dual writes and shadow reads to ensure zero downtime and data loss. Critical lessons learned include managing consistency complexities and the importance of presplitting for range-based partitioning. Kafka played a crucial role in maintaining eventual consistency during the migration process. Mussel V2 successfully merges bulk ingestion, high-speed writes, and low-latency reads in a single platform, simplifying data infrastructure for engineering teams.