Achieving High Availability wi... Note

Achieving High Availability with distributed database on Kubernetes at Airbnb

Organizations traditionally used costly standalone servers with sharding for database scaling, but this approach proved problematic for maintenance as data demands grew. Running horizontally scalable, open-source databases reliably in the cloud with high availability, low latency, and scalability at a reasonable cost is a significant challenge. Airbnb adopted an innovative strategy of deploying a distributed database cluster across multiple Kubernetes clusters for improved reliability and operability. Managing stateful services like databases on Kubernetes is difficult, especially concerning node replacement and upgrades, as Kubernetes lacks data distribution awareness. To mitigate this, Airbnb attached storage volumes to nodes using AWS EBS, enabling automatic reattachment to new virtual machines via Kubernetes Persistent Volume Claims. Custom Kubernetes operators were developed to manage node replacement events, categorizing them into database-initiated, proactive infrastructure, and unplanned failures. For database-initiated and proactive events, operators ensure all nodes are running before replacement and intercept pod evictions to coordinate safe deletions. Unplanned failures cannot be coordinated, but ongoing maintenance is protected by blocking replacements until failed hardware is fixed. To ensure high regional availability, Airbnb deploys each database across three independent Kubernetes clusters in different AWS availability zones, limiting the blast radius of issues. Overprovisioning database clusters guarantees sufficient capacity even if an entire AZ, Kubernetes cluster, or all storage nodes in a zone go down. AWS EBS provides rapid reattachment for node replacements and superior durability, allowing a highly available cluster with only three replicas. Tail latency spikes in EBS are mitigated by implementing storage read timeouts and allowing reads from replicas to reduce latency and avoid cross-AZ costs, with stale reads further optimizing read performance. This multi-cluster Kubernetes strategy, leveraging AWS EBS and custom operators, allows open-source distributed storage systems to achieve high availability, low latency, and scalability in cloud environments, enabling robust data management.
CdXz5zHNQW_5ISBolXzut.jpeg