Building a fault-tolerant metrics storage system at Airbnb
Airbnb developed an internal storage system to handle 50 million samples per second and 2.5 petabytes of time series data. This shift was necessary due to the immense volume of data generated by extensive code instrumentation across their evolving products and infrastructure. The primary engineering challenge was to persist and serve this massive dataset performantly.To manage this scale, Airbnb adopted a multi-tenant architecture, isolating tenants by service or process for stable grouping and attribution. They implemented shuffle sharding to isolate tenant workloads, improving fault tolerance by ensuring tenants only write to and are queried from a subset of nodes. Operational complexity, especially tenant onboarding and configuration management, was addressed by a consolidated control plane that automated onboarding and simplified configuration updates.Key requirements for the system included handling over 50 million samples per second, supporting numerous dashboards and alerts, and maintaining low query execution times. Initial validation using shadow clusters revealed reliability issues, compaction delays, and slow query performance, especially with large data payloads. Addressing these challenges began with ensuring the reliability of a single cluster, focusing on stabilizing writes, reads, and compaction through benchmarking, guardrails, and isolation of query paths.The system was made fault-tolerant with zone-aware stateful components deployed across three zones. Per-replica limits and tenant-level controls were implemented for effective fleet management and system protection. Subsequently, a multi-cluster architecture was adopted to reduce the blast radius of failures and enhance flexibility.This multi-cluster approach, however, introduced complexities in metrics discovery, querying, and operational overhead. These were mitigated by tooling for tenant-cluster mapping and automated deployment strategies using Kubernetes operators. The introduction of Promxy with custom enhancements facilitated cross-cluster querying and alerting.Key learnings from this journey include the significant cost of cross-cluster querying and the importance of deployment consistency, achieved through automation and standardized deployments. The philosophy evolved towards treating clusters as disposable resources, similar to "cattle," rather than critical, unique "pets," allowing for easier scaling and maintenance. Ultimately, building this platform required a blend of architectural innovation, operational rigor, and a cultural shift in managing expectations.