Beyond synthetic testing: Capturing and replaying real database workloads at Airbnb
Airbnb relies heavily on MySQL-compatible databases for its operations. Managing these databases at scale presents challenges in capacity planning, ensuring consistency during upgrades, and debugging production incidents. To address these issues, Airbnb developed a system for capturing and replaying real production database traffic.Previously, Airbnb's system for database query capture and replay was fragmented and lacked sufficient detail for accurate replaying. It also had a heavy maintenance burden and scaled poorly. The new system was designed with three key goals: load testing with real traffic, ensuring query compatibility across upgrades, and aiding performance debugging.The system architecture leverages ProxySQL, an existing proxy that sits between applications and databases. This provides a single, client-agnostic capture point requiring no application changes. The system comprises three in-house components: a Log Mover, a Log Processor, and a Log Replayer.The Log Mover, deployed as a sidecar to ProxySQL, collects query logs and transfers them to cloud object storage. The Log Processor is an offline job that parses, groups, and orders these logs into replayable datasets for each cluster. It also buckets queries by timestamp for fine-grained control and rewrites queries to maintain auto-increment behavior consistency.The Log Replayer replays captured traffic against target databases. Users specify the source cluster, time range, target endpoints, and replay mode. The replayer supports "Replay only" for load testing and capacity planning, and "Replay and compare" for compatibility testing.The Replay Task Scheduler breaks jobs into tasks and manages their execution to preserve original timing and concurrency. Replay Task Workers scale horizontally to execute the logged queries against target databases. This framework has been critical for testing database upgrades, like MySQL 5.7 to 8.0, by identifying performance regressions and correctness issues early. It allows teams to ensure a preserved performance baseline and backwards-compatible behavior before production deployments.