Sitar-agent: Building a reliable dynamic configuration sidecar at scale
Airbnb developed the sitar agent, a lightweight Kubernetes sidecar, to reliably deliver dynamic configuration changes to thousands of service instances. The configuration delivery begins with developers creating or updating values through Git or a UI, which are then stored in the Sitar Service. Periodically, the full state of configurations is packaged into compressed snapshots and uploaded to AWS S3. When a service pod starts, the sitar agent first downloads these S3 snapshots to a mounted disk, enabling a quick bootstrap. It then synchronizes with the Sitar Service for any changes made since the snapshot was created, signaling readiness to the main application container. After startup, the agent continuously polls the Sitar Service for updates every few seconds. The main application container reads configurations from the mounted disk using a Sitar client library that caches values and detects file changes. A key design decision was maintaining the sitar agent as a separate sidecar container rather than integrating it into the main container, prioritizing reliability, operational safety, and multi-language support over minor cost savings. The system uses a pull model where the agent polls the Sitar Service, optimized with a server-side cache and token-based database access to reduce load. For its local on-disk key-value store, Airbnb opted for SQLite over the legacy Sparkey-backed implementation due to SQLite's superior concurrency, performance, and multi-language support. SQLite's built-in Write-Ahead Logging allows concurrent reads during writes, and its simpler operational model was preferred over RocksDB's higher performance but greater complexity. This robust sidecar design ensures that critical configurations are delivered quickly and reliably across Airbnb's vast service fleet.