Airbnb has successfully upgraded its Istio service mesh 14 times, managing tens of thousands of pods across dozens of Kubernetes clusters and thousands of VMs. Their upgrade process prioritizes zero downtime and gradual rollouts, allowing independent upgrades without user intervention. The architecture involves a management cluster for Istiod and multiple workload clusters. Upgrades follow a canary model, running current and new Istio versions concurrently.This is achieved by coordinating control plane (Istiod) and data plane (istio-proxy) updates. Crucially, older istio-proxy versions are not used with newer Istiod; they are updated atomically. A central management file, rollouts.yml, dictates the desired Istio version distribution across namespaces. For Kubernetes, an in-house tool called Krispr injects Istio revision labels into deployments during CI and pod admission.This mechanism ensures workloads are upgraded even if they don't deploy frequently. For virtual machines, upgrades are managed by an on-host daemon, mxagent, which installs artifacts based on VM tags. A central controller, mxrc, updates these tags to align with rollouts.yml. Mxrc also monitors VM health, ensuring a controlled upgrade process. This approach effectively decouples infrastructure upgrades from application deployments. Airbnb’s continuous investment in maintainability and safety has enabled these complex, large-scale Istio upgrades.