Kubernetes v1.36: Staleness Mi... Note

Kubernetes v1.36: Staleness Mitigation and Observability for Controllers

Kubernetes controllers can suffer from staleness, leading to incorrect or delayed actions due to outdated cached data. Staleness arises from the controller's local cache being out of sync with the cluster's actual state. Kubernetes v1.36 introduces features to mitigate staleness and improve controller behavior. These improvements include atomic FIFO processing in client-go, enhancing queue consistency. The kube-controller-manager has integrated these client-go improvements in several key controllers like DaemonSet and ReplicaSet. These controllers now check cache resource versions before acting, preventing actions on stale data. Informer authors can use a ConsistencyStore to track and manage resource versions, mitigating staleness in their controllers. The ConsistencyStore provides functions for recording writes, checking cache readiness, and clearing stale object entries. Kubernetes v1.36 also offers new metrics to monitor controller health, including the number of skipped syncs due to staleness. Client-go also now emits metrics exposing the latest resource versions of shared informers. The Kubernetes team plans to expand these staleness mitigation features to more controllers, and integrates it in controller-runtime. They encourage user feedback and future development.