Kubernetes v1.36: Mutable Pod ... Note

Kubernetes v1.36: Mutable Pod Resources for Suspended Jobs (beta)

Kubernetes v1.36 promotes the ability to modify container resource requests and limits in the pod template of a suspended Job to beta. This feature, introduced as alpha in v1.35, enables queue controllers and administrators to adjust resource specifications like CPU, memory, and GPUs for a Job while suspended, before it executes. Previously, resource requirements were immutable once set, forcing deletion and recreation of Jobs to change them, losing valuable metadata. This new capability addresses situations where resource needs are not precisely known at Job creation or when cluster capacity fluctuates. For example, a queue controller can now reduce a machine learning Job's GPU request from four to two if only two are available. The Kubernetes API server relaxes immutability constraints on specific resource fields for suspended Jobs, requiring the Job's spec.suspend to be true and all active Pods to be terminated if it was previously running. In the beta version, the MutablePodResourcesForSuspendedJobs feature gate is enabled by default in v1.36. Users can test this by creating a suspended Job, editing its resources, and then resuming it. It's crucial to ensure all active Pods terminate before modifying resources for a suspended running Job to prevent inconsistencies.