Kubernetes Blog - TheNote.app

Kubernetes Blog
Follow

The official homepage of Kubernetes, a container orchestration system for automating deployment, scaling, and management of containerized applications. This platform offers comprehensive documentation on Kubernetes, a project maintained by Cloud Native Computing Foundation. It includes details about running stateless and stateful applications, batch jobs, and CI/CD workflows using Kubernetes. The site includes detailed guides, tutorials, reference material, API documentation, and community engagement initiatives to help users get started with Kubernetes and leverage its features effectively to manage cloud-based applications efficiently.

Kubernetes Blog kubernetes.io

RSS kubernetes.io

RSS Hunter • Aug 19, 2024

Thread Of Notes

Why Is My Kubernetes Job Still Pending? Tracing Kueue Admission with Headlamp

When a Kubernetes Job is pending, it might be due to Kueue still determining resource availability. The Headlamp Kueue plugin helps investigate this by integrating Kueue resource views into the UI. This allows users to trace a Workload's admission path through LocalQueues, ClusterQueues, ResourceFlavors, and Cohorts without extensive command-line usage. The investigation begins by examining the Workload's status and conditions to see if Kueue has admitted it. If the Workload is not admitted, the issue lies with Kueue's admission process. If it is admitted but Pods are pending, the problem shifts to Kubernetes’ scheduler. The Workload view in Headlamp shows requested resources, and following the LocalQueue link reveals the entry point for the Job's request. Inspecting the linked ClusterQueue is crucial, as its quota and policies dictate resource allocation. In a specific example, a ClusterQueue with limited CPU and Pod slots caused a Job requesting more to remain suspended by Kueue. ResourceFlavors and Cohorts are examined for more complex quota or hardware class issues. If the Workload is admitted, troubleshooting then moves to Headlamp's standard Job and Pod views to check scheduler-related events. This Headlamp-first approach aims to provide a faster, more integrated troubleshooting experience for Kueue-related delays.

https://kubernetes.io/blog/2026/10/14/tracing-kueue-admission-with-headlamp/ kubernetes.io

RSS Hunter • Oct 14

The Shift to cgroup v2 in Kubernetes: What You Need to Know

Kubernetes uses Linux cgroups to manage container resources, with a significant shift towards cgroup v2. Cgroup v1 support is now in maintenance mode and will be removed entirely in future Kubernetes releases. Cgroup v2 offers a unified hierarchy, a more consistent interface, and enhanced resource isolation. Kubernetes v1.31 marked cgroup v1 management as maintenance, while v2 has been stable since v1.25. Newer Kubernetes versions, like v1.35, default to failing kubelet startup on cgroup v1 nodes unless explicitly overridden. System verification checks in kubeadm will also flag cgroup v1 usage starting with v1.35. Cgroup v2 enables new features like Memory QoS with tiered protection, offering better memory management than cgroup v1. It also improves container-aware OOM handling by ensuring all processes within a container are killed together. Rootless container support is also officially better handled with cgroup v2's delegation capabilities. Migrating requires ensuring compatibility with kernel versions, container runtimes, and the kubelet configuration.

https://kubernetes.io/blog/2026/10/06/kubernetes-cgroups-v2-shift/ kubernetes.io

RSS Hunter • Oct 6

Scaling Kubernetes Workloads with Node Swap

Kubernetes clusters often face memory limitations before CPU constraints, a problem exacerbated by new agentic AI workloads. These AI workloads require significant memory to initiate and run untrusted code, then remain idle. This idle memory is costly and limits the number of pods a node can host. Swap memory, especially when backed by fast NVMe SSDs, can address this by paging out dormant memory. Kubernetes support for swap, now generally available, allows nodes to host more pods. Benchmarking across CI/CD builds, sandboxed browsers, and Python runtimes demonstrated density gains of up to three times with minimal latency impact. Historically, swap was avoided due to memory accounting issues and slow disk latencies. However, cgroup v2 and fast NVMe SSDs resolve these concerns, making swap a viable solution. Traditional workloads like kernel builds can tolerate reduced memory limits when swap is enabled. High-density agent workloads, such as headless browsers, see significant pod density increases with swap. Sandboxed Python runtimes also benefit from swap, enabling three times more concurrent sessions. Enabling node swap involves configuring the kubelet and using cloud provider local disks. Workloads with Burstable QoS can leverage swap for dynamic memory balancing. Ultimately, fast swap offloads idle memory states, increasing pod density and node utilization for agentic workloads.

https://kubernetes.io/blog/2026/10/05/scaling-kubernetes-workloads-with-node-swap/ kubernetes.io

RSS Hunter • Oct 5

Spotlight on SIG Apps

Kubernetes adoption has led to managing complex application lifecycles, supporting diverse workloads and ensuring reliability. SIG Apps, comprising controllers like Deployments and StatefulSets, forms the bedrock of application management in Kubernetes. The group focuses on enhancing workload resilience, refining lifecycle management, and solving operational challenges. Janet Kuo and Maciej Szulik, SIG Apps chairs, discuss the evolution of workload management and future directions. SIG Apps is fundamental for how applications run on Kubernetes, managing their creation, updates, scaling, and recovery. As Kubernetes supports more varied workloads like AI and batch processing, SIG Apps evolves its APIs for reliability and simplicity. Current efforts prioritize serving workloads, improving rollout and scaling performance, and addressing user-reported issues. A major focus is improving workload resilience, particularly concerning node failures and their impact on workloads like DaemonSets and Jobs. New subprojects like JobSet and LeaderWorkerSet aim to handle failures in large-scale AI training and inference. Successful delivery of these improvements will reduce manual intervention for platform teams operating Kubernetes. A key challenge for SIG Apps is balancing aggressive controller behavior, necessary signals, and critical backward compatibility for existing user automation. To avoid breaking legacy systems, SIG Apps prefers opt-in features and introducing new paradigms via CRDs. The group also revisits proposals like KEP-4443 to enhance the Job API with more specific failure reasons for improved automation. New contributors are encouraged to join the #sig-apps Slack channel or contribute to evolving subprojects like the Agent Sandbox.

https://kubernetes.io/blog/2026/09/22/sig-apps-spotlight/ kubernetes.io

RSS Hunter • Sep 22

Kubernetes v1.37: Tracking When a PersistentVolumeClaim Was Last Used (Beta)

Kubernetes v1.37 introduces the PersistentVolumeClaimUnusedSinceTime feature gate to Beta, enabled by default. This enhancement adds an "Unused" condition to each PersistentVolumeClaim (PVC) within its status. This condition directly indicates whether any running pod currently references the PVC. Previously, identifying unused PVCs was complex, requiring cross-referencing multiple Kubernetes objects and extensive custom scripting. The new feature simplifies this by natively reporting PVC usage status. A PVC is marked as Unused if no non-terminated pods reference it, and as Used if at least one running or pending pod is attached. Terminated pods, including those from completed batch jobs, do not count as actively using a PVC. The "Unused" condition also includes a lastTransitionTime field, which records when a PVC became idle. This allows administrators to easily identify PVCs that have been unused for specific durations, aiding in cost optimization and storage management. The feature, previously in Alpha in v1.36, has undergone extensive testing and is now on by default for Beta. Users can directly inspect the PVC status via kubectl to observe these changes. The Kubernetes project plans to move this feature to General Availability in a future release based on user feedback.

https://kubernetes.io/blog/2026/09/21/kubernetes-v1-37-pvc-last-used-time/ kubernetes.io

RSS Hunter • Sep 21

Kubernetes v1.37: Hardening Container Storage with Bind Mount Options and EmptyDir Permissions

Kubernetes v1.37 introduces crucial storage security enhancements to bolster workload protection. These features enable the configuration of emptyDir volume permission modes and bind mount options directly within Kubernetes. This allows developers and security professionals to implement stringent security policies without complex workarounds. The improvements are built upon fundamental Linux security mechanisms like bind mount flags and directory permissions.Bind mount flags, such as noexec, nosuid, and nodev, control actions permitted on mounted filesystems. Directory permissions, including the sticky bit, regulate access and prevent unauthorized file deletions in shared directories. The primary motivation for these features is to enhance Kubernetes workload security by allowing security-relevant bind mount options. Previously, volumes were mounted without these protective flags, creating security vulnerabilities.A significant gap addressed is the inability to set mount options for emptyDir volumes and the default 0777 permission mode for directories. This allowed any process to access, modify, or delete files within emptyDir volumes, regardless of ownership. The new features provide native solutions, for instance, allowing the sticky bit to be set on emptyDir for secure shared workspaces, mirroring /tmp behavior. These enhancements empower application developers to prevent privilege escalation on writable mounts and enforce the principle of least privilege for application data.Use cases include preventing arbitrary binary execution from writable volumes and securing shared scratch space in multi-container pods. Developers can now confidently lock down access to temporary storage by setting specific modes like 0750. Both features are currently Alpha in v1.37 and require enabling feature gates: VolumeBindMountOptions and EmptyDirVolumeMode. These new capabilities are Linux-specific and do not affect Windows nodes.

https://kubernetes.io/blog/2026/09/16/kubernetes-v1-37-hardening-container-storage/ kubernetes.io

RSS Hunter • Sep 16

Kubernetes v1.37: Pod-Level Resource Managers graduated to Beta

Kubernetes v1.37 introduces Pod-Level Resource Managers to Beta status, allowing Kubelet's managers to utilize pod-level resource declarations. Previously, exclusive NUMA alignment for critical applications required dedicating cores to all containers, which was inefficient for sidecars. This new feature enables hybrid allocation, reserving NUMA-aligned resources for primary containers while shared pools accommodate sidecars. Sidecars gain NUMA alignment and protection without consuming dedicated cores. The Beta release includes the PodLevelResourceManagers feature gate, which is disabled by default, and enhancements to the PodResources API. The API now reports top-level cpu_ids and memory for pods, preventing double-counting in monitoring. Users can opt into this feature to manage resources more effectively. The goal is to provide better performance for critical workloads and more efficient resource utilization. Detailed documentation and tutorials are available for users to get started. Feedback is crucial as the feature progresses towards General Availability.

https://kubernetes.io/blog/2026/09/15/kubernetes-v1-37-pod-level-resource-managers-beta/ kubernetes.io

RSS Hunter • Sep 15

Kubernetes v1.37: Memory QoS Graduates to Beta

Kubernetes v1.37 marks the Beta graduation of Memory QoS, now enabled by default on Linux nodes using cgroup v2. This feature enhances container memory management by leveraging the kernel's memory controller for better guidance. Originally introduced as Alpha in v1.22, it received expanded functionality in v1.36 with tiered memory reservation. The Beta promotion means the MemoryQoS feature gate is active in all v1.37 kubelets without requiring explicit configuration changes. Crucially, the default kubelet configuration does not enable memory throttling or reservation unless users explicitly opt-in.Users can enable specific Memory QoS behaviors through kubelet configuration fields. Setting memoryThrottlingFactor, for instance, activates memory.high throttling for Burstable and BestEffort containers. Alternatively, configuring memoryReservationPolicy to TieredReservation enables tiered memory protection via memory.min and memory.low. A significant change in v1.37 is the default for memoryThrottlingFactor shifting from 0.9 to null, preventing automatic memory throttling for existing clusters upon upgrade. This ensures that runtime behavior remains consistent unless explicitly configured otherwise.To configure Memory QoS, cluster operators can set memoryThrottlingFactor for throttling and memoryReservationPolicy for tiered reservation. Both can be enabled simultaneously or independently. Disabling Memory QoS entirely is possible by setting the feature gate to false and adjusting the kubelet configuration accordingly. A known limitation is that memory reservation is node-wide, applying consistently to all pods regardless of their desired behavior, impacting individual pod control. The next development phase for Memory QoS is its graduation to General Availability, with ongoing feedback shaping future adjustments.

https://kubernetes.io/blog/2026/09/14/kubernetes-v1-37-memory-qos-graduates-to-beta/ kubernetes.io

RSS Hunter • Sep 14

Kubernetes Changed Block Tracking API - Beta Differences

Changed Block Tracking (CBT) for Kubernetes storage CSI drivers initially launched as Alpha in September 2025. This feature enhances storage operations by tracking changes within block volumes. The Alpha announcement details its motivation, core components, and usage. CBT currently only supports block volumes, excluding file-based storage. The recent v1.0.0 release of the external-snapshot-metadata project has promoted this functionality to Beta.The primary update in the Beta phase is the upgrade of the SnapshotMetadataService Custom Resource Definition (CRD) from v1alpha1 to v1beta1. While the schema remains the same, the old v1alpha1 version has been removed, requiring users to adapt. To transition from Alpha to Beta, users must re-apply the CRD definition, update their manifests to use the new API version, and adjust any client or controller code. This migration is a manual, one-time process.The Beta version necessitates a minimum Kubernetes version of 1.33 and CSI spec version 1.10 or newer. The compatible container image is csi-snapshot-metadata:v1.0.0. To test the feature, ensure your CSI driver supports snapshots and includes the external-snapshot-metadata sidecar. Install the v1beta1 CRD and create a SnapshotMetadataService resource for your driver. Then, use a lister tool or custom implementation to interact with the GetMetadataAllocated and GetMetadataDelta functions.The development team is now focusing on broader CSI driver adoption and gathering operational feedback during the Beta period. They encourage CSI driver maintainers to evaluate adding support and backup application developers to provide input on streaming clients and the iterator package. Further information can be found in the CSI developer documentation, KEP-3314, the external-snapshot-metadata repository, the gRPC schema, and the snapshot-metadata-lister example. Contributions to this feature have been significant, and interested individuals are encouraged to engage with SIG Storage and the Data Protection Working Group.

https://kubernetes.io/blog/2026/09/14/csi-changed-block-tracking-beta/ kubernetes.io

RSS Hunter • Sep 14

Kubernetes v1.37: Native Histograms Graduates to Beta

Native histograms are graduating to Beta in Kubernetes v1.37, offering high-resolution and low-cardinality observability. This advancement replaces older, static Prometheus histograms which suffered from bucket guessing, high cardinality, and interpolation errors. Prometheus native histograms utilize dynamic, exponential buckets stored within a single time series, reducing time series by up to 90%. This provides high resolution automatically and offers accurate quantile calculations. In Kubernetes, native histograms are integrated into shared metrics with dual exposition, ensuring compatibility with existing observability stacks. Classic buckets are still emitted alongside new native spans for a zero-breaking-change transition. Standard exponential options are automatically applied, ensuring bounded error for quantile calculations. This feature automatically supports major Kubernetes components like the API server, scheduler, and Kubelet. Upgrading to Kubernetes v1.37 enables this functionality, with Prometheus scrape configurations adapting to ingest the new format. Migrating dashboards and alerts involves updating PromQL queries from classic bucket references to direct metric name usage. A recommended migration workflow involves enabling both formats initially, then updating queries, verifying, and finally disabling classic format scraping for storage savings. Opting out or rolling back is flexible by adjusting Prometheus scrape configurations or component feature gates. This feature is moving towards General Availability, with community involvement encouraged.

https://kubernetes.io/blog/2026/09/11/kubernetes-v1-37-native-histograms-beta/ kubernetes.io

RSS Hunter • Sep 11

Kubernetes v1.37: Scheduler Preemption for In-Place Pod Resize (Alpha)

Kubernetes v1.35 introduced in-place Pod resizing, allowing dynamic adjustments to CPU and memory without restarts. However, scaling up a Pod on a fully utilized node could lead to a "Deferred" status if insufficient capacity existed. This meant the Pod would wait indefinitely for resources to become available. To address this, Kubernetes v1.37 introduces scheduler preemption for in-place Pod resizing, currently in Alpha. This feature enables the Kubernetes scheduler to proactively free up node capacity. If a running Pod's in-place resize request exceeds available resources, the scheduler can preempt lower-priority Pods. This ensures critical applications can scale up even on busy nodes. The preemption is localized to the Pod's current node, targeting eligible lower-priority workloads. This allows cluster operators to maximize resource utilization by bin-packing lower-priority tasks. Without this, operators faced a dilemma between underutilization or risking critical application scaling failures. The feature integrates with the scheduler's existing tracking mechanisms. It distinguishes resize preemption from standard Pod placement preemption. Resource reservations are handled safely to prevent race conditions during resizing. The feature requires Kubernetes v1.37 or later and the InPlacePodVerticalScalingSchedulerPreemption feature gate enabled. Administrators can disable resize preemption for specific nodes using the podPreemptionPolicy field.

https://kubernetes.io/blog/2026/09/10/kubernetes-v1-37-scheduler-preemption-for-in-place-pod-resize-alpha/ kubernetes.io

RSS Hunter • Sep 10

Kubernetes v1.37: Introducing Node Lifecycle Conditions

Kubernetes traditionally lacked a unified way to describe a Node's lifecycle states beyond basic status information. This often led to fragmented understanding and potential conflicts between different cluster components. Kubernetes v1.37 introduces five well-known Node conditions to address this gap. These new conditions are DrainInProgress, Drained, MaintenancePlanned, MaintenanceInProgress, and GracefulNodeShutdownInProgress. Each condition uses a True, False, or Unknown status to indicate its current state. Administrators or authorized controllers are responsible for setting and clearing these conditions. Currently, core workload controllers do not act on these conditions, but they provide operational clarity by signaling Node lifecycle events. For instance, automation can set MaintenancePlanned before work begins and MaintenanceInProgress once it starts. Similarly, drain automation can set DrainInProgress and Drained accordingly. The recommended pattern is to use these conditions for status reporting while still utilizing existing Kubernetes mechanisms for actual operations like cordoning and draining. This shared signal is crucial because various cluster components, from the scheduler to external maintenance systems, need to understand Node status. Without it, components might make conflicting decisions, such as a DaemonSet controller replacing a Pod intentionally terminated during graceful shutdown. These new conditions provide a stable foundation for future enhancements in Node lifecycle management. The Kubernetes community is actively seeking feedback on these new conditions to improve scenarios like graceful node shutdown and maintenance.

https://kubernetes.io/blog/2026/09/09/kubernetes-v1-37-node-lifecycle-conditions/ kubernetes.io

RSS Hunter • Sep 9

Kubernetes v1.37: Advancing Workload-Aware Scheduling

Kubernetes v1.37 marks a significant milestone for Workload-Aware Scheduling, enhancing support for complex batch and AI/ML workloads. The core Workload and PodGroup APIs, essential for gang scheduling, have graduated to Beta. This release introduces the CompositePodGroup API to manage hierarchical scheduling requirements for heterogeneous Pod groups. New controller integration APIs and the workloadbuilder Go library simplify integration of out-of-tree controllers with workload-aware scheduling features. The native Job controller now fully utilizes these expanded APIs for advanced scheduling policies and topology awareness. Gang scheduling and Workload/PodGroup APIs are now more robust with Beta graduation and updated versioning. PodGroup queueing has been improved, making PodGroups first-class citizens in the scheduling queue. The minCount field for PodGroups is now mutable, allowing dynamic adjustments for elastic workloads. Workload-aware preemption is now integrated into the GenericWorkload feature and respects PodGroup disruption modes. The CompositePodGroup API enables multi-level topology-aware scheduling, allowing for complex co-location constraints across different infrastructure levels.

https://kubernetes.io/blog/2026/09/08/kubernetes-v1-37-advancing-workload-aware-scheduling/ kubernetes.io

RSS Hunter • Sep 8

Kubernetes v1.37: KubeletInUserNamespace (aka Rootless mode) Graduates to Beta

Kubernetes v1.37 promotes KubeletInUserNamespace to beta, allowing node components like kubelet and runtimes to run as non-root users. This rootless mode, an evolution from a 2018 experiment, enhances security by confining potential container breakout vulnerabilities. Unlike pod user namespaces, this focuses on the host's node components. The feature mitigates risks from vulnerabilities such as CVE-2022-0811 and CVE-2023-27561, which historically could grant attackers root privileges. By running in a user namespace, an attacker's impact is limited to the non-root user's account, preventing kernel or firmware modification. User namespaces do not protect against kernel vulnerabilities themselves and should be combined with other hardening measures like seccomp. This feature has various use cases, including production clusters, shared machines, laptops, and AI sandboxes. It works by mapping a host non-root user to a fake root inside the namespace, which suffices for most node component tasks. The user namespace must be created externally, for example, using Rootless Docker. The KubeletInUserNamespace feature gate, now enabled by default, allows kubelet to ignore specific permission errors. Recent improvements include better support for idmapped tmpfs, user-namespaced pods, and writable cgroups, enabling nested Kubernetes deployments. Tools like kind, minikube, Usernetes, and k3s offer easy ways to create rootless Kubernetes clusters. The project plans to graduate this feature to General Availability in the future and welcomes community involvement.

https://kubernetes.io/blog/2026/09/04/kubernetes-v1-37-rootless-beta/ kubernetes.io

RSS Hunter • Sep 4

Kubernetes v1.37: DRA Updates

Kubernetes 1.37 introduces significant advancements for Dynamic Resource Allocation (DRA), with the Extended Resource support reaching General Availability. This allows DRA drivers to manage traditional extended resources without separate device plugins. ResourceClaims now include a devices field in their status, enabling better visibility into device states like network interface names and IP addresses. DRA device taints and tolerations are now stable, mirroring node taints to manage device availability and maintenance. A standardized numaNode device attribute ensures consistent naming across different drivers.The ResourceClaim workload support has advanced to Beta, allowing a single claim to be shared across multiple Pods, overcoming previous limitations. The DRA Device Attributes Downward API, now in Beta, facilitates injecting device attributes directly into workloads, simplifying custom controller needs. Alpha features include list types for attributes, enabling more complex device matching, and node allocatable resource requests to prevent node oversubscription.Resource availability visibility is in its second Alpha, offering point-in-time snapshots. DRA's Optional Node Operations allows drivers to skip unnecessary node setup calls. Derived Attributes, a new Alpha feature, uses CEL expressions for flexible device matching based on custom rules. DRA Device Compatibility Groups prevent scheduling incompatible device combinations.The PreQueueingHint extension point, also in Alpha, optimizes pod scheduling by reducing the scope of scans during scale-ups. DRA Consumable Capacity now supports fractional values, enhancing precision in resource allocation. These ongoing developments highlight Kubernetes' commitment to improving resource management capabilities.

https://kubernetes.io/blog/2026/09/03/kubernetes-v1-37-dra-updates/ kubernetes.io

RSS Hunter • Sep 3

Kubernetes v1.37: Scale Workloads to Zero with HorizontalPodAutoscaler

Kubernetes v1.37 now offers Beta API support for scaling workloads down to zero replicas by default. This enables HorizontalPodAutoscalers to reduce idle resources to zero and then scale back up when needed. Previously, this functionality required external add-ons or alpha feature gates. Scaling to zero is particularly beneficial for workloads like queue consumers, where resource savings can be significant, especially with expensive dedicated hardware.The primary trade-off is an increased cold-start time as the system needs to observe metrics, schedule a pod, and initiate the application. This approach is most effective when work can be temporarily held in a durable queue. Unlike CPU or memory metrics which cease to exist when pods are gone, object and external metrics persist independently. This allows the HorizontalPodAutoscaler to continue monitoring metrics such as queue length even when no worker pods are running.To scale from zero, an external metrics adapter like the Prometheus Adapter is necessary to expose metrics such as queue consumer lag to Kubernetes. The HorizontalPodAutoscaler is then configured with a minimum of zero replicas and targets an external metric to dictate scaling. A special ScaledToZero status condition distinguishes automatic scaling down from a manual pause, ensuring the HorizontalPodAutoscaler can resume control. This feature, previously in Alpha, has graduated to Beta in v1.37 and is enabled by default for improved resource efficiency.

https://kubernetes.io/blog/2026/09/02/kubernetes-v1-37-hpa-scale-to-zero-beta/ kubernetes.io

RSS Hunter • Sep 2

Kubernetes v1.37: etcd RangeStream Cuts Memory Use on Large List Reads

etcd RangeStream is graduating to beta in Kubernetes v1.37, working with etcd v3.7. This feature significantly reduces memory usage for the API server and etcd when reading large collections of resources. Previously, populating the API server's watch cache involved expensive full reads from etcd. While pagination existed, it was based on key count, not object size, leading to unpredictable memory spikes and potential Out-Of-Memory errors. etcd's unary Range RPC assembled entire pages before sending, exacerbating memory pressure on both etcd and the API server.The new RangeStream RPC in etcd v3.7 addresses this by streaming results in chunks. The chunk size is adaptively tuned to value sizes, bounding reads by bytes rather than key count. This allows memory to be freed incrementally as the stream progresses. When the EtcdRangeStream feature gate is enabled, the API server utilizes RangeStream for both watch cache initialization and fallback list requests. It decodes and releases chunks as they arrive, preventing the entire collection from being held in memory simultaneously. This feature requires Kubernetes v1.37+ and etcd v3.7+. The API server automatically detects etcd's support and falls back if RangeStream is not implemented. To confirm RangeStream is active, monitor the etcd_request_duration_seconds_count{operation="listStream"} metric for non-zero values.

https://kubernetes.io/blog/2026/09/01/kubernetes-v1-37-etcd-range-stream/ kubernetes.io

RSS Hunter • Sep 1

Kubernetes v1.37: Storage Version Migration Enabled by Default

Storage Version Migration (SVM) has reached General Availability in Kubernetes v1.37, now enabled by default. This feature addresses the challenge of updating the storage version of Kubernetes API resources. Resources are stored using a specific schema representation, and changes to this representation require re-writing existing objects. A common use case is promoting a Custom Resource Definition (CRD) to newer API versions while deprecating older ones. Without SVM, removing old versions was unsafe if resources still used them for storage. Similarly, encryption at rest or key rotation necessitated re-writing stored data to apply new encryption. Previously, this required manual scripting or external tools, which were error-prone and difficult to manage. SVM simplifies this by introducing a declarative StorageVersionMigration API object. The built-in controller watches for these objects and automatically migrates resources to the designated default storage version. Initiating a migration involves creating a StorageVersionMigration object, which the controller then processes. The status of the migration can be monitored, and a successful completion indicates all resources are updated. CRD authors can now include migration manifests directly with CRD upgrades for a streamlined process.

https://kubernetes.io/blog/2026/08/31/kubernetes-v1-37-storage-version-migration-ga/ kubernetes.io

RSS Hunter • Aug 31

Kubernetes v1.37: Pod Certificates and Cluster Trust Bundles

Kubernetes has introduced Pod Certificates and Cluster Trust Bundles as a new production identity mechanism. This feature, now generally available, integrates X.509 certificate issuance for TLS and mTLS directly into core Kubernetes. This is a significant step beyond the previous primary mechanism, service account JWTs. While JWTs offer ease of use and broad compatibility, they suffer from a crucial weakness: they are bearer tokens, making them susceptible to unauthorized access if compromised. Pod Certificates, on the other hand, leverage proof-of-possession credentials based on asymmetric cryptography, primarily X.509 certificates. This enhances security by requiring a demonstration of possession rather than simply sharing the credential itself.The goal is to make using X.509 certificates from Kubernetes workloads as straightforward as JWTs while maintaining high security standards. Pod Certificates are designed for flexibility, allowing for various certificate types within a single cluster through a pluggable interface. This contrasts with the single, standardized flavor of service account JWTs. Future Kubernetes versions are expected to offer built-in certificate providers for service DNS names and SPIFFE client certificates.The architecture involves the application requesting certificates, Kubelet issuing requests and managing trust bundles, and a signer controller fulfilling these requests and publishing trust bundles. The issuance process is chronological, starting with the pod's scheduling and culminating in the application accessing its keys and certificates. Automatic rotation is built-in, with applications needing to handle these updates. Security checks are integrated into the kube-apiserver to enforce isolation and prevent compromised nodes from gaining undue access. To experiment with these features, users need to install a third-party signer like Tinycert, which provides example signers and a library to help applications load SPIFFE certificates.

https://kubernetes.io/blog/2026/08/28/kubernetes-v1-37-pod-certificates-and-cluster-trust-bundles/ kubernetes.io

RSS Hunter • Aug 28

Kubernetes v1.37: Metrics API graduates to stable

Kubernetes v1.37 has promoted the metrics.k8s.io API to stable v1. This API provides essential CPU and memory usage data for nodes and Pods. It underpins critical functionalities like kubectl top and resource-metrics-based autoscaling. For cluster operators and developers, this graduation signifies the API's guaranteed stability. The v1 API retains the same resource types and fields as its v1beta1 predecessor. Introduced as alpha in v1.6, the Resource Metrics API became beta in v1.8 and has been reliably used in production for years. The API exposes NodeMetrics for node resource usage and PodMetrics for Pod resource usage, including per-container breakdowns. It deliberately offers only the metrics necessary for autoscaling and basic inspection, not a full monitoring solution. The v1 API surface is identical to v1beta1, excluding only the API version itself. While kubectl top supports both versions, preferring v1, the Horizontal Pod Autoscaler controller currently only supports v1beta1. No feature gate needs to be enabled, as the API is served via API aggregation by implementations like metrics-server. To make v1 available, implementations must serve it and have an associated APIService registered. During this transition, implementations should continue serving both v1 and v1beta1 for backward compatibility.

https://kubernetes.io/blog/2026/08/27/kubernetes-v1-37-metrics-api-ga/ kubernetes.io

RSS Hunter • Aug 27

Kubernetes v1.37: Garhwal

Kubernetes v1.37, themed Garhwal, has been released with 67 enhancements. Sixteen features have graduated to Stable, 23 to Beta, and 27 are entering Alpha. The release theme symbolizes the interconnectedness of community contributions. A key Stable update is resilient watch cache initialization, enhancing API server startup and recovery by preventing etcd overload. Horizontal Pod Autoscaler scale to zero support, now in Beta, allows workloads to scale down when idle for cost reduction, except for CPU and memory metrics. Manifest-based admission control configuration also graduates to Beta, enabling policies and webhooks to be loaded from disk. Alpha introductions include Pod-level checkpoint and restore, extending CRI with new RPCs for container runtimes. Notable Stable graduations include KYAML for safer YAML subsets and the metrics.k8s.io API for resource usage. SELinuxMount and SELinuxChangePolicy flags are now Stable and enabled by default for volumes. Several Device Resource Allocation (DRA) features have reached Stable, including ResourceClaim status for device configuration, handling extended resource requests, device taints and tolerations, and a standardized NUMA node device attribute. Node declared features, a framework for declaring feature-gated Node capabilities, also graduates to Stable.

https://kubernetes.io/blog/2026/08/26/kubernetes-v1-37-release/ kubernetes.io

RSS Hunter • Aug 26

How to Pretty-Print Your Kubernetes YAML as KYAML and Why You'd Want To

KYAML offers a stricter, more consistent way to write Kubernetes manifests, addressing issues found in standard YAML. Designed as a subset of YAML, KYAML maintains compatibility with existing parsers while narrowing stylistic choices. It aims to prevent common traps, such as whitespace sensitivity and silent type coercion, which can lead to unexpected behavior. Unlike standard YAML, KYAML explicitly defines structure and types, using braces for maps and brackets for lists. It mandates quoting string values to avoid silent type conversions, like the "Norway Bug" where "NO" is parsed as boolean false. KYAML also supports comments and trailing commas, improving the configuration writing experience compared to JSON. Kubernetes 1.34+ allows generating KYAML output via kubectl -o kyaml. Tools like sigs.k8s.io/yamlfmt and google/yamlfmt can convert existing YAML files to KYAML format. Adopting KYAML is a choice for greater consistency and fewer errors, especially in team environments, without requiring changes to existing Kubernetes tools or pipelines. It is a new habit for writing more robust configurations, not a mandatory migration.

https://kubernetes.io/blog/2026/08/11/how-to-pretty-print-kubernetes-yaml-as-kyaml/ kubernetes.io

RSS Hunter • Aug 11

Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard

The Kubernetes Gateway API has released version 1.6.0, enhancing its capabilities for service networking. This new release marks a significant advancement by standardizing layer 4 protocol routing. Previously, Gateway API's stable foundation was primarily for HTTP and TLS traffic. Version 1.6.0 now brings TCPRoute and UDPRoute resources to the stable, version 1 API. These resources allow routing for raw TCP and UDP traffic, addressing a gap for workloads like databases and IoT telemetry. Experimental innovation is also being streamlined with clearer API boundaries. New experimental resources now reside in a separate API group, gateway.networking.x-k8s.io, and use an 'X' prefix. This distinction makes experimental and standard resources easier to identify. The XBackend resource, for instance, is a new experimental addition enabling extended backend configurations. It allows for destinations like ExternalHostname, which were previously problematic for the stable Service resource. This experimental feature is designed for use cases like egress traffic to cloud APIs. The graduation of TCP and UDP routing to standard solidifies Gateway API as a universal networking solution. Users can now adopt version 1.6.0 with various compatible Gateway controller implementations. The Gateway API community encourages feedback and contributions from users and developers.

https://kubernetes.io/blog/2026/08/03/gateway-api-v1-6-release/ kubernetes.io

RSS Hunter • Aug 3

Kubernetes v1.37 Sneak Peek

Kubernetes v1.37 is nearing its release with several planned changes. The --filename/-f flag for kubectl run will be deprecated. Static Pods can no longer directly reference Secrets or ConfigMaps due to a bug fix. Kube-proxy's support for IPVS mode is being deprecated and will be removed by v1.43. Support for cgroup v1 is being phased out, and users should migrate to cgroup v2. SELinux volume relabeling, known as SELinuxMount, is graduating to General Availability. The metrics.k8s.io API is also graduating to GA after being in Beta for nearly nine years. Kubelet in User Namespace, or Rootless Mode, is expected to graduate to Beta. A volume health monitor feature is being reintroduced to Alpha with new CSI RPCs. These changes aim to improve the maintenance and health of Kubernetes environments.

https://kubernetes.io/blog/2026/07/31/kubernetes-v1-37-sneak-peek/ kubernetes.io

RSS Hunter • Jul 31

How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server

Kubernetes controllers, commonly written in Go with kubebuilder and controller-runtime, offer efficient development for distributed workloads. However, as load increases, understanding controller-runtime's internal workings becomes crucial to avoid production issues. The core concept is that r.Get() and r.List() within a reconciler do not directly query the API server; instead, they access a local, in-memory cache. This cache is populated initially by a list operation and subsequently maintained through watches.This design makes local reads exceptionally fast, avoiding strain on the control plane. The trade-off, however, is that the local cache can consume significant memory and may return stale data. Writes, conversely, are always directed to the API server, bypassing the cache. The size of this local cache and the indexing strategies directly impact memory usage. An inefficient List() operation can inadvertently lead to slow linear scans of numerous objects.The reconciliation loop continuously compares desired states with actual states to bring them into alignment. Events are queued, leading to a reconcile function that reads the current state, determines necessary actions, and potentially generates new events. The cache's primary purpose is to provide a rapid, up-to-date view of object states for these reconciliation loops. This watch-based model, originating from client-go, prevents constant API polling by maintaining a single, long-lived connection for updates.controller-runtime abstracts the complexities of lower-level client-go components like Reflector, DeltaFIFO, and Indexer. The Manager orchestrates the shared cache, controllers, and other services. An Informer monitors a specific GVK, maintaining a local store (Indexer) and dispatching events to subscribers. ResourceEventHandlers process these events, updating the store, which is then reflected in the controller's view. A workqueue manages object keys for processing by workers, and Predicates filter events before enqueueing.The Reflector is the sole component directly interacting with the API server, performing an initial list and then establishing a watch. It leverages resourceVersion to resume watches and ensure no events are missed. If a watch connection is lost or the API server indicates an outdated resourceVersion, a relist is performed. DeltaFIFO acts as an ordered buffer for changes, grouping deltas by object key. It provides all accumulated changes for an object since the last Pop() call, ensuring ordered, batched processing. However, it does not collapse consecutive "Added" or "Updated" events, which could lead to multiple processing rounds for rapidly changing objects.

https://kubernetes.io/blog/2026/07/29/controller-runtime-cache-explained/ kubernetes.io

RSS Hunter • Jul 29

Building a Custom Metrics Exporter for Kubernetes

Kubernetes scales based on CPU and memory, but real-world demands often require custom metrics like queue depth or active connections. A Prometheus metrics exporter bridges this gap by exposing application state as plain text on an HTTP /metrics endpoint for Prometheus to scrape.Exporters can be standalone services or embedded directly within an application, depending on the data source and control over application code. Prometheus expects metrics in specific formats: counters for ever-increasing values, gauges for fluctuating values, and histograms for distributions. Metric names should follow a clear " __ " snake_case convention. Building an exporter in Go involves initializing a module, setting up the Prometheus client library, and registering metrics like counters, gauges, and histograms. These metrics are then updated by a polling loop that reads from your application's data sources. An HTTP server exposes the /metrics endpoint, along with a /healthz endpoint for liveness probes. The exporter is packaged into a minimal container image using a multi-stage Docker build, ensuring a small footprint. Deployment to a Kubernetes cluster involves a Deployment for pod management and a Service to provide a stable address for Prometheus. Prometheus discovers these metrics via a ServiceMonitor if using the Prometheus Operator, or through annotation-based discovery on the Pod template. Verification involves checking the Prometheus targets page to ensure the exporter is "UP" and querying the metrics in the expression browser. This foundation allows for autoscaling with the HorizontalPodAutoscaler, which uses a metrics adapter (like the Prometheus Adapter) to integrate custom metrics with the Kubernetes Custom Metrics API. This enables workloads to scale based on business-relevant signals, not just basic resource utilization.

https://kubernetes.io/blog/2026/07/14/custom-metrics-exporter-kubernetes/ kubernetes.io

RSS Hunter • Jul 14

Operating AI/ML Workloads on Kubernetes: A Headlamp Plugin for Kubeflow

Kubernetes is increasingly the default platform for AI and machine learning workloads. Kubeflow is a popular tool that leverages Kubernetes' Custom Resource Definitions (CRDs) for managing these ML tasks. While Kubeflow's specialized dashboards help data scientists, they often obscure the underlying Kubernetes layer, forcing operators to use kubectl for troubleshooting. The Headlamp Kubeflow plugin bridges this gap by displaying Kubeflow's custom resources directly within a general-purpose Kubernetes UI. This allows operators to observe and manage ML resources using the same primitives as other cluster components. Headlamp itself is an extensible Kubernetes web UI maintained under Kubernetes SIG UI. The plugin provides cluster-level insights by directly querying the Kubernetes API server, offering details on Pod conditions, failure reasons, and resource states across namespaces. It supports various Kubeflow components like Notebooks, Pipelines, Katib for hyperparameter tuning, Training, and Spark. The plugin allows inspection of notebook Pod details, Katib trial statuses, and pipeline states without relying on Kubeflow's backend database. It also maps ML resources as graph nodes within Headlamp, visualizing relationships between them. This approach demonstrates a pattern for any CRD-heavy platform to provide operators with essential cluster-level visibility directly within their familiar tools.

https://kubernetes.io/blog/2026/07/13/introducing-headlamp-plugin-for-kubeflow/ kubernetes.io

RSS Hunter • Jul 13

Kubernetes Dashboard to Headlamp: A Step-by-Step Guide

Kubernetes Dashboard and Headlamp offer cluster visibility but function differently. Headlamp is a desktop or in-cluster client that uses kubeconfig to connect to multiple clusters, respecting user identity. Kubernetes Dashboard is an in-cluster web app typically accessed via token authentication. To migrate, document current Dashboard usage, verify kubeconfig works, and choose a rollout plan like parallel rollout. Decide whether Headlamp will run on your desktop or within the cluster, with desktop being faster for individual use and in-cluster better for shared access. Desktop installation is straightforward via package managers, while in-cluster installation uses Helm or YAML manifests. Authentication for desktop Headlamp relies on kubeconfig, while in-cluster setups often use OpenID Connect (OIDC) or an external auth layer for sign-in. RBAC permissions are respected by Headlamp, meaning users only see and can perform actions they are authorized for. Headlamp excels at multi-cluster management, allowing users to switch between clusters directly within the UI using their kubeconfig. This provides a more integrated experience compared to the single-cluster focus of Kubernetes Dashboard.

https://kubernetes.io/blog/2026/07/13/kubernetes-dashboard-to-headlamp/ kubernetes.io

RSS Hunter • Jul 13

Announcing etcd v3.7.0

SIG etcd announces the release of etcd v3.7.0, a significant update to the distributed key-value store. This release introduces the much-anticipated RangeStream feature, enabling efficient streaming of large result sets. It also delivers various performance enhancements, including faster lease operations and optimized keys-only range requests. The legacy v2 store dependency has been fully eliminated, with etcd now booting exclusively from v3store, simplifying operations. A comprehensive protobuf overhaul replaces outdated libraries with supported versions, improving security and maintainability. The release also updates core dependencies, bbolt to v1.5.1 and raft to v3.7.0. Further improvements include support for Unix sockets for local development and testing. etcdutl commands now feature timeout arguments to prevent indefinite blocking. Client v3 offers more authentication flexibility with direct JWT setting and the ability to retrieve AuthStatus without prior authentication. New watch and request duration metrics have been added for enhanced observability. etcdctl commands have been reorganized for clarity, and deprecated experimental flags have been removed. Significant cleanup of legacy v2 API packages and code has also been performed.

https://kubernetes.io/blog/2026/07/08/announcing-etcd-3.7/ kubernetes.io

RSS Hunter • Jul 8

Open source maintainership in the age of AI

AI is transforming software development, enabling more contributors through code generation. However, this advancement outpaces improvements in code maintenance, posing a challenge. The Kubernetes community is proactively adapting to AI-assisted coding by establishing a comprehensive AI policy. This policy aims to balance innovation with accountability, ensuring code quality and human oversight. A core principle is transparency, requiring contributors to disclose AI usage in their pull requests.Crucially, human accountability remains paramount; AI cannot be listed as a co-author or co-signer. Contributors must also personally explain any AI-generated code, preventing knowledge gaps. The project enforces Contributor License Agreements for all co-authors, including AI-assisted ones, flagging incomplete PRs. Automated AI reviews are being explored to enhance code quality and provide initial feedback.Tools like GitHub Copilot and CodeRabbit are being evaluated and tested within specific Kubernetes projects. These tools can act as quality gates, offering quick spot checks before human review. The community is actively seeking help to tune review tools, evaluate emerging AI technologies, and explore AI's potential in reducing maintainer burnout and assisting with test triage.

https://kubernetes.io/blog/2026/06/26/open-source-maintainership-in-the-age-of-ai/ kubernetes.io

RSS Hunter • Jun 26

Introducing the Cluster API plugin for Headlamp

Headlamp is an open-source Kubernetes UI project for managing cluster resources from a browser. Cluster API (CAPI) provides declarative, Kubernetes-style APIs for cluster lifecycle management. The Headlamp Cluster API plugin simplifies managing CAPI resources, eliminating the need for raw kubectl commands. This plugin adds a dedicated CAPI section to Headlamp, offering full visibility into core CAPI resources through consistent list and detail views. Key features include a cluster overview, Machine visibility, and a centralized Cluster API dashboard for health monitoring. Users can track KubeadmControlPlane replicas, scale MachineDeployments and MachineSets, and visualize owned resource hierarchies. The plugin also allows inspection of KubeadmConfig without raw YAML and provides topology awareness. A map view visualizes Cluster, Control Plane, and Worker relationships, supporting both v1beta1 and v1beta2 Cluster API versions. Prometheus metrics are integrated for live performance data inline on detail pages. Developed during the CNCF LFX Mentorship program, the plugin aims to enhance the Cluster API management experience. This is an Alpha release, and community feedback is encouraged for future development.

https://kubernetes.io/blog/2026/06/25/headlamp-cluster-api-plugin/ kubernetes.io

RSS Hunter • Jun 25

Inspect Volcano workloads faster with Headlamp

Volcano is a cloud-native batch scheduler for Kubernetes, designed for high-performance computing, AI/ML, and other batch workloads. Kubernetes was originally built for long-running services, whereas batch workloads often require dynamic job arrival, resource competition, and co-starting multiple workers. Volcano extends Kubernetes with concepts like queues, priorities, quotas, and gang scheduling, treating workloads as a whole rather than independent Pods. The Volcano plugin for Headlamp, an extensible Kubernetes web UI, brings these scheduling details into a single interface.The plugin provides dedicated views for Volcano Jobs, Queues, and PodGroups, making it easier to operate and troubleshoot batch workloads. The Job view displays workload status, task details, Pod status, and allows direct actions like suspending/resuming and accessing logs. The Queue view offers insights into resource allocation, capacity, and reservation details. The PodGroup view clarifies gang scheduling states and potential blockers.A key feature is the map view, which visually represents how Jobs, Queues, PodGroups, and Pods are interconnected, helping to quickly identify issues in pending or non-progressing workloads. This plugin enhances the interactive troubleshooting experience by centralizing related resources, structured details, and runtime output, without replacing CLI tools for automation. Future enhancements may include Prometheus integration and richer scheduling insights. Users can install the plugin via Headlamp's Plugin Catalog and provide feedback to shape its development.

https://kubernetes.io/blog/2026/06/25/visual-context-volcano-headlamp-plugin/ kubernetes.io

RSS Hunter • Jun 25

See your serverless: introducing the Headlamp plugin for Knative

Headlamp is an open-source project designed to manage and debug Kubernetes resources. Knative enables serverless workloads on Kubernetes but can be complex to operate across multiple tools. To address this, a Headlamp plugin for Knative has been developed. This plugin allows users to manage Knative extensively from within Headlamp. It integrates Knative resources into Headlamp's resource mapping view, showing relationships between KServices, Revisions, and DomainMappings. The plugin offers a detailed view for KServices, enabling live edits of traffic splits and autoscaling configurations. Users can also restart pods and access logs directly from the KService header. The plugin facilitates granular traffic splitting across revisions for phased rollouts and A/B testing. It visually displays traffic distribution, readiness status, and tags for each revision. Autoscaling configurations are clearly presented, indicating whether settings are explicit or inherited from cluster defaults. When combined with the Prometheus plugin, it provides metrics like request rates and latency for KServices and Revisions. The plugin also offers list and detail views for other Knative CRDs, including Revisions, DomainMappings, and a networking overview. Installation involves searching for and installing the Knative plugin from Headlamp's Plugin Catalog. Feedback and bug reports can be submitted via GitHub issues or the Kubernetes Slack channel.

https://kubernetes.io/blog/2026/06/25/headlamp-knative-plugin/ kubernetes.io

RSS Hunter • Jun 25

Spotlight on WG Device Management

The Device Management Working Group addresses the increasing need for specialized hardware management in Kubernetes. Traditional resource allocation methods are insufficient for AI, Edge, and Telecommunications workloads requiring GPUs, TPUs, and specific network interfaces. The group's primary project, Dynamic Resource Allocation (DRA), has reached General Availability, signifying a major advancement. DRA provides a structured four-stage framework for device management: modeling, requesting, scheduling, and actuation. This new approach allows vendors to advertise granular hardware capabilities and users to specify precise hardware needs. The Kubernetes scheduler then intelligently matches these requirements to available hardware. DRA replaces the legacy Device Plugin API, which treated devices as simple integers, with a flexible and declarative API. The working group is a cross-SIG effort involving several SIGs to ensure comprehensive integration across Kubernetes components. Current initiatives focus on enhancing DRA's expressiveness, supporting operational visibility, and improving multi-node and complex hardware topology management. Future work includes device health monitoring and better support for grouped device usage. The working group aims to create a more programmable and hardware-aware future for Kubernetes.

https://kubernetes.io/blog/2026/06/24/wg-device-management-spotlight-2026/ kubernetes.io

RSS Hunter • Jun 24

Spotlight on SIG Storage

The article highlights SIG Storage, the Kubernetes Special Interest Group responsible for persistent data and volume management. Xing Yang, a co-chair of SIG Storage, discusses the group's evolution from handling basic persistent volumes to advancing complex storage features. Initially designed for stateless workloads, Kubernetes now supports stateful applications, necessitating dedicated storage solutions. SIG Storage formed to address these challenges, introducing primitives like PersistentVolumes and PersistentVolumeClaims.A significant advancement was the Container Storage Interface (CSI), which enables third-party storage providers to integrate their systems without core Kubernetes modifications. Current work includes Volume Group Snapshot for crash-consistent multi-volume snapshots and Changed Block Tracking for efficient backups, both recently graduated to stable versions. The Container Object Storage Interface (COSI) is also progressing to standardize object storage integration.Recent wins for users include the graduation of VolumeAttributesClass to General Availability, allowing dynamic tuning of storage properties like IOPS. Future roadmaps feature Volume Health for improved operational visibility and potential automated remediation. SIG Storage seeks community help with bug fixes, tests, reviews, and feedback on features like Mutable PV Affinity and volume replication.Challenges for stateful workloads include data gravity, day-2 operational complexity, and data mobility. As AI workloads grow, storage in Kubernetes is expected to become more intelligent, with object storage gaining prominence. High performance, low latency storage, and data-aware scheduling are also anticipated trends. SIG Storage invites community involvement to tackle these evolving storage demands.

https://kubernetes.io/blog/2026/06/15/sig-storage-spotlight-2026/ kubernetes.io

RSS Hunter • Jun 15

From Kubernetes Dashboard to Headlamp: Understanding the Transition

The Kubernetes Dashboard, once a primary visual interface for Kubernetes, has been archived. It served as an important onramp for many users, simplifying cluster visibility and resource inspection. Headlamp now carries this legacy forward, building on the Dashboard's foundation. It offers a clear visual interface while incorporating modern Kubernetes usage patterns. Headlamp provides multi-cluster visibility, application-centric views through Projects, and extensibility via plugins. The transition aims to honor the user-centered legacy of Dashboard and offer a growing UI solution. Many familiar workflows from Kubernetes Dashboard are retained in Headlamp, ensuring continuity and ease of use. Headlamp expands capabilities by allowing multi-cluster management from a single interface, reducing friction for distributed environments. Projects within Headlamp offer application-centered views, grouping related resources for better understanding and troubleshooting. The platform is also extensible through plugins, such as the Flux plugin for GitOps workflows or an AI Assistant for guidance. Headlamp offers flexible deployment options, usable as an in-cluster tool or a desktop application. Understanding current Dashboard usage, including clusters, namespaces, and authentication, aids a smooth transition to Headlamp.

https://kubernetes.io/blog/2026/06/01/dashboard-to-headlamp/ kubernetes.io

RSS Hunter • Jun 1

Reconciling the Past: Correcting Records for Unfixed Kubernetes CVEs

Kubernetes is improving transparency by refining its CVE records for better accuracy. They discovered discrepancies in older CVE records, with some incorrectly listing fixed versions. The Kubernetes Security Response Committee will correct these records on June 1, 2026. This might lead to vulnerability scanners identifying previously undetected issues. This post offers technical details about three unfixed vulnerabilities: CVE-2020-8561, CVE-2020-8562, and CVE-2021-25740. The updates ensure correct vulnerability scanning and clarify persistent administrative mitigation needs. CVE-2020-8554, also unfixed, will receive a standardized version number format. The identified vulnerabilities remain unfixed because fixing them would disrupt core Kubernetes functionality. Each vulnerability has specific mitigations that administrators should implement to secure their clusters. These actions are crucial given the vulnerabilities’ architectural nature. The project emphasizes a "secure by configuration" approach to manage these risks. Updating these records shows a maturing security ecosystem promoting transparency and accurate risk assessment.

https://kubernetes.io/blog/2026/05/26/reconciling-unfixed-kubernetes-cves/ kubernetes.io

RSS Hunter • May 26

Announcing etcd 3.7.0-beta.0

SIG-Etcd has released the first beta of etcd v3.7.0, a significant update for the distributed database. This version introduces RangeStream, a feature designed to improve handling of large result sets, enhancing latency and memory management. The release also includes refactoring and cleanup of legacy components and interfaces, improving overall performance. The developers encourage users to test the beta and report any issues found in the etcd repository. A key highlight is the removal of the last vestiges of etcd v2store, completing the transition to v3store. This transition may introduce breaking changes, particularly for users not on v3.6.11, so feedback is requested regarding any problems encountered. This beta release also incorporates updates to bbolt and raft libraries. Furthermore, the release timeline is linked to the End of Life (EOL) for etcd v3.4, which will cease updates after May. The community is prepared to release an additional security patch for v3.4 if required, before its ultimate deprecation. Users are urged to upgrade from v3.4. Future betas are planned, potentially with further protobuf refactoring leading to release candidates and the final version in June or early July. Feedback is actively solicited through GitHub issues, the Kubernetes Slack channel, and the etcd-dev mailing list.

https://kubernetes.io/blog/2026/05/20/etcd-370-beta/ kubernetes.io

RSS Hunter • May 20

Kubernetes v1.36: New Metric for Route Sync in the Cloud Controller Manager

This article, originally misdated, now reflects a May 15, 2026, publication date. Kubernetes v1.36 introduces a new alpha metric, route_controller_route_sync_total, for the Cloud Controller Manager's route controller. This metric tracks route sync operations with the cloud provider, aiding in monitoring the CloudControllerManagerWatchBasedRoutesReconciliation feature gate. This feature, introduced in v1.35, switches the route controller to a watch-based approach. This change reduces API calls by only reconciling routes when nodes change. To test the new feature, compare the metric's behavior with the feature gate disabled and enabled. With the feature gate disabled, the counter increments at a fixed interval. Conversely, with the feature enabled, the counter only increments upon node changes. This difference is most noticeable in stable clusters with infrequent node modifications. Feedback can be provided through Kubernetes Slack, a GitHub issue, and the SIG Cloud Provider community page. Further details are available in KEP-5237.

https://kubernetes.io/blog/2026/05/15/ccm-new-metric-route-sync-total/ kubernetes.io

RSS Hunter • May 15

Kubernetes v1.36: Mixed Version Proxy Graduates to Beta

The Mixed Version Proxy (MVP) enhances Kubernetes cluster upgrades by safely routing requests for unknown resources to newer API servers, preventing 404 errors. Initially introduced as an Alpha feature in Kubernetes 1.28, MVP is now moving to Beta in version 1.36 and will be enabled by default. MVP addresses the issue of API servers with differing versions during upgrades, where requests for new resources might fail on older servers. Instead of the incorrect 404, the request is proxied to a server capable of handling it. The Beta version of MVP uses aggregated discovery instead of the StorageVersion API for determining peer capabilities, improving functionality. This update also includes peer-aggregated discovery, providing clients with a unified view of all available APIs. To enable MVP, API servers require the --peer-ca-file flag, along with --peer-advertise-ip and --peer-advertise-port if needed. With kubeadm, you can include those flags in your ClusterConfiguration file to streamline the process. Users are encouraged to test MVP in staging environments and provide feedback to the SIG API Machinery as part of the 1.36 upgrade.

https://kubernetes.io/blog/2026/05/15/kubernetes-1-36-feature-mixed-version-proxy-beta/ kubernetes.io

RSS Hunter • May 15

Kubernetes v1.36: Deprecation and removal of Service ExternalIPs

The .spec.externalIPs field in Kubernetes Services, initially designed for non-cloud load-balancer functionality, is now deprecated due to security vulnerabilities identified in CVE-2020-8554. This field allows specifying additional IP addresses a Service responds to, but it has inherent security risks because it assumes trust among all users. Kubernetes 1.21 already recommended disabling .spec.externalIPs, and an admission controller was introduced to enforce this. Alternatives, like manually managed LoadBalancer services or non-cloud load balancer controllers such as MetalLB, offer better security and control. MetalLB allows administrators to control IP address assignments, mitigating security concerns. The Gateway API also provides a secure solution, giving administrators control over the IP through a Gateway resource. Kubernetes 1.36 officially deprecated .spec.externalIPs and started issuing warnings about its usage. Kube-proxy support for the feature will be disabled in a future release, with full removal planned in subsequent versions. Users are encouraged to migrate away from this insecure feature.

https://kubernetes.io/blog/2026/05/14/kubernetes-v1-36-deprecation-and-removal-of-service-externalips/ kubernetes.io

RSS Hunter • May 14

Kubernetes v1.36: Advancing Workload-Aware Scheduling

Kubernetes v1.35 introduced workload-aware scheduling improvements, including the Workload API and basic gang scheduling for identical Pods. Kubernetes v1.36 refines this architecture by separating the Workload API (static template) from the new PodGroup API (runtime state). This separation streamlines the kube-scheduler, enabling it to directly read PodGroup information for enhanced performance.A new PodGroup scheduling cycle allows atomic processing of workloads, evaluating entire groups as a unified operation to prevent deadlocks. If a valid placement is found and group constraints are met, Pods are bound together; otherwise, the entire group is considered unschedulable and retries later. This forms the foundation for gang scheduling, ensuring all-or-nothing placement for strict workload requirements.Topology-aware scheduling in v1.36 enables defining topology constraints on PodGroups, co-locating Pods within specific physical or logical domains to reduce network latency. This involves generating, evaluating, and scoring candidate placements based on scheduling constraints.Workload-aware preemption is introduced to support the PodGroup scheduling cycle, preempting Pods from multiple Nodes simultaneously to make space for an entire PodGroup. It treats the PodGroup as a single preemptor unit, with PodGroup priority and disruptionMode fields controlling preemption behavior.Finally, v1.36 integrates Dynamic Resource Allocation (DRA) with the Workload API, allowing PodGroups to request and share specialized hardware resources through ResourceClaims. These advancements lay a robust foundation for building advanced workload scheduling capabilities in future Kubernetes releases.

https://kubernetes.io/blog/2026/05/13/kubernetes-v1-36-advancing-workload-aware-scheduling/ kubernetes.io

RSS Hunter • May 13

Kubernetes v1.36: PSI Metrics for Kubernetes Graduates to GA

Pressure Stall Information (PSI) has been integrated into the Linux kernel since 2018, providing high-fidelity signals for identifying resource saturation before it leads to outages. Unlike traditional utilization metrics, PSI quantifies stalled tasks and lost time across CPU, memory, and I/O. With Kubernetes v1.36, a stable interface for observing resource contention at node, pod, and container levels is now available. PSI offers cumulative totals of stalled time and moving averages (10s, 60s, 300s) to distinguish between transient spikes and sustained resource tension.Extensive performance testing by SIG Node on high-density workloads (80+ pods) proved PSI's readiness for production. Kubelet overhead, measured by toggling the KubeletPSI feature gate, showed negligible impact on resource usage. The Kubelet's collection logic proved lightweight, blending seamlessly into standard housekeeping cycles, consuming less than 0.1 cores or 2.5% of total node capacity.Regarding kernel overhead, enabling PSI on the Linux kernel (psi=1 vs psi=0) resulted in a consistent delta of 0.037 to 0.125 cores (0.925% - 3.125% of node capacity) under heavy load. The kubelet process, as the primary collector, also maintained remarkably low CPU usage, with spikes not exceeding 0.25 cores (6.25%) for more than a second.Improvements in v1.36 include smarter metric emission; the Kubelet now detects OS-level PSI support via cgroup configurations before reporting, preventing misleading zero-valued metrics. To use PSI, nodes must run Linux kernel 4.20+, use cgroup v2, and have PSI enabled at the OS level (CONFIG_PSI=y, no psi=0 boot parameter).PSI metrics are generally available in v1.36 and require no feature gate opt-in. Users can scrape the /metrics/cadvisor endpoint or query the Summary API. PSI is a Linux-kernel feature and is not available on Windows nodes. Proxying to the Kubelet's HTTP API via the control plane's API server allows real-time pressure data from the Summary API but is a privileged operation.

https://kubernetes.io/blog/2026/05/12/kubernetes-v1-36-psi-metrics-ga/ kubernetes.io

RSS Hunter • May 12

Kubernetes v1.36: Moving Volume Group Snapshots to GA

Kubernetes v1.36 introduces General Availability (GA) for volume group snapshots, a feature that was previously an Alpha and then Beta enhancement. This functionality leverages extension APIs to enable crash-consistent snapshots of multiple volumes simultaneously. The system groups PersistentVolumeClaim objects using label selectors, allowing for the restoration of workloads to a consistent recovery point. This feature is exclusively supported for CSI volume drivers, offering a significant advantage for applications utilizing multiple volumes that require write order consistency.Previously, individual volume snapshots could lead to inconsistencies if taken at different times, particularly for multi-volume applications. Group snapshots eliminate the need for manual application quiescence, providing crash consistency across all volumes in the group without tedious, sequential individual snapshots. Kubernetes manages group snapshots through three custom API kinds: VolumeGroupSnapshot, VolumeGroupSnapshotContent, and VolumeGroupSnapshotClass. These CRDs, now promoted to v1 in the GA release, allow users to request group snapshots, track their provisioned resources, and define their creation policies, respectively.The GA release brings enhanced stability, bug fixes, and improved restoreSize reporting based on feedback from prior beta versions. To use this feature, users must label their PersistentVolumeClaims to be grouped and then define a VolumeGroupSnapshot object with a selector matching these labels, along with a VolumeGroupSnapshotClass. For restoration, new PersistentVolumeClaims are created from individual VolumeSnapshot objects that are part of a larger VolumeGroupSnapshot. Storage vendors can add support by implementing new group controller services and RPCs within their CSI drivers.

https://kubernetes.io/blog/2026/05/08/kubernetes-v1-36-volume-group-snapshot-ga/ kubernetes.io

RSS Hunter • May 8

Kubernetes v1.36: More Drivers, New Features, and the Next Era of DRA

Dynamic Resource Allocation (DRA) in Kubernetes v1.36 introduces significant advancements, extending its capabilities beyond specialized hardware to native resources like CPU and memory. Driver support for various hardware types, including networking, is expanding, making DRA a more hardware-agnostic solution. Several key features have graduated, enhancing scheduling flexibility and cluster utilization. The Prioritized list feature enables fallback preferences for device requests, improving resource allocation efficiency. Extended resource support allows a gradual transition to DRA by enabling resource requests via traditional extended resources. Partitionable devices provide native DRA support for dynamically carving physical hardware into smaller, logical instances. Device taints empower administrators to manage hardware more effectively by preventing faulty devices from being allocated or reserving specific hardware. Device binding conditions improve scheduling reliability by delaying Pod commitment until external resources are fully prepared. Resource health status exposes device health information directly in Pod status, aiding in quick identification and reaction to hardware failures. New alpha features include ResourceClaim support for workloads, optimizing large-scale AI/ML by managing shared resources across PodGroups. Node allocatable resources integrate CPU and memory allocation under the DRA umbrella, allowing for fine-grained performance tuning. DRA resource availability visibility provides administrators with real-time device capacity information for better planning. Deterministic device selection allows drivers to influence scheduling through lexicographical ordering. Discoverable device metadata in containers provides a standard protocol for drivers to expose device attributes to containers. The future roadmap focuses on maturing existing features, enhancing performance, scalability, and integration with workload-aware and topology-aware scheduling, with a strong emphasis on migrating users from Device Plugin to DRA.

https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/ kubernetes.io

RSS Hunter • May 7

Kubernetes v1.36: Server-Side Sharded List and Watch

Kubernetes controllers face scaling challenges as cluster sizes increase, particularly when watching high-cardinality resources. Client-side sharding, while functional, doesn't reduce the data volume from the API server, causing inefficiency. Server-side sharded list and watch, introduced in Kubernetes v1.36 as an alpha feature (KEP-5866), addresses this inefficiency. The API server filters events based on the controller's specified hash range, sending only relevant data to each replica. Controllers use informers to list and watch resources, incorporating the shardSelector through WithTweakListOptions. The shard selector is used to filter resources based on the object's metadata.uid or metadata.namespace. The API server returns a shardInfo field in the list response metadata to confirm if the shard selector was applied correctly. If absent, the client must handle the full, unfiltered collection, potentially resorting to client-side filtering. This feature requires enabling the ShardedListAndWatch feature gate. The Kubernetes community seeks feedback from controller authors and operators, especially those managing large clusters. This approach is designed to improve controller performance and scalability in demanding Kubernetes environments.

https://kubernetes.io/blog/2026/05/06/kubernetes-v1-36-server-side-sharded-list-and-watch/ kubernetes.io

RSS Hunter • May 6

Kubernetes v1.36: Declarative Validation Graduates to GA

Kubernetes v1.36 introduces Declarative Validation for native types, now generally available. This shifts from handwritten Go code to IDL tags for defining validation rules, enhancing API reliability and predictability. The previous reliance on handwritten code led to technical debt, inconsistencies, and opaque APIs. The solution utilizes validation-gen, a code generator that parses tags to automatically generate Go validation functions. This framework includes various marker tags for presence, constraints, collections, unions, and immutability. A key benefit is "ambient ratcheting", allowing immediate tightening or loosening of validation without breaking existing objects. Declarative validation makes API reviews easier and more consistent with tools like kube-api-linter. The project plans to migrate remaining legacy code and mandate declarative validation for new APIs. This also unlocks future ecosystem benefits, like client-side validation by tools such as kubectl and integration with tools such as Kubebuilder. The migration is ongoing, with opportunities to contribute to the Kubernetes codebase. The document concludes with acknowledgments to contributors, welcoming the declarative future of Kubernetes validation.

https://kubernetes.io/blog/2026/05/05/kubernetes-v1-36-declarative-validation-ga/ kubernetes.io

RSS Hunter • May 5

Kubernetes v1.36: Admission Policies That Can't Be Deleted

Kubernetes' manifest-based admission control, introduced in v1.36, addresses security policy enforcement gaps during cluster bootstrap. Existing API-based admission control has vulnerabilities: policies are API objects and can be removed, creating a security window. This new feature allows defining admission webhooks and CEL-based policies as files loaded by the API server at startup. This ensures policies are active before any requests are served, protecting against unauthorized modifications. It uses a staticManifestsDir field in the AdmissionConfiguration file to specify a directory containing policy YAML files. These files must have names ending in .static.k8s.io to distinguish them from API-based configurations. The feature can protect the admission configurations themselves from deletion or modification. Changes to the manifest files are automatically updated at runtime. The API server enforces strict validation during startup and handles runtime updates atomically. To implement it, you'll need to enable the ManifestBasedAdmissionControlConfig feature gate.

https://kubernetes.io/blog/2026/05/04/kubernetes-v1-36-manifest-based-admission-control/ kubernetes.io

RSS Hunter • May 4

Kubernetes v1.36: Pod-Level Resource Managers (Alpha)

Kubernetes v1.36 introduces Pod-Level Resource Managers as an alpha feature, enhancing resource management for performance-sensitive workloads. It extends kubelet's Topology, CPU, and Memory Managers to a pod-centric resource allocation model, moving beyond per-container specifications. This addresses the challenge of providing exclusive, NUMA-aligned resources for primary application containers while supporting lightweight sidecars efficiently. Previously, achieving predictable performance often meant allocating exclusive resources to all containers, which was wasteful for sidecars. Alternatively, not doing so sacrificed the pod's Guaranteed QoS. Pod-level resource managers enable hybrid allocation, allowing high-performance workloads to achieve NUMA alignment without wasting resources. For example, a latency-sensitive database pod can have its main container receive exclusive CPU and memory, while sidecars share a distinct pod shared pool, isolated from other node resources. Another use case involves ML workloads where the training container gets exclusive NUMA-aligned resources, and a service mesh sidecar runs in the general node-wide shared pool. CPU isolation is managed by disabling CFS quota enforcement for exclusive containers and enforcing it at the pod level for shared pool containers. Enabling requires specific kubelet feature gates, Topology Manager policies, and static CPU and Memory Manager configurations. New kubelet metrics provide observability into resource allocations and container assignments. This feature is currently in alpha, with known limitations and caveats, and user feedback is encouraged through Kubernetes community channels.

https://kubernetes.io/blog/2026/05/01/kubernetes-v1-36-feature-pod-level-resource-managers-alpha/ kubernetes.io

RSS Hunter • May 1

Kubernetes v1.36: In-Place Vertical Scaling for Pod-Level Resources Graduates to Beta

Kubernetes v1.36 introduces In-Place Pod-Level Resources Vertical Scaling, now in Beta and enabled by default. This feature allows users to dynamically adjust the aggregate resource limits of a running Pod. This is particularly useful for Pods with shared resources and no container-specific limits. The Kubelet determines the update method based on each container's resizePolicy, deciding between in-place updates or restarts. When a resize happens, the Kubelet first checks if the node has enough resources. It then sequences cgroup updates to prevent resource overshoot, expanding the pod-level cgroup before individual container cgroups are enlarged. Pod conditions, like PodResizeInProgress, track the resize's progress and status. The feature requires cgroup v2, CRI support, specific feature gates, and Linux-based nodes. The next step is integrating this with the Vertical Pod Autoscaler (VPA). Users are encouraged to test this feature and provide feedback through the community channels.

https://kubernetes.io/blog/2026/04/30/kubernetes-v1-36-inplace-pod-level-resources-beta/ kubernetes.io

RSS Hunter • Apr 30