Kubernetes Blog Note

Kubernetes Blog

The official homepage of Kubernetes, a container orchestration system for automating deployment, scaling, and management of containerized applications. This platform offers comprehensive documentation on Kubernetes, a project maintained by Cloud Native Computing Foundation. It includes details about running stateless and stateful applications, batch jobs, and CI/CD workflows using Kubernetes. The site includes detailed guides, tutorials, reference material, API documentation, and community engagement initiatives to help users get started with Kubernetes and leverage its features effectively to manage cloud-based applications efficiently.

Thread Of Notes

Volcano is a cloud-native batch scheduler for Kubernetes, designed for high-performance computing, AI/ML, and other batch workloads. Kubernetes was originally built for long-running services, whereas batch workloads often require dynamic job arrival, resource competition, and co-starting multiple workers. Volcano extends Kubernetes with concepts like queues, priorities, quotas, and gang scheduling, treating workloads as a whole rather than independent Pods. The Volcano plugin for Headlamp, an extensible Kubernetes web UI, brings these scheduling details into a single interface.The plugin provides dedicated views for Volcano Jobs, Queues, and PodGroups, making it easier to operate and troubleshoot batch workloads. The Job view displays workload status, task details, Pod status, and allows direct actions like suspending/resuming and accessing logs. The Queue view offers insights into resource allocation, capacity, and reservation details. The PodGroup view clarifies gang scheduling states and potential blockers.A key feature is the map view, which visually represents how Jobs, Queues, PodGroups, and Pods are interconnected, helping to quickly identify issues in pending or non-progressing workloads. This plugin enhances the interactive troubleshooting experience by centralizing related resources, structured details, and runtime output, without replacing CLI tools for automation. Future enhancements may include Prometheus integration and richer scheduling insights. Users can install the plugin via Headlamp's Plugin Catalog and provide feedback to shape its development.
CdXz5zHNQW_lVauTs2nj6.png
CdXz5zHNQW_uA3Yt1MH58.png
The article highlights SIG Storage, the Kubernetes Special Interest Group responsible for persistent data and volume management. Xing Yang, a co-chair of SIG Storage, discusses the group's evolution from handling basic persistent volumes to advancing complex storage features. Initially designed for stateless workloads, Kubernetes now supports stateful applications, necessitating dedicated storage solutions. SIG Storage formed to address these challenges, introducing primitives like PersistentVolumes and PersistentVolumeClaims.A significant advancement was the Container Storage Interface (CSI), which enables third-party storage providers to integrate their systems without core Kubernetes modifications. Current work includes Volume Group Snapshot for crash-consistent multi-volume snapshots and Changed Block Tracking for efficient backups, both recently graduated to stable versions. The Container Object Storage Interface (COSI) is also progressing to standardize object storage integration.Recent wins for users include the graduation of VolumeAttributesClass to General Availability, allowing dynamic tuning of storage properties like IOPS. Future roadmaps feature Volume Health for improved operational visibility and potential automated remediation. SIG Storage seeks community help with bug fixes, tests, reviews, and feedback on features like Mutable PV Affinity and volume replication.Challenges for stateful workloads include data gravity, day-2 operational complexity, and data mobility. As AI workloads grow, storage in Kubernetes is expected to become more intelligent, with object storage gaining prominence. High performance, low latency storage, and data-aware scheduling are also anticipated trends. SIG Storage invites community involvement to tackle these evolving storage demands.
Kubernetes v1.35 introduced workload-aware scheduling improvements, including the Workload API and basic gang scheduling for identical Pods. Kubernetes v1.36 refines this architecture by separating the Workload API (static template) from the new PodGroup API (runtime state). This separation streamlines the kube-scheduler, enabling it to directly read PodGroup information for enhanced performance.A new PodGroup scheduling cycle allows atomic processing of workloads, evaluating entire groups as a unified operation to prevent deadlocks. If a valid placement is found and group constraints are met, Pods are bound together; otherwise, the entire group is considered unschedulable and retries later. This forms the foundation for gang scheduling, ensuring all-or-nothing placement for strict workload requirements.Topology-aware scheduling in v1.36 enables defining topology constraints on PodGroups, co-locating Pods within specific physical or logical domains to reduce network latency. This involves generating, evaluating, and scoring candidate placements based on scheduling constraints.Workload-aware preemption is introduced to support the PodGroup scheduling cycle, preempting Pods from multiple Nodes simultaneously to make space for an entire PodGroup. It treats the PodGroup as a single preemptor unit, with PodGroup priority and disruptionMode fields controlling preemption behavior.Finally, v1.36 integrates Dynamic Resource Allocation (DRA) with the Workload API, allowing PodGroups to request and share specialized hardware resources through ResourceClaims. These advancements lay a robust foundation for building advanced workload scheduling capabilities in future Kubernetes releases.
Pressure Stall Information (PSI) has been integrated into the Linux kernel since 2018, providing high-fidelity signals for identifying resource saturation before it leads to outages. Unlike traditional utilization metrics, PSI quantifies stalled tasks and lost time across CPU, memory, and I/O. With Kubernetes v1.36, a stable interface for observing resource contention at node, pod, and container levels is now available. PSI offers cumulative totals of stalled time and moving averages (10s, 60s, 300s) to distinguish between transient spikes and sustained resource tension.Extensive performance testing by SIG Node on high-density workloads (80+ pods) proved PSI's readiness for production. Kubelet overhead, measured by toggling the KubeletPSI feature gate, showed negligible impact on resource usage. The Kubelet's collection logic proved lightweight, blending seamlessly into standard housekeeping cycles, consuming less than 0.1 cores or 2.5% of total node capacity.Regarding kernel overhead, enabling PSI on the Linux kernel (psi=1 vs psi=0) resulted in a consistent delta of 0.037 to 0.125 cores (0.925% - 3.125% of node capacity) under heavy load. The kubelet process, as the primary collector, also maintained remarkably low CPU usage, with spikes not exceeding 0.25 cores (6.25%) for more than a second.Improvements in v1.36 include smarter metric emission; the Kubelet now detects OS-level PSI support via cgroup configurations before reporting, preventing misleading zero-valued metrics. To use PSI, nodes must run Linux kernel 4.20+, use cgroup v2, and have PSI enabled at the OS level (CONFIG_PSI=y, no psi=0 boot parameter).PSI metrics are generally available in v1.36 and require no feature gate opt-in. Users can scrape the /metrics/cadvisor endpoint or query the Summary API. PSI is a Linux-kernel feature and is not available on Windows nodes. Proxying to the Kubelet's HTTP API via the control plane's API server allows real-time pressure data from the Summary API but is a privileged operation.
CdXz5zHNQW_xWB13lRlZh.png
Kubernetes v1.36 introduces General Availability (GA) for volume group snapshots, a feature that was previously an Alpha and then Beta enhancement. This functionality leverages extension APIs to enable crash-consistent snapshots of multiple volumes simultaneously. The system groups PersistentVolumeClaim objects using label selectors, allowing for the restoration of workloads to a consistent recovery point. This feature is exclusively supported for CSI volume drivers, offering a significant advantage for applications utilizing multiple volumes that require write order consistency.Previously, individual volume snapshots could lead to inconsistencies if taken at different times, particularly for multi-volume applications. Group snapshots eliminate the need for manual application quiescence, providing crash consistency across all volumes in the group without tedious, sequential individual snapshots. Kubernetes manages group snapshots through three custom API kinds: VolumeGroupSnapshot, VolumeGroupSnapshotContent, and VolumeGroupSnapshotClass. These CRDs, now promoted to v1 in the GA release, allow users to request group snapshots, track their provisioned resources, and define their creation policies, respectively.The GA release brings enhanced stability, bug fixes, and improved restoreSize reporting based on feedback from prior beta versions. To use this feature, users must label their PersistentVolumeClaims to be grouped and then define a VolumeGroupSnapshot object with a selector matching these labels, along with a VolumeGroupSnapshotClass. For restoration, new PersistentVolumeClaims are created from individual VolumeSnapshot objects that are part of a larger VolumeGroupSnapshot. Storage vendors can add support by implementing new group controller services and RPCs within their CSI drivers.
Dynamic Resource Allocation (DRA) in Kubernetes v1.36 introduces significant advancements, extending its capabilities beyond specialized hardware to native resources like CPU and memory. Driver support for various hardware types, including networking, is expanding, making DRA a more hardware-agnostic solution. Several key features have graduated, enhancing scheduling flexibility and cluster utilization. The Prioritized list feature enables fallback preferences for device requests, improving resource allocation efficiency. Extended resource support allows a gradual transition to DRA by enabling resource requests via traditional extended resources. Partitionable devices provide native DRA support for dynamically carving physical hardware into smaller, logical instances. Device taints empower administrators to manage hardware more effectively by preventing faulty devices from being allocated or reserving specific hardware. Device binding conditions improve scheduling reliability by delaying Pod commitment until external resources are fully prepared. Resource health status exposes device health information directly in Pod status, aiding in quick identification and reaction to hardware failures. New alpha features include ResourceClaim support for workloads, optimizing large-scale AI/ML by managing shared resources across PodGroups. Node allocatable resources integrate CPU and memory allocation under the DRA umbrella, allowing for fine-grained performance tuning. DRA resource availability visibility provides administrators with real-time device capacity information for better planning. Deterministic device selection allows drivers to influence scheduling through lexicographical ordering. Discoverable device metadata in containers provides a standard protocol for drivers to expose device attributes to containers. The future roadmap focuses on maturing existing features, enhancing performance, scalability, and integration with workload-aware and topology-aware scheduling, with a strong emphasis on migrating users from Device Plugin to DRA.
Kubernetes v1.36 introduces Pod-Level Resource Managers as an alpha feature, enhancing resource management for performance-sensitive workloads. It extends kubelet's Topology, CPU, and Memory Managers to a pod-centric resource allocation model, moving beyond per-container specifications. This addresses the challenge of providing exclusive, NUMA-aligned resources for primary application containers while supporting lightweight sidecars efficiently. Previously, achieving predictable performance often meant allocating exclusive resources to all containers, which was wasteful for sidecars. Alternatively, not doing so sacrificed the pod's Guaranteed QoS. Pod-level resource managers enable hybrid allocation, allowing high-performance workloads to achieve NUMA alignment without wasting resources. For example, a latency-sensitive database pod can have its main container receive exclusive CPU and memory, while sidecars share a distinct pod shared pool, isolated from other node resources. Another use case involves ML workloads where the training container gets exclusive NUMA-aligned resources, and a service mesh sidecar runs in the general node-wide shared pool. CPU isolation is managed by disabling CFS quota enforcement for exclusive containers and enforcing it at the pod level for shared pool containers. Enabling requires specific kubelet feature gates, Topology Manager policies, and static CPU and Memory Manager configurations. New kubelet metrics provide observability into resource allocations and container assignments. This feature is currently in alpha, with known limitations and caveats, and user feedback is encouraged through Kubernetes community channels.
Kubernetes v1.36 has been released, featuring 70 enhancements with 18 graduating to stable and 25 entering beta. The release theme, "Haru," symbolizes spring, clear skies, and distant horizons, with the logo inspired by Hokusai's "Red Fuji." This release emphasizes community collaboration, with many individuals and teams contributing to its success.Key stable features include fine-grained kubelet API authorization for improved least-privilege access control. Resource health status for allocated devices has entered beta, offering unified reporting for hardware failures. Alpha introduces Workload Aware Scheduling, treating related pods as a single logical entity for better resource management.Volume group snapshots are now stable, enabling crash-consistent snapshots across multiple PersistentVolumeClaims. Mutable CSI node allocatable limits also reach stability, allowing dynamic updates to node volume capacities. The external ServiceAccount token signer feature is now stable for offloading token signing to external systems.Dynamic Resource Allocation (DRA) admin access and prioritized lists are now stable, providing a secure framework for resource management. Declarative mutating admission policies are stable, offering a native alternative to webhooks for resource mutations. Declarative validation for Kubernetes native types with validation-gen has also graduated to stable, streamlining custom resource development. The removal of the gogo protobuf dependency for Kubernetes API types marks a significant step forward for security and maintainability.
Kubernetes v1.36, slated for late April 2026, will introduce significant removals, deprecations, and numerous enhancements. The project adheres to a strict deprecation policy, ensuring stable APIs are only removed after a newer stable version is available and have a minimum lifetime. A recent example of this policy is the retirement of the Ingress NGINX project as of March 24, 2026, with no further support or security updates. For v1.36, the .spec.externalIPs field in Service is being deprecated due to security concerns (CVE-2020-8554), with full removal planned in v1.43, urging migration to LoadBalancer, NodePort, or Gateway API. The gitRepo volume driver, deprecated since v1.11, will be permanently disabled in v1.36 due to a critical security vulnerability allowing root code execution. Workloads currently using gitRepo must migrate to alternatives like init containers or external git-sync tools.Key enhancements in v1.36 include the General Availability (GA) of faster SELinux labeling for volumes, which uses mount options for consistent performance and reduced Pod startup delays. This feature, introduced as beta in v1.28, now defaults to all volumes with Pods specifying spec.SELinuxMount. External signing of ServiceAccount tokens, a beta feature, is expected to graduate to stable in v1.36, allowing clusters to integrate with external key management systems for improved security. Dynamic Resource Allocation (DRA) also sees advancements, with Device taints and tolerations graduating to beta, enabling specialized hardware resources to be restricted to specific workloads. Additionally, DRA will support partitionable devices, allowing a single hardware accelerator to be split into multiple logical units, improving resource utilization for costly resources like GPUs. These changes highlight a continued focus on security, efficiency, and advanced resource management within Kubernetes.
Kubernetes will retire Ingress-NGINX in March 2026, and users need to migrate to other solutions like Gateway API. Ingress-NGINX has several surprising defaults and side effects that can cause outages if not considered during migration. The blog post highlights these behaviors to help users migrate safely and make conscious decisions about which behaviors to keep. One of the key issues is that Ingress-NGINX treats regex patterns as prefix and case-insensitive matches, which can lead to unexpected routing. Gateway API, on the other hand, uses implementation-specific regex matching, and users need to check with their implementation to verify the semantics of regex matching. The post also discusses how to preserve Ingress-NGINX behavior in Gateway API, including using HTTP path matches with a type of RegularExpression and configuring redirects using the HTTP request redirect filter. Additionally, the post notes that Ingress-NGINX and NGINX Ingress are two separate Ingress controllers, and the blog post only discusses Ingress-NGINX. The nginx.ingress.kubernetes.io/use-regex annotation applies to all paths of a host across all Ingress-NGINX Ingresses, and the nginx.ingress.kubernetes.io/rewrite-target annotation silently adds the nginx.ingress.kubernetes.io/use-regex annotation, along with all its side effects. Ingress-NGINX also redirects requests missing a trailing slash to the same path with a trailing slash, which can cause outages if not explicitly configured in Gateway API. Overall, the post aims to help users understand the quirks of Ingress-NGINX and migrate to Gateway API safely. Users need to be aware of these behaviors and take steps to preserve them in Gateway API to avoid outages.
The Kubernetes Steering Committee and Security Response Committee have announced the retirement of Ingress NGINX, a critical infrastructure component used by about half of cloud native environments, effective March 2026. The project has been in dire need of contributors and maintainers for years, and despite public warnings, it has not received the necessary support. After the retirement, there will be no more releases for bug fixes, security patches, or updates, leaving users vulnerable to attack if they do not migrate to alternative solutions. The committee emphasizes the severity of the situation and the importance of beginning migration to alternatives like Gateway API or third-party Ingress controllers immediately. Choosing to remain with Ingress NGINX after its retirement will leave users vulnerable to attack, and none of the available alternatives are direct drop-in replacements, requiring planning and engineering time. Existing deployments will continue to work, but users may not know they are affected until they are compromised, and they can check their reliance on Ingress NGINX by running a specific command with cluster administrator permissions. The Ingress NGINX project has been maintained by only one or two people working in their free time, and despite its widespread use, it has not received the necessary contributors to maintain it securely. The committee did not make the decision to retire Ingress NGINX lightly, but it is necessary for the safety of all users and the ecosystem as a whole due to the technical debt and fundamental design decisions that exacerbate security flaws. The committee urges users to check their clusters now and begin planning for migration if they are reliant on Ingress NGINX to avoid serious risk. The retirement of Ingress NGINX is a significant change that affects a large percentage of Kubernetes users, and it is imperative that users take immediate action to address the issue.
This document provides a guide for setting up a local experimental environment for learning Gateway API concepts using kind. It emphasizes that this setup is not for production use. The process involves creating a kind Kubernetes cluster and deploying cloud-provider-kind, which offers LoadBalancer Services and a Gateway API controller. Users will then create a Gateway, deploy a demo echo application, and configure an HTTPRoute to direct traffic to this application. The guide includes steps to test the Gateway API configuration and provides troubleshooting tips for common issues. Finally, it outlines the cleanup process to remove all created resources and suggests next steps for exploring production-ready implementations and advanced Gateway API features. This local setup is specifically designed for understanding Gateway API principles without production complexities. It requires Docker, kubectl, kind, and curl to be installed. The cloud-provider-kind component simulates a cloud-enabled environment by providing necessary controllers and CRDs. Creating a Gateway involves defining a GatewayClass and listener configurations that accept specific hostnames and protocols. Deploying the echo application involves creating a namespace, Service, and Deployment. Configuring the HTTPRoute links the Gateway to the echo application for a specific hostname. Testing involves using curl to send a request to the Gateway's IP address with the defined hostname. Checking resource statuses and controller logs are recommended for troubleshooting. Cleanup involves deleting namespaces, stopping the cloud-provider-kind container, and deleting the kind cluster.
Kubernetes v1.35 introduces a beta feature, CSI Driver Opt-in for Service Account Tokens via Secrets Field. Previously, service account tokens for CSI drivers were passed through the volume_context field, which is not ideal for sensitive data and has led to tokens being accidentally logged. This new feature allows CSI drivers to receive these tokens via the secrets field in NodePublishVolumeRequest, the designated place for sensitive information in the CSI specification. Existing CSI drivers will continue to receive tokens via volume_context by default, as the new serviceAccountTokenInSecrets field in the CSIDriver spec defaults to false.To adopt this feature, CSI driver authors should first implement fallback logic in their driver code. This logic checks both the secrets field and volume_context for tokens, ensuring compatibility with both older and newer Kubernetes versions. After deploying this updated driver, the cluster must be upgraded to Kubernetes v1.35 or later, including both kube-apiserver and kubelet. Once the cluster and driver are upgraded, the CSIDriver manifest can be updated to set serviceAccountTokenInSecrets: true.It is crucial to follow a specific rollout sequence to avoid breaking existing volumes. The driver update with fallback logic must be deployed and fully rolled out before updating the CSIDriver object to enable the new behavior. This opt-in mechanism eliminates the risk of accidental token logging, uses the correct CSI specification field for sensitive data, and is managed by the protosanitizer tool without needing driver-specific workarounds. CSI driver authors are encouraged to adopt this feature and provide feedback.