Kubernetes Blog
Follow
Why Is My Kubernetes Job Still Pending? Tracing Kueue Admission with Headlamp
When a Kubernetes Job is pending, it might be due to Kueue still determining resource availability. The Headlamp Kueue plugin helps investigate this by integrating Kueue resource views into the UI. This allows users to trace a Workload's admission path through LocalQueues, ClusterQueues, ResourceFlavors, and Cohorts without extensive command-line usage. The investigation begins by examining the Workload's status and conditions to see if Kueue has admitted it. If the Workload is not admitted, the issue lies with Kueue's admission process. If it is admitted but Pods are pending, the problem shifts to Kubernetes’ scheduler. The Workload view in Headlamp shows requested resources, and following the LocalQueue link reveals the entry point for the Job's request. Inspecting the linked ClusterQueue is crucial, as its quota and policies dictate resource allocation. In a specific example, a ClusterQueue with limited CPU and Pod slots caused a Job requesting more to remain suspended by Kueue. ResourceFlavors and Cohorts are examined for more complex quota or hardware class issues. If the Workload is admitted, troubleshooting then moves to Headlamp's standard Job and Pod views to check scheduler-related events. This Headlamp-first approach aims to provide a faster, more integrated troubleshooting experience for Kueue-related delays.