AI21 Labs, a leading AI company, encountered significant challenges in managing its GPU compute resources. High utilization of their Google Kubernetes Engine (GKE) clusters led to manual scheduling and prolonged wait times for critical model training jobs. Teams previously negotiated capacity through a Slack channel, causing bottlenecks and fragmentation of available GPUs. This situation resulted in high-priority jobs experiencing delays of up to 72 hours.To address these issues, AI21 Labs adopted Google Cloud AI Hypercomputer and integrated it with the open-source scheduler Kueue. This solution streamlined the scheduling process, significantly reducing manual interventions. The adoption of Kueue allowed for automatic orchestration of various workloads, including training and inference. Furthermore, AI21 Labs collaborated with the Kueue team to develop new features like Admission Fair Sharing (AFS).AFS helps ensure equitable distribution of compute resources across teams. Topology Aware Scheduling was also implemented, preventing jobs from being placed where they cannot run. These improvements resulted in a substantial decrease in high-priority job wait times, from 72 hours to just 12. GPU fragmentation also reduced, and the problem of "zombie jobs" was eliminated. The company's total cost remained the same, but their operational efficiency and researcher productivity greatly increased. This shift freed up valuable personnel time, enabling faster experimentation and iteration.