Netflix relies heavily on GPUs for its personalized content recommendations and other machine learning applications. These GPUs are crucial for training and serving content, making their reliability a business imperative for both availability and cost efficiency. Basic cloud provider health checks are insufficient because they do not detect subtle GPU or network fabric failures that can cripple long-running training jobs. Such failures can manifest as slow performance or complete job stoppage, impacting user experience and wasting resources.Netflix employs a layered approach to ensure GPU infrastructure health. The first layer focuses on detecting and remediating failing hardware like GPUs and network adapters. The second layer involves the scheduler and platform, which retry tasks and validate machines before they are used. The third layer enables long-running jobs to survive machine loss through checkpointing and elastic workers. The fourth layer is observability, which monitors the effectiveness of the other three.Health checks operate in two tiers: device-level checks for individual GPUs and node-level checks performed by an agent. These checks report results as Kubernetes node conditions. When a problem is detected, the machine's condition is updated, and controllers take action. Initial issues are tolerated, but persistent failures escalate to machine evacuation, draining, and replacement.Remediation is graded, with actions matched to the fault severity, ranging from restarting a single GPU to terminating and replacing the entire machine. For persistently failing hardware, immutable identifiers are recorded to prevent them from rejoining the pool. Machines are validated before being released for work, ensuring they pass all diagnostics. Finally, long-running jobs are made resilient through checkpointing, allowing them to resume from the last stable state if a machine is lost.