Shipping a production language model system requires rapid iteration on improvements, which is challenging due to the non-deterministic nature of models and evaluation processes. Airbnb addressed these challenges by building reliable LLM infrastructure across four layers, focusing on engineering enhancements and integration. The core principle is that the seams between components are where issues arise, and comprehensive end-to-end testing is crucial.Layer 1 focuses on diagnostic framing of evaluation noise, distinguishing between data and judging uncertainty to understand performance variations. Noise in LLM evaluations can stem from judges scoring inputs differently or LLM-generated references changing, making it difficult to discern true model improvements. Separating epistemic uncertainty (model/judge limits) from aleatoric uncertainty (task ambiguity) is key to accurate diagnosis.Layer 2 establishes a deterministic evaluation foundation by stabilizing inputs to judges. This is achieved through a per-sample cache for references and judge scores, ensuring identical inputs yield cached results and making evaluation reproducible and efficient. This deterministic measurement is essential for Layer 1's diagnostic capabilities.Layer 3 enables bounded, scoped model mutation through micro adapters. These are small LoRA patches trained on specific bugs, allowing for quick, hour-long training and hotfix-style deployments. Three lifecycle rules—fusing co-triggering patches, retraining on accumulation, and unloading unused patches—prevent the stack of adapters from degrading.Layer 4 provides end-to-end validation at the seams of the system. Despite individual components appearing fine, their interactions can cause unexpected behavior. This layer involves running representative inputs through the entire production path to measure combined quality and latency, ensuring that bugs at the seams are surfaced before deployment. These four layers form a dependency stack where each layer's effectiveness relies on the others.