Airbnb emphasizes treating evaluation as a crucial engineering discipline for building trustworthy Generative AI products. Traditional software testing assumptions are challenged by the non-deterministic and subjective nature of LLM outputs, often requiring AI to evaluate AI. Product teams at Airbnb build LLM-powered features, supported by an infrastructure team providing tooling and best practices. A foundational principle is understanding that evaluation must be planned from the outset to avoid false confidence and undetected regressions. The "one rule" is to manually review data and build intuition for success by examining prototype outputs. This habit evolves into eval-driven development, where failure modes are discovered, encoded, and continuously tested. Key principles of eval-driven development include defining goals, letting real errors guide metrics, using small, sharp evaluator sets, appointing a decision-maker, and continuous collaboration. Three evaluation methods form layers: programmatic checks for obvious failures, LLM-as-a-Judge for nuanced quality, and human evaluation for edge cases and validation. Programmatic checks use deterministic code-based tests, while LLM-as-a-Judge employs stronger LLMs against rubrics, requiring calibration for trustworthiness. Human evaluation serves as the gold standard for ground truth and high-stakes decisions. For agentic systems, evaluation extends across multiple layers, examining reasoning paths and tool calls, not just the final output. A practical walkthrough demonstrates exploring initial failures, building layered evaluations with programmatic checks and calibrated virtual judges, and scaling with production monitoring. Key takeaways highlight the importance of looking at data, avoiding generic metrics, starting small, calibrating judges, using layered defenses, and mirroring evaluations in production. Ultimately, successful AI product development hinges on evaluation being a collaborative team sport that shapes the definition of product success.