The AI industry is shifting how it evaluates agents, moving from scoring individual conversations to comparing groups of users against a baseline. This change addresses the gap where a single conversation might score well but still indicate a product issue. Experts advocate for evaluating AI agents based on user cohorts rather than isolated traces. This new approach treats evaluation criteria as a dynamic product specification, similar to a product requirements document. Teams are realizing that exhaustive pre-launch testing may not catch all real-world failures. Instead, continuous, broad monitoring is crucial for identifying problems as they arise. Contrastive analysis, which compares user groups to a baseline, reveals issues missed by evaluating single interactions. For instance, increased clarification questions or purchases made outside a conversation might go unnoticed otherwise. This analysis helps pinpoint specific, category-related problems. The industry is also moving towards using smaller, cheaper judge models for evaluating AI agents. These evaluations should start with the most capable models to confirm solvability, then progressively use smaller ones. Additionally, guardrails can be implemented using simpler methods like regular expressions, not just complex AI models. Despite advancements in AI judging, the need for human oversight remains critical. Humans are essential for accountability, especially in sensitive sectors like legal, finance, and healthcare. Human review also builds trust and facilitates memory and learning within AI systems.