A company is developing an AI test framework for Postgres AI Hybrid Manager, a product that integrates traditional Postgres management with AI tools like Langflow. This chatbot aims to unify product features, help documentation, and AI workflows into a single conversational interface. Testing such an LLM-backed agent is challenging because its responses are non-deterministic and can be subtly wrong even when fluent. The framework evolved to address this by focusing on testing the entire chat trajectory, not just final answers.Key concepts are introduced, including "Goldens" (perfect desired outputs) and "Rubrics" (qualitative checklists for good answers). For AI testing, LLMs are used as judges to score outputs against Goldens or Rubrics, translating complex responses into pass/fail results. Task Completion Rate (TCR) is used to aggregate multiple eval passes. The testing strategy began with simple routing evals, checking if the chatbot directed prompts to the correct specialized agent or skill.As the system evolved from agents per tool to a consolidated orchestrating agent with skills, routing evals adapted to check for the visibility and selection of the right skill. Subsequently, TCR evals were implemented to assess if the chatbot's full response successfully completed the user's task, using LLM-as-a-judge with specific rubrics. This led to two execution modes: Direct mode for prompt and rubric development, and Proxy mode to test the full production path.The framework also had to handle multi-step conversations, where the test unit became the entire conversation, not just single turns. This required preserving conversation state across multiple API calls in Proxy mode or simulating it in Direct mode. Scoring now includes per-step checks and an overall conversation score. The underlying testing library used is deepeval, upon which the company built its pipeline, plugin system, CI integration, and Langfuse push. Routing is implemented as a custom BaseMetric in deepeval, demonstrating how specific business rules can be integrated into the LLM evaluation process.