Telecommunications company KDDI faced a challenge balancing quality and speed in their generative AI application, Buffmee. Buffmee is designed to help users grow by providing information grounded in over 100 diverse sources, ensuring reliability and preventing hallucinations. Initially, KDDI struggled with latency issues and the need to process a vast amount of proprietary content. To overcome these obstacles, they implemented an automated evaluation framework and performance optimization techniques.This systematic approach led to significant improvements, reducing total application response latency by 38% and improving Time To First Token (TTFT) by nearly 18%. KDDI utilized the Gemini Enterprise Agent Platform Evaluation Service for automated testing, replacing manual efforts with a data-driven process that improved groundedness scores by 25%. They adopted a binary evaluation system, strategic content sampling based on format and media composition, and calibrated thresholds based on product judgment.Bottlenecks were identified and resolved using BigQuery Agent Analytics and the Agent Development Kit (ADK) log analysis agent, which helped pinpoint prompt bloat and skill division issues. Large system prompts were modularized into ADK Skills to avoid attention drift and optimize response times. These techniques allowed KDDI to create a RAG system where ingested content instantly functioned, providing users with a safe and personalized way to explore media. The success of Buffmee demonstrates the importance of systematic evaluation and performance analysis for reliable generative AI applications.