Voice assistants have evolved from being simple, rule-based systems to advanced conversational agents driven by large language models (LLMs). Early versions of voice assistants could only handle specific tasks with pre-defined commands. In contrast, modern LLM-powered assistants can now engage in long and open-ended conversations, follow complex instructions, and perform multi-step reasoning. These improved capabilities bring new evaluation challenges. Traditional metrics like intent classification accuracy, slot-filling accuracy/recall, and goal completion rates can no longer capture the overall quality of a voice assistant.  Assistant responses can sound fluent and plausible, even when they contain factual errors or unsafe content. For example, an LLM assistant might correctly identify a user’s request to “find Italian restaurants” (intent) and extract the location “downtown” (slot), but then respond with a restaurant name that doesn’t even exist. Traditional benchmarks would mark the intent/slot task as successful, without accounting for the factual error. Therefore, new metrics and techniques are needed to assess factuality, safety, reasoning ability, instruction following, and user experience.