DEV Community
Follow
DRS for your AI: making a support bot wait for a person before it changes (Switch + Slack)
The author builds voice bots, noting the significant time and manual effort involved in testing prompt changes and ensuring policy compliance. Current methods are slow, prone to rule breaks, and lack proper sign-off, as exemplified by a complex sales bot and the Air Canada chatbot liability case.To address this, the author developed "ThirdUmpire," a system inspired by cricket's third umpire. This system automatically reviews bot changes by running 40 customer conversations three times on both old and new prompts. It grades replies against business rules and posts a verdict card in Slack for a named person to approve or reject.ThirdUmpire uses Switch to integrate with Slack, ensuring approvals pause commands and that separate agents (builder, umpire, rules, release) have distinct identities and permissions. This prevents AI from overriding approval rules and enforces accountability.The system includes a "NOT TESTED" verdict, refusing releases if new prompt lines lack covering rules or tests, requiring compliance sign-off on policy, rules, and test cases. It also prevents partial reviews, edits to live prompts, and re-reviews of already released changes, ensuring integrity.ThirdUmpire saves significant time, providing a verdict in about three minutes compared to a full day of manual testing. Fixed issues include false "OUT" verdicts, missing approval cards, script approval failures, and the critical "NOT TESTED" gap.Limitations include reviewing only prompt changes, potential AI errors in coverage and judging, and a focus on text bots. Future plans involve nightly re-checks, voice bot support, and addressing multi-language capabilities.The project is open-source and provides a demonstrable solution for structured AI development and deployment. The author highlights the critical role of external enforcement for AI governance, emphasizing that the rule is enforced by the room, not inside the AI. This approach ensures robust, auditable, and accountable AI development workflows.This system guarantees that no bot changes bypass human oversight or compromise established policies.