A healthy security operations center alerting system requires more than just fine-tuning false positives; it also needs to ensure critical but infrequent detections are functional. GitLab's Signals Engineering team developed a framework called WATCH (Weekly Attack Testing for Continuous Health) to address this gap. WATCH automates the validation of security detections by simulating real malicious behavior on their infrastructure. This process verifies the end-to-end alerting pipeline, from log source to SIEM and security orchestration.WATCH works by scheduling scripted attack simulations in a staging environment, followed by verification that expected alerts propagate through the monitoring stack. Before a test runs, WATCH notifies the SOAR system with expected detections, creating trackable records. The simulated malicious behavior is then executed, and the SIEM processes logs to fire detection rules. Alerts arriving in the SOAR are correlated with registered tests to prevent false escalations.A verification stage checks if all expected detections fired, updates detection status metadata, and deploys results to a GitLab Pages dashboard. Failures trigger immediate notifications to the team. WATCH is orchestrated using GitLab CI/CD across three stages: scheduling, test execution, and verification/reporting. The framework is designed for ease of use, allowing team members to create new tests by subclassing a base class and defining setup, execution, and cleanup procedures.The configuration of expected detections, mapping SIEM rule names to expected alert arrival times, is a key aspect. WATCH tests can be readily scaffolded with GitLab Duo, an AI assistant, by providing prompts for specific malicious behaviors. This significantly lowers the barrier to entry for creating new tests. Duo Agent Skills further enhance consistency by providing detailed outlines of good test practices and helper functions.WATCH also provides two interactive dashboards deployed via GitLab Pages, offering real-time visibility into detection health. One dashboard, the Detection Status Dashboard, summarizes the current test status of all detection rules. The other, the Detection Test Results Dashboard, offers a deep dive into individual test outcomes. This comprehensive approach ensures the reliability and effectiveness of the security alerting system.