Trail of Bits Blog
Follow
Mutation testing for the agentic era
Code coverage is a dangerous metric because it measures execution, not verification, potentially hiding untested critical functionality. Mutation testing, which systematically introduces bugs to see if tests catch them, addresses this by flagging untested code. Historically, mutation testing tools were slow and language-specific, hindering adoption, especially in blockchain. Universalmutator, using regex, gained traction but had limitations with multi-line statements and inefficient mutant prioritization. Slither-mutate improved speed through mutant prioritization and a cleaner testing cycle but remained Solidity-specific.
The new tools, MuTON and mewt, aim to overcome these challenges. MuTON offers first-class support for TON blockchain languages by utilizing the tree-sitter parser for better language comprehension and handling multi-line statements. Mewt serves as a language-agnostic core, also supporting Solidity, Rust, and Go. Both tools store results in a SQLite database, enabling persistence and flexible filtering. AI agents can now efficiently configure campaigns and triage results using specialized skills.
Future development focuses on AI-guided test generation that encodes requirements, not just bug detection. The goal is to equip AI agents with skepticism and demand external validation for robust test suites. Users are encouraged to install MuTON and mewt, contribute to the open-source projects, and watch for new AI skills that will streamline mutation testing. These advancements promise to transform mutation testing into a more routine and effective part of software development.
