Trail of Bits Blog
Follow
How we use /goal to find bugs in Patch the Planet
The goal feature in Codex amplifies bug hunting by allowing it to work independently towards a success condition, but getting good results requires the right prompt, scope, and number of outcomes per run. For the Patch the Planet initiative, Codex was pointed at widely used codebases like Rust, curl, and zlib, and it found every Rust bug submitted, including a soundness hole and a miscompilation. The engineers converged on three techniques for using the goal feature, including letting Codex write the goal, defining the outcome, not the path, and assigning one outcome per agent. Letting Codex write the goal works because it knows the target and its own tendencies better than humans can specify up front, and it can translate a threat model into concrete, testable success criteria. Defining the outcome, not the path, means spending tokens on defining the outcome and almost none telling the model how to get there, and a good goal names the outcome, defines it precisely, and enforces persistence. Assigning one outcome per agent results in uneven optimization if two competing outcomes are put in one prompt, and it is better to move coverage out of the prompt entirely and ask Codex to identify the most promising attack surfaces. The most effective resource for goal bug hunting is a threat model file, which precisely defines what valid bugs look like without explaining how to find them. The goal feature is a powerful tool for amplifying bug-hunting work, but its effectiveness still depends on an expert knowing where to look, verifying its results, and knowing what the maintainers want to see as a valid vulnerability disclosure. Overall, the goal feature has been successful in finding bugs, including every bug in Rust, and it has the potential to create custom security infrastructure that takes a security researcher weeks to build in under a day.