1Password's AI patching benchm... Note

1Password's AI patching benchmark is misleading

1Password's recent report on AI patching is flawed and presents a misleading 26% clean-fix rate. This figure is skewed by experiments that deliberately instructed AI agents to apply incorrect fixes and trials where agents could not compile or test their patches. The report's methodology included a difficult sample of vulnerabilities, prompts that told agents to apply wrong fixes 22% of the time, and a third of trials prohibiting testing. It also used inconsistent reasoning settings for models and penalized intended behavior changes during grading.Our reanalysis of 1Password's data, excluding trials with bad instructions or no testing, shows that 86% of patches blocked the supplied exploit, demonstrating useful patching capability. This contradicts the report's headline and indicates a significant ability for AI under reasonable conditions. The automated grading also exhibited issues, with low agreement with human reviewers and instances of accepting incomplete repairs or flagging valid fixes as regressions.Human developers, even under ideal conditions with detailed reports and expert reviews, also make mistakes, with 12.5% of their first fixes failing to fully resolve issues in our Trail of Bits security assessments. In real-world projects, our joint initiative with OpenAI, Patch the Planet, saw maintainers merge 67.7% of AI-assisted pull requests, with 72.2% of those accepted without security-relevant revisions. Notably, in one case, both a maintainer and an AI agent independently introduced the same crash when addressing a freenginx memory-safety bug, highlighting common pitfalls.We have since reviewed 33,500 subsequent commits in Patch the Planet projects and found functional bugs, build/test/release automation bugs, and performance issues introduced by our patches, but no exploitable security vulnerabilities. To improve AI patching, we are releasing two new agent skills: "post-patch-validation" to help agents rigorously test security fixes and "review-walkthrough" to assist engineers in reviewing code changes. The post-patch-validation skill guides agents through reproducing original bugs, testing alternative failure paths, and checking for regressions, aiming to catch incomplete fixes earlier.
CdXz5zHNQW_Lgu2ORz1CG.webp