DEV Community
Follow
Why evidence matters more than model memory in AI pentesting
AI findings that cannot be reproduced are considered liabilities rather than definitive results. Darkmoon addresses this by appending exact commands and raw tool output to every finding, enabling human peer review. Traditional AI security tools often provide only a confidence score and a textual description, which is insufficient for offensive security. The ability to demonstrate the specific command that confirmed a vulnerability is crucial for defending findings in reports and meetings. Darkmoon stores the executed command, its raw output, and the reasoning connecting them for each finding. This approach transforms findings into reproducible artifacts, eliminating the need for blind trust. While larger models can reduce errors, they don't eliminate them entirely. Darkmoon's evidence trail empowers humans to identify any remaining errors, making it a core design principle. This methodology streamlines the review process, shifting it from full re-verification to spot-checking the evidence trail. Consequently, reports can be automatically generated from real data, bypassing the need to paraphrase model output. Users can experience this evidence trail firsthand by cloning the open-source Community Edition and testing it on a lab environment. This open-source tool is built by pentesters for pentesters, welcoming feedback on its methodology.