How I Replaced LLM-Based Code Analysis with Static Analysis (And Got Better Results)

The author initially employed an LLM to build a security scanner for AI-generated code. This LLM-based approach proved unreliable, producing inconsistent vulnerability assessments with varying severity scores. The scanner's results were highly variable, failing to consistently identify known vulnerabilities across multiple runs. This inconsistency, mirroring p-hacking, made the LLM unsuitable for security applications. The author realized the LLM's limitations when it sometimes classified a critical command injection as low risk. Consequently, the author switched to a static analysis method. The new static analysis approach offered deterministic results, consistently identifying vulnerabilities correctly. Static analysis proved significantly faster and cheaper, allowing for more frequent scans. It also demonstrated better coverage, detecting more vulnerabilities more reliably. The static analysis system employed 93 rules across 14 categories. The author gained precision in identifying composite risks and AI-specific vulnerabilities, which the LLM struggled with. While lacking natural language explanations and context awareness, the static analysis offered pre-written descriptions and confidence levels to compensate. The static analysis approach showed improvements across all measured metrics compared to the LLM. The author concludes that static analysis is superior for automated scanning, emphasizing its speed, consistency, and reliability. The author suggests LLMs are better suited for code review and threat modeling but not for automated vulnerability detection. Ultimately, the simpler approach proved better for security scanning.