DEV Community
Follow
Why you shouldn't let the model review its own AI code
A recent arXiv paper highlights structural weaknesses in AI code review patterns. It introduces a two-gap framework for evaluating software implementation against requirements and a deployment environment. The requirement gap exists between stakeholder needs and documented requirements, while the model gap is the difference between the assumed and real deployment environments. AI hallucination exacerbates both these gaps by fabricating information.The paper argues that when the same AI model that generates code also reviews it, it does so using the same flawed requirements and environment model. This leads to a false sense of verification because the AI is essentially re-checking its own assumptions and blind spots. Self-review only catches errors the model already recognizes as problematic, while passing code that shares its incorrect assumptions.To effectively narrow these gaps, the paper proposes two key strategies. Cross-model review involves a second, independent AI model re-deriving requirements and environment assumptions from scratch, mitigating shared blind spots. The other crucial strategy is to run the code in a production-like environment, as reality is the ultimate verifier.Pre-deployment evaluations are proxies, and execution-based checks are superior to static assessments because they demand observable behavior. The paper frames human judgment as the scarce resource for the requirement gap and accurate evaluation as the bottleneck for the model gap. Given the volume of AI-generated code, a sensible approach involves AI-generated diffs being reviewed by an independent model, followed by execution-based checks. Human review should then focus on the code that passes these initial stages, maximizing the return on human attention. While same-model AI review can act as a linter, it should not be mistaken for true verification.