DEV Community
Follow
Two "Codex CLI" models on the same benchmark: the harness hides the model
Specific Labs' Real-SWE benchmark exposes a critical issue in evaluating coding agents. Model names like "Claude Code" or "Codex CLI" are misleading because they represent a harness, not the underlying AI model. The same harness can show wildly different performance depending on the integrated model. For instance, GPT-6 Astra on Codex CLI achieved 33.8% resolution, while GPT-5.6 Sol on the same harness only reached 16.2%. This more than twofold difference highlights how the harness is merely a routing layer, not the component doing the thinking.Similarly, Fable 5.1 on Claude Code scored 38.8%, while GLM 5.3 on Claude Code scored 28.8%, a ten-point discrepancy. Real-SWE correctly presents results as model-and-harness combinations, an honest practice often avoided by vendors due to potentially unfavorable numbers under their own tooling. For teams seeking coding agents, simply "using Claude Code" provides no insight into the agent's actual skill. The model swap is the most significant performance lever, yet it remains hidden in most marketing.When reviewing benchmark scores, it's crucial to identify both the model and the harness used. Essential details include conventions, context carry-over, tool loop, and the judge. A vendor's reluctance to disclose the specific model linked to a score should be a red flag. The scaffold, or harness, can influence a score more profoundly than the agent's actual reasoning capabilities.