Martin Fowler
Follow
Are Harnesses Worth the Cost?
Naresh Jain compares coding agents building the same
API service with and without a behavioral harness. A frontier model, GPT-6
Sol, passed every external evaluation either way. But a cost-optimized
model, GPT-6 Luna, succeeded in only 3 of 10 unharnessed runs and all 10
with the harness. While adding the harness increased token usage and
runtime for Luna, the harnessed Luna runs achieved the same 10-for-10
reliability as GPT-6 Sol while costing 33% less per run ($0.273 vs
$0.406). This experiment suggests a behavioral harness might be a
practical way to get dependable results from smaller, cost-optimized
models.