How Small Can an Agent Model G... Note

How Small Can an Agent Model Get? The Nemotron Floor

This analysis compares NVIDIA's Nemotron models on real-world coding tasks to determine their effectiveness and value. A key finding is that model size acts as a threshold, rather than a linear improvement in quality for agent tasks. The smallest model, Nemotron 12B, failed to perform at all because it fell below a critical capability floor. Above this floor, the choice shifts to which model meets the specific work requirements. Nemotron 30B is a cost-effective option for simple, well-defined jobs. Conversely, Nemotron 120B is necessary for more complex, multi-step agent operations. The evaluation uses real-world tasks and skills, assessing both instruction following and task completion. Unlike contrived benchmarks, these tasks reflect actual development work, highlighting fundamental capability gaps. The Nemotron 120B model mostly completes tasks but struggles with precise instruction following. The Nemotron 30B model, however, has issues with reliable completion, often failing entirely. This suggests that model scale dramatically changes outcomes, leading to either near-perfect results or significant failures. Skills provide significant benefits, but only on models that are already capable enough to act on them. Forcing a skill onto a model below the capability floor does not improve performance proportionally. The cost per task can be misleading, as failures and retries add hidden expenses, making the seemingly cheaper model more costly overall. Therefore, the optimal approach is to first define the required quality level for the work and then select the model that meets that standard.