The author's infrastructure bill was exorbitant due to high costs from a single coding assistant API. This prompted an investigation into alternative models and their actual performance. They benchmarked ten models across five common engineering tasks to determine cost-effectiveness. Marketing claims about "best in class" models were found to be unreliable. The goal was to find models that delivered the best return on investment for their specific workloads.The benchmark involved scoring models on correctness, code quality, documentation, and edge cases, then dividing by their output cost per million tokens. This "score-per-dollar" metric proved crucial in identifying efficient models. The author discovered that specialized and cheaper models often outperformed expensive, general-purpose ones on common tasks. For complex reasoning or critical code reviews, however, premium models demonstrated their value.The testing revealed that different models excel at different types of tasks, leading to a routing strategy. The author now directs prompts to specific models based on their complexity and nature. This approach significantly reduced their API spend by an estimated 87%. A routing model like Ga-Standard was also highlighted as an option for those who don't want to build their own routing logic.A key takeaway is the importance of avoiding vendor lock-in. By using a unified API layer, the author can easily switch models if prices change or performance degrades. This abstraction ensures flexibility and prevents costly migrations. The author advocates for standardizing on an OpenAI-compatible interface to minimize integration costs. Ultimately, the benchmarking process led to substantial cost savings and a more robust, adaptable AI coding assistant strategy.