Alibaba's new Qwen 3.8-Max shows conflicting performance results, with Alibaba claiming it's second only to Claude Fable 5, while an independent benchmark places it mid-pack. This discrepancy arises from differing token and time budgets used in testing. Alibaba's benchmarks utilized significantly longer timeouts, explaining the better scores.The article argues that price per token is an insufficient metric for evaluating reasoning models. Instead, a "cost per successful task" metric is proposed, which accounts for all expenses, including failed attempts, divided by successfully completed tasks. This approach provides a more realistic view of model efficiency and cost.Furthermore, the article highlights that failure rates are heavily influenced by configuration settings, particularly time or token budgets. Benchmarks often fail to distinguish between outright incorrect answers and tasks that simply time out, with budget exhaustion being a dominant cause of failure. It suggests that clearer reporting of failure reasons is necessary.The authors advocate for making time or token budgets explicit acceptance criteria rather than hidden details. This is crucial for building effective agent systems, as optimizing for speed without success can lead to increased costs and poor outcomes. Several organizations are already adopting cost-per-successful-task metrics.To improve model evaluation, it's recommended to emit failure reasons separately, compute cost per successful task per effort level, and cap on tokens rather than wall clock time unless latency is critical. Finally, checking default effort settings on deployed models is advised, as higher effort settings do not always yield better results.