The cheapest token is not always the cheapest task
Production AI costs include retries, retrieval, tool calls, human correction and failures. A useful benchmark measures cost per accepted result, not only dollars per million tokens.
Build a task-level rubric
Define correctness, completeness, evidence, format and safety. Run the same representative cases across models, record trajectories and inspect errors. Separate model problems from data, prompt and tool problems.
Turn evaluation into a routing policy
Use a fast model for routine work, a stronger model for complex judgment and a human for high-risk actions. Re-run the evaluation whenever the model, data or workflow changes.