If model A gets a larger prompt budget and more retries than model B, the chart does not compare models. It compares two different experiments.
Freeze what counts as the experiment
Write down the model identifier, date, prompt, system message, tools, temperature or sampling settings, context, retry policy and maximum time. Save the inputs. If one provider's interface silently retries a failed call and the other does not, count that difference instead of hiding it.
Choose tasks from a real workload, not whichever prompt made your favourite model look sharp. Include typical cases, hard cases, and a few cases where the model should abstain or ask for clarification. Define acceptance criteria before looking at the outputs.
For coding agents, freeze the repository commit, environment, test commands and permissions. For retrieval, use the same documents and search budget. For speech, use identical audio and a documented reference. Comparison only becomes meaningful when the conditions match.
Measure the work, not just the response
Pick a quality measure tied to user success. Then track latency, cost, retries, human repair time and failure severity. Use per-request values as well as averages. P95 latency matters when a person is waiting; a few dangerous errors matter even when the average accuracy is high.
A reasonable cost per usable result includes calls that are retried or rejected. Count tool calls, search, input and output tokens, preprocessing and review time. When the provider's pricing changes, save the retrieval date and assumptions so the calculation can be updated.
Repeat stochastic tasks. If you cannot afford many runs, be transparent about the small sample and avoid treating a tiny lead as certain. For subjective outputs, blind the reviewer to the model label where possible and make disagreement visible.
| Dimension | Example measure | Common trap |
|---|---|---|
| Quality | Task-specific pass rate or reviewer rubric | Using fluency as a proxy for truth |
| Reliability | Success across runs and held-out tasks | Reporting the best of several attempts |
| Latency | Median and upper percentile end-to-end time | Reporting only model generation speed |
| Cost | Expense per acceptable completed task | Ignoring retries and human review |
| Risk | Severity-weighted failures and abstentions | Averages that hide a rare high-impact error |
Write a result someone else could reproduce
Publish the test conditions beside the result. Keep the raw output where privacy rules permit, or provide a safe synthetic test if the input is sensitive. Separate provider claims from your own measurements. A vendor's reported score is useful context, not a measurement you performed.
If the model version is closed or changes over time, label the test date and API configuration. Re-run the evaluation when behaviour changes. Store the test prompts and scoring rubric so future comparisons have a baseline.
The result may be that a smaller, slower or less fashionable model works better for your use. That is a useful result. A comparison should help someone make a decision, not stage a race the test was not designed to settle.
Choose with a threshold, not a trophy
Before running the test, set minimum quality, maximum acceptable latency and budget boundaries. Among systems that meet them, choose the one whose operational tradeoffs fit your project. A model that misses a safety threshold should not win because it saves a few milliseconds.
Decide what will trigger another evaluation: a model update, a change to the prompt, a larger corpus, a new user group or an increase in error reports. That makes comparison a living engineering practice instead of a launch-day spreadsheet.
A fair test cannot tell you everything. It can make your uncertainty explicit and prevent a benchmark chart from making the decision for you.
FOLLOW THE EVIDENCE
Sources & further reading
Process note: AI tools assisted with research and drafting. Sources are linked so you can inspect the evidence. Unless an essay says otherwise, it does not claim hands-on testing or independent reproduction.
