You do not need a public leaderboard to know your feature fails. Ten examples from the real task can reveal more than a hundred model comparisons with somebody else's prompts.
Collect examples from the task, not the model demo
Start with the user's actual inputs. Remove sensitive details and get permission to use the material. Include ordinary examples, the awkward ones that arrive in practice, and examples that should make the system pause. A summariser needs short and long documents. A classifier needs each category and cases where category boundaries are unclear.
Do not let a model generate all the evaluation questions and expected answers. It can help organise a set, but if both tests and output are made by the same system, you risk rewarding its own habits. Have someone who knows the job define what a correct and useful result looks like.
Keep the examples small enough that a person can inspect them. A table with input, acceptable result, evidence, severity and notes is a solid first format.
Define success before you see the output
Write a short rubric with observable criteria. For a document answer, criteria might be factual correctness, relevant source citation, no unsupported detail and a clear 'not found' response when evidence is missing. For a draft message, you may score whether it addresses the request and preserves required facts. Avoid 'sounds good' as the only criterion.
Separate quality from safety and cost. A response can be fluent but wrong; correct but too slow; mostly useful with one high-impact failure. Track what matters for this use. Label critical failures separately so averages do not hide them.
If you compare models with human judgement, keep the reviewer blind to which model produced which response where practical. Write down disagreement and resolve the rubric before judging every case by intuition.
Freeze the setup so the comparison means something
Record model ID, prompt version, tools, temperature or sampling settings, input sources and date. Keep one fixed set for comparison and a separate development set for debugging. If the prompt changes, version it. If the model changes, rerun the same tasks.
For agents, examine the trajectory and outcome. Count the retries and tool calls, note whether the right evidence was available, and inspect side effects. Run repeated attempts for stochastic workflows when you can afford it. One successful screenshot is a demo, not a reliability estimate.
Small samples do not give precise guarantees. They give you readable evidence about the examples you chose. Expand the set when the product reaches new users or when you discover a new failure class.
Let failures teach you what to fix
Classify failures before adjusting the prompt. Was required data missing? Did retrieval fail? Did the model misread an instruction? Did validation accept an invalid response? Did the interface hide a needed confirmation? Different causes need different fixes.
Add a failure example to the regression set after you understand the cause. Change one major thing at a time, then rerun the original set and the new case. A prompt that fixes one task by breaking another is a tradeoff to see, not a win to hide.
Your first evaluation does not need a complex dashboard or a universal score. A small spreadsheet, a rubric and a repeatable script are enough to start making decisions with evidence. Keep it boring and keep it runnable.
FOLLOW THE EVIDENCE
Sources & further reading
Process note: AI tools assisted with research and drafting. Sources are linked so you can inspect the evidence. Unless an essay says otherwise, it does not claim hands-on testing or independent reproduction.
