An agent can close a benchmark issue and still be the teammate you do not trust with your repository. The missing part lives between passing tests and owning the change.
Issue resolution is a useful slice
SWE-bench asks a model to resolve real GitHub issues in a repository, then checks whether the patch passes tests. That is a concrete, valuable task. Verified narrows the benchmark with human filtering because some original tasks were ambiguous or did not reliably test the requested fix.
But passing the provided tests is not a complete definition of responsible software work. Tests may miss a regression. The patch may implement a brittle workaround. A human maintainer might reject an answer that technically passes but ignores project conventions or expands the change unnecessarily.
The benchmark measures a slice of work under a specific environment. It says something useful about that slice. The mistake is promoting it into 'this agent can do software engineering'.
The agent is bigger than the model
The system includes the model, prompt, tools, context management, retry policy, sandbox, and budget. Give a model search and terminal tools and it can inspect a repository in ways a bare prompt cannot. Change the harness and the outcome may change even when model weights stay fixed.
That makes benchmark comparison harder. Reports should name the full configuration and resource limits. How much wall time? How many model calls? Could it run tests repeatedly? Did it receive a prepared description or discover files itself? Without these details the reader cannot tell whether the advantage came from reasoning, scaffolding, or simply more attempts.
| Benchmark result | Good evidence for | Still untested |
|---|---|---|
| Tests pass | The patch met the supplied tests | Uncovered regressions or maintainability |
| High task completion | Solves examples under the stated setup | Reliability outside the chosen tasks |
| Fast completion | Lower time for successful benchmark runs | Cost, retries, setup or review time |
| Large context | Can process large inputs | Whether it used relevant context well |
Software quality has more than one axis
A useful local evaluation asks whether a patch is correct, small enough, readable, secure, and easy to maintain. It also asks what it costs to get there. Count the first-pass success rate and the number of interventions. Record when a model edits unrelated files, invents APIs, or claims to run checks it did not run.
Review the failed cases carefully. An agent that fails loudly is safer than one that confidently hides failure. A green score can cover a dangerous false success, such as a test weakened to pass. Your evaluator needs to notice those behaviours, not just whether a task ended with a patch.
- Use recent issues from your codebase, after removing private material.
- Freeze the repository state, tools, prompt, model version, and time limit.
- Score test results, patch quality, unrequested edits, and human review time.
- Keep original and hidden tests so the agent cannot pass by memorizing a task.
- Track cost and failure severity alongside completion.
Where the number belongs
Benchmark scores help shortlist tools and generate questions. They do not answer whether an agent fits your codebase. Use the benchmark to locate capability, then run the same controlled task set on your work and review its patches like you would review any new contributor.
If the agent saves a developer fifteen minutes but creates twenty minutes of cleanup, its benchmark reputation does not rescue the economics. If it produces a safe first draft that an engineer can validate quickly, even a lower benchmark score may be irrelevant to its usefulness.
The hard part is not a model writing code. It is knowing whether the change should be there, and taking responsibility when the answer is wrong.
FOLLOW THE EVIDENCE
Sources & further reading
Process note: AI tools assisted with research and drafting. Sources are linked so you can inspect the evidence. Unless an essay says otherwise, it does not claim hands-on testing or independent reproduction.
