Agents with Tools Fail Together

SWE-bench Verified is 500 GitHub issues from twelve Python repositories, each with a hidden test suite that decides whether a patch resolves it. Submissions to the leaderboard live in the public repository SWE-bench/experiments; for each one the team stores a results file with the instance ids the system resolved, and for the mini-SWE-agent series a per-instance file with the same information plus cost. I took the repository at commit 40f164d5 (2026-10-09), 182 submission directories, and kept the results files and the metadata: 135 carry the results list, 40 carry the per-instance file, one both, seven neither. Nothing else was fetched: no trajectories, no logs, no patches. The cell is 1 if the instance is in the resolved list and 0 otherwise; a system that generated no patch for an instance counts as 0.

The 175 systems span October 2023 (the RAG baselines of the original paper, 0.4–4.4%) to September 2026, with a median accuracy of 56.4% and a best of 79.2%, reached twice with Claude 4.5 Opus in two scaffolds (Sonar Foundation Agent and live-SWE-agent). Metadata gives the model, the scaffold, whether the submission used one attempt or an internal best-of-n, and whether the SWE-bench team checked it. None of the top seven is marked checked by the team; 121 of the 175 are not. The matrix, the script and the pre-registration are in the working repository and will be released with the note.

The pre-registration carried five predictions from the text corpus: pairwise φ 0.3–0.6 among the top seven (lower than text, since scaffolds differ more than samples do); n_eff of the seven between 2 and 3; the majority at or under the best single at every k; the oracle ten to twenty points above the best single; all-wrong across every system under 5% of 500.

Take the k most accurate systems and ask three things of them: how correlated their resolved sets are, whether a majority of them resolves more bugs than the best one alone, and how many bugs at least one of them resolves. The last number is what a perfect verifier would collect; on SWE-bench it is also what the hidden tests would collect if one could run them, which is exactly what a submission cannot do.

Majority: a bug counts as resolved if more than half of the k systems resolved it. Δ is majority minus best single, with a bootstrap over the 500 items (1,000 draws); the best single is re-chosen inside each draw. Oracle: resolved by at least one. All-wrong: resolved by none, out of 500.

The seven agree at φ 0.68 pairwise, in a narrow range (0.60–0.84), and together carry 1.38 effective votes. The majority of the seven lands on 79.4% against 79.2% for the best alone, two tenths of a point with zero inside the interval, and the picture does not change at three, five or fifteen; at thirty the majority is already losing, and over all 175 it loses twenty points, because the median system is at 56% and the vote is dragged to the median. The oracle is where the pool differs from its best member: ten points at seven, twelve at fifteen, fifteen and a half over all 175. Those points exist, they are not reachable by voting, and on SWE-bench they are not reachable by a judge either, because the judge that knows whether a patch passes is the test suite the submission cannot see.

The seven above mix models and scaffolds, so their correlation could come from either. The asset allows both controls.

One scaffold, 41 models. The mini-SWE-agent series runs the same hundred-line agent over 41 models, from a Gemini 3 Pro run that resolved nothing to Claude 4.5 Opus at 76.8%. Its seven best: φ 0.68, n_eff 1.39, majority 75.8% against 76.8% for the best (−1.0 [−3.4, +0.4]), oracle 88.0%. The same numbers as the mixed pool, to the second decimal on φ and n_eff. Model diversity inside a fixed scaffold buys the same nothing that family diversity bought on text.

One model, many scaffolds. Where the metadata names the same model in at least three submissions with accuracy 30% or higher:

Eleven scaffolds around Claude 3.5 Sonnet, from 34% to 63%, are the clearest case: the majority of the eleven is ten points under the best of them, the oracle sixteen points above. A scaffold moves which bugs a model resolves, and across eleven of them the union is wide, but no vote collects it. The newer the model the tighter the pairs: Claude 4.5 Opus in four scaffolds is at φ 0.75 and 1.23 votes, with a spread of five points between the best and worst scaffold.

Across all pairs of systems at 30% or better, same-model-different-scaffold pairs sit at φ 0.61, same-scaffold-different-model pairs at 0.56, pairs sharing neither at 0.51. Read naively, the model is the operator and the scaffold the smaller perturbation; but same-model pairs are also closer in accuracy, and φ rises with matched marginals, so this is a direction and not a measurement. Submissions that declare an internal best-of-n (81 of 175) are no less correlated with one another than single-attempt ones (top seven of each: φ 0.65 against 0.69).

Three of the eleven Claude 3.5 Sonnet scaffolds have been read by someone else. An empirical study of failures in automated issue solving (arXiv:2509.13941) takes the same public logs for OpenHands CodeAct 2.1, Tools + Claude and Agentless 1.5, all on Claude 3.5 Sonnet, and reports that the first two share 205 resolved issues with 60 and 40 of their own, and that Agentless resolves 30 that neither does; it reads the complementarity as a reason to ensemble, and does not count what an ensemble would do. The matrix here reproduces those four numbers exactly and adds the count: the three sit at pairwise φ 0.53–0.60 and 1.4 effective votes of three, their majority resolves 51.6% against 53.0% for the best of them, and their union is 67.0%, fourteen points above the best. The complementarity is real and is the oracle column; the vote that would collect it without the test suite does not exist here either. Added 11 October 2026.

The matrix has one run per system, so it cannot say how much of an agent's failure is the agent and how much is the dice. τ-bench can: the Sierra team ships, under MIT, the full logs of each model run four times (eight in the first version) over the same customer-service tasks, with the benchmark's own evaluator deciding each run. τ-bench v1 gives GPT-4o and Claude 3.5 Sonnet on airline (50 tasks) and retail (115); τ²-bench gives Claude 3.7 Sonnet, GPT-4.1, GPT-4.1 mini and o4-mini on airline, retail and telecom (50, 114, 114), all against the same simulated user. The same pre-registration discipline: predictions first, then the script.

Eight of the eighteen default runs; the rest sit inside the same ranges (φ 0.40–0.64, n_eff 1.37–1.82). An agent run four times is worth about one and a half independent runs, the number the text models gave when resampled at temperature one. Running again adds coverage, fourteen to twenty-five points from one run to any-of-four, and the coverage is as wide as a panel's: on the same τ² tasks, four runs of the best model reach 0.76, 0.93 and 0.70 (airline, retail, telecom) where four different models once reach 0.78, 0.97 and 0.63. Across models on the same domain the pairwise φ is 0.29–0.47, below the within-model 0.44–0.59, so the shared task and simulated user explain less of the co-failure than the policy does; the panel's gain over its best member is again the oracle only, seventeen to twenty points.

One thing the text corpus had and this one does not: a vote. A τ-bench run is a conversation that ends in a database state; there is no answer to count, and the evaluator is hidden from the agent. "Majority of four runs" can still be computed, and it shows what the operator is worth: with ties counted as failures it sits half a point to eight points under a single run, with ties counted as passes two to thirteen points above. Ten points that depend on a rule nobody can apply without the evaluator is the whole gap between pass@k and pass@1, priced in verifier.

Two predictions missed, both toward more correlation. I expected scaffolds that differ in tools, retries and prompting to be less correlated than samples from the same seven text models (φ 0.3–0.6, n_eff 2–3); they are at 0.68 and 1.38, tighter than the three larger text layers of Where the Swarm Pays (n_eff 2.1–2.3 of 7). One reading is that a bug's difficulty is more of a fixed property than a math problem's: the test suite is the same, the repository is the same, and the failing agents fail at the same locality. Another is leaderboard selection: everyone submits their best run, and best runs of strong models resolve the same easy 316. The 26 never-resolved bugs are 5.2% of 500 against the predicted under-5%; the benchmark has been attacked for two years by 175 systems and still has a floor, as the text corpus did before its label audit, and the audit here has not been done. A third reading was tested after the first draft, on 11 October: fit a one-dimensional item-response model to the matrix and look at what is left once every bug has a difficulty and every system an ability. The seven best keep a residual pairwise correlation of 0.05, and the model name explains more of that remainder than the scaffold (Limitations, last item).

A trajectory-level study published after the matrix was drawn says where the remainder sits. Coherence Collapse (arXiv:2603.24631) maps 16,758 SWE-bench and PolyBench trajectories of seven models in three scaffolds onto the files viewed, functions read and functions edited, and finds that for the capable models 60–69% of failures come after the agent has reached and edited the functions the reference patch changes: the right place, the wrong patch, and in five cases a patch identical to the reference that the agent then undid. The stages a scaffold controls, search and reading, are not where the strong agents fail; the stage the model controls is. That is the same ordering the residuals give, and it says what a different scaffold around the same model can and cannot move: which bugs are found, not which found bugs are fixed. The study also reports that successful agents converge on the same files 93.7% of the time and uses six other models' viewed files as a hint to a seventh; whether the six miss the same files on the bugs none of them resolves is in its list of things not tested, and its released sets of viewed files would answer it without running an agent. Added 11 October 2026.

This is also the number the orchestrator note asked Sakana for and did not get: the co-failure rate of a pool on the tasks the orchestrator is scored on. For a pool of top SWE-bench agents it is 11% at seven and 5.2% at 175. A router over the seven cannot exceed 89%; a workflow that rewrites the patch from several agents' work is not bound by that in principle, and nothing here measures it.

The within-agent number has one use already. The failure study above (arXiv:2509.13941) builds an Expert agent whose prompt carries its failure taxonomy, has it supervise an OpenHands Executor on the same model, and reports that the pair resolves 22.2% of 108 issues OpenHands had failed. There is no second plain run of OpenHands to compare with, the round budget was raised from the original runs to 40, and the taxonomy was built on the issues it is scored on. Section 4 gives the missing control from another benchmark: one agent rerun is worth 1.4–1.9 independent runs, and a second run alone recovers 14–25 points of the tasks the first one failed. A fifth of previously failed issues is inside what rerunning the Executor by itself would be expected to recover, so the number does not yet say whether the Expert adds anything to the dice. Added 11 October 2026.

• Not solo versus team. The scaffolds differ in model, tools, retries, prompts and date at once, so neither asset says whether a team of agents beats one agent on the same task. CooperBench (arXiv:2601.13295) is the only public setting with that contrast and publishes aggregates only: on 652 tasks GPT-5 resolves 48.3% alone and 28.0% as a pair, Claude Sonnet 4.5 47.1% and 25.9%, Gemini 3 Flash 48.6% and 26.2%; its repository holds the tasks, not the runs. HAL's traces are 113 GB, encrypted and unlicensed. The contrast stays open.
• Selected submissions. Leaderboard entries are what their authors chose to submit; 121 of 175 are not marked checked by the SWE-bench team, none of the top seven is checked, and 21 directories carry the team's flag for suspicious commits. Nothing was excluded on that basis.
• Two benchmarks, two scaffolds. The within-agent numbers come from τ-bench, a single tool-calling scaffold on customer-service tasks with a simulated user and a model-assisted evaluator; four τ² models and two τ¹ models, 50–115 tasks per domain, so airline intervals are ±6–10 points. The SWE-bench matrix has one run per system and no within-system number of its own.
• Model names are free text. "Claude 4.5 Opus" and "Claude Opus 4.5" are two strings in the metadata; the same-model table uses the strings as given, so some groups are undercounted (the OpenHands Claude 4.5 Opus run, 77.6%, is not in the four-scaffold row).
• No-generation as wrong. A system that produced no patch is scored 0, which raises correlation among weak systems; the top-k tables are unaffected, since the top systems generate for every instance.
• Labels unaudited. The 26 never-resolved bugs have not been read; on the text corpus the never-resolved items were mostly the benchmark's fault.
• Interval on φ. Intervals are reported for majority minus best only; φ and n_eff are point values over 500 items.
• Difficulty explains most of the 1.4. A 2PL item-response model with the 175 systems as persons, fitted on the 474 instances that at least one and not every system resolves (the never-resolved 26 dropped), beats a Rasch model held-out by 0.015 nats per response and leaves the seven best systems with a residual pairwise correlation of +0.05 [+0.01, +0.10] (bootstrap over instances) against φ 0.68 on all 500 instances and 0.61 on the 474; Kish n_eff on the residuals is 5.3 of 7, and 7.8 of 11 for the eleven Claude 3.5 Sonnet scaffolds (φ 0.52, n_eff 1.8 raw). Residual correlation within a scaffold exceeds between-scaffold by +0.055, within a model name by +0.085, within a model family by +0.017 (permutation p < 0.001 each; ICC 0.07, 0.11, 0.03). So the agents fail together mostly because the bugs are hard for everyone in one order, and the model carries more of the remainder than the scaffold, as the fixed-scaffold and fixed-model rows above already suggested. The number that matters for a vote is still 1.4: a vote does not get to condition on difficulty. Pre-registered on 11 October 2026 (irt-swebench-residual in the working repository); the predicted family excess of +0.02 to +0.12 came in under, at +0.017.

Data: github.com/SWE-bench/experiments, evaluation/verified/*/results/results.json, per_instance_details.json and metadata.yaml at commit 40f164d5, fetched 9 October 2026 by sparse checkout. Benchmark: Jimenez et al., SWE-bench: Can Language Models Resolve Real-world GitHub Issues?, ICLR 2024, arXiv:2310.06770; the Verified split is OpenAI and SWE-bench, August 2024. Definitions of φ, Kish n_eff, majority and oracle follow Where the Swarm Pays; the pre-registration with its five predictions was written before the script ran and is released with the matrix. τ-bench: Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv:2406.12045, logs in sierra-research/tau-bench historical_trajectories/ at 59a200c6; Barres et al., τ²-Bench, arXiv:2506.07982, logs in sierra-research/tau2-bench data/tau2/results/final/ at 4ce7c039; both MIT, fetched 9 October 2026, with their own pre-registration. CooperBench: arXiv:2601.13295, leaderboard at cooperbench.com read 9 October 2026. Failure study of the three Claude 3.5 Sonnet tools: An Empirical Study on Failures in Automated Issue Solving, arXiv:2509.13941. Trajectory stages: Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code, arXiv:2603.24631.

Earlier notes: Where the Swarm Pays (the axes this one completes), What a Model Orchestrator Can and Cannot Add (the router ceiling this one measures for a pool), Effective Independence (n_eff), There Is No Free Lunch After All (why the union is the boundary).

Archived with a DOI: 10.5281/zenodo.23299738 (all versions: 10.5281/zenodo.23299735). Cite as: Latipov, A. (2026). Agents with Tools Fail Together. Zenodo. https://doi.org/10.5281/zenodo.23299738

← notes · github.com/Alatip