Paper draft · v10 · Verification

Draft Paper (v10). What an Unreliable Verifier Collects

On agent benchmarks the share of tasks that at least one run solves sits 10 to 32 points above the best single run, and only a selector with an external check can reach it. This draft asks what an imperfect check collects: a formula with two error rates per task, one released verifier, and two pre-registered runs of our own on the same 1,600 candidates. Model judges' false-reject rate, not their false-accept rate, rises with task difficulty; model-written tests, even after reading the code, accept 46% of the wrong patches. The rule with the largest point estimate, found after the pre-registered endpoints, gates on tests and ranks by a judge: 0.28 of the prize at eight candidates, with an interval that overlaps the tests alone.

13 October 2026draft v1031 pages, 30 referencesPDF on Zenododoi:10.5281/zenodo.23336060

Abstract A selector that picks among n candidate runs with a verifier can reach the oracle, the share of tasks that at least one candidate solves, only if the verifier never accepts a wrong candidate. We replace the single error rate of the earlier paper draft by two rates per task, false accept α_i and false reject β_i, and derive what follows: the selector's accuracy tends to the verifier's precision at the task's base rate, not to 1; false rejects heal with more candidates, false accepts hurt; and rates that vary by task matter more than their mean. On the R2E-Gym release (500 SWE-bench Verified bugs, about 25 candidates each, a trained model verifier) the per-task formula fits the selection curves within half a point in-sample and removes a third to two thirds of the pooled formula's error out of sample. The verifier collects half of the 32-point prize at two candidates and about a third at sixteen to twenty-five. Its false-reject rate rises with difficulty (Spearman +0.46 against item-response difficulty); its false-accept rate does not. In two pre-registered runs on 200 contested SWE-rebench issues with 8 Qwen3-Coder rollouts each, a Claude Haiku judge (α 0.20, β 0.54) collects 0.15 of the prize at eight candidates by threshold and 0.19 by ranking. Tests written by Haiku from the issue text accept 39% of wrong patches; tests written after reading the repository accept 46%, although they are valid against the reference fix on 163 of 200 issues and collect 0.23. The agent's own tests, on the 30,740 released rollouts that carry them, accept 69%. Model-written tests supply recall, not a small false-accept rate; the one strict execution-based verifier in any of the data uses the hidden fix. Tests as a gate before the judge's ranking, a rule examined after the pre-registered endpoints, collect 0.28 [0.15, 0.40] at eight candidates, the largest point estimate in this series, with an interval that overlaps the tests alone. The residual error correlation between model families on the SWE-bench leaderboard, after one difficulty factor, is at most 0.09, so choosing a verifier from another family moves little; the false-accept rate decides.

Results

The frame. On agent benchmarks the oracle, the share of tasks that at least one run solves, sits 10 to 32 points above the best single run (9.8 among the top seven SWE-bench Verified submissions, 19 to 32 in mid-ranked bands, 14 to 25 on τ-bench), and only a selector that checks candidates against something outside the solvers can reach it. A rule's share of the prize is (its accuracy − a random candidate) / (oracle − random) at the same number of candidates n. The draft replaces the single verifier error rate of the earlier working paper by two rates per task: false accept α_i and false reject β_i.

The formalism: three consequences.

A precision ceiling: as n grows, the selector's accuracy on a task tends to the verifier's precision at the task's base rate, p(1 − β) / [p(1 − β) + (1 − p)α], not to 1; the prize sits on hard tasks, where p is small.
α and β are not symmetric: false rejects heal with more candidates (β^k), false accepts grow with them. At n=4, raising β from 0 to 0.3 costs 20 points of share; raising α by the same amount costs 49.
Rates that vary by task collect less than their mean says; the per-task version replaces (α, β) by (α_i, β_i).

A released model verifier (R2E-Gym, 500 SWE-bench Verified bugs, 12,695 candidates): the formula holds per task.

Prize 32.5 points (pass@1 0.319, pass@n 0.644). The verifier's argmax collects 0.51 [0.46, 0.56] of it at two candidates, 0.38 at eight, 0.33 [0.26, 0.41] at twenty-five; at a strict threshold (0.7) the share rises with n instead, as the asymmetry predicts.
One pooled (α, β) overpredicts the threshold rule by 3.8–5.7 points; per-bug rates estimated on a held-out half remove 0.36–0.65 of that error, the rest being estimation noise.
Errors cluster by bug (intraclass correlation 0.26–0.43). The false-reject rate rises with item-response difficulty (Spearman +0.46 [+0.36, +0.55]); the false-accept rate does not (+0.09).

Our own verifiers on the same 1,600 Qwen3-Coder candidates (200 contested SWE-rebench issues, 8 rollouts each, prize 42.4 points; two pre-registered runs).

Claude Haiku judge: α 0.20, β 0.54; 0.15 [0.08, 0.22] of the prize at eight candidates by threshold, 0.19 by ranking (0.32 at two). Same pattern as R2E: errors clustered by issue (ICC 0.65), β, not α, tracks difficulty.
Tests written by Haiku from the issue text: valid against the reference fix on 98 of 200 issues, α 0.39, share 0.11 at eight.
Tests written after reading the repository: valid on 163 of 200, β falls 0.48 → 0.25, but α 0.46 [0.39, 0.53] does not fall; share 0.23 [0.16, 0.30] at eight, the largest of any single verifier. The pre-registered prediction that tests would be the strict verifier failed both times.
The agent's own tests (30,740 released rollouts that carry them): α 0.69, share 0.04. The one execution-based check with a small α in any of the data (0.12) is restricted by the hidden reference fix.
Tests as a gate, then the judge's ranking, a rule examined after the pre-registered endpoints: 0.42 [0.31, 0.52] at two candidates, 0.28 [0.15, 0.40] at eight, the largest point estimate, with an interval that overlaps the tests alone. A lead, not a result.

Independence of families: moves hundredths. On the SWE-bench Verified leaderboard (91 submissions with a known family), the residual error correlation after one difficulty factor is at most 0.09 within a family (Qwen +0.092) and −0.03 to +0.03 between families, against raw correlations of 0.4–0.5 from difficulty alone. Choosing a verifier from another family moves little; the false-accept rate decides.

In one sentence. A verifier's two error rates, measured per task, predict what a selector collects; on agent tasks the prize goes to a verifier with a small false-accept rate, and neither model judges nor model-written tests, blind or reading the code, supply one, so the best deployable rule found collects 0.28 of the prize at eight candidates by gating on tests and ranking by a judge.

Download

The PDF (31 pages, 30 references) is on Zenodo: doi:10.5281/zenodo.23336060. Version 10 of 13 October 2026: the terms, the prior results, the formalism with its derivations, the six hypotheses with their pre-registered thresholds and verdicts, every experiment (E1 to E5b), a results section by question, and the conclusion. Sections 0–4 are the plan before the experiments, 5–6 the hypotheses and plan with verdicts, 7–9 and the Conclusion the results. A changed text gets a new Zenodo version.

What the draft is built from

Data: R2E-Gym, Jain, Singh, Shetty, Zheng, Sen and Stoica, R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents, arXiv:2504.07164, release R2E-Gym/32B_predictions on Hugging Face; Nebius, nebius/SWE-rebench-openhands-trajectories (CC-BY-4.0) and the swerebench environment images; the SWE-bench Verified leaderboard matrix and item-response fit of Agents with Tools Fail Together. Theory: Dorner, Chen, Cruz and Yang, ROC-n-reroll, arXiv:2507.12399; Stroebl, Kapoor and Narayanan, arXiv:2411.17501. Compute: 4,000 cloud sandboxes, under $10 in all.

The draft supplements the working paper Exploring the Boundaries of Multi-Agent Ensembles (doi:10.5281/zenodo.23311676), whose bound for a selector assumed a verifier with one error rate, and builds on these notes:

Status and what is still open

This is a working draft, not peer reviewed. The two runs of our own were pre-registered in full; of their five primary endpoints the two about tests being strict failed, two were undecided, and the per-task-rate endpoint held; the gate-then-rank rule was not among them. The R2E analysis was partly pre-registered, after a first look at the same data. Known limits: two benchmarks and two solvers, one verifier per kind, one test-writing model with at most five tests per issue; the 200 issues are contested by construction, so shares are shares on contested issues (29% of issues with eight candidates); per-task rates rest on about 13 candidates per half on R2E and 4 on our runs; "correct" means "passes the benchmark's hidden tests". Two data sets could not be used (the LLM-as-a-Verifier release ships no judge scores; the W&B Programmer submission stores only the submitted patch). The code and pre-registrations (sim_select.py, the E1/E3 analyses, the E5/E5b runs with PREREG.md, run.sh and verbatim model outputs, the family-residual matrices) are to be released as a companion repository. The verifier the formula asks for, one with a small false-accept rate that does not use the hidden tests, is still to be found.

Cite

Latipov, A. (2026). Paper Draft. What an Unreliable Verifier Collects (v10). Zenodo. https://doi.org/10.5281/zenodo.23336060. Archived with a DOI on 13 October 2026.

← notes · github.com/Alatip