Debate Is Not Sampling

The first paper treats the n attempts a model makes at one problem as a cluster sample. With ρ the intraclass correlation of correctness, the effective sample size is n/[1+(n−1)ρ], bounded by 1/ρ; that is the correlation ceiling, and it was the subject of two earlier notes on this site.¹ The paper adds a second result about selection. Plurality voting over exchangeable samples converges to the rate at which a problem's most common answer is correct, π_mode, and on a problem whose mode is wrong every further sample makes the plurality more confidently wrong. Coverage rises; selection has a ceiling; the gap between them is the identifiability gap.

The second paper is TUMIX, a test-time scaling system: fifteen heterogeneous agents (text, code, search) answer a question in parallel, read each other's answers, answer again for a few rounds, an LLM judge decides when to stop, and a majority vote closes. On Humanity's Last Exam it reports the identifiability gap almost verbatim: at least one agent is right on 65% of questions while the final answer is right on 34%, and the authors note that over-refinement discards correct answers and ask for a principled stopping rule based on the marginal value of another round. They never measure the correlation between agents.

A literature-based-discovery pass over a graph of 125 papers paired the two and wrote the hypothesis down: in refinement, majority accuracy after round r is bounded by π_mode; refinement helps only on problems whose first-round mode is correct and hurts on the rest; and a rule that stops when the effective sample size stops changing should match the judge. The non-obvious step was that the theory assumes exchangeable draws from a fixed distribution, while rounds of debate actively raise ρ, so each round would be a new cluster sample. That step is where the bridge breaks, and this note is about how.

The theorem is about aggregation. Fix a problem and a per-problem answer distribution; draw n answers; take the plurality. As n grows the plurality finds the mode of that distribution, and nothing else, because nothing else is there to find. Three things follow for a refinement process, if it is sampling in disguise:

• Round 0, where agents have not yet seen each other, is the only round whose answers come from the undisturbed distribution. If later rounds only aggregate, the final majority cannot do better than the round-0 modal-hit rate except by noise.
• On problems whose round-0 mode is wrong, the agents should converge on the wrong mode: per-agent accuracy on that stratum should fall from round to round.
• The correlation between agents should rise toward 1, and the effective number of voters toward 1, as the panel collapses onto its mode.

The third is a prediction about ρ and is what the earlier notes would expect. The first two are predictions about π_mode, and they hold only if the per-problem distribution is the same object in every round. In debate it is not: an agent in round 2 is answering a different prompt than in round 0, one that contains two other answers and, in most systems, their reasoning. Whether that difference matters is an empirical question, and the data to answer it already existed.

TUMIX has not released its logs. The closest public substitute is DebateLedger,² released in September 2026 with a paper that counts collapses (a correct initial majority that ends wrong) and corrections (the reverse): three copies of one model answer an MMLU-Pro question independently (round 0), then for three rounds each agent sees all current answers and revises. The open tier keeps every parsed answer letter per agent per round and the gold label; model text is withheld. We used the primary cohort, 6,925 debates across Gemini 3-flash, GPT-5.4-mini, Llama-3.1-8B, Phi-4-mini, Qwen3-4B and Qwen3-8B, and three extension sets as robustness: three heterogeneous panels of three different models, three GSM8K runs with free-form answers, and four Qwen3 runs with reasoning on and off. Our loader reproduces the paper's own headline counts exactly (6,925 debates, 253 collapses).

The pre-registration, committed before any outcome was computed,³ fixed three hypotheses, each with a statistic, a cluster bootstrap over questions (2,000 resamples) and a decision rule over the six primary files:

• H1, ceiling. D1 = MV₃ − MV₀, the final majority accuracy minus the round-0 modal-hit rate, has its interval inside ±0.03 on at least 4 of 6 files. Falsified upward if the interval is entirely above +0.03 on 3 or more.
• H2, anti-scaling. On problems whose round-0 plurality is a wrong letter (S⁻), per-agent accuracy falls from round 1 to round 3: D2 = A₃(S⁻) − A₁(S⁻) < 0 with interval clear of zero on 4 of 6. Round 0 itself is excluded from the comparison because conditioning on it caps A₀(S⁻) at one third by construction. Falsified if D2 > 0 with interval clear of zero on 3 or more.
• H3, correlation. (a) The one-way ANOVA intraclass correlation of agent correctness within a problem is non-decreasing over rounds 0–3 and its rise has an interval clear of zero. (b) A rule that stops after the first round where the implied n_eff = 3/(1+2ρ) changes by less than 0.05 keeps majority accuracy within 0.01 of three fixed rounds while using at most two.

One hypothesis in the discovery system's statement, that per-round majority accuracy tracks that round's mean π_mode, was dropped before the run because for a single panel of three it is an identity: the modal-hit rate of a round is that round's majority correctness. The honest prior written down was 65% for H1, 55% for H2, 85% for H3a and 40% for H3b.

Majority accuracy by round and the three pre-registered statistics on the six primary files. Intervals are 95% cluster-bootstrap percentiles over questions. S⁻ is the wrong-mode stratum; ICC is the within-problem inter-agent correlation of correctness; n_eff is the effective number of voters in the panel of three.

^* This row's round-0 accuracy is 0.245 where the same model scores 0.615–0.630 at round 0 in the reasoning-on/off runs; see Limitations.

H1 is falsified upward. Three of six intervals lie entirely above +0.03; one of six lies inside the band. The excess is largest where round-0 accuracy is lowest: across all sixteen files the correlation between MV₀ and D1 is −0.74. On the strong model, on GSM8K (round-0 accuracy 0.95–0.96, D1 between −0.003 and +0.010) and on Qwen3-32B with reasoning on, the ceiling holds numerically, because there is almost nothing left to correct.

H2 is falsified. No file shows anti-scaling. On the wrong-mode stratum, per-agent accuracy from round 1 to round 3 is non-negative on all sixteen files and positive with interval clear of zero on three of the six primary ones; the majority-correct rate on the same stratum, which the theory says should stay at zero, rises to 0.106 on Phi-4-mini, 0.370 on Qwen3-4B and 0.488 on Qwen3-8B.

H3a is confirmed on all six files, and on all sixteen. The rise in ICC is monotone everywhere and its interval is clear of zero everywhere, with the smallest rise on the model that starts nearly saturated (Gemini, 0.914 → 0.987) and on Llama-3.1-8B, which barely revises (0.504 → 0.615).

A majority that was wrong at round 0 can become right in two ways. If one of the three agents had the right answer, the other two can be talked into it; that is persuasion by a correct minority, and it is compatible with a story in which the information was already in the panel and debate only aggregated it better than a vote. If none of the three had it, no aggregation of round-0 answers can produce it: the agents have to generate an answer that was not there. The split was not pre-registered; it is the first thing we looked at once H1 and H2 had failed.

The weak models dig themselves out of unanimous error at rates the modal ceiling says are zero: 31% of 793 such problems on Qwen3-4B, 45% of 110 on Qwen3-8B. That is not aggregation. It is re-sampling under a changed prompt, in which three identical wrong letters evidently prompt a model to reason again, and the per-problem distribution it samples from in round 1 is not the one it sampled from in round 0. Collapses exist on the same models, 14–15% of two-against-one correct panels lose the correct majority on Llama and Phi, but they are outnumbered by corrections wherever round-0 accuracy is low.

Whether the peers matter, or any second look would do, has a control in the same dataset: Qwen3-8B runs in which each agent revises for three rounds seeing only its own previous answer. Same model, same 200 questions, same rounds.

Self-revision alone is worth one to one and a half points of majority accuracy and does nothing to the correlation. Seeing the peers is worth three and a half to four points and takes the correlation from 0.72 to 0.97. On this properly configured Qwen3-8B the all-wrong stratum is corrected at 0–10% in either mode, which is the number to carry, not the 45% from the anomalous primary row; the one-correct stratum is where peer visibility does its work, 40–50% against 0–10% without it.

The correlation ceiling is intact and sharper than in the sampling case. By round 3 the effective number of voters in a panel of three is 1.0–1.3 on every file (last column of the table in Section 4; 1.02–1.20 on the extensions). The panel is one voter; further rounds cannot add votes, only re-samples, and the three heterogeneous panels, which the earlier notes would expect to decorrelate, end at ICC 0.75–0.78 from 0.37–0.47, the largest rises in the data. Heterogeneity buys a lower starting correlation and the same destination.

TUMIX asked for a stopping rule based on the marginal value of another round. The pre-registered candidate was the plateau of n_eff: stop when it changes by less than ε between rounds. It is label-free, which is its merit, and on the weak models it does not fire, because the correlation is still climbing at round 3 (Figure 1). On three of six primary files it stops at round 3, which is no rule at all; where it stops earlier it loses at most 0.005. The rule that works is the one every debate paper already uses, applied per problem: stop when the three agents agree.

Majority accuracy at the stopping round / mean rounds used. "Best round per problem" is an oracle upper bound. Across all sixteen files unanimity is within −0.015 of three fixed rounds and above it on three.

It does not say debate is a good idea. The gain is 1–6 points on the models anyone would deploy, zero on GSM8K, and scales inversely with how badly round 0 went. It does not say TUMIX's 34% is not a selection ceiling; it says the ceiling is not π_mode of the first round, and the system's own observation that coverage falls while accuracy rises is the correlation ceiling at work. And it does not touch the question the first paper actually studied, independent samples from one session, where the theorem applies as stated.

• The primary Qwen3-8B row is anomalous: round-0 majority accuracy 0.245 against 0.615–0.630 for the same model at round 0 in the reasoning runs (the dataset paper's own table lists it at 28%). Round 0 there was probably under-elicited; the open tier has no text to check. Without that file, two of five primary intervals lie above +0.03, which is "not within the band, not formally falsified" for H1. H2's rejection and the all-wrong corrections on Qwen3-4B (n = 2,161) do not depend on it.
• Round 0 and rounds 1–3 use different prompts (no peer block against a peer block), so part of the round-1 jump may be elicitation rather than information. The self-only control bounds the pure self-revision share at +0.010 to +0.015 on one model. A generation-matched independent re-sample control does not exist in the data; it is the next experiment.
• Three homogeneous agents on multiple-choice letters, three rounds, no LLM judge. TUMIX has fifteen heterogeneous tool agents, up to six rounds and a judge. The three heterogeneous panels here (200 debates each) behave like the strong homogeneous ones. The judge rule could not be scored.
• Three rounds means a plateau can be detected at round 2 or 3 at best; the n_eff rule was given little room.
• Answers and gold labels are DebateLedger's parsed fields. Our plurality with their tie-break disagrees with their stored majority on 18 of about 27,700 round majorities; not investigated.
• Two post-hoc additions, the all-wrong/one-correct split and the self-only control, are labelled as such above and were not part of the pre-registration. Nothing pre-registered was removed or re-thresholded. The reformulation of the degenerate per-round hypothesis is in the committed pre-registration.
• The rising correlation is not an accuracy artefact. Accuracy also rises between rounds on most files, and a correlation of correctness can rise for that reason alone. Recomputed on 11 October 2026 with the accuracy-adjusted exact-match agreement of Goel et al. (arXiv:2502.04313; chance = both right by their own accuracies, wrong answers never agreeing by chance) the picture is unchanged: on the sixteen peer files the adjusted agreement rises from round 0 to round 3 by 0.13 to 0.43, on the two self-only controls by 0.00 and 0.04; Geirhos' error consistency moves with the ICC on every file; and where a pair is wrong at round 3, the two give the same wrong answer on 61–100% of those problems on the peer files against 58% and 67% on the controls. Pre-registered (debate-capa in the working repository), all four predictions held.

Code, pre-registration and every reported number: Alatip/tech-note-4. One numpy script plus two post-hoc scripts, no scipy, no API calls; the full run with a 2,000-resample bootstrap over sixteen files takes about fifteen seconds. Data: the open tier of XINLI1997/DebateLedger on Hugging Face (CC-BY 4.0, snapshot of 30 September 2026), fetched by data/fetch.sh. If TUMIX releases per-agent, per-round logs, the same three statistics apply with n = 15 and up to six rounds, and this note will be updated with the numbers on the system the bridge was actually about.

Archived with a DOI: 10.5281/zenodo.23133060. Code release v1.0: 10.5281/zenodo.23133114. Cite as: Latipov, A. (2026). Debate Is Not Sampling. Zenodo. https://doi.org/10.5281/zenodo.23133060

1. Effective Independence, 31 August 2026, and The Control Ate the Signal, 2 October 2026. The second left the question of whether the 1/ρ ceiling survives majority voting over discrete answers; this note's answer for refinement is that it does, with the panel reaching the ceiling in two or three rounds, while the separate selection ceiling of the sampling theory does not. ↩
2. Li, X., Liu, M., Yuen, C. (2026). Measuring Collapse and Correction in Homogeneous-Panel LLM Debate. NeurIPS 2026 Datasets & Benchmarks, arXiv:2609.35279; data at XINLI1997/DebateLedger. The theory: When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling, arXiv:2606.28661. The system: Chen et al. (2025), TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture, arXiv:2510.01279. The closest prior result on the correlation half is The Ringelmann Effect in Multi-Agent LLM Systems, arXiv:2606.02646, which reports pairwise agreement rising over debate rounds and an ICC-based effective team size on other data, with fixed rounds and no per-problem split; Section 4's H3a is a replication of it with a different estimator. ↩
3. The pre-registration (PREDICTION.md) and the prior-art check (ALREADY-DONE.md) are in the companion repository with the commit that precedes the first run. The bridge itself was proposed by a literature-based-discovery pipeline over a corpus of agent-swarm papers, judged 6/10 by an LLM judge before testing; that pipeline is the subject of a later note. ↩

← notes · github.com/Alatip