Where the Swarm Pays

Three earlier notes argued that a group of LLM agents is bounded by its effective number of independent voters, and that debate raises that correlation rather than escaping it. This note is the ledger. We built one corpus of answer traces so that most of the questions the agent-swarm literature asks could be put to the same data: seven reasoning models from five families (Claude Opus 5.5, Sonnet 5.5 and Haiku 4.5, DeepSeek V3.2, GLM 5.3 Flash, Qwen 3.7 Plus, Mistral Large 2512) each answer 240 problems from MMLU-Pro, GPQA Diamond, GSM-Hard and the 2024–25 AIME; three of them draw 8–16 answers each at temperature 1; six of them debate for three rounds with every turn stored; DeepSeek runs the same debate as six copies of itself, with and without seeing its peers, and answers again under four role prompts; five of them judge each other's candidates, first by picking one from a list and then by certifying each one alone; and one of them writes a final answer from the others' answers and reasoning excerpts. The first finding is about the benchmarks, not the models. Every problem in the current run files on which all seven models were scored wrong was re-solved by a model, eight of them by hand as well and the three GPQA Diamond items by hand only: of the 48 items the audit touched, 21 had a wrong gold label and were relabelled, 22 were ill-posed and excluded, 2 were failures of our own grader, and on 3 the models were wrong and the gold stands. After the audit the seven models fail together on 1 problem of the 733 on which all seven finished an answer. That denominator hides the hard items: two of the models hit an 8,192-token cap on a fifth to two fifths of GPQA Diamond and half of the olympiad problems, and when a cut-off answer is a missing vote rather than a reason to drop the item, every answering model is wrong on 20 problems of 987, which is 19 of the 229 problems on which some model was cut off, 8.3%, 18 of them not yet audited. The second finding is about counting. The first two versions of this note scored a cut-off answer as a wrong vote in the debate and judge stages; this version scores it as absent, keeps the old tables in a sub-section for comparison; the 13-point gain of every agent, the decorrelation story and the "spine" ranking of the earlier versions go with the change, while the majority's gain survives it and shrinks instead under the tie rule and the within-item control (Section 10). On the answer side no operator that pools different models beats the best single model except where that model was cut off: plurality over the seven models is 0.8–1.9 points below the best member on three layers and ties it on the fourth; the seven are 2.1–2.3 effective voters on the three larger layers. Resampling one model, judged against its mean draw since a best draw is not a control, gains nothing to two points on MMLU-Pro and GSM-Hard, four to seven on GPQA Diamond and fifteen on the olympiad problems DeepSeek finished sixteen times, the one place on the answer side where more calls clearly pay. Debate across families gains 4.9 points of majority accuracy over the round-0 vote when ties go to the first agent in the fixed order, 3.9 with ties split and 2.2 when the strongest agent breaks them; ten of its eleven fixes are on problems where an agent had been cut off before answering, though on only four of them does that agent join the winning majority, and seven are round-0 ties; on the 162 problems where all six agents answered it gains one item, and on the items the best agent of each layer answered it gains 2.0 points on GPQA Diamond and nothing or less elsewhere. It takes unanimity from 152 to 219 of 225 items and the panel from 2.2 effective voters to 1.2, and it ends unanimously wrong on 4 items where it started on 0, all four on problems the audit had not reached and all four carrying the marks of a label fault in the audit file. Six copies of one model gain 3.5 points, half of which come from re-reading one's own answer with no peers at all. Role prompts change the error correlation by no amount that 13–59 items can resolve. A synthesiser recovers 0 of 4 all-wrong problems, the four being the unaudited items whose labels look wrong. On the verifier side the picture is mixed. Asked to pick the correct candidate from the solvers' disagreeing answers on 74 contested problems, five judges whose picking errors correlate at φ = 0.16 when a cut-off call is a wrong pick and 0.29 when it is absent reach 0.838 against 0.757 for the solvers' plurality when ties go to the first agent in a fixed order, 11 items against 5, p = 0.21, nine of the eleven on problems where the solvers were tied; with ties split the lead is 3 items, and with the strongest solver and the strongest judge breaking ties the solvers lead by one. Where more than half of the judges agree, 54 problems, the panel is right on 0.944 against 0.796 for the solvers' plurality on those problems. Over the whole item set, plurality plus panel lands half a point below or a point above the best single solver when a cut-off answer scores 0, and, on the items the best solver answered, 0.6–2.6 points below it. For one model, DeepSeek, disagreement with five other models from four families predicts its error at AUROC 0.98; disagreement among fifteen of its own resamples, 0.93, and one strong peer alone, Sonnet, matches the fifteen. Asked instead to certify a single candidate, the same judges accept their own wrong answers 36–100% of the time, and a unanimous committee reaches a 0–2% false-acceptance rate only by rejecting 15–32% of the correct candidates, the range being how a judge's conflicting verdicts on two formats of one answer are read. The swarm is a warning light and, within a tie-break rule of nothing, a comparator, not a second brain; on these benchmarks a good part of what it appeared to add was fixes on problems an agent had not finished and a tie rule that favoured the group. Decomposition into subtasks, which is what industrial agent systems mostly do, is a different operator and is not measured here.

Nothing in the framing of this note is new; the measurements are. The premises come from four places, and it is worth naming them because the vocabulary of the LLM-swarm literature borrows from all four without always saying so.

Juries. Condorcet's jury theorem (1785) says that a majority of independent voters, each right with probability above one half, is right with probability that goes to one as the jury grows. Boland (1989) and Ladha (1992) worked out what happens when the votes are correlated: the majority's accuracy no longer goes to one but to a plateau set by the correlation, and above a threshold correlation adding voters adds nothing. The theorem is about a binary choice; its extension to a plurality over many candidate answers is List and Goodin (2001). Every "N agents beat one" claim is a Condorcet claim, and every vote ceiling in this note is a Boland–Ladha plateau; the coverage ceiling of Section 4 is a different bound.

Surveys. Kish (1965) gave the design effect for a cluster sample: n correlated observations with intraclass correlation ρ carry the information of n/[1+(n−1)ρ] independent ones, which is bounded by 1/ρ however large n grows. The first note on this site applied this to LLM resamples as the effective number of independent voters, n_eff. It is a formula about the variance of a mean; Section 8 shows what it does and does not say about a majority of binary votes.

Latent correlation. Two agents' correctness is a 2×2 table, and the Pearson correlation of two 0/1 variables, φ, cannot reach 1 unless their accuracies are equal. Pearson (1900) defined the tetrachoric correlation as the correlation of two latent normal variables that would produce the table, and gave the cos-π approximation to it; Fisher's intraclass correlation (1925), in the one-way analysis-of-variance form of Shrout and Fleiss (1979), is the panel version of φ on the observed scale, not a latent quantity; it is used here because the debate note used it. The one-factor Gaussian model used in Section 7 is Vasicek's (1987) single-factor model of correlated defaults, in which every borrower loads on one latent economic state; here every model would load on one latent item difficulty.

LLM ensembles. Self-consistency, the plurality over sampled chains of thought, is Wang et al. (2022, arXiv:2203.11171). Multi-agent debate in the form used here, agents that read each other's answers and revise, is Du et al. (2023, arXiv:2305.14325). Mixture-of-agents, a model that reads several answers and writes one, is Wang et al. (2024, arXiv:2406.04692). The LLM-as-a-judge paradigm and its self-preference bias are Zheng et al. (2023, arXiv:2306.05685) and Panickssery et al. (2024, arXiv:2404.13076). The 2025–26 papers whose claims this corpus was built to test are named in Section 17; three of them frame the whole note: the co-failure ceiling (arXiv:2606.27288), which bounds routing, voting and any scheme that returns one member's answer by the rate at which all members fail together; the modal and correlation ceilings of test-time scaling (arXiv:2606.28661), which the previous note tested on the public debate traces of arXiv:2609.35279 (the DebateLedger release); and Nine Judges, Two Effective Votes (arXiv:2605.29800), which found a panel of nine judges to be about two effective votes; Section 13 measures the same quantity for five judges on a different task and finds two to three, depending on how a cut-off judging call is scored.

The three earlier notes on this site¹ established, on public data, that LLM pools are one to two effective voters, that diversity metrics do not predict vote gain once accuracy is controlled, and that debate raises correlation while escaping unanimous error on weak models. This note puts all of that on one corpus with the same items for every operator. It also found, on the way, that the co-failure rate those premises are built around cannot be measured on these benchmarks without first auditing the benchmarks.

The hypotheses in the swarm literature can be sorted into eight questions; this is one such sorting, fixed as axes before the corpus was built, and it is a working taxonomy rather than a claim about the field. The note covers five of them.

A. Parallel answers and aggregation

Does pooling independent answers add intelligence? Sections 7–8.

B. Debate

Is a multi-round exchange a different operator from sampling, and what does it transmit? Sections 10–11, with synthesis in Section 12.

C. Selection and verification

Can a verifier reach the ceiling that voting cannot, and how independent is it from the solvers? Sections 13–15.

D. Sources of diversity

Can independence be bought with families, roles, prompts or temperature? Sections 7–9.

G. Cost at equal budget

What does each operator cost in tokens, and when does it pay against one model given the same budget? The token side is in Section 16; the cost-matched comparison is the next note.

Three axes are not in this note. E, iterated chains: swarms in time, self-training and iterated in-context learning, and whether they converge to the prior. F, organisation with tools: agents that decompose a task, take roles, keep memory and call code or search, which is what industrial agent systems do; the corpus has no decomposition and no tools. H, theory as bound: which of the classical results above is tight on which data; the Kish check in Section 8 is a first instalment, not a treatment.

A literature-based-discovery pipeline over about a thousand agent-swarm papers paired one paper's stated limitation with another paper's result wherever the two did not cite each other, scored each pair with an LLM judge, and produced 947 candidate hypotheses with a score of 4 or more out of 10, 140 of them at 6 or more. A handful were tested the usual way, one bridge at a time, two of them in the previous notes. Reading the rest made the economics obvious: most ask for one of five or six measurements on one kind of data, a correctness matrix across model families, resamples of one model, multi-round debate traces with every intermediate answer kept, a judge panel, and the token count of each call. A bridge test that costs ten dollars is not worth running; a corpus that answers forty bridges for fifty dollars is.

So the corpus was designed from the hypotheses, built in two days, and each of the 140 was then read against its tables by a model instructed to say "supported", "not supported", "mixed" or "the corpus does not measure this". After three passes, each adding the cuts the previous pass had asked for, 27 hypotheses had an answer and 113 asked for things the corpus does not contain. Section 17 re-reads sixteen of the 27 against the audited tables. The rest of this note is the measurements themselves, with the definitions first because half the disagreements in this literature are about what the words mean.

Item, layer, correct

An item is one problem with one gold answer. A layer is a source dataset. A model's answer is correct or not. A truncated answer (the model hit its completion cap, 8,192 tokens for the OpenRouter models, before writing a final line; eight Claude rows with no final line, five "unable to determine" on MMLU-Pro and three empty on GSM-Hard, are coded the same way although the Claude models had no cap) is missing: it is not a vote, not a dissenter, not a row in a correlation, and an item is not dropped because of it unless the table says so. Where a single agent is the control, its truncated answer scores 0, because a model that does not answer has not answered. The first two versions of this note counted truncated rows as wrong in the debate and judge stages; Section 10 keeps those tables in a sub-section.

Answer identity

Two answers are the same answer when they are equal after normalisation: numbers as numbers to seven significant digits, the grader's own tolerance (24, 24.00, $24, 5,072,530 and 5072529.6 are one answer each), LaTeX fractions and roots in one form. Every plurality, unanimity and candidate count in this note uses this identity. The first version compared raw strings; the second rounded to six decimals, which still split a handful of large GSM-Hard answers by formatting and so counted six items as contested that were not.

Best single

The highest accuracy among the agents on the same items, a truncated answer scoring 0. The standard control in the ensemble literature. It is the floor of the box: any aggregation that lands below it has cost accuracy. Where it matters, the note also gives the best agent's accuracy on the items that agent answered, with the group's majority taken on the same items, which is the fairer control for a capped model; the earlier versions set that accuracy beside the majority on all items, which is not a comparison. Both are in-sample maxima over six or seven agents and flatter the single model by a point or so on the small layers.

Plurality

The answer given by the largest number of agents, ties broken by first occurrence in a fixed agent order (for the seven-model matrix, the order of the data files; Section 5), as in self-consistency; Sections 10 and 13 show what the two results that depend on ties look like when ties are split or go to the strongest agent. "Majority" is used for the same thing when the agents are a debate panel. Plurality is computed on answer identity, so two agents who are both wrong in different ways do not vote together.

Ceiling

The share of items on which at least one agent is correct, 1 − P(all wrong); the co-failure ceiling 1 − β of arXiv:2606.27288, the "oracle" or "coverage" of the test-time-scaling literature. No scheme that chooses among the agents' answers can exceed it; a scheme that writes a new answer can. The box between best single and ceiling is where every answer-side result in this note lives. After the label audit the ceiling is 0.998–1.000 on the items all seven models finished and 0.97–0.99 when a truncated answer is a missing vote (Section 7).

Unanimous, unanimous-and-wrong

All agents gave the same answer; that answer is wrong. A unanimously wrong item is invisible to any rule that uses agreement as a confidence signal. Before the audit, every unanimously wrong matrix item turned out to have a wrong or ambiguous gold label.

Fixed, broken

In a debate, an item whose plurality was wrong at round 0 and correct at the last round, and the reverse; arXiv:2609.35279 calls them corrections and collapses.

φ

The Pearson correlation between two agents' 0/1 correctness vectors over items, averaged over the pairs on which it is defined (the phi coefficient, Yule 1912). Each pair is computed on the items both agents answered. It is exact but bounded away from 1 when the two accuracies differ, undefined when either agent is constant, and it understates the correlation of the latent errors.

Tetrachoric ρ

The correlation of two latent normal variables that would produce the observed 2×2 table of correct/incorrect. Pearson (1900); we use his cos-π approximation, which is poor when the margins are skewed and degenerates to ±1 when a cell of the table is empty. Pairs with an empty cell are dropped from the average, and the number of pairs that survive is printed next to it; after the audit few survive on the small layers.

n_eff

Kish's (1965) effective sample size, n/[1+(n−1)φ], as used in Effective Independence. A design-effect formula from survey sampling, used here as a descriptive summary of how much independence n agents bought; Section 8 compares the majority it predicts with the majority observed.

ICC

Fisher's intraclass correlation in the one-way ANOVA form ICC(1) of Shrout and Fleiss (1979), of correctness within an item, the share of the variance of "this agent was correct on this item" that is explained by the item. An observed-scale quantity, close to φ on the debate panel and below it on the matrix; reported because the debate note used it.

Contested item

Introduced here. For the judge panel: the candidates proposed by the agents that answered disagree by answer identity, and at least one of them is correct and one wrong. A cut-off proposer's partial text is not a candidate, although the judges saw one as a candidate on about ten items. On a contested item a perfect selector scores 1 and the plurality scores less; it is the only kind of item on which a verifier can earn anything.

Pick vs certify

Introduced here as names; both tasks exist in the LLM-as-a-judge literature as pairwise/listwise and pointwise judging. Pick: see all distinct candidates with a reasoning excerpt each, choose one or say "none". Certify: see one candidate alone, say correct or incorrect.

AUROC

Area under the receiver operating characteristic curve: the probability that a randomly chosen error receives a higher warning score than a randomly chosen correct answer; 0.5 is no information, 1 is perfect.

Design. The unit of analysis is an item answered by several agents, and every stage of the corpus is a different way of producing several answers to the same item: by different models, by resampling one model, by rounds of debate, by role prompts, by judging. The stages were chosen from the list of measurements the 140 hypotheses asked for, in descending order of how many hypotheses each would serve, and each was given a spend cap before it ran. Nothing was added after seeing a result except the cuts and the three later stages described below.

Controls. For every operator the comparison is to the best single agent on the same items and to the ceiling on the same items; both are computed from the same rows, never from a separate run. Debate has two controls built in: the round-0 plurality, which is what voting would have done, and the self-only arm, in which copies revise without seeing peers, which is what re-elicitation alone does. Judges are compared to the solvers' plurality on the same contested items, to each other, and, over the whole item set, to the best single solver. Comparisons are within-item: wherever an agent's accuracy on the items it answered is the control, the majority is taken on the same items (the earlier versions set it beside the majority on all items). Two comparisons still cross item sets and say so: copies against families in Section 11, and each model's matrix accuracy against its layer-wide value in Section 7.

Statistics. Accuracies are plain proportions. φ is the mean pairwise Pearson correlation of 0/1 correctness; the tetrachoric estimate is the cos-π approximation; ICC is the one-way ANOVA estimator; n_eff is Kish's formula applied to φ. The Kish-predicted majority is the probability that a majority of n_eff independent votes at the mean accuracy is correct: the exact binomial majority, ties split, at the two integers around n_eff, interpolated linearly. The first version of this note rounded n_eff to an integer, which made the prediction p² whenever it rounded to 2; the second used a normal approximation, which at n < 3 is off by up to four points in a direction set by p, not by φ, and produced a pattern Section 8 of that version mistook for a finding. Intervals, where given, are Wilson 95% intervals on proportions, or percentile bootstrap intervals over items (2,000 resamples, seed 0) for differences; paired comparisons are exact McNemar tests on discordant items, uncorrected for the four layers. Every plurality over the six debate agents breaks ties by the first agent in one fixed order (DeepSeek, GLM, Mistral, Qwen, Haiku, Sonnet), which puts GLM, the agent with the lowest round-0 accuracy once a cut-off answer scores 0, second and Sonnet, the highest, last; the seven-model plurality of Section 7 uses the order of the data files (Haiku, Opus, Sonnet, DeepSeek, Mistral, Qwen, GLM), which the earlier versions did not say, and Section 7 gives its range over orders. The debate gain and the judge-panel lead are the two results that depend most on it, and Sections 10 and 13 give both with ties split and with the strongest agent first; the third version broke the solvers' ties in Section 13 by the order in which the candidates had been shown to the judges, which is a per-item shuffle, and Section 13 says what that changed. The strata of interest are small enough that the reader can see the counts, and they are reported as counts. Spearman correlations over model triples are over 35 triples per layer (30 on the olympiad, where φ is undefined on some), which share models and are not independent.

Labels. After the first version of this note was written, the items on which the models had all been scored wrong were re-solved by Claude Opus 5.5 with explicit arithmetic, eight of them re-solved by hand as a check, the three GPQA Diamond items by hand only. The audit file holds 48 items. Forty-one of them are the items on which all seven models had finished an answer and been scored wrong; six more are items on which every debate agent that finished was wrong while one or two agents had been cut off; one, a GSM-Hard item with a negative count of toys, was added on inspection although four models had matched its gold. The decisions: 21 gold labels were wrong and were relabelled with the demonstrably correct answer (9 MMLU-Pro, 11 GSM-Hard, 1 GPQA Diamond); 22 items were ill-posed or had two defensible options and are excluded from every table (15 MMLU-Pro, 6 GSM-Hard, 1 GPQA Diamond); on 3 the models were wrong and the gold stands; and 2 olympiad items had been scored wrong because the matrix stage had graded expressions with the numeric grader (1/576 against \frac{1}{576}). Every row of every stage was re-graded with the normalising grader, which changed 41 further matrix rows on 13 other HMMT items (an earlier draft of this version said 61). The first revision reported these as 45 items in a 24/16/2/2 split; that arithmetic did not match the file and is withdrawn. An independent re-check of the file for the third version agreed with all 11 GSM-Hard relabels and 20 of the 21 overall, and would have excluded rather than relabelled one MMLU-Pro item whose two options say the same thing; the earlier versions also quoted a range of 36–42 items for that re-check in the Limitations, which is not in the audit file and is withdrawn. The relabels are not of one kind: the GSM-Hard ones are arithmetic that a reader can redo (a program substituted large numbers into some quantities of a word problem and not others) and the two olympiad ones are grading; the nine MMLU-Pro relabels and the fifteen MMLU-Pro exclusions are judgments about knowledge questions, made by a model, and Section 7 gives the ceiling with them reverted. The audit, with the reason for each decision, is label-audit.json in the companion repository, and the re-grading script makes no model calls. Three things the audit does not cover. Items on which some model matched a wrong gold label were not read, so individual accuracies may still carry label noise in both directions. Items on which every answering model was wrong while another model had been cut off by its token cap were not selected, because the selection rule asked for seven finished answers; there are 18 such items after the audit, plus one all-seven-wrong MMLU-Pro item that sits in an older run file outside the tables, and they are listed in the audit file as unaudited. And the four items on which the six-agent debate is unanimously wrong at round 3 meet the stated rule but were not read either; the audit file's own note says that two of them are GSM-Hard items whose gold has the hallmarks of partial substitution and the other two MMLU-Pro items with two defensible options. All of these stay unrelabelled in this version and the note says so where they matter. The auditor is one of the audited models, and the new gold agrees with Opus's own first answer on every relabel; a reader who distrusts that should start from the file.

Truncation. The four OpenRouter models ran under an 8,192-token completion cap; the Claude models, through the CLI, had none. GLM and Qwen hit the cap on 68 and 19 of 566 MMLU-Pro items, 76 and 44 of 197 GPQA Diamond, 8 and 8 of 144 GSM-Hard and 42 and 50 of 90 olympiad items at temperature 0, and on 86 of the 1,350 round-0 rows of the debate (GLM 50, Qwen 36); DeepSeek was cut off on 2 matrix rows, neither among the debate items. Mistral was never cut off but has no matrix row at all on 32 GPQA Diamond, 6 olympiad and 2 MMLU-Pro items, for reasons the run logs do not record; 18 of the GPQA Diamond and 4 of the olympiad items missing from the seven-finished tables are missing for that reason, not for a cap, and the earlier versions did not say so. The matrix tables of Section 7 use the items every model finished, which drops the hard items and is said so there. In every other stage a truncated row is absent: it does not vote, it is not a dissenter, it is not in φ, ICC or the unanimity count, and an agent's accuracy is over the rows it answered; where a single agent is the control, its truncated answer scores 0. The first two versions of this note treated a truncated row as a wrong answer in the debate and judge stages, which depressed round-0 accuracy and correlation on exactly the layers where the capped models truncate, and produced the debate gains those versions reported; Section 10 keeps the old tables in a sub-section so that the two policies can be read side by side. Items are included in a stage's table when every agent of that stage has a row for it, truncated or not, which is why item counts differ between tables and are printed in each.

Exclusions. The 22 audited items are dropped everywhere. Five of the first-pass layers (MATH-500, GSM8K, CommonsenseQA, StrategyQA, Knights & Knaves) were dropped before any aggregation was computed. The first two versions said the criterion was that every model exceeded 0.95 on them; the matrix file does not bear that out (CommonsenseQA is 0.79–0.93 for the seven, Mistral is 0.66 on Knights & Knaves), and the honest description is that they were dropped as knowledge or near-saturated layers by judgment, which is a researcher degree of freedom. They remain in the matrix file. Nine of the 240 debates were never run because Mistral had no matrix answer to seed them with, and one debate lost Mistral on a rate limit; the six-agent tables use the 225 items with a row for every agent in every round, and a five-agent recount without Mistral is in the companion tables.

Post-hoc cuts. The first reading of the hypotheses against the tables asked for cuts the first tables did not have: the all-wrong stratum of debate, the transition table by round-0 correct count, the flip table, the copula and two-state fits, the judge committee thresholds, the error-detection AUROC. These were computed from the existing rows, with no new calls. The three homogeneous-debate arms, the certify stage and the synthesis stage were new runs made after the first reading, each with its own cap; they are the only data collected after a result had been seen. The label audit, the tie analysis of the judge panel and the whole-set accounting of Section 13 were added in the second version, after an external reading of the first, with no new calls. The third version, after a second external reading, changed the truncation policy, merged candidate answers within the grader's tolerance, replaced the Kish estimator, re-counted the audit, reported judge coverage, and replaced the reversible GPQA answer hashes in the companion data; again no new calls, and the old debate and judge tables are kept for comparison. This fourth version, after a third reading that compared the text with the code, corrected the places where the two disagreed: the solvers' tie rule in Section 13, a cut-off agent counted as having abandoned its answer in the flip table of Section 10, letters outside the candidate list counted as named picks, a planted decoy that the audit had made correct, the role correlation computed from one draw per role, the triple gains computed on a majority of correct votes rather than a plurality, and a sample table that silently dropped the items a model had truncated; it also added the audit sensitivity of Section 7, moved every number the note quotes into the companion script, and replaced the GPQA tokens again, per item this time. No new calls. A fourth reading, folded into this version before it was deposited, found that the 'best agent on its answered items' control had been set beside the majority on a different item set, that the fixed tie order favoured the group in both results that depend on it, that Mistral's missing matrix rows had gone unmentioned, and that the four unanimously wrong debate items are flagged as probable label faults in the audit file; the tables of this version carry those corrections, again with no new calls. A fifth reading, also folded in before deposit, recomputed the counts the text quotes from the rows and found that the 254 "cut-off" items were 229 plus 25 that Mistral never answered, that the seven-model plurality of Section 7 broke ties by the order of the data files rather than the stated one, that the cut-off agent itself joins the winning majority on four of the ten fixes credited to it, that nine of the panel's eleven wins are solver ties, that the synthesiser's four all-wrong items are the four probable label faults, and that the debate panel spans five families, not four; the text carries those corrections, again with no new calls. A sixth reading, a blind recount of every number the text quotes from the stored rows with independently written code, reproduced the tables and found three misdescriptions rather than wrong numbers: the cost table of Section 16 selects items by whether every row carries a token count, not by whether every operator finished, so its item counts and the "45 GPQA items" were described wrongly; the certify stage's "82% of the calls" is the share of candidate rows the OpenRouter judges were called on against the Claude judges, left over from the second version's wording; and the earlier versions' triple gains were quoted as a range that dropped the GPQA value. It also found rules the text had used without stating them, now stated: the role tie order of Section 9, which row of Haiku's duplicated olympiad draws is counted in Section 8, what counts as a named pick and which judge breaks ties in Section 13, and which decoys the certify table counts in Section 15. No new calls. A seventh reading, against the collection code and the rows, found that the certify excerpt is the first proposer's rather than another proposer's, that the cost table also requires the single answer and the draws to be untruncated, that the abstract had credited the truncation policy with what the tie rules and the within-item control had done, and one sentence in Section 17 that contradicted Section 11; it added the plurality test of Section 7 with the relabels reverted, the debate's fixes and breaks under the strongest-first rule, Opus as an outside control on the debate items, and the rule behind the last column of the judge table. No new calls. The fifth version is a readability edit of the fourth: the abstract was rewritten to state the results, the old abstract became the Introduction, and the byline was shortened; no number, table or claim changed, so it was not deposited or frozen.

Reading the hypotheses. The 140 hypotheses were read against the first-version tables by Claude Opus 5.5 at low effort, three times, with instructions to quote only numbers present in the tables and to return "insufficient" whenever the quantity was not measured. The verdicts in Section 17 are the author's re-reading of those sixteen hypotheses against the audited tables.

Items. 240 problems with a single checkable answer, sixty from each of four layers: MMLU-Pro (ten-option multiple choice, stratified over subjects), GPQA Diamond (four-option graduate science), GSM-Hard (GSM8K word problems with large numbers substituted by a program; Section 5 says what that did to its gold labels), and the 2024 and 2025 AIME (integer answers), with the February 2025 HMMT (short expressions) in the matrix only; all public before the models' training cutoffs, so contamination is possible and unmeasured. A first pass over 1,576 items had included MATH-500, GSM8K, CommonsenseQA, StrategyQA and Knights & Knaves puzzles; they were dropped from every table (Section 5 says on what grounds) and remain in the matrix file. The seven-family matrix covers 566 MMLU-Pro items (573 for Opus, whose current run file holds seven items the others answered in an older one), 197 GPQA Diamond, 144 GSM-Hard and 90 olympiad after the exclusions; the debate, sampling and judging stages use the sixty per layer, and their sixty olympiad items are the AIME ones.

Models. Seven models from five families (three of them Anthropic's), all current reasoning models in October 2026 and all allowed to think before answering, though not equally: Claude Opus 5.5, Sonnet 5.5 and Haiku 4.5 through the Claude Code CLI in print mode with tools and settings disabled and the system prompt replaced, where temperature is not controllable, the effort setting was low and there is no completion cap, so that Opus and Sonnet averaged 54 and 19 reasoning tokens per MMLU-Pro answer while Haiku averaged 2,200 and ran to 25,000 on the olympiad; and DeepSeek V3.2, GLM 5.3 Flash, Qwen 3.7 Plus and Mistral Large 2512 (GLM 5.3 and Qwen 3.7 in the tables) through OpenRouter under an 8,192-token completion cap, with the serving provider logged per call and no separate reasoning-token count returned. GLM hit the cap on 39% of GPQA Diamond and 47% of the olympiad problems, Qwen on 22% and 56%, so that, with Mistral's missing rows, only 91 of the 197 GPQA Diamond items and 29 of the 90 olympiad items have seven finished answers (Section 5). Four further models answered subsets of the matrix and are not used here. Opus is in the matrix but not in the debate or judge panels; the panel was fixed at six agents when the stage was configured, and the design notes do not record why Opus was the one left out, so the reader should treat its absence as unexplained rather than principled. Its matrix answers exist for every debate item, so it serves as an outside control: on the 225 six-agent items Opus alone, a cut-off answer scoring 0, is at 0.956 against the round-3 majority's 0.973, 7 items against 3 (p = 0.34), behind the majority on GSM-Hard (0.912 against 0.965) and GPQA Diamond (0.964 against 0.982) and level with it on MMLU-Pro and the olympiad.

Grading. Letters by exact match; numbers by numeric comparison with a relative tolerance of 10⁻⁶; expressions by a normaliser (fractions, roots, dollar signs, trailing approximations) plus an LLM arbiter (Opus, decisions cached) where the normaliser and the gold string still disagreed. Every answer was extracted from a required last line, FINAL: <answer>. The first version of this note had graded the matrix stage with the numeric grader alone, which is the grading failure described in Section 5.

Every row stores the parsed answer, its correctness, the provider that served the call and the token counts (prompt, completion, reasoning). The debate stages also store the full text of every turn and what each agent was shown. The released rows (companion repository) carry no texts: GPQA Diamond answers are per-item random tokens, and a cut-off answer that was a fragment of reasoning is a digest. Nine debates were never run for lack of a Mistral seed answer and one lost Mistral mid-way; after the label exclusions the six-agent tables use 225 items and a five-agent recount on 226 is in the companion repository. The OpenRouter calls cost about $50; the Claude calls went through a subscription and would have cost about $335 at list price, which the byline of the first two versions left out. The budget-forcing rows were collected for the cost-matched comparison of the next note and are not used here.

The oldest question first: does the plurality of seven different models beat the best of them? On the items all seven finished, on no layer does it. On MMLU-Pro, the one layer large enough to test the margin, the best model is right and the plurality wrong on 13 items and the reverse on 4, an exact McNemar p of 0.05 before any correction for four layers (five MMLU-Pro items and one GPQA Diamond item are seven-way ties, broken here by the order of the data files; under the six-agent order of Section 5 with Opus last the MMLU-Pro plurality is 0.953, 14 against 3, p = 0.01, with Opus first 0.961, 10 against 3, p = 0.09, and with ties split 0.958, and GPQA Diamond runs 0.967–0.978); on the other three layers the plurality is behind by one item or tied, which is within what the in-sample choice of the best model can produce.

Items are those all seven finished without truncation, after the 22 exclusions, which is why these layers are larger than the sixty-item debate subsets and the olympiad is smaller: GLM and Qwen hit their token cap on 42 and 50 of the 90 olympiad items and on 76 and 44 of the 197 GPQA Diamond items. The selection is not neutral. Each model's accuracy on the items it answered is lower than on this subset, by 0.6–5.4 points on MMLU-Pro and by a lot where the cap bites: Haiku is 0.69 on the GPQA Diamond items it answered against 0.84 here, DeepSeek 0.61 on the olympiad against 0.93, Mistral 0.46 against 0.83. The per-model columns are accuracies on the problems two capped models could finish, not on the layer. They are 1–12 points higher than in the first version of this note, about one point on GPQA Diamond and nine to eleven on GSM-Hard, and that difference is the audit and the grader: the gold was wrong, not the models.

The last column was the surprise of the second version, and it needs the same caveat. On the items all seven finished, the share on which someone was right is 0.998 on MMLU-Pro and 1.000 on the other three layers: of 733 items, the seven models all fail on one. The first version reported all-wrong rates of 1–10% and fitted a one-factor latent model to them; those rates were made of wrong gold labels, ill-posed questions and a grader that could not read a fraction. But the items the capped models could not finish are the hard ones. Count a cut-off answer as a missing vote and keep every item that at least five models answered, and every answering model is wrong on 12 of 566 MMLU-Pro items, 2 of 191 GPQA Diamond, 4 of 143 GSM-Hard and 2 of 87 olympiad: 20 of 987, 2.0% pooled over easy and hard items, and 19 of the 229 items on which some model was cut off, 8.3%, sixty times the headline rate on the finished items (19 of the 254 items with fewer than seven answers, 7.5%, if the 25 items Mistral never answered are counted in). Eighteen of the twenty are outside the audit, because the audit selected on seven finished answers, and one further MMLU-Pro item on which all seven finished and failed sits in an older run file outside these tables and was not read either. If those behave like the 48 the audit did read, most of them are the benchmark's fault; nothing here says so yet. The honest statement is narrower than the second version's: on the problems these models finish, joint failure after the audit is one item in 733; on the problems that make a model run out of tokens it is about 8%, unaudited; and the question the co-failure literature asks cannot be answered on public benchmarks of this kind without auditing the benchmark first and without a token budget the hard items fit in. The audit itself has a soft part. Its GSM-Hard relabels are arithmetic and its two olympiad corrections are grading, but the nine MMLU-Pro relabels and the fifteen MMLU-Pro exclusions are a model's judgments about knowledge questions, and the new gold on every relabel is the answer the auditing model had given as a solver. A reader who rejects those judgments gets a different ceiling: with the MMLU-Pro relabels reverted to the published gold, the seven models fail together on 10 of the 485 MMLU-Pro items, 2.1%; with the excluded items restored as well, on 24 of 499, 4.8%, and on 5 of 133 GSM-Hard and 1 of 92 GPQA Diamond items. The plurality comparison above survives the same reversal: with the MMLU-Pro relabels reverted Opus is still the best member, at 0.957 against Qwen's 0.940, the plurality is 0.940, and the discordant items are 12 against 4, p = 0.08. The one-in-733 figure is the audit taken whole. What survives either way is the correlation of the errors that remain, which is far from independence:

Seven models from five families are 2.1–2.3 effective voters on the three larger layers (3.2 on the 29 olympiad items, where five models are at or near 1.0 and φ is defined on ten of the 21 pairs). The tetrachoric ρ is the mean over the pairs on which the cos-π estimate is defined; on the olympiad one pair survives and it is not shown. The one-factor copula² at that ρ over-predicts the all-wrong rate by a factor of five to infinity, which is what one expects when the estimate of ρ is driven by the few remaining shared errors; it is reported for continuity with the first version, not as a fit. Family still does not buy a vote anything: among the 35 triples of models the plurality's gain over its best member is negative on average on every layer (−1.4, −3.5, −0.1 and −0.8 points), and on the two multiple-choice layers it is positively correlated with the triple's mean φ (Spearman +0.49 and +0.45, tie-averaged ranks; on GSM-Hard it is −0.05 and on the 30 olympiad triples −0.57, under the file order; under the other orders the two multiple-choice correlations stay positive, +0.26 to +0.59 and +0.40 to +0.45, and the GSM-Hard one runs from −0.05 to +0.23; the earlier versions computed this on a majority of correct votes rather than a plurality, which gave −1.9, −3.7, −0.4 and −1.3 points and +0.41 on GPQA Diamond). That is the wrong sign for any story in which diversity is what a vote buys, but it is also the sign that φ's mechanical dependence on the spread of accuracies within a triple would produce, which is the confound the previous note on this site was about; the triples share models, and this corpus does not separate the two readings.

Self-consistency, the plurality over k draws of one model at temperature 1, is the within-model version of the same question, and it has the same shape with one exception.

Items are those with k complete, untruncated draws, the first k by index, which is a selection the note's own policy does not make elsewhere: on all sixty olympiad items, with a cut-off draw a missing vote, DeepSeek's plurality of sixteen is 0.833 and its pass@16 0.883 (on GPQA Diamond 0.850 and 0.917), so the fifteen-point gain below is on the 49 items the model could finish sixteen times, and the earlier versions did not say so. Plurality uses answer identity and breaks ties by draw order, which matters at k = 8 and on one MMLU-Pro item at k = 16. GLM is omitted: on GPQA and the olympiad it truncated most of its draws, and the 26 and 19 items that survive are the easy ones, so its rows would overstate it. Haiku's olympiad file holds two draws under the same index on 415 of its item–index pairs, 61 of them disagreeing; the row counted is the later one in the file for each of the indices 0 to 7, and the two items on which those eight are not all present are dropped, which is why the row has 58 items; the second version of this note averaged both draws into its mean draw. The Kish prediction is the exact majority of n_eff independent votes at the mean-draw accuracy, which for n_eff ≤ 2 is the mean-draw accuracy itself.

Draws of one model correlate with each other at φ = 0.45–0.75 and are worth 1.3–2.1 independent draws. Over the mean single draw, the plurality gains nothing to two points on MMLU-Pro and GSM-Hard, four to seven on GPQA and fifteen on the olympiad for DeepSeek; the exception is real and it is the layers where φ is lowest. The "0 of k" column, the share of problems the model never solves at this temperature, is 0–9%, and it bounds pass@k and plurality alike. A two-state fit, in which each item is either unsolvable or solvable at one fixed rate, over-predicts the plurality by 3–9 points on each of the six sixteen-draw rows: items are not binary, and the middle of the solve-rate distribution is where the plurality is lost.

The last column is the third estimator this note has tried for the same quantity, and the lesson is about the estimator. Kish's n_eff was used in the earlier notes as a ceiling on what a vote could achieve; the first version of this note called it a floor, on an estimator that rounded n_eff to 2 and so predicted p²; the second version used a normal approximation and reported that the Kish majority over-predicted the plurality where φ was high and under-predicted it where φ was low. That pattern was the approximation. At n below 3 the normal majority is off by up to four points in a direction set by p, not by φ, and the high-φ rows are also the high-p rows. With the exact binomial, a majority of 1.3–2.1 independent votes is the single-draw accuracy for n_eff ≤ 2 and barely more above it, so Kish predicts that resampling buys nothing, and the observed plurality is above that prediction on six of eight rows, by one to fourteen points, most where φ is lowest. A design-effect formula built for the variance of a survey mean does not predict a plurality of correlated binary votes at these n_eff; it is a descriptive summary of how much independence the draws bought, and the two hypotheses in the literature that bet on its direction get no verdict from it (Section 17).

If temperature cannot buy independence, perhaps a prompt can: the "many minds" literature gives each copy a persona and reports gains. DeepSeek answered the same items four times under each of four one-sentence roles, a skeptical reviewer, a formal mathematician, a patient teacher and a pragmatic engineer, and the question is whether four roles are less correlated with each other than four plain draws.

The role correlation is the mean over pairs of different roles and over every pair of their four draws; the plain correlation uses four draws of the same prompt; the pluralities average over the four draw indices, a tie among the four roles going to the first in the alphabetical order of their files (engineer, formal, skeptic, teacher) and a tie among plain draws to the earliest draw. GPQA is absent because the role runs hit their spend cap before reaching it, and the olympiad row is thirteen items. Four roles are 1.2–1.5 independent answers, against 1.2–1.6 for four plain draws, and cost one to four points of single-draw accuracy. The difference in φ, +0.17, −0.03 and +0.07, has a bootstrap interval over items that includes zero on every layer ([−0.39, +0.31], [−0.22, +0.16], [−0.08, +0.23]; on MMLU-Pro and the olympiad a column with no errors makes φ undefined in about 2% of the resamples, which are dropped), so what the table says is that roles bought no measurable independence, not that they cost some; the roles' plurality is above the plain one on MMLU-Pro, below it on GSM-Hard and level on the olympiad. The second and third versions computed the role correlation from one draw per role, which gave +0.06, +0.22 and +0.07 and an interval of [−0.82, +0.24] on MMLU-Pro that was Monte Carlo noise from dropped resamples; the +0.22 on GSM-Hard was that one draw. On what 13–59 items can see, a persona is a temperature setting with a hat on. This is consistent with, and no stronger than, a one-model result from an earlier bridge test on this site.

The previous note found on public traces of three copies of one model that debate is not sampling: it moves the answers between rounds, and weak models escape unanimous error at rates a sampling theory puts at zero. Here the agents are strong, the families are mixed, and the texts are ours. One fact about the texts governs everything below. Six agents read each other's round-0 answers; two of them, GLM and Qwen, had been cut off by their token cap before writing one on 86 of the 1,350 round-0 rows, and what their peers saw in those cases was an empty answer. This version counts such a row as absent. The sub-section at the end of this section shows the same tables with it counted as wrong, which is what the first two versions of this note did without noticing.

225 items with a row for all six agents in every round, after the exclusions. Unanimity is among the agents that answered; φ is over the items each pair answered; ICC is over the items with six answers in that round (162, 185, 203 and 207). The five-agent recount without Mistral on 226 items gives .938 → .973, 9 fixed, 1 broken. "Fixed" and "broken" are cumulative against round 0.

Taken alone the first three columns are a success: the majority gains 4.9 points, the mean agent's accuracy gains 9 on the rows it answers, eleven items are fixed and none is broken, and the gain plateaus at round 2. The remaining columns are the price and the mechanism. Unanimity rises from 152 items to 219; the panel goes from 2.2 effective voters to 1.2 within two rounds; the cut-off rows fall from 86 to 21, which is consistent with an agent that has read a peer's answer writing its own without running out of tokens, though rounds 1–3 also run at temperature 1 under a different prompt and this corpus has no arm that separates the two; and the two columns that were empty or nearly so at round 0 fill up. Before anyone spoke there was no item on which the answering agents were unanimously wrong, and one on which all of them were wrong. After three rounds there are four of each. On all four the wrong answer already held the round-0 plurality and one or two agents held the right one; debate talked the minority round. None of the four was in the audit (Section 5); the audit file's own note says that two are GSM-Hard items whose gold has the look of a program's partial substitution and the other two MMLU-Pro items with two defensible options, so "unanimously wrong" here means unanimous against four labels that look wrong and have not been re-read. If the four behave like the all-wrong items the audit did read, debate was manufacturing agreement with the truth against the benchmark; until they are audited the count can be read either way, as four wrong consensuses or as four wrong labels.

Where the eleven fixes come from is the question the stored rows answer. Split the 225 items by how many of the agents that answered at round 0 were right:

The one all-wrong item is the olympiad problem on which the four agents that answered were wrong and two were cut off; three agents converge on the right answer, which none of them had written. Five of the nine items on which exactly one answering agent was right end correct, twelve of the fourteen with two, and all twenty-one with three. But the strata are not what they look like: of the 24 items in the zero-, one- and two-correct strata, 21 had at least one agent cut off at round 0, so "one of six" is mostly "one of four". Ten of the eleven fixes are on items with a cut-off round-0 row. On the 162 items on which all six agents answered at round 0, the majority goes from 0.975 to 0.981: one fix. Seven of the eleven fixes are items on which the right answer was tied for the round-0 plurality and lost the tie-break to an earlier agent, and the fixed order puts the two agents weakest on the hard layers first. With ties split, the round-0 vote is 0.934 and the debate gain is 3.9 points; with ties going to the strongest agent by round-0 accuracy (Sonnet, then Haiku), it is 0.951 and the gain is 2.2, and on MMLU-Pro and GSM-Hard the round-3 majority then equals the round-0 vote (0.948 and 0.965). Under that rule the debate fixes 5 items and breaks none, against 11 and none under the fixed order (exact McNemar p = 0.06 and 0.001). The headline 4.9 is the tie rule least favourable to the vote; the companion tables give all three. A difficulty split says the same from the other side: using DeepSeek's own solve rate over its sixteen draws as the item's difficulty, eight of the eleven fixes are on problems DeepSeek solves less than a quarter of the time, two on problems between a quarter and a half, one above, and none above three quarters, where the round-0 majority was already right on 168 of 169. Debate is a procedure by which a correct minority wins, and, on this corpus, one whose fixes sit on the problems an agent could not finish: of the ten such fixes, the formerly cut-off agent joins the winning majority on four, and on the other six it is still cut off or wrong at round 3, so the fix came from the agents that had answered. The wrong round-0 majorities are also where the cut-offs are, 13 items against 4 with six answers, so the 162-item figure is one fix of four possible against ten of thirteen. On these items debate did little else.

The texts also say how the price is paid. For every agent that was correct at some round, count the answering peers who held a different answer and look at the next round:

Against one dissenter a correct agent is safe; against three or more it folds in 15 of 35 cases, and the cases are few: 24 with three answering dissenters, 5 with four, 6 with five. An agent that was cut off in the next round is not counted either way; the third version counted it as having folded, which put the three-dissenter rate at 0.39. The second version of this note ranked the families by "spine" from 35 alone-against-five cases. Twenty-nine of those 35 had either a cut-off peer counted as a dissenter or a peer whose answer differed only in format; with both corrected there are six genuine cases, and the lone correct agent kept its answer in one of them. No ranking survives that, and the sentence that matched it to the judge ranking of Section 13 is withdrawn.

Per layer:

Read against the best agent that finished every item, debate wins on the two layers where it starts least correlated and loses on the two where it starts most correlated, which is what the second version of this note reported. Read against the best agent on the items it answered, with the majority taken on those same items, it wins one item on GPQA Diamond (Qwen's 49 items, 0.980 → 1.000), ties on the olympiad and loses one item on each of the other two; the earlier versions set the agent's accuracy on its items beside the majority on all items, which gave +0.3 and 0 and was not a comparison. GLM and Qwen, the strongest models on GPQA Diamond, are at 0.97 and 0.98 on the GPQA Diamond items they finished (35 and 49 of 56) and at 0.91 and 1.00 on the olympiad ones (33 and 26 of 54), and were cut off on the rest, and the low round-0 correlation on those two layers is in part the same thing seen from the other side: the items the capped models could not finish are the ones the others get wrong. A low starting correlation is still a precondition for debate to pay rather than something it produces, and debate consumes it, taking the ICC to 0.92–0.94 on MMLU-Pro and GSM-Hard within three rounds and, on the other two layers, to a value that cannot be computed because by round 3 almost every answering agent is right on almost every item. The one place in this corpus where plain voting beats the best panel member, GPQA Diamond at round 0, is the place where the best member that answered every item is Sonnet at 0.893 and the stronger members are missing on a third of the items: the vote pays because it can skip the missing votes.

Two practical numbers. Stopping each item at its first round of unanimity among the answering agents, and taking round 3 otherwise, gives majority accuracy 0.973, the same as three fixed rounds, at 0.42 rounds per item on average, because 152 items were unanimous before anyone spoke and 57 more after one round; counted the old way, with a cut-off agent breaking unanimity, it is 0.68 rounds. Three rounds of six agents cost 21,900 completion tokens per item on top of the first answers (1,439 per agent-turn in round 1, falling to 1,088 in round 3); the stopping rule spends about a seventh of that for the same accuracy, on the same items it was tuned on. Subsets of the panel tell the same story in miniature: the mean plurality over all pairs of agents is 0.849 at round 0 and 0.969 at round 3, over all triples 0.874 and 0.971, and the full six 0.924 and 0.973; debate makes a pair almost as good as a sextet, which is what one would expect if its job is to transmit the one or two correct answers in the room and to let a cut-off agent answer.

If a cut-off answer counts as wrong

The first two versions of this note scored a cut-off round-0 answer as a wrong vote. The debate itself was not affected, since a peer saw an empty answer and an empty answer was never counted in a plurality; the tables were. Every number that depends on an agent's correctness rather than on the vote moves: agent accuracy, φ, ICC, n_eff, unanimity, the all-wrong stratum, the flip table and the per-layer best single agent. The majority, the fixes and the stopping accuracy do not. For the record, the same tables under that policy; the companion repository prints all of them under "legacy".

Under this policy the second version read the per-layer table as "debate pays where the panel starts decorrelated", ranked the families by spine, and reported that every agent gained 13 points. The decorrelation on GPQA Diamond and the olympiad was in part 86 empty answers scored as wrong and correlated with nothing; the spine cases were in part empty peers scored as dissenters; the 13 points were in part agents finishing their answers. The reader can take either policy; the note takes the one under which an empty answer is empty, and the one under which an agent that could not finish alone scores 0 as a single-agent control.

Two things are entangled in a debate gain: the information each agent receives from its peers, and the fact that it was asked to think again. The public traces in the previous note had a self-only control on one model; here it is built in. Six copies of DeepSeek, seeded with six of its own temperature-1 draws as round 0, debate the items three ways. In the plain arm each copy sees the other five as in Section 10. In the self-only arm each copy sees only its own previous answer and reasoning, with the instruction to reconsider on its own. In the pressure arm the system prompt adds that the other agents are usually right and the copy should defer to the majority unless certain.

Cut-off rows are absent here too (66–77 of the roughly 5,600 rows in each arm). On the 230 items all three arms completed, the round-0 majority is 0.865 and the round-3 majorities are 0.900, 0.883 and 0.909, so the comparison in the text uses those: +3.5, +1.7 and +4.3 points. (The second version put the round-0 majority at 0.861 and the plain gain at 3.9; the round-0 answers are the same six draws in every arm, and 0.865 is what they vote.) Three readings. First, about half of the homogeneous gain, 1.7 points of 3.5, is self-revision with no peers at all, and that part costs nothing: the self-only arm leaves the correlation at 0.58 and takes the count of unanimously wrong items from 6 to 2. The other half, the information, takes that count to 22, on items that, like the four of Section 10, have not been audited. The two arms are 11 fixes and 3 breaks against 8 and 4 on the same items, which is a contrast 230 items cannot pin down; "half" is a point estimate. Second, telling the copies to defer changes little they were not already doing; the pressure arm is a point above the plain arm with fewer wrong unanimities, within what 236 items can resolve. Third, and this is the number to carry from the previous note: with a strong model talking to itself, the all-wrong stratum is 0 for 15 in the plain arm and 1 for 13 and 1 for 15 in the other two, and the one escape is a GSM-Hard item whose gold is one of the two the audit has not reached, so it may be no escape at all. The 31–45% escape rates on weak models in the public traces were real and were specific to models whose first answers were under-elicited; a 2026 reasoning model that is wrong six times is wrong because the problem is hard for it, or because the label is wrong, and six copies of it have almost nothing to tell each other.

Copies gain less than families, 3.5 against 4.9 points, though the families' gain is 3.9 with ties split and 2.2 with the strongest agent first, below the copies', and the copies' gain has not been recomputed under other rules, with the further caveat that the two debates start from different round-0 answers (temperature-1 draws against temperature-0 matrix answers) and different item sets, so the comparison is one contrast, not a controlled one. Section 10 offers two readings: the copies start at φ = 0.58 where the families start at 0.34, so there is less decorrelation to consume; and six copies under the same token cap cannot give each other a second chance to finish, which is where most of the mixed panel's fixes came from. The homogeneous majority at round 3, 0.894, is still 4 points under the ceiling for those copies, 0.936 (1 − 15/236, the "someone was right" rate across the six seed draws), and the mixed majority 0.973 is 2 points under its ceiling of 0.996 (1 − 1/225); neither majority leaves its box, although single items do: the one all-wrong mixed-debate item was fixed, and one all-wrong item in each of the self-only and pressure arms.

The generative objection to everything above is that a vote is the wrong aggregator: a model that reads six answers can write a seventh, and the mixture-of-agents literature reports exactly that. We gave DeepSeek, at temperature 0, the six round-3 answers of the mixed debate, each with the last 900 characters of that agent's reasoning, and the instruction to pick one or derive a different answer if all were wrong; separately, five of its own draws with their reasoning tails. The first version of this note called these inputs "transcripts"; they are final answers with excerpts, about 1,400 input tokens per item.

Over the debate answers the synthesiser does half a point worse than the plurality; over its own draws, 1.3 points better; in both cases it recovers nothing from the all-wrong stratum and writes a novel answer once in 462 tries; the four all-wrong debate items are the four on which the debate ends unanimously wrong, unaudited and flagged as probable label faults (Section 10), and on all four the synthesiser wrote the agents' answer, so that half of the test says nothing until they are read. Two caveats on the design. The debate-answer test is weak as a test of mixture-of-agents, because by round 3 the answering agents agree on 219 of 225 items and there is little to mix; the own-draws test, which does start from diverse inputs, is the fairer one, and it says the same thing. An aggregator with the freedom to leave the box sits one point under its ceiling on the debate answers and six under it on its own draws, which is where a vote on the same inputs sits. On these models and these problems, reading the others' reasoning is not a way to get a new answer; it is a way to pick among the old ones.

Everything so far asks the group to produce the answer, and the box says the prize is in selection: on a contested item some model is right and the plurality may not be, and a verifier that finds the right one collects the distance between plurality and ceiling, which on the sixty-item subsets is 3–11 points at round 0. The earlier notes predicted that an LLM verifier cannot, because its errors are the solvers' errors. Part of that prediction holds; the part that fails is the useful part, it is smaller than the first two versions of this note made it, and it is no larger than a tie rule.

On the 75 items where the answering debate agents' matrix answers disagreed by answer identity and at least one was correct and one wrong, each of five models was shown the distinct candidates, each with a 700-character excerpt of one proposer's reasoning, in a shuffled anonymised order, and asked to pick one or say none. (The candidate set is the six debate agents' answers, so a judge that also solved the item may find its own answer among them; on 51 of the 75 items one or more agents had proposed nothing, having been cut off, and on nine of them the judges were shown a cut-off agent's partial text as a candidate, which this version does not count as one.) The contested items are 31 olympiad, 24 GPQA Diamond, 12 GSM-Hard and 8 MMLU-Pro. The second version counted 82 and the third 77: five GSM-Hard items whose candidates differ by less than the grader's tolerance were merged in the third version, and the fourth dropped one item that was contested only by a cut-off agent's partial text and one on which both candidates were graded correct.

A judge that named no candidate either hit its token cap before the PICK: line, 20 of Qwen's 75 calls, 29 of GLM's and 7 of DeepSeek's, or wrote a letter that pointed outside the candidate list, usually an option letter of the question: 10 of DeepSeek's, 6 of Sonnet's, 2 each for Haiku and GLM. The second version called these "a handful", scored them as wrong picks, and scored a judge's cut-off own answer as a wrong solve; the third version still counted the stray letters as named picks. A judge that answered "none", or that picked a cut-off agent's partial text, counts as having named a candidate and as wrong, but a "none" is not a pick of a wrong candidate, so the last column is over the picks that named a listed candidate and is not one minus the "on those" column for DeepSeek and GLM; "picks its own answer" is over the 75 contested items; in the panel counts below a cut-off call, a stray letter and a "none" are all absent. Like for like, on the picks it named against its solving on the items it answered, two judges pick better than they solve, Haiku by 19 points and DeepSeek by 22, and three pick about as well as they solve, within 2 points: Sonnet, Qwen and GLM, the three strongest solvers on these items, gain nothing by judging rather than answering. Over all contested items, with an unnamed pick scored wrong, Sonnet at 0.80 is the only judge above the solvers' own plurality of 0.747 on the same 75 items; Qwen reaches 0.89 on the 55 items where it managed to name a candidate.

The result is the panel, and it depends on how one counts a panel. Judges' picking errors correlate across judges at φ = 0.16 (tetrachoric 0.27) on the 74 items all five saw, a cut-off or stray pick scored as wrong, which departs from the rule used for solvers and folds GLM's 29 and Qwen's 20 cut-off calls into the error vectors: five judges are 3.1 effective voters by that count. The solvers' errors on the same items cannot be set beside that directly, because selecting items on disagreement deflates their correlation; on the whole sixty-item layers the six solvers' round-0 φ is 0.2–0.5 (Section 10). The plurality of the five judges' picks is correct on 0.838 of the contested items when a tie at the top is broken by the first judge in the list (Haiku, Sonnet, DeepSeek, Qwen, GLM), which is the rule the first version used without saying so. Ties are not rare: 13 of 74 items; and on 49 of the 74 the panel is two, three or four judges rather than five, because a cut-off call, a stray letter or a "none" is absent. The solvers' plurality under the same kind of tie rule, first proposer in the fixed agent order wins, is 0.757, so the panel leads by 11 items to 5, an exact McNemar p of 0.21. If a tie counts as no answer, the panel is at 0.757 and the solvers at 0.743. If one asks only for a strict majority, three or more of the five judges on one candidate whatever the number that named one, the panel answers 54 of the 74 items and is right on 0.944 of those, where the solvers' plurality is 0.796; on the other 20 it has nothing to say. The mean over all ten triples of judges is 0.801 with first-judge ties and 0.758 without; the mean single judge is 0.638. The second version read the panel's lead as seven points and the third as four. The third version's solvers' plurality broke ties by the order in which the candidates had been shown to the judges, a per-item shuffle, which happened to favour the solvers on three tied items; the text called it "first proposer wins", and this version makes it so. Eight points, 11 against 5, is still within noise, and it is the tie rule's number. The fixed order gives the solvers' ties to DeepSeek, the weakest solver on these items, and the panel's to Haiku, the second-best judge; what favours the panel is the first. Eleven of the 74 contested items are solver ties, the fixed order picks a wrong answer on ten of them and the panel is right on ten, and nine of the panel's eleven wins are those items; on the 63 contested items with no solver tie the solvers' plurality is 0.873 and the panel 0.825. With ties split evenly on both sides the panel is 0.842 against 0.802, three items; with the strongest on each side breaking them, Sonnet for both, the solvers' plurality is 0.865 and the panel 0.851, four items won against five lost. The comparator's lead runs from eight points to minus one with the tie rule and nothing else, and to twelve, 0.878 against 0.757, under the rule behind the whole-set 0.953 below, a tied panel falling back to the solvers' plurality; the companion tables give all of them.

The honest accounting is over the whole item set, because a verifier is only ever used on top of a vote. Over the 235 items all five judges saw: the solvers' plurality alone is 0.915; plurality on unanimous items plus the panel on contested items is 0.940 with first-judge ties and 0.953 when a tied panel falls back to the solvers' plurality; the best single solver with a cut-off answer scoring 0 is Sonnet at 0.945, and the best on the items it answered is Qwen at 0.980 on 196 of the 235. The panel moves the vote by two and a half to four points, to half a point below Sonnet or a point above it. On the 196 items Qwen answered the plurality alone is 0.959 and plurality plus panel 0.954 or 0.974, so the pool sits 0.6–2.6 points below Qwen there; the earlier versions set Qwen's 0.980 beside the pool's whole-set figure, which gave "three to four points" and was not a comparison. Under the strongest-first tie rule the whole-set plurality alone is 0.949 and plurality plus panel 0.945. That is the whole of the comparator result: on the contested items the judges' picking errors are less correlated with each other than the solvers' errors are on whole layers, though the two numbers are not on the same items; the gain on contested items is 11 items against 5 under one tie rule and nothing under another; and it is at most the size of the gap between a vote and its best voter.

Two things qualify this further. On the 2 items where no candidate was correct, the judges said "none" once in 10 opportunities; a picker cannot abstain. And the result rests on 74 contested items from one corpus, 13 of them decided by a tie-break and 49 judged by a panel of fewer than five. It is the only positive number on the verifier side of this note, it is within noise and within a tie rule, and it is the one that most needs a second dataset.

The same independence shows up in a cheaper form. Take DeepSeek's matrix answer on each of the 225 debate items, wrong on 17% of them, and ask how well two warning signals rank the wrong ones: the number of the other five debate agents, from four families, whose answer differs from it, and the number of DeepSeek's own temperature-1 draws that differ from it, the usual self-consistency signal. Peer disagreement, counted over the peers that answered and by answer identity, reaches AUROC 0.98; disagreement among three own draws, 0.89, among five, 0.90, among fifteen, 0.93, counted as the share of the draws that answered. The comparison is not quite between "others" and "self": one strong peer alone, Sonnet, reaches 0.93, so part of the peer signal is that the peers are better solvers. Unmatched, five one-shot calls to other models beat fifteen extra calls to oneself by five points, with a bootstrap interval over items of [+0.01, +0.10]; matched to one strong peer, Sonnet alone against fifteen own draws, the gap is 0.00 and its interval spans zero ([−0.07, +0.07]). The third version counted a cut-off own draw as a disagreeing one, which put the fifteen-draw figure at 0.94. Five one-shot calls to other models, not cheap ones where Haiku is among them, are at least as good an abstention signal as fifteen to oneself, for DeepSeek, the only target tested. This is the mechanism behind the panel in Section 13: a judge's "this candidate is right" is partly "this candidate agrees with what I would have said", and that signal is, on the contested items, less correlated across judges than the answers are across solvers on whole layers, two different item sets.

The part of the prediction that holds is the other judging task. Shown one candidate at a time, with its excerpt, and asked only whether it is correct, with one planted wrong answer per item, taken from a wrong draw some model had made on that item in another stage and that no proposer had put on the candidate list, the same five judges behave more like solvers again.

The three OpenRouter judges have fewer verdicts for two reasons: the stage hit its spend cap when the three OpenRouter judges had been called on about 82% of the candidate rows the Claude judges saw (248–252 rows against 301–305), and 38, 77 and 96 of the calls DeepSeek, Qwen and GLM did make returned no verdict, nearly all of them cut off at the cap and ten of DeepSeek's not (Haiku returned none on 9), so they ruled on 70%, 55% and 50% of the candidates Sonnet ruled on. Two things stand out. Sonnet is a good certifier, 6% false acceptance and no decoy in 64 (the one it had accepted in the earlier versions was the correct answer of a relabelled GSM-Hard item that the decoy list still carried as wrong; the decoy column counts the decoys that are wrong, which leaves that one out, and the "others' wrong answers" column leaves the decoys out); the rest accept a quarter to a half of wrong candidates, and a planted wrong answer that no proposer had offered is accepted 9–24% of the time by the other four. And the largest term, for four of the five, is self-preference: every judge accepts its own wrong answer more often than someone else's, by a factor of 1.1 for DeepSeek, 2.8 for Haiku, 3.4 for GLM and Qwen, and 6.6 for Sonnet, on 5–35 own wrong answers each. The candidate was anonymised and shown with the excerpt of its first proposer in the fixed agent order (DeepSeek, GLM, Mistral, Qwen, Haiku, Sonnet), so whether a judge saw its own reasoning under its own wrong answer depends on its place in that order: DeepSeek did on all 32 of its own wrong candidates, GLM on all 8, Haiku on 20 of 35, Sonnet on 4 of 14 and Qwen on 1 of 5 (the earlier versions said the excerpt was another proposer's whenever several models shared the answer, which is so only for the later agents in the order). The judge that always saw its own excerpt, DeepSeek, has the smallest self-preference factor and the one that rarely did, Sonnet, the largest, which argues against style recognition of the excerpt without settling it; whether it is recognition of the answer or re-derivation of the same error this corpus cannot say (Section 18), and "own" here means "equals the judge's own matrix answer", which is consistency rather than recognition.

A committee repairs this only at a price. The false acceptances correlate at φ = 0.38 across the five judges, which is the solvers' 0.34–0.39 on whole layers; the 2.0 effective judges against the solvers' 2.1–2.3 is mostly the panel size, five against seven, and against the picking panel's 2.3–3.1 it is a difference under only one of the two ways of scoring a cut-off pick. Over the 94 candidates all five ruled on (48 of them GSM-Hard, 20 of the 54 wrong ones planted decoys):

Unanimity lets through 1 of 54 wrong candidates, 2%, at the cost of 15% of the correct ones; four of five lets through 11% and keeps 95%. The table reads a judge as accepting an answer when it accepted any of the formats it was shown in: 73 of the candidate–judge pairs were shown twice, as "24" and "24.00" or with and without thousands separators, and 17 of those got conflicting verdicts. If a judge must accept every format, unanimity lets through none at the cost of 32% of the correct ones and four of five lets through 5 at the cost of 18%; the third version took whichever row came first in the file, which gave 1 of 54 at a 30% cost. The one wrong candidate that passes unanimously is a GSM-Hard answer whose gold is one of the two the audit has not reached. The first version of this note said that no threshold reached a 5% false-acceptance rate; six of the eight "wrong" candidates that a unanimous committee had accepted were right, and the gold was wrong. Put next to Section 13: the same five models and nearly the same candidates, though not the same mix of items (the committee's 94 candidates are 51% GSM-Hard, the contested pick items 16%), and in one framing they are 3.1 judges whose strict majority is right 94% of the time where it exists, in the other they are 2.0 correlated solvers who wave through their own mistakes unless all five are asked. The difference is, at least in part, whether the judge compares or re-solves; the item mix is the other part, and this corpus does not separate them. The swarm can rank, a little; it can certify only when unanimous and at a recall cost of 15–32%; and it cannot abstain.

Every row carries its token counts, so the operators can be put on one scale. For DeepSeek, on the items of each layer on which every row of every operator carries a token count, completion tokens (reasoning included) per item and the accuracy of each operator's output:

The item counts in the header are the items on which every row of all four operators carries a token count and the single answer and the sixteen draws are untruncated: in the two debate rows a row that hit the cap is in, with its 8,192 tokens, and does not vote, while a cut-off single answer or draw drops the item, as does any row the provider returned without a token count. That is why the one-answer accuracies differ from the matrix table and why GPQA has 39: the copies' debate has a token count on every row on 45 GPQA items and DeepSeek's sixteen draws are all untruncated on 55 (with cut-off single answers and draws admitted the counts would be 42 and 53). Three of the 39 GPQA items and six of the 43 olympiad ones carry a cut-off row in the copies' debate, eleven and thirteen in the families' debate; the earlier versions described these items as the ones every operator completed, which they are not. The copies' debate includes its six seed draws; the families' debate excludes round 0 because the matrix calls were made by other models. On MMLU-Pro and GSM-Hard, ten to thirty times the tokens buy two to five points or nothing; on the olympiad, sixteen draws buy nine points and six copies in debate twelve for a third fewer tokens, which is the one place in this corpus where spending on sampling is clearly worth it (four and five net items of 43). The mixed-family debate's GPQA and olympiad accuracies are not DeepSeek's: they are the panel's, with Sonnet and Qwen in it. A cost-matched comparison of these operators against a single model given the same budget is the subject of the next note; the table is here so that the reader has the raw numbers.

Five things this does not say. It does not say that a swarm of agents working on different subtasks, with tools, memory and a plan, adds nothing; that is a different operator, its gains would come from context length and parallel time rather than from independent errors, and the right control for it, one agent doing the same decomposition sequentially at the same token budget, is not in this corpus. It does not say anything about base or non-reasoning models, where error correlation may be lower and voting may pay; every model here was allowed to think, though two of the Claude models at low effort barely did, and the previous note's weak models behaved differently. It does not say debate is useless: a 2.2–4.9-point gain over the round-0 vote, depending on the tie rule, with a stopping rule that costs a seventh of three rounds is a real engineering result, and the fact that most of it sits on the problems a capped model could not finish is itself a result about what debate does. It does not say the comparator result is large or general: it is two to four points on 235 problems from one corpus, 11 items against 5 on the contested ones under one tie rule and 4 against 5 under another, and within noise. And it does not say that strong models never fail together: it says that on four public benchmarks the cases where they appeared to were, on 43 of the 48 problems the audit read the benchmark's fault and on 2 the grader's, by an audit whose MMLU-Pro half is a model's judgment and whose reversal would put the rate at 2–5% on that layer, that eighteen more such problems sit behind a token cap and have not been read, and that any measurement of joint failure on such data has to start with an audit and a budget the hard problems fit in.

The hypotheses the corpus was built for are bridges written by the discovery pipeline: each pairs a stated limitation of one paper with a result of another, and the prediction is the pipeline's, not either paper's. They were judged before any of this data existed, and the companion repository of the discovery pipeline holds all 140 with their three verdict passes against the first-version tables. The sixteen the tables bear on, re-read by the author against the audited tables, by each hypothesis's own falsification criterion:

• Correlated judge panels (a bridge from CollabEval, arXiv:2603.00993, to Nine Judges, Two Effective Votes, arXiv:2605.29800). Predicted: discussion rounds raise φ and cut n_eff, and items that reach consensus after discussion are wrong more often than items unanimous before it. Supported on correlation: φ 0.34 → 0.85, n_eff 2.2 → 1.2; not yet on wrong consensus: unanimous-and-wrong 0 of 152 → 4 of 219 (Section 10), but all four items look like label faults and are unaudited. A further caveat: these are solver debates, not judge panels.
• Pre-debate independence predicts debate gain (OptAgent, arXiv:2510.18032, read against Effective Independence). Predicted: gain over the best single agent tracks the panel's pre-debate n_eff, and debate drives n_eff toward 1 whatever the topology. Not separable on this corpus. The two layers with the highest round-0 n_eff (2.8–2.9, the lowest φ) are the two on which the capped models were cut off most; against the best agent that finished every item debate gains there (+8.9, +3.7) and loses on the high-φ layers, which is what the second version reported, but against the best agent on the items it answered, majority on the same items, it gains +2.0 and 0, and on the 162 items with six answers it gains one item (Section 10). That n_eff falls toward 1 is supported in every arm in which the agents see peers, and the self-only control of Section 11, where φ stays at 0.58, shows that seeing peers is what drives it, for copies of one model. Four layers is four data points.
• Relay gains live inside the selection ceiling (WILC, arXiv:2607.29087, against the co-failure ceiling paper, arXiv:2606.27288). Predicted: iterative collaboration gains almost entirely on items where some worker was already right. Supported, with a smaller denominator than before: the gain sits in the one- and two-correct strata, mostly on items where a worker had been cut off; the all-wrong stratum has one item after the audit, and debate rescued it.
• Mixture-of-agents synthesis beats the ceiling (a bridge from arXiv:2605.01566 to arXiv:2606.27288). Predicted: an aggregator's surplus over voting comes from items where every proposer was wrong. Not supported on the own-draws test, 0 of 16; the debate-answers test, 0 of 4, is on the four unaudited items whose labels look wrong and says nothing (Section 12); the surplus is selection inside the ceiling.
• Role prompts decorrelate (Dynamic Role Assignment, arXiv:2601.17152, against the Blackwell-informativeness paper, arXiv:2605.06028). Predicted: role-diversified proposals have lower inter-agent error correlation than temperature samples. Not supported, and not refuted: φ is higher under roles on two layers, by 0.07 and 0.17, and lower on one, by 0.03, with bootstrap intervals that include zero on all three (Section 9). A sibling hypothesis from the same bridge, that a reported debate gain is score-only selection and the subsequent debate erases diversity (against The Interaction Tax, arXiv:2608.23541), is also not supported: debate raised accuracy above the round-0 vote on the full set, and the best single judge beats the round-0 vote only on the contested items (0.80 against 0.76) and only by naming a candidate the vote already held.
• Kish over-predicts the vote (two hypotheses pairing Kish 1965 with Boland 1989 and with arXiv:2607.23931). Predicted: n_eff = 1/φ overstates what a majority can reach. No verdict. With an exact estimator the Kish majority at n_eff ≤ 2 is the single-draw accuracy, which the observed plurality exceeds on six of eight rows; the hypothesis assumed the formula would over-state a vote, and on these rows it predicts no vote at all (Section 8). The first version reported the reverse of the hypothesis on an estimator that rounded n_eff to an integer, and the second a mixed verdict on a normal approximation whose error tracked accuracy rather than φ; both readings are withdrawn. The companion hypothesis from arXiv:2602.08003, that a probit one-factor bound beats the Kish prediction, has no table behind it in this corpus and gets no verdict either.
• A one-factor model under-predicts joint failure by 2–3× (a bridge from the Honest Quorum paper, arXiv:2607.16109, to Effective Independence; the 2–3× figure is the co-failure paper's 2.5×, arXiv:2606.27288). Insufficient after the audit: on the items all seven finished the observed all-wrong rate is one in 733 and the copula at the tetrachoric ρ over-predicts it; on the items with cut-off answers it is about 8% and unaudited, 2% pooled over all items (Section 7). There is no measured joint-failure tail on these benchmarks to compare a model against. The same bridge's claim that provider-family diversity does not make a certifying committee feasible is supported in part by Section 15: false endorsements correlate at 0.38, and only unanimity reaches a 5% bound, at 68–85% recall depending on how conflicting verdicts across formats are merged.
• Pooled independence ≲ 2 (CTTS, arXiv:2508.03333, against No Free Lunch, the note of 8 September 2026 on this site). Predicted: a heterogeneous pool has about two effective members and selection stays within the union of candidates. Supported in part: n_eff 2.1–2.3 of 7 on the three larger layers, every aggregator inside the ceiling; the paper's own method was not run.
• Debate rounds add nothing beyond round-0 voting (TUMIX, arXiv:2510.01279, against the modal-ceiling paper, arXiv:2606.28661), the hypothesis the previous note tested on public traces. Not supported on the full item set, on a second dataset with different models: majority 0.924 → 0.973, 11 fixed against 0 broken, under the fixed tie order; 0.951 → 0.973 when the strongest agent breaks ties. Supported on the 162 items on which every agent had answered at round 0, where the gain is one item. The verdict turns on whether a round that lets a cut-off agent answer counts as refinement. (TUMIX's own abstract reports gains from refinement; the "nothing" is the bridge's reading of its halting result.)
• Majority accuracy peaks before consensus (the Koopman-certification paper, arXiv:2608.05956, against Ladha 1992). Not supported: majority accuracy is non-decreasing to near-unanimity, with no interior peak (Section 10), within the resolution of four rounds.
• Debate gains when errors are already correlated (arXiv:2511.07784 against arXiv:2605.29800). Not supported, reversed in the per-layer table: the gain over the round-0 vote is largest where φ starts lowest, which is in line with that paper's own abstract. The reversal is not clean, because the low-φ layers are the layers with cut-off answers.
• Prompted deference does not control convergence (Thinking with Many Minds, arXiv:2501.02348, against BOUNDARY_SYNC, arXiv:2607.01600). Supported with two prompt levels: exposure drives convergence, the stated weight barely moves it (Section 11).

Each of these is one reading of one table, and each hypothesis was written about a specific system this corpus does not reproduce. They are listed as what a careful reader of these tables would conclude about those claims, not as replications of the papers. A reader who follows the links should also expect the bridge's premise to be the pipeline's reading rather than the paper's: OptAgent, the Honest Quorum paper and CTTS, for instance, do not themselves make the predictions attributed to their bridges.

One paper published after the audit measures the benchmarks from the other side and meets the same problem. ErrorMap (arXiv:2601.15812) has an analyst model diagnose about 7,000 wrong answers from 83 models on 35 datasets and build a taxonomy of the causes; its most common category is a missing required element (15.6%), and on MMLU-Pro, GPQA and Omni-MATH about 44% of the diagnosed errors are calculation, omission or counting rather than reasoning. That agrees with what the audit found underneath the all-wrong items here. The pipeline, though, takes the reference answer as true and leans on correct answers from other models as its reference points, and it checks neither the gold label nor whether the answer was cut off. On an item whose gold is wrong there are no correct answers to lean on, and every model is still assigned an error; a truncated answer is a missing element by construction. Of the 48 all-wrong items audited here, 43 had a wrong gold or an ill-posed question, and 229 problems had some cut-off answer. Part of what ErrorMap calls the failure landscape is, on these benchmarks, the landscape of their labels and caps, and its scale, 83 models over 35 datasets, makes that a property of the atlas rather than of one dataset. Its per-item diagnoses also agree with a meta-judge 91% of the time while agreeing with themselves across three prompt variants on about half of the items, which is the pattern Section 15 reports for certifying judges: agreement is not stability. Added 11 October 2026.

• Labels. The audit covered 48 items, selected mostly on all seven models having finished and failed; it found the gold wrong or the item ill-posed on 43 of them, by the author's reading, and an independent re-reading agreed with 20 of the 21 relabels (the earlier versions quoted a 36–42 range for it that is not in the audit file); the MMLU-Pro decisions are judgments about knowledge questions, and Section 7 gives the ceiling with them reverted. Items on which some model agreed with a wrong gold label have not been audited, so per-model accuracies may still be biased; 18 items on which every answering model was wrong while another model was cut off, one all-seven-wrong item in an older run file, and the four items on which the debate ends unanimously wrong have not been audited either, and are listed in the audit file. The auditor is one of the audited models and the new gold agrees with its own first answer on every relabel; eight items were checked by hand, the three GPQA Diamond items by hand only. GSM-Hard's gold is produced by a program that substitutes large numbers into GSM8K; eleven of its 150 matrix items had the substitution applied to some quantities and not others (six of them among the sixty), and six more are not well-posed after substitution (three among the sixty).
• Truncation. The four OpenRouter models ran under an 8,192-token completion cap and two of them hit it on a fifth to two fifths of GPQA Diamond and about half of the olympiad problems. The matrix tables are on the items every model finished, which are the easier ones; the debate and judge tables count a cut-off row as absent; the first two versions counted it as wrong, and Section 10 shows what that changed. The cap is a property of this corpus, not of the models, and a re-run with a larger cap would move every number that depends on it.
• Size. Sixty items per layer, 225–236 in the debate stages, 74 contested items for the picking result, 54 wrong candidates for the certifying one. The strata are small: the all-wrong stratum of debate is one item, six agents were alone against five answering dissenters six times, the panel result turns on 13 tied items and is 11 against 5 on the discordant ones, and the homogeneous arms differ by 11 fixes and 3 breaks against 8 and 4.
• Models. Reasoning models of one vintage only, the Claude models at an uncontrolled temperature and low effort, so that two of the seven barely reasoned; the OpenRouter calls return no reasoning-token count, so "allowed to think" is a setting, not a measurement. The homogeneous debate, the personas, the synthesiser and the error-detection result rest on one family, DeepSeek V3.2. Opus is in the matrix but not in the debate or judge panels.
• Controls. Best single is an in-sample maximum over six or seven agents, which flatters it by about a point on the small layers; on GPQA Diamond and GSM-Hard the one-item margins of Section 7 are inside that. The "best agent on its answered items" control is compared with the majority on the same items; the earlier versions compared it with the majority on all items. Round 0 of the debate is the temperature-0 matrix answer and the rounds are at temperature 1 for the OpenRouter agents (the Claude agents' temperature is not controllable in either), as in the public traces of the previous note; the self-only arm of Section 11 is the control for the share of the round-1 jump that is re-elicitation, about half for copies of DeepSeek, and there is no self-only arm for the mixed-family debate, so its first-round jump, including the fall in cut-off rows, mixes peers, re-elicitation at temperature 1 and a new prompt.
• Mistral. Mistral has no matrix row on 32 GPQA Diamond, 6 olympiad and 2 MMLU-Pro items, which removes 18 GPQA Diamond and 4 olympiad items that every other model finished from the seven-finished tables; nine of the 240 debates were never run because Mistral had no matrix answer to seed them with, and one lost Mistral to a rate limit. The six-agent tables use the 225 complete items; the five-agent recount recovers one item and matches at round 3.
• Estimators. Tetrachoric correlations are the cos-π approximation, averaged over the pairs on which it is defined, which after the audit is few on the small layers; φ and ICC are exact, φ on the items each pair answered. The copula is a simulation at one ρ per layer and is no longer a fit to anything. n_eff is Kish's formula used descriptively; the Kish majority is an exact binomial interpolated between integers, the third estimator this note has used for it. The debate stopping rule was evaluated on the items it was chosen on. Plurality ties go to the first agent in a fixed order for the six debate agents and to the order of the data files for the seven-model matrix; seven of the eleven debate fixes are round-0 ties, nine of the judge panel's eleven wins are solver ties, and Sections 10 and 13 show the debate gain running from 4.9 to 2.2 points and the panel's lead from +8 to −1 across tie rules. The certify committee depends on how a judge's conflicting verdicts on two formats of one answer are merged; Section 15 gives both readings.
• Judging design. Candidates were anonymised but shown with a reasoning excerpt; a judge could recognise its own style, and the excerpt is the first proposer's in the fixed agent order, so DeepSeek always saw its own and Sonnet rarely did (Section 15). In the pick task a judge's own answer was among the candidates whenever it had solved the item; in the certify task "own" means the candidate equals the judge's matrix answer. The self-preference rates are therefore a mixture of style recognition, answer recognition and re-derivation, and this corpus cannot split them. A judge that ignored the list and solved the item, writing its own final answer, was mapped to the candidate its answer matched; on the contested items that is about a quarter of DeepSeek's named picks and a seventh of Haiku's, so for those two "picking" is partly re-solving. The certify committee is 51% GSM-Hard candidates where the contested pick items are 16%, so the pick-against-certify contrast of Section 15 is also a contrast of item mix. The judges' φ in Section 13 counts a cut-off call as a wrong pick, against the rule used for solvers.
• Coverage. In the pick task GLM and Qwen named a candidate on 44 and 55 of 75 items; in the certify task the OpenRouter judges ruled on 50–70% of the candidates Sonnet ruled on; the role runs did not reach GPQA; GLM's hard-layer samples are mostly truncated; Haiku's olympiad samples were drawn twice under many of the same indices and the later row for each index is the one counted.
• Contamination. All four layers are public and predate the models; the olympiad problems are from 2024 and 2025 competitions. Nothing here measures memorisation.
• Pre-registration. None. The stages were designed to serve hypotheses written by other people, and the verdict passes that read those hypotheses against the first tables were done by a model; both are in the discovery repository, not in this note. The first revision was prompted by an external reading of the first version, the second by an external reading of the second, the third by a reading that set the text against the code, the fourth, folded into this version before it was deposited, by a reading that recomputed the controls from the rows, and the fifth, also folded in, by a reading that recomputed the counts the text quotes from the rows; each found errors the author had not; the sixth, a blind recount of every quoted number from the rows, found the three misdescriptions listed in Section 5 and no wrong table, and the seventh, against the collection code, found the certify excerpt rule and the misstatements listed in Section 5; the reader should assume an eighth reading would find more.

Every table in this note, and about twenty more that it quotes from, is rebuilt from the stored answers by a single standard-library script in under a minute with no network access, under both truncation policies; the McNemar tests, the bootstrap intervals and the counts of Sections 7, 10 and 13 that the earlier versions had computed outside the script are in it now, and so are the tie-rule and within-item tables of Sections 10 and 13 (section T of the companion tables). The dataset and that script will be released as tech-note-5 within about a month of this note: every row of every stage with the parsed answer, its correctness, the serving provider and the token counts, but not the model texts; GPQA Diamond answers replaced by random tokens drawn per item and per option, which preserve answer identity within an item and nothing across items, and no question text or gold label stored, per that dataset's terms (the second version's hashes were an unsalted digest of the option letter and could be inverted; the third version's tokens were drawn per letter across items, which kept the letter up to a permutation; both have been replaced). The release also carries label-audit.json, the item-by-item record of the audit with the reason for every relabel and exclusion and the lists of unaudited items, and regrade_all.py, the script that applied it, which imports the grader from the collection repository and will run once that is released. Figures are drawn inline from the same numbers. The collection scripts, graders, and the discovery pipeline that proposed the hypotheses and read the verdicts live in a separate repository that will accompany a later note on the pipeline itself. Until the release, the numbers here are the author's word; after it, they are a command.

Archived with a DOI: all versions 10.5281/zenodo.23229042; this fourth version is 10.5281/zenodo.23241005, version 3 is 10.5281/zenodo.23235028 and version 1 is 10.5281/zenodo.23229043. Cite as: Latipov, A. (2026). Where the Swarm Pays (version 4). Zenodo. https://doi.org/10.5281/zenodo.23241005 The companion data and code (every answer row without the texts, the label audit, the table builder) are at github.com/Alatip/tech-note-5, archived as 10.5281/zenodo.23252753, CC BY 4.0.

1. The earlier notes: Effective Independence (31 August 2026), The Control Ate the Signal (2 October 2026) and Debate Is Not Sampling (4 October 2026). The pipeline follows Swanson's literature-based discovery: paper A states an open question or limitation, paper B reports a result that bears on it, A and B do not cite each other and are otherwise dissimilar, and the pair is a candidate hypothesis. Candidates were sieved by embedding similarity between A's stated needs and B's stated offers and scored 0–10 by an LLM judge (Claude Opus 5.5, with a second opinion from Claude Fable 5.1 on the strongest). Papers and foundations, from Condorcet and Kish to the 2025–26 preprints, were summarised into fixed-schema cards first; the cards, not the papers, were what the judge read. ↩
2. One-factor Gaussian copula: model i is correct on an item when z_i = √ρ·F + √(1−ρ)·e_i < Φ⁻¹(acc_i), with F shared across models within the item and e_i private, both standard normal. Simulated with 20,000 items per layer. At the tetrachoric ρ the all-wrong rate is 0.011 / 0.003 / 0.004 against observed 0.002 / 0.000 / 0.000 (MMLU-Pro / GPQA / GSM-Hard), and the simulated majority-correct rate is 0.941 / 0.957 / 0.975 against observed 0.946 / 0.967 / 0.984. The model has no answer identities, so its "plurality" is a majority of correct votes; the observed figures in this footnote are on the same definition and differ slightly from the answer-identity plurality of Section 7. In the first version of this note the same model matched all-wrong rates of 1–10% to within a few tenths of a point; those rates were label noise, and the match was a fit to noise. ↩

← notes · github.com/Alatip