Paper draft · v8 · Ensembles

Paper Draft (v8). Exploring the Boundaries of Multi-Agent Ensembles: An Evaluation of Aggregation Operators

The ten technical notes on this site, assembled into one paper with one yardstick: for every aggregation operator, how much of the distance between the best single member and the oracle it collects, on which data, and at what price. A working draft, not peer reviewed; the PDF is the deliverable.

12 October 2026draft v815 sections, 20 tables, 87 referencesPDF on Zenododoi:10.5281/zenodo.23311676

Abstract An ensemble of language models can be no better than the share of problems that at least one of its members solves. Every way of combining members, which we call an aggregation operator, collects some part of the distance between the best single member and that ceiling. The paper measures that share, on the same data and with the same yardstick, for plurality voting, repeated sampling of one model, multi-round debate, synthesis, selection and certification by judges, routing by an orchestrator, and decomposition into subtasks. The data are a text corpus of 733 problems answered by seven models from five vendors, with every jointly failed problem re-audited; the same corpus with the frontier models removed; 9,600 public debate traces; the SWE-bench Verified leaderboard matrix of 175 agent submissions, at the top and in mid-ranked accuracy bands; two public leaderboard matrices; released τ-bench and OS-Harm logs; and a cost-matched study of one model's draws. Operators that only rearrange answers already produced collect nothing, whether the room above the best member is 2 points or 32, because the members fail on the same hard problems: one latent difficulty factor reproduces their joint failures and their votes, and the independent Condorcet model over-predicts the vote by up to 27 points. Operators that add computation collect a few points under conditions the paper names: repeated sampling where a single draw is below 0.9 at about ten times one answer's tokens; debate where the members' errors are unshared, at a loss of coverage; a judge panel whose sign against the best member depends on the tie rule. The one large prize nobody collects is the oracle on agent tasks, 10 to 32 points above the best submission, which a selector with an external verifier might reach.

Results

The frame. The ceiling of any operator that returns a member's answer is the oracle, the share of problems that at least one member solves. The whole prize is oracle minus best member. The question is not "does the ensemble help" but "what share of that distance does the operator collect", with the best member chosen on held-out problems wherever the data allow. One distinction organises the results: operators that only rearrange answers already produced (plurality voting, routing) collect nothing, because the members fail on the same hard problems; operators that add computation (repeated draws, debate, a judge or synthesis call) collect a few points, only where the single answer is weak and the members' errors are not already shared, at several times the tokens.

Plurality voting: collects nothing over a held-out best member.

Full pool of seven models (best 0.975–1.000, oracle 0.998–1.000, so at most 2.3 points of room): indistinguishable from the best member once it is chosen on held-out problems (split-half +1.8 [0.0, +3.3] for the best on MMLU-Pro, ≤ 0 elsewhere; Opus fixed in advance 714 vs 708 of 733, p = 0.26); in sample it trails by 0–1.9 points.
Pool without the frontier models (four models, 1–9 points of room): collects none of it; where one member dominates (GLM 0.989 vs 0.76–0.87 on GPQA Diamond) the vote loses to it by 8.9 points with the interval clear of zero.
SWE-bench submissions at accuracy 0.4–0.7 (19–32 points of room, 58–93 bugs nobody in the band resolves): the majority is 1.6–4.4 points below the best member; random panels of seven beat their own best in 1–17 % of draws. Chen MMLU-Pro and Leaderboard MMLU in the 0.5–0.7 band (14–25 points of room): a tie or 4–6 points below the held-out best.
The reason is the same everywhere: 1.4–2.3 effective votes, and after a fitted difficulty axis the residual correlation is zero (−0.009…+0.009 in every SWE-bench window below the top). The independent Condorcet model over-predicts the majority by up to 27 points where it promises most; a one-factor copula at the tetrachoric ρ predicts it within 1–3 points.
What voting does give is insurance: the plurality of three to five mid-ranked models scores about where the best of them scores, without knowing in advance which that is (it beats DeepSeek fixed in advance, 10 won / 33 lost, p = 0.001).

Self-consistency (k draws of one model): pays in a narrow regime.

Pays where a single draw is below 0.9 and the problems are hard: GPQA Diamond +4.3…+6.7, olympiad up to +14.5 (49 of 60 problems; +6.1 over the first draw); MMLU-Pro and GSM-Hard −0.6…+1.7; no intervals.
At equal tokens it loses to one answer at every budget a draw can use (by 0.3–31 points even at the best n); it wins only when the budget is about ten times an answer's natural length: olympiad +7…+12 at 32,768 tokens, one GSM-Hard cell +4.3.
A cheap model's ensemble against an expensive model: Haiku reaches Opus only on MMLU-Pro at k = 15 (ten times the price) and on GSM-Hard at k = 5; at the measured draw correlation (ρ 0.74–0.93) nine copies add at most 2.7 points, so the replacement works only when the gap is under about three points and the price ratio is at least the number of copies.

Debate: depends on whether the members have something to exchange.

Full pool: +4.9 under one tie rule, +2.2 under another (p = 0.06); 10 of 11 fixes are on answers an agent had not finished within its token cap; in the six-copy control half the gain appears with no peers at all ("think again"); on the 162 problems every agent answered, one fix. The panel collapses to about one effective voter.
Pool without the frontier models, pre-registered (errors on the contested problems uncorrelated, φ̄ −0.10): +6.8 on the 146 contested problems (14 fixed / 4 broken, p = 0.031), corpus 0.947 → 0.963 against the best member's 0.950 (p = 0.052), about 1.3 of the 3.6 points of room. The price: eleven contested problems end with every member wrong, coverage 1.000 → 0.925; a correct agent facing three disagreeing peers switches 46 % of the time. All four pre-registered "nothing will happen" predictions failed, each toward a larger effect.
9,600 public traces of weak models: +6…+20 over the round-0 modal-hit rate, with the panel again collapsing toward one voter.
Reading: debate collects one to two points over the best member where the members fail on different problems; where their errors are shared difficulty it collects zero; it always destroys part of the coverage.

Judges (best-of-n with a posterior check): a few points; the sign against the best member is set by the tie rule.

Full pool: +8.1 / +4.0 / −1.4 over the solvers' plurality under three tie rules; over the whole set within a point of the best solver.
Pool without the frontier models: +3.4…+6.8 over the plurality, 0.939–0.958 on the corpus against the best member's 0.950; the panel is not clearly above its best judge. DeepSeek picks 0.829 while solving 0.575 of the same problems.
Certification: only unanimity of five keeps false accepts under 5 % (1 of 54, Wilson [0.3 %, 9.8 %]), and that rests on the two strictest judges; the other three alone pass 8 of 54.
Synthesis in one call: 0.849 on the contested problems against 0.870 for three rounds of debate; 0 of 16 all-wrong problems recovered on the saturated pool, 1 of 10 on the weak one, with an answer none of the members had given.
The oracle (a perfect selector): +9.8 on the SWE-bench top seven, +14…+25 on τ-bench, +19…+32 in the mid-ranked bands. This is the unmeasured prize: no partial verifier (tests, execution) was run, and the 82 submissions that declare an internal best-of-n selector are indistinguishable from their peers.

Routing: not measured by us. By the published tables routers collect about a tenth to a quarter of their pool's gap to the oracle (Trinity ≈ 0.15 and under its oracle on every task; Chen's deployable routers ≈ 0.09; the Conductor and Fugu report no oracle). On our pool the router ceiling is at most 2.4 points and a per-problem-set route collects 0.7 of them.

Decomposition: nothing, except a budget effect. On 14 hard numeric problems split pass@4 is 0.21 against 0.21 (0 of 7 never-solved problems solved in eight tries); +0.34 [+0.18, +0.50] on 38 problems where one pass runs out of tokens (mechanism not isolated); −0.41 on medium and easy problems for DeepSeek; handing the parts to other models is never better than splitting within one.

In one sentence. Operators that rearrange finished answers give nothing, because the members fail on the same hard problems; operators that add computation give a few points, only where the single answer is weak and the members' errors do not coincide, at several times the tokens and with part of the coverage lost; the one large unclaimed prize is a selector with an external verifier on agent tasks, which this paper does not measure.

Download

The PDF (1.6 MB, 15 sections, 20 tables, 87 numbered references) is on Zenodo. Version 8 of 12 October 2026, archived as doi:10.5281/zenodo.23311676 (PDF and HTML; all versions under 10.5281/zenodo.23311675). Earlier drafts are kept locally; a changed text gets a new Zenodo version.

What the draft is built from

Every number in the paper comes from one of the notes below or from their companion data; the paper adds the common yardstick, the cross-validated best member, two new measurements (voting, debate and judges on the corpus without its frontier models; voting among mid-ranked members of three public matrices), and the one-difficulty-factor reading that ties the operators together.

Status and what is still open

This is a working draft: the results are the notes' results, re-read with the controls a referee would ask for (held-out best member, pre-registered runs with their missed predictions reported, tie rules and in-sample choices marked). It has not been peer reviewed. Known open items, listed in the paper's Section 13: the text corpus is saturated and four of its models ran under an 8,192-token cap; the label audit was one-sided; several tie rules were chosen in sample; the agent matrices are leaderboard submissions; the partial-verifier measurement on agent tasks has not been made. Volume and page details of the six classical references are to be verified before any submission.

Cite

Latipov, A. (2026). Exploring the Boundaries of Multi-Agent Ensembles: An Evaluation of Aggregation Operators (working paper, draft v8). Zenodo. https://doi.org/10.5281/zenodo.23311676 (all versions: 10.5281/zenodo.23311675). Archived with a DOI on 12 October 2026; the deposit holds the PDF, the HTML and this page.

← notes · github.com/Alatip