The Control Ate the Signal

Take any three models, vote by correct count, compare the vote's accuracy with that of the best of the three. Call the difference the gain. Kim's finding is that the gain is positive in only 10% of size-3 subsets on MMLU-Pro, and that the measures ensemble practice uses to pick members, strict diversity, disagreement, double-fault, Jaccard error overlap, focal diversity, mostly re-express the members' accuracies: strict diversity is collinear with one minus mean accuracy at ρ = 0.99, and the rest are 84–99% explained by ranked best and mean accuracy. The paper then residualizes each measure on those two accuracies in rank space and asks what is left. What is left is a residual pairwise co-failure term at Spearman −0.43, AUC 0.597 held out, weakening to −0.2 on hard items.

The audit is careful and the algebraic identities are correct. Our interest was in the estimation question it leaves open, and which the previous note on this site¹ raised from the other side: binary contingency measures such as double-fault are bounded by the marginals, so they cannot separate dependence from accuracy. The estimator built to do that separation is the tetrachoric correlation, and the model that turns correlation plus heterogeneous accuracies into a predicted vote accuracy is the correlated Condorcet jury of Boland and Ladha. The hypothesis, written down before any number was computed, was that this pair would predict gain better than the residualized double-fault.

Before the data, the arithmetic. For three independent voters with accuracies p₁ ≥ p₂ ≥ p₃, the majority is correct with probability

The gain is positive when p₂ and p₃ are close to p₁ and negative when they fall away from it. It is a cubic in the three accuracies, with a maximum depending on p₁, and it cannot be written as a·best + b·mean + c in any monotone transform of the two. Residualizing a predictor on ranked best and ranked mean accuracy therefore does not "remove capability": it removes the linear part of a nonlinear function and leaves the predictor to explain the curvature, which it does badly, with the sign that happens to correlate with the curvature. Dependence enters only afterwards, as a correction to P_maj whose size is the question.

This is not a criticism Kim's own numbers hide. The paper reports that the surviving association "varies with difficulty", flips sign between control sets for strict diversity, and vanishes in a nine-model subpool. Those are the symptoms of a misspecified control. What was missing was a predictor that uses the right functional form, so that the dependence term could be measured against it rather than against a straight line.

Kim's correctness matrices are promised but not yet released. We used four public ones.² Three come from Chen's co-failure release: MMLU-Pro, 52 models with full coverage on 200 items; MATH-Hard, 67 on 298; MATH-500, 52 on 200; all zero-shot at temperature 0, 2026 frontier models. The fourth is the archived Open LLM Leaderboard: 28 open base models on all 14,042 MMLU items, older and weaker, included because it has enough items to make difficulty strata stable.

Protocol, fixed in writing before the first run:

1. Twenty seeds. Each splits items in half; everything that defines a predictor uses the first half, the target uses the second. The best member is chosen on the first half, as in Kim's held-out variant.
2. Each seed also splits models in half. One half defines item difficulty, by its mean correctness; subsets of size 3 are drawn from the other half. This is Kim's "non-circular split" and avoids defining difficulty with the voters themselves.
3. Target: held-out accuracy of the correct-count majority minus held-out accuracy of the chosen best member.
4. Predictors, all from the first half of the items. (b) per-stratum tetrachoric correlation matrix of the voters, projected to the nearest positive-semidefinite matrix, Gaussian-copula simulation with each model's per-stratum accuracy as its threshold, 20,000 draws per stratum, predicted gain = simulated majority accuracy minus the best member's accuracy. (c) Kim's measure: mean pairwise double-fault over the subset, rank-residualized on ranked best and ranked mean accuracy, sign flipped so that less shared error predicts more gain. (a) the Kish effective-vote count from mean pairwise φ, the quantity the previous note was about. Controls: minus best accuracy, mean accuracy.
5. Metric: Spearman correlation between predictor and held-out gain across subsets; AUC for gain > 0; seed percentiles and a model-level bootstrap for the difference between predictors.

Falsification was fixed in advance: (b) had to beat (c) and (a) with a bootstrap interval excluding zero, and had to beat the accuracy-only controls, or the hypothesis failed. After a two-seed smoke test, and before the main run, two controls were added that the pre-registration should have contained from the start: (e) the identical Condorcet model with the latent correlation forced to the identity, and (f) the identical model with φ instead of tetrachoric as the latent correlation. Without (e) the headline below would have been misread.

Spearman correlation with held-out gain over the best member, size-3 subsets, mean over 20 seeds. Predictors are oriented so that positive means "predicts more gain".

Model-level bootstrap of the differences on MMLU-Pro, 300 resamples of the evaluator models:

By the pre-registered criteria the hypothesis is confirmed: (b) beats (c), beats (a), beats the accuracy-only controls, intervals nowhere near zero, on all four datasets, at size 4, and under the circular protocol with all 52 models. It is also calibrated: predicted gain is on the accuracy scale, mean signed error below 0.01, R² between 0.70 and 0.99 against observed gain. And it is confirmed for the wrong reason. The independent jury, which contains no estimate of dependence at all, is 0.011 behind. The whole 0.6-point gap to Kim's metric is the functional form.

What the dependence estimate adds, once the marginals are modelled properly, is +0.011 on MMLU-Pro, +0.001 on MATH-Hard, −0.004 on MATH-500 and +0.023 on the leaderboard pool. The sign and size line up with sample size: Chen's matrices give about 30 training items per difficulty tercile, the leaderboard gives over 2,000. Three details follow.

φ and tetrachoric are interchangeable here. The previous note argued, and still holds, that φ substituted into Kish's formula overstates a panel's independent votes, because the formula's meaning rests on a latent model that prescribes the tetrachoric. In a copula that reproduces the marginals exactly, the choice of latent correlation moves the predicted gain by less than the Monte-Carlo noise. The estimator question matters for how many votes you report; it does not matter for predicting whether this trio beats its best member.

Thirty items per stratum do not support a latent correlation matrix. On the Chen data the per-stratum tetrachoric matrices of 26–33 models are badly non-positive-definite; the projection changed entries by up to 1.0–1.2. The matrix that enters the simulation is a heavily shrunk version of the estimate, which is one reason (b) and (e) coincide. On the leaderboard pool the projection changed nothing. Per-stratum estimation is right in principle, and the strata do differ (φ about 0.2 on the hardest tercile, 0.03–0.09 on the others), but it needs thousands of items, not hundreds, and the strata contribute +0.003 to +0.009.

The effective-vote count anti-predicts gain. Kish n_eff correlates with held-out gain at −0.42 on MMLU-Pro and −0.80 on the leaderboard pool. A subset with low error correlation is, in a heterogeneous pool, a strong model paired with weak and dissimilar ones; its majority is dragged below the strong member. n_eff is a property of a pool, and a confounded one, not a selection criterion for a committee. Kim's raw-correlation sign flips are the same phenomenon seen through a different statistic.

It does not say majority voting is a good idea. Gain is positive in 9–22% of size-3 subsets on the Chen data and 0.1% on the leaderboard pool; being able to predict which subsets those are is useful for committee design and changes nothing about the base rate. It does not say that error dependence is unimportant for ensembles: the all-wrong floor, the subject of Chen's paper and the previous note, is a dependence quantity and is not touched by this analysis. And it does not say anything about Kim's own matrices, which were not available. Chen's MMLU-Pro slice reproduces Kim's qualitative findings, the positive raw double-fault sign, a weak residual, a 9% gain-positive rate against Kim's 10%, so the comparison is fair in kind, not in instance.

• Subsets overlap heavily; the only uncertainty statements are the seed percentiles and the model-level bootstrap, as in Kim. No per-subset p-values.
• The leaderboard pool consists of 2023–24 base models at 24–77% MMLU accuracy; positive gain is almost never observed there, so its AUC is degenerate and its Spearman is the informative number.
• The Gaussian copula is known to underprice tail co-failure (Chen). Here it is asked to predict a majority event, not a tail event, and is calibrated to within 0.01; the tail objection stands for other uses.
• Twenty thousand copula draws per stratum put Monte-Carlo noise on the predicted gain near 0.003, below every difference reported except (b) − (f), which should be read as no difference.
• One deviation from the pre-registration, the two added controls, is disclosed above. Nothing was removed and no criterion was changed.

Code, pre-registration and every reported number: Alatip/tech-note-3. One numpy script, no scipy, no API calls. The estimators are the ones from the previous note, with the vectorised tetrachoric solver. Data: Chen's matrices from the Hugging Face dataset josefchen/co-failure-67-models (CC-BY 4.0, snapshot of 25 September 2026), and the per-instance leaderboard files already described in the previous note. The full run over four datasets, 20 seeds and a 300-resample bootstrap takes about twelve minutes on a laptop; the circular variant with all 22,100 subsets takes another twelve. When Kim's matrices appear the rerun is one command, and this note will be updated with the numbers on the data the audit was actually run on.

Archived with a DOI: 10.5281/zenodo.23133020. Code release v1.0: 10.5281/zenodo.23133111. Cite as: Latipov, A. (2026). The Control Ate the Signal. Zenodo. https://doi.org/10.5281/zenodo.23133020

1. Effective Independence, 31 August 2026. The two open questions it left, how to estimate ρ for language-model errors and whether the 1/ρ ceiling survives majority voting over discrete answers, are the ones this note takes up. The answer to the second, on this evidence: the ceiling is a statement about averaging, and the quantity that governs majority-vote gain over the best member is not ρ but the spread of the accuracies. ↩
2. Chen, J. (2026). When Does Combining Language Models Help? A Provable Co-Failure Ceiling Across 67 Frontier Models. arXiv:2606.27288; data at josefchen/co-failure-67-models. Kim, D. (2026). Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles. arXiv:2607.20768. The closest prior use of the same machinery is Li & Hai (2026), State-dependent error correlations shape voting thresholds in committees of AI agents, arXiv:2607.23931: tetrachoric inversion into a heterogeneous Gaussian copula for k-of-n screening rules, without a comparison against diversity metrics or a gain-over-best target. Boland, P. J. (1989), Majority systems and the Condorcet jury theorem, and Ladha, K. K. (1992), The Condorcet jury theorem, free speech, and correlated votes, for the correlated heterogeneous jury. ↩

← notes · github.com/Alatip