Effective Independence

When several models judge or answer the same items, the natural question is how much independent information the panel actually carries. The standard answer borrows Kish's design effect from survey statistics:

with ρ̄ the mean pairwise correlation of errors. The same limit is independently familiar as the diversification floor in portfolio theory and as the Spearman–Brown ceiling in test theory. In ensemble learning it is the reason Random Forests randomise features rather than only bootstrapping rows.

What makes n_eff interpretable is a latent model. Correctness is treated as a threshold on a continuous, unobserved quantity — ability meets difficulty — and ρ is the correlation of that latent, not of the recorded zeros and ones:

Here c_i is the component shared by every model on item i — shared pretraining, shared blind spots, or nothing more than the item being hard for everyone — and e_ij is idiosyncratic. The model does not tell these apart: in item-response terms c_i is the item's difficulty and ρ its variance share, so independent solvers that face one ordering of the items produce the same ρ (Limitations, last item). Practice, however, estimates ρ̄ with the φ coefficient: a Pearson correlation computed directly on the binary correctness vectors.

Dichotomising a continuous variable discards information about where within a class each observation fell, and the resulting Pearson correlation is attenuated relative to the latent one. This is not a subtlety of language models; it is a century-old result in psychometrics, and the estimator built to undo it is the tetrachoric correlation — the correlation of the bivariate normal whose thresholded version reproduces the observed 2 × 2 table.

The point is narrow and worth stating precisely. The published analyses are not wrong about their data. φ is a perfectly good descriptive measure of association between two binary vectors. But the moment φ is substituted into Kish's formula and read as “independent votes”, the latent model has been invoked — and that model prescribes a different estimator. Using φ there is internally inconsistent: the interpretation assumes a latent that the estimator refuses to model.

The estimator point itself is not ours. Chen (arXiv:2606.27288) states it for co-failure ceilings — the single-factor copula “must be calibrated with the tetrachoric (latent) correlation, not the Pearson correlation of 0/1 correctness indicators … a century-old psychometric point we claim no novelty for, but which the LLM-evaluation literature routinely elides” — and credits Turkmen et al. (arXiv:2602.08003) with fitting a tetrachoric copula to binary LLM errors before that. What we do here is apply the same correction to the published effective-vote figures themselves, and measure its size on per-instance leaderboard data.

Attenuation is not a fixed factor. It vanishes as marginals approach 0.5 and grows as they move to the extremes, so the size of the correction depends on the accuracy of the models being compared. Its direction does not: φ is never larger than the tetrachoric value, so n_eff computed from φ is never smaller than the truth. Every error of this kind flatters the panel.

Before applying an estimator to data where the answer is unknown, it should be pointed at data where it is known. We built a synthetic world in which the latent correlation is fixed by construction: each agent's error decomposes into a per-item shared component, a constant per-agent offset, and fresh per-draw noise, so ρ = var_c / (var_c + var_e) exactly. Correctness is a one-sided threshold, matching the model's assumption.

At ρ = 0.500 with nine agents, the truth is n_eff = 1.80. Tetrachoric returns 1.79; φ returns 2.52 — forty per cent more votes than the panel actually carries. In a second synthetic world that adds a constant per-agent bias the same substitution reports 3.48 against a true 1.80: the flattery only grows as the world departs from the cleanest case.

The correction can be applied to real models without running any of them. The archived Open LLM Leaderboard publishes per-instance correctness for several thousand models, evaluated through a common harness. We assembled pools of 24–28 open models spanning 13–15 families — from bloom-7b1 to Mixtral-8x7B and Qwen1.5-72B — and recomputed both estimators pairwise from the actual 2 × 2 tables, with each pair's own marginals.

On ARC-challenge, twenty-four models drawn from thirteen families carry roughly one independent model's worth of information. Restricting the pool to its strongest third makes this worse rather than better: tetrachoric ρ rises from 0.865 to 0.908 and coverage falls, consistent with the observation that more accurate models are also more correlated.

Two recent papers report effective independence for cross-family panels. Neither releases the underlying 2 × 2 tables, so the correction can only be projected from the summary φ̄ under an assumed common marginal. On our own data that projection sits within 1% of the honest pairwise recomputation, which suggests it is informative about magnitude, though it remains a projection rather than a recomputation.

The direction is uniform. A nine-judge panel reported as carrying “two effective votes” carries closer to one and a half; a twenty-four model pool reported at 2.65 sits nearer 1.78. Every downstream quantity derived from these — ensemble ceilings, expected gains from adding a model, cost-benefit arguments for panel size — inherits the same optimistic bias.

A single number for “how correlated are model errors” does not exist. Drawing on the same archive of models and changing only the benchmark moves tetrachoric ρ from 0.350 to 0.967 — effective independence by a factor of 2.6. (Pools overlap heavily but are not identical across benchmarks; the decisive ARC–MMLU comparison below is on one common pool of 24 models.)

A format explanation, and its refutation

The obvious reading of three benchmarks was that multiple choice supplies shared distractors which pull every model the same way, while free generation offers nothing to be pulled toward. Three benchmarks differed simultaneously in format, guessing floor, subject matter and difficulty, so three points could not separate four explanations.

MMLU supplies the control: 57 subjects, all four-option, all with a 0.25 floor, varying only in content. Pooled across all 14 042 items it returns ρ = 0.373 — adjacent to GSM8K at 0.350, not to ARC at 0.865. On a single common pool of 24 models evaluated on every task, the gap persists: ARC 0.865 against MMLU 0.474. Format does not explain the ordering.

Two further explanations were pre-registered and also fail. Restriction of range — a single subject being more homogeneous than a mixed benchmark — predicts that pooling the 57 subjects should raise ρ; it does not (0.373 pooled against a subject median of 0.362). Sample size predicts that small subjects bias ρ; slicing MMLU and ARC into random chunks of equal size leaves both unmoved (0.375 and 0.866), so few items add noise but no bias.

Within MMLU the subjects hold together tightly: interquartile range 0.058, and a between-subject standard deviation of 0.074 which falls to 0.068 once bootstrap noise is removed — against 0.239 across tasks on the same pool. Subjects scored the same way cluster roughly three times more tightly than tasks cluster with one another.

• Tetrachoric carries its own assumption. It presumes a bivariate normal latent — exactly the assumption that makes n_eff meaningful, so the estimator is internally consistent, but it is not assumption-free. Where the latent is plainly not normal, as under a guessing floor, our figures are upper bounds with unknown bias.
• The corrected figures are still likely optimistic. Chen (arXiv:2606.27288) measures the all-models-wrong tail at ~2.5× what even a correctly tetrachoric-calibrated single-factor model predicts — a common-mode component that pairwise correlation of any kind cannot represent. The direction is one-way: the truth sits at or below our corrected n_eff, not above it.
• Kish's design effect was derived for means of continuous outcomes. Applying it to binary correctness is a further approximation, inherited from the practice being corrected rather than introduced here.
• Published numbers are projected, not recomputed. Without the original 2 × 2 tables the correction assumes a common marginal; measured projection bias on our own data was −1%, but that is one regime.
• Pool composition. Open models from 2023–2024 up to 72B. No frontier closed models appear in the archive, and effective independence among stronger, more recent models is not measured here.
• Stratifying by item difficulty does not fix anything. We tried it. In this model item difficulty is the shared component; conditioning on a leave-out estimate of it removes the quantity being measured, driving recovered ρ from 0.51 to 0.04 on data with a known ρ of 0.50, and negative on real data through collider bias. Unstratified tetrachoric is the correct estimator.
• ρ bounds voting; it does not measure how alike the models are. The same ρ would arise among any independent solvers that share one ordering of the items by difficulty, so n_eff is a property of the pool and the benchmark together and says nothing by itself about shared architecture or training data. Removing one common factor and comparing residual correlation within and between model families gives a family excess of 0.02 on ARC and 0.07–0.09 on MMLU-Pro and MATH-Hard (permutation p < 0.01 on all three); the rest cannot be split into item difficulty and a common blind spot from correctness matrices alone. A one-dimensional item-response model says the same in standard language: a 2PL fitted to the same matrices with models as persons (ability per model, difficulty and discrimination per item) leaves the residual correlation between models of different families at the null level, −0.02 to −0.04 where −1/(k−1) is expected, and within a family at +0.03 to +0.09; the within-minus-between excess is +0.06 to +0.11 with permutation p < 0.001 on MMLU-Pro, MATH-Hard, MATH-500 and ARC. The statistic is the person-side analogue of Yen's Q3 (Yen 1984; critical values in Christensen, Makransky & Horton 2017), and tinyBenchmarks (arXiv:2402.14992) shows that such a model fits leaderboard matrices well enough to predict a new model's score from 100 items. So the ρ reported above is, to first order, the difficulty axis; what the families add on top is real and small. Added October 11, 2026.

Everything here runs on public data with no model inference. The archived leaderboard is ungated; the current leaderboard requires a token and is not needed. Code: github.com/Alatip/tech-note-1; per-instance correctness downloads on first run and is cached locally.

The tetrachoric implementation is self-contained: the bivariate normal CDF is integrated numerically and the inverse normal uses Acklam's approximation, so the result depends on no external numerical library and every step is auditable. Per-model, per-item correctness is cached after a single download.

Practical notes for anyone rebuilding the matrix: some archived runs are truncated to a handful of rows; the correctness column appears variously as acc, acc_norm, mc1, or a metrics struct, and occasionally as the strings 'True'/'False'; and the prompt field is formatted differently across runs, so hashing it does not align models. Align positionally and verify the question text matches, or the matrix will be silently joined out of register.

Archived with a DOI: 10.5281/zenodo.23132592. Code release v1.0: 10.5281/zenodo.23133109. Cite as: Latipov, A. (2026). Effective Independence. Zenodo. https://doi.org/10.5281/zenodo.23132592