Selection operators in language-model ensembles: how much of the gap to the oracle they collect

Abstract Voting, routing and judge panels over a pool of language models return an answer a member already gave. The share of problems solved by at least one member (the oracle) is a ceiling, and the gap between the oracle and the best member's accuracy (the prize) is all a selection operator can add. We measure the share operators collect on a 941-problem corpus of four benchmarks (MMLU-Pro, GPQA Diamond, GSM-Hard, olympiad mathematics) answered by seven models from five vendors, its subsets without frontier models, two public correctness matrices, and the SWE-bench Verified leaderboard, with prizes from 1 to 32 points. Voting is level with or below the best member chosen on held-out data; routing by dataset collects nothing on held-out data; a judge panel in the full pool solves 1.4–3.1 points less than the best member; judge unanimity lets through 8.3% of wrong candidates. Where operators beat the best member, the gain is usually insurance against a member giving no answer; only a judge panel in a pool of three without a strong model collects about a fifth of the prize, which survives no multiplicity correction under the main tie rule. The data are consistent with one shared difficulty factor linking members' errors: it predicts voting accuracy within 0–3.5 points, whereas independent errors promise up to 27 points, never reached. We recommend reporting the oracle and the prize, choosing the best model on held-out data, and separating problems where it gave no answer.