Generative operators in language-model ensembles: what they add to their starting point and what they cost

Abstract Self-consistency, debate, answer synthesis and task decomposition promise that extra language-model calls give a better answer than one. We call such operators generative and ask two questions of each: how much it adds to its starting point — a single answer or a vote of the same members — and whether the gain holds at the same cost. All four are measured on the same problems, with one pool of seven previous-generation models and cost in dollars. Hypotheses from set A, the nearly saturated corpus of paper 1, were tested on a preregistered set B of 200 problems. The freedom of a generative operator is barely used: a correct answer that no member gave appeared on 0–2 of 200 problems. At first sight GLM 4.7 Flash self-consistency is the one gain that survives the Holm correction (+19.0 points). A closer look at how this comparison was scored shows that its starting point, the first sample, is often empty; measured instead against the first sample that gave an answer, voting adds only 4.5 points, not significant after correction. So no operator shows a gain over its own starting point that survives correction; the design resolves about 5–6 points, so smaller gains are not excluded. Gains over the vote larger than about 6.5 points are unlikely given the data. Seven-model debate and synthesis add about 2 points to the vote of the same answers, with intervals that include zero, at noticeably higher cost. The answers themselves explain why: each problem has its own chance of success for a model, close to 0 or 1, so repeated samples can change the outcome only on the few problems in between; in debate the members converge (mean member accuracy on the debated problems from 0.564 to 0.773), mostly onto the answer that already led the vote; and part of the vote's advantage is the tie rule: with ties split evenly the vote solves 0.755 instead of 0.770, and debate and synthesis then add +3.5 and +4.0 points, still not a gain that survives correction. Decomposition helped none of the three models; on set A only the length limit made it help. At the same cost five operators beat the best model of the pool, but for debate and synthesis 10 of those 12 points already come from the vote of the members' first answers; GLM voting and GLM decomposition lose to the best model. The problems are almost all mathematical, with a short answer.