Technical notes
Senior software engineer working on AI security. Notes on measurement, ensembles of language models, and what their failure modes mean for the agents built on them.
Authorship and AI-generation disclosure
The research program behind these notes is mine: the questions, the hypotheses and pre-registered predictions, the design of each experiment, the reading of the results and the conclusions drawn from them.
I work with Anthropic's Claude models (via Claude Code) as a research assistant. They draft most of the prose and write most of the code and analysis scripts; I direct the work, rewrite and edit text myself, reject what does not hold, send numbers back for recomputation and independent re-checks, and decide what gets published. Every note goes out under my name and I am responsible for it.
This records how the notes are made; it does not imply that Anthropic endorses them. Most notes link their code and data where relevant.
19 October 2026
The first paper of the series split out of the working draft, as one clean study: voting, routing and judge panels over a pool of language models, measured by the share of the gap between the best member and the oracle they collect, on 941 re-audited problems, public correctness matrices and the SWE-bench leaderboard. Voting and routing collect nothing over a best member chosen on held-out data; where operators do beat the best member, the gain is usually insurance against a member that gives no answer. Every number is reproduced by released code. · read · pdf, doi · code, doi
13 October 2026
A selector with an external check is the one operator that can reach the oracle on agent tasks, and this note measures what an imperfect check collects: a formula with false-accept and false-reject rates per task, checked on a released critic (R2E-Gym, 500 SWE-bench Verified bugs) and on pre-registered runs of a Claude Haiku judge and two test writers over the same 1,600 Qwen3-Coder candidates. Model verifiers err by rejecting correct patches on hard tasks; model-written tests accept 39–69% of wrong patches even after reading the code. Tests as a gate before a ranking judge collect 0.28 of the prize at eight candidates. · read · full text, doi
12 October 2026
The ten notes assembled into one working paper with one yardstick: for each aggregation operator, voting, repeated sampling, debate, synthesis, judges, routing, decomposition, how much of the distance from the best member to the oracle it collects and when. Operators that only rearrange answers collect nothing, at 2 points of room or 32; operators that add computation collect a few points under named conditions. PDF with 87 references on Zenodo. · read · pdf, doi
11 October 2026
175 software agents, each a model wrapped in its own tools and workflow, scored on the same 500 SWE-bench Verified bugs: the seven best carry 1.4 independent votes, their majority ties the best of them, and only a perfect selector would gain ten points. One scaffold over 41 models and one model over 11 scaffolds give the same shape. On τ-bench logs one agent rerun four times is worth 1.4–1.9 runs, and four runs of the best model cover as much as four models once. Pre-registered, zero inference, public results only. doi
11 October 2026
Five computer-use agents from four vendors on 150 OS-Harm tasks, pre-registered, at zero cost. When the user asks for harm, the agents that comply are mostly the same ones (tetrachoric ρ 0.67, about 1.4 effective votes of five); when a web page asks for harm, which agent obeys is close to idiosyncratic (ρ 0.28, no task breaks all five). On the same tasks safety breaches are more shared across models than task failures. doi
10 October 2026
Cut a problem into steps and let the same model solve them one at a time, at the cost of one whole attempt. What it could not solve whole it still cannot: on the hard numeric problems none is gained in eight tries. The gain lands only on the problems where a single pass is cut off by the token cap, and in dollars a split attempt costs three to nine whole draws, so the single model ties or wins at equal spend. Three other models solving the parts cost more and solve no more. Pre-registered, 560 runs, $5.73. code · doi
9 October 2026
Sakana's orchestrators learn in training which model to trust and when to stop, and check nothing when they answer. That lifts them above the best model in their pool, and never above what the pool can do together. A reading of the four papers behind Fugu, released ten days after Fable 5 went offline under US export controls. doi
8 October 2026
One corpus of answer traces, seven reasoning models from five families, 240 hard problems, every cheap way of multiplying agents. The first finding is about the benchmarks: on the problems where all seven models were scored wrong, the gold label, the question or the grader was at fault on 45 of 48, and the "co-failure ceiling" on the problems the models finish is 0.998–1.000. The second is about counting: two of the models hit a token cap on the hard problems, and a cut-off answer scored as a wrong vote had manufactured most of debate's gain. Inside the box, plurality is below the best model on every layer; debate's gain is mostly a second chance for an agent that ran out of tokens, it gains one item on the problems every agent answered, and it ends unanimously wrong on four problems whose labels are unchecked; about half of a homogeneous debate's gain is re-reading one's own answer; personas add no measurable independence; synthesis recovers nothing. On verification the swarm pays a little: five judges picking among candidates lead the solvers' plurality by a margin that depends on the tie rule and is within noise, their strict majority is right 94% of the time where it exists, and peer disagreement flags errors at AUROC 0.98 against 0.93 for fifteen self-resamples. Asked to certify a single candidate they are solvers again, trustworthy only when unanimous. Revised three times the same day, after a label audit, a truncation audit and a reading of the code against the text, with every version kept and comparable; measured, not pre-registered; data and code released. code · doi
4 October 2026
The modal ceiling of test-time scaling says plurality voting cannot beat the rate at which the most common answer is right. On 9,600 public debate traces, three-agent refinement beats the round-0 modal-hit rate on every weak model and corrects 31–45% of problems where all three agents started wrong; the exchangeability premise is what fails. The correlation ceiling does transfer: inter-agent correlation rises every round and the panel is one effective voter by round 3; stopping at unanimity beats the n_eff-plateau rule. Pre-registered, zero inference cost. code · doi
2 October 2026
Kim 2026 finds that diversity metrics barely predict majority-vote gain once accuracy is controlled for. On four public correctness matrices a heterogeneous Condorcet model with no dependence term predicts the same gain at Spearman 0.80 against 0.20 for the residualized metric; the tetrachoric dependence estimate adds 0.01. Gain over the best member is a nonlinear function of the accuracies that a linear control throws away. Pre-registered, zero inference cost. code · doi
8 September 2026
Notes before writing down a theorem. The boundary of what a model can solve is the closure of its training set under composition of bounded depth; swarms, chains, debate and short RLVR move mass inside it, only external data extends it. The claim, why it is nearly a tautology, where the content is, what threatens it, and the plan to test it. Nothing measured yet. doi
31 August 2026
Panels of language models carry fewer independent votes than the literature reports, and the gap is an estimator artefact rather than a disagreement about data: φ where the latent model implies the tetrachoric. Recomputed on per-instance leaderboard data, zero inference cost. code · doi