What Is Hard for LLMs in Our Problem Sets: Interim Results

Every earlier note ended up at the same place: what matters happens on the hard problems. In Where the Swarm Pays, on the problems all seven models got wrong, the gold label, the question or the grader was at fault on 45 of 48. A panel of models carries fewer independent votes than its size suggests (Effective Independence), and the same holds for agents with tools (Agents with Tools Fail Together). Decomposition buys budget at best, so it cannot help where a model runs out of ideas rather than tokens (Decomposition Buys Budget at Best, Nothing Usually). There Is No Free Lunch After All states, before any measurement, a claim about where the boundary of what a model can solve lies. All of these lean on "the hard problems" of our sets as if that were a well-defined thing. This log checks what that set contains for current models: how much is real difficulty, how much is broken items, and where today's strongest models stop.

The earlier notes lean on a "single difficulty factor": items are hard or easy for every model at once, which is why model errors are so correlated. A natural follow-up is whether the models also line up on one scale. Is the best model always the best, in voting, in judging, in solving, on every benchmark?

On fitted IRT abilities over the public matrices of Chen (2026; 52–67 models), the order is stable inside one domain and moves across domains:

Across the three Chen sets the median model moves 9 places of 52 between its best and worst set, and 22 models move more than 10. Claude Opus 4.8 is 17th on MMLU-Pro and 1st on MATH-500. Role reorders models more than the benchmark does. In our own corpus Haiku 4.5 is 5th–6th of 7 as a solver and 3rd of 6 as a judge picking the right candidate. Opus 5.5, added as a sixth judge, picked the right candidate on 0.968 of 62 contested items, against 0.919 for Sonnet 5.5 and 0.66–0.69 for DeepSeek V3.2, Qwen 3.7 Plus and GLM 5.3 Flash.

We took the hardest 5% of each Chen set by solve rate: 53 items, out of 1,010 with at least 20 models each. Opus 5.5 and Fable 5.1 audited them independently. Each solved the item without seeing its solve rate, then judged the gold, the wording and the answer format. They agreed that an item is or is not a defect on 44 of 51. Both flagged a wrong gold in, for example, an arithmetic-series item whose gold has the wrong sign (−3, not 3; Claude re-derived it in a two-line computation).

The 13 unlabelled items in the hard tiers are all AIME problems. Counting them as genuinely hard, 14–15 of the 37 hard items (38–41%) are defects. The two items nobody solved are both defects.

Defects also leave a trace in who solves them. For each item we correlated "this model solved it" with the model's overall accuracy:

A permutation test puts the gap between genuine and defect items at +0.24 (p = 0.003). It holds at matched solve rates: with ≤ 10% solving, +0.14 against −0.05 (p = 0.02). Hard items are solved, rarely, by strong models. Defects are "solved" by whoever happens to land on the gold. That gives a label-free detector: an item almost nobody solves, whose solving is unrelated to model strength, is almost certainly broken. The detector was found post hoc and still needs a pre-registered test on another matrix.

Removing the defects makes the picture cleaner, but less than it looks. Against a control that drops equally many clean items at matched solve rate, the error correlation falls by 0.02–0.08, with every 95% interval including zero. The gain of majority voting over the best model does not move (±0.005).

Next, sets with verifiable answers and dates after the models' training: final-answer olympiads 2025–26 from MathArena (AIME, HMMT, BRUMO, SMT, CMIMC, APEX), the public AIMO reference problems, Reasoning Gym at its "hard" presets, and Project Euler 943–992. Every item went through a cascade, one attempt per step and the next model only on failure: Haiku 5.5 → Opus 5.5 → Fable 5.1. Answers came in three conditions: no tools (A), a calculator (B), and sandboxed Python (C, Euler only). All runs used subscription Claude via the CLI with effort max.

Each item no one solved, in every set, was broken:

• cmimc_2025:7. All five of our attempts answer 932, and the gold is 930. A short exact computation, written and run by Claude, gives 932. MathArena had scored 27% of its models (42% of its top 10) correct, for answering 930.
• aimo2:192e23. All six attempts give the exact count, 4046·4045!!, and the gold 250 is that number mod 1000 (computed the same way). The AIMO rules ask for answers mod 1000, but the problem text does not.
• apex_shortlist:35 is the one honest miss: unsolved without tools (Opus 28, gold 31), solved by Haiku with the calculator.

The calculator helped Haiku on 9 MathArena items and hurt it on none. Part of that is not arithmetic. Without tools, 13 Haiku attempts ran into the 128k-token ceiling on one response. With the calculator the model thinks in pieces between calls, and only 7 did. On Reasoning Gym's hard presets nothing differed by task or by calculator: every task was 9–10 of 10 for Haiku. Raising the generators' parameters until Haiku drops to about half takes 9-word cryptarithms, bitwise difficulty 8, or 12–15-person knights-and-knaves. There most misses are again the 128k ceiling, not wrong answers. Two generators cannot go harder at all: zebra puzzles stop at 7 people and futoshiki at 9×9.

Project Euler has no published answers, so an answer counted when two different models agreed. Haiku and Opus agreed on 46 of 50, and on the other 4 Fable sided with Opus. With Python available these are not a reasoning test, so the no-tool stage was skipped.

MathArena also builds final-answer problems out of new arXiv papers. Its best models (GPT-6.x, Fable 5.1, Opus 5.5) solve 82–96% of each month, so we took the hard tail of June, August and September 2026: 51 items where fewer than half of MathArena's models were correct. The cascade went Haiku A→B→C → Opus A→B→C → Fable A→B→C.

The run was stopped after 7 items. Two are solved, both by Opus: a problem on rainbow cycles in triangle-free graphs, without tools, and one on a dimension threshold for Batyrev's non-negativity conjecture, with Python. The other five resisted every model and condition tried. Their failures include Haiku spending 11M tokens with the calculator to answer "+∞", Opus spending 3.2M tokens with Python to a wrong function, and Fable answering that the exact value is not established. Four of the five were solved by no MathArena model either. Before calling them hard they need the defect check from section 2.

To measure reasoning depth without defects or contamination we wrote a generator. It has 8 people, made-up property words, facts, and rules of the form "Everyone who is A and B is C". A goal property is reachable by exactly one person in exactly d rule applications (depth). Each chain rule has w premises (width), the extra ones being facts that must be matched to the person. The other seven people are near misses, each lacking one fact in the second half of the chain. There are 2d noise rules and dead decoy branches. Every item is checked by forward chaining. The answer is a name, so a guess is right one time in eight. No tools are given: with Python this is a 20-line script.

Neither Claude model gave a wrong name anywhere. Output grows with depth and faster with width. For Opus it is 3k tokens at d10 w2, ~90k at d70 w5, and ~106k at d100 w8, close to the 128k ceiling on one answer. Haiku writes 3–5 times more than Opus on the same item. We stopped the sweep there: 100 steps while matching seven facts per step across eight candidates is already well beyond what a person does in their head.

Qwen3.8-27B breaks between 20 and 30 steps. A control at d30 with width 2 fails as often as width 3 does, so depth is what breaks it, and it breaks by running out of output or looping, never by naming the wrong person. Models released around the same time, run on the same items within a $5 cap, split into two styles of failure:

Some models walk the chain until the output runs out. Others answer after a few thousand tokens and pick a plausible near miss; DeepSeek V4 Pro once wrote that a person "appears uniquely satisfying all conditions in the intended solution". With three items per point these are impressions, not rates.

• The 128k ceiling is per response and comes from the API. Raising CLAUDE_CODE_MAX_OUTPUT_TOKENS in Claude Code does not lift it. Without tools it bounds every attempt, whatever budget is set.
• Thinking can be saved, as a summary. A flag missing from claude --help, --thinking-display summarized, with --output-format stream-json --verbose, returns a summary of the thinking for Haiku 5.5, Opus 5.5 and Fable 5.1. The raw thinking is not available (allowed values: summarized, omitted, highlights).
• Headless runs saw the account's connectors. In the first smoke test one model reported a tool result asking it to create a document. From then on every run used --strict-mcp-config --safe-mode, and the contaminated attempts were discarded.
• Graders lie in one direction. Five classes of correct answers were first marked wrong: π read as a symbol, prefixes like C= and f(n)=, \dbinom, and \sqrt5 without braces. All flips went from wrong to right, and Claude re-checked every one.
• Safety filters fire on harmless long problems. Opus 5.5 and Fable 5.1 refused a handful of made-up-word puzzles and arXiv problems as policy violations. These are recorded as errors and are not scored.

For the strongest current models, "hard" in public benchmarks is first a question of data quality. The bottom of a solve-rate ranking collects broken items, and the models' disagreement with the gold there tells you more about the gold. Fresh competition mathematics, even its hardest tail, is within reach of Haiku plus Opus at one attempt each.

For long deductive chains the binding constraint is how much a model may write, not whether each step is right. Writing the reasoning out turns depth into length. What separates models is whether they keep walking or start guessing, and how many tokens a step costs them.

The problems that resisted are research-level ones built from new papers. That is where the next measurement belongs, together with the defect check, since four of those five items were solved by no model at all.

• A log, not a study. Individual runs had pre-registrations (each folder has PREREG.md with predictions and logged deviations), but the sequence was decided as results came in. The defect detector was found post hoc.
• Small cells. The deduction runs have 3 items per point and the cascade sets 30–60 items. Only the hard-tail correlation has intervals.
• Labels are model audits. Defect calls come from Opus and Fable. Where possible Claude checked them with a short computation (cmimc_2025:7, aimo2:192e23, the arithmetic series). No item was checked by a person.
• Effort max, one attempt, subscription CLI. No sampling variance was measured, and the 128k ceiling applied to all Claude runs.
• Project Euler answers are agreements, not checks. They were not submitted to projecteuler.net.
• The arXiv run is partial (7 of 51), and the Sonnet 5.5 comparison was started and stopped.

Matrices: Chen 2026, HF josefchen/co-failure-67-models (CC-BY 4.0). Competition and arXiv problems with model outputs: HF MathArena/*. AIMO reference problems: HF mirrors of the Kaggle sets. Procedural tasks: Reasoning Gym 0.1.25. Runs, prompts, raw answers with thinking summaries, and scripts are in the swarm-lbd repository under experiments/ (difficulty-components, matharena-hard, aimo-public, procedural-rg, procedural-hard, euler, arxivmath-hard, deduction-depth).

Earlier notes: Where the Swarm Pays, Agents with Tools Fail Together, Decomposition Buys Budget at Best, Nothing Usually.

← notes · github.com/Alatip