What Is Hard for LLMs, Part 2: Optimization Instead of an Answer (Interim Results)

Part 1 looked for problems hard enough to separate today's strongest models and kept finding the same three things. Problems that almost no model solves are often broken. We audited the 37 problems with the lowest solve rates in a public matrix of 52–67 models. Nine of them (24%) have a confirmed defect: a wrong answer key or ambiguous wording. Between a quarter and 38% of the hardest items are not hard but broken; the upper figure also counts cases still open (details in Part 1, section 2). Recent competition sets, their hardest tail included, fell to Haiku 5.5 and Opus 5.5 at one attempt each; Fable 5.1, last in the cascade, was not needed. A synthetic deduction of 100 steps was solved without error, and the limit in sight was the 128k-token ceiling on one answer. Only research-level problems built from new arXiv papers resisted, and four of the five that did were solved by no MathArena model either, so they still need a defect check.

We read all three as consequences of one format: a problem with a single checkable answer. Such a problem is solved or not. Once the strongest models solve it, it stops ordering them, and the items that remain unsolved are disproportionately those whose answer key is wrong. Harder sets meet the same wall later: they saturate in turn, and each has to be checked for defects before its misses mean anything.

This part tries a different kind of task. The agent gets a working interpreter for a small virtual machine and is asked to make it do less work without changing what it does. The result is a number with no fixed top: how many times fewer machine instructions the interpreter executes on workloads the agent has not seen. Correctness is kept out of that number and checked by a separate gate of hidden tests against a reference interpreter. Any failure sets the score to 0, and the failures are reported beside it. The virtual machine and its interpreter were written for this bench, so they are not in any model's training data. Spending is counted in dollars at official prices, so a run gives a curve of result against budget rather than one point.

This report asks two questions. Does such a task rank models the same way from run to run? And does a larger budget bring fewer errors? The runs so far give partial answers to both: the order of the models changes with the effort setting, and none of the correctness failures from the medium series went away with a larger budget. These rest on three attempts per cell and five longer runs, so both need more runs before they can be stated with confidence. Section 3 lists which parts of this design already exist in other benchmarks.

The task in plain words: Quern is a small virtual machine made up for this bench, and the agent receives a correct but slow interpreter for it, written in C. The agent has to change that interpreter so that it runs Quern programs with less work, while every program still prints exactly the same output and ends with the same exit code. What counts as "less work" is how many processor instructions the interpreter executes, and the result is checked on programs the agent never sees.

Several different things in this bench could each be called a program, and "instruction" could mean two different things, so the words below are used in one sense only:

• Quern: the virtual machine defined by the specification.
• Interpreter: an implementation of Quern in C. The starting interpreter is the one the agent receives; the agent's interpreter is the one it hands in.
• Reference interpreter: an implementation of Quern in Python. Its output and exit code define correct behaviour.
• Quern program: code in Quern's assembly language that an interpreter runs. There are three kinds. A test checks behaviour and has an expected output and exit code. A workload is a longer program whose cost is measured. A random program is generated for fuzzing.
• Opcode: one Quern operation, such as LOAD or LT.
• Machine instructions: the processor instructions an interpreter executes, as counted by cachegrind¹. The score is built from these.
• Gate: the hidden tests and hidden workloads that decide whether the agent's interpreter counts as correct.
• Fuzzing: running the agent's interpreter on many generated random programs and comparing it with the reference interpreter. It is done after each series and does not change the score.

Quern is a dynamically typed stack machine with eight value types (nil, bool, 64-bit int, binary64 float, byte string, array, map and function), 66 opcodes, a binary file format that is checked before it runs, and exact rules for errors, limits and the text of printed numbers. All of it is fixed by an 835-line specification. Quern programs are written in a small assembly language and assembled into bytecode. This one, from the specification, computes a Fibonacci number recursively and prints 6765:

.func fib 1 1
    LOAD 0
    ICONST 2
    LT
    JF rec
    LOAD 0
    RET
rec:
    LOAD 0
    ICONST 1
    SUB
    CALL fib
    LOAD 0
    ICONST 2
    SUB
    CALL fib
    ADD
    RET
.end
.func main 0 0
    ICONST 20
    CALL fib
    PRINTLN
    HALT
.end

Example 2 of the worked examples in section 13 of the specification.

The starting interpreter has 1,268 lines, conforms to the specification and is slow in ordinary ways. It handles one opcode at a time, in a plain loop around one large switch:

for (;;) {
    const Func *fn = &funcs[cur_func];
    const uint8_t *code = image + fn->off;
    uint8_t op = code[pc];
    uint32_t npc = pc + op_size(op);
    switch (op) {
    ...
    case OP_LT: case OP_LE: case OP_GT: case OP_GE: {
        Value a = vs[sp - 2], b = vs[sp - 1];
        int r = 0, nan = 0;
        if (is_num(a) && is_num(b)) r = num_order(a, b, &nan);
        else if (a.tag == T_STR && b.tag == T_STR) r = str_cmp(a, b);
        else die(EXIT_TYPE);
        int res = !nan && (op == OP_LT ? r < 0 : op == OP_LE ? r <= 0 : op == OP_GT ? r > 0 : r >= 0);
        vs[sp - 2] = vbool(res);
        sp--;
        break;
    }
    ...
    case OP_JT:
    case OP_JF: {
        int t = truthy(vs[--sp]);
        if (t == (op == OP_JT)) npc = rd_u32(code + pc + 1);
        break;
    }
    ...
    }
    pc = npc;
}

The starting interpreter (excerpt). It was written by Haiku 5.5 from the specification and the open tests only.

Below are the same opcodes in the interpreter produced by Sonnet 5.5 (effort high, first attempt). The run ended by itself at $4.38, passed the gate, and executes 5.02 times fewer machine instructions on the hidden workloads.

#define NEXT(n) do { ip += (n); goto *ip->h; } while (0)
...
L_LT_LI_JF: {
 Value a = fp[ip->b], b; b.tag = T_INT; b.u.i = (int64_t)(int16_t)ip->c;
 int res;
 if (LIKELY(a.tag == T_INT && b.tag == T_INT)) res = a.u.i < b.u.i;
  else if (a.tag == T_FLOAT && b.tag == T_FLOAT) res = a.u.f < b.u.f;
  else res = cmp_slow(OP_LT, a, b);
 if (!res) { ip += ip->a; goto *ip->h; }
 NEXT(4);
}

The same four opcodes in the agent's interpreter (excerpt from a 4,382-line file).

The starting interpreter handles the four opcodes that begin fib one by one. The agent's interpreter fuses them into one step and takes a short path when the values are integers.

The stand was built so that no part checks itself. Sonnet 5.5 at effort high wrote the specification. A separate Sonnet 5.5 session, given only the specification and never any C, wrote the assembler, the reference interpreter, the tests and the workloads, open and hidden. Writing the reference interpreter turned up nine unclear or contradictory places in the specification; two were corrected before any interpreter in C was written, and the rest are resolved in a list of fixes kept with it. Haiku 5.5 then wrote the starting interpreter from the specification and the open tests alone. Fable 5.1 reviewed the whole stand for critical bugs, and Opus 5.5 reviewed the tool layer and the isolation. Before any data was collected, Fable 5.1 found a way for an interpreter to hide its work from the machine-instruction count by running it in a child process; that is now a rule violation, detected in the source and in the binary.

Correctness is checked twice, by two different means. The gate runs 382 hidden tests, many of them on edge cases the open ones leave out, and 8 hidden workloads, and compares their output byte for byte, and their exit code, with the reference interpreter. It decides the score. After each series every final agent's interpreter also goes through random programs: about 10,000 from a generator frozen, with its hash recorded, before any agent's code was opened (one in eight of them is a deliberately damaged bytecode file, which the interpreter must reject correctly), and about 9,600 from a second generator that walks a grid of opcodes, operand values and the opcodes that produce and consume them. Each random program is run on the agent's interpreter and on the reference interpreter, and their output and exit code are compared; any mismatch is a defect in the agent's interpreter. The starting interpreter is run too, as a check on the generators themselves: it agreed with the reference interpreter on every random program (three damaged files, the same three in every series, on which the reference interpreter ran out of time, were dropped), so a mismatch cannot be blamed on a Quern program whose behaviour the specification leaves undefined. Fuzzing does not change the score, but it once found what the gate missed: the top result of the high series, 8.24, has a real memory error, which the gate passed and 43 of 10,000 random programs exposed (section 6).

The agent works in a Linux container. Its working directory holds the specification, the file vm.c to edit (the starting interpreter), a read-only copy of it, the assembler, a test runner, 101 open tests and 6 open workloads with their expected output, and a measure script that builds the interpreter and counts its machine instructions on the open workloads under cachegrind. The build command is fixed (gcc -std=c11 -O2 -o vm vm.c -lm). The source may not change code generation by pragmas or attributes, and it stays one self-contained file. Computed goto and any other plain C are allowed. The container has internet access, but no agent used it: across all runs there was no web search and no command that fetched anything from the network. The prompt says that the result is judged on hidden tests and hidden workloads of the same kind, and that any wrong result makes the score zero. A run has two phases: the task prompt, and once the agent stops, a fixed "Continue optimizing /work/vm.c. The same task and rules apply."

The agent is Claude Code 2.1.281, run headless (claude -p) on a subscription. Its built-in file and shell tools are switched off and replaced with MCP tools that run inside the run's own container. It has no CLAUDE.md, memory, skills, hooks or other MCP servers, and a preflight check confirms that it cannot read files on the host, directly or through a subagent. The hidden tests and hidden workloads never enter the agent's container. Each checkpoint is measured over a snapshot of the agent's interpreter in a separate container with no network.

Three things set this bench apart from the benchmarks of code optimization we know of, and each comes at a price.

• The code is new. Quern and its interpreter did not exist before this bench, so no model has seen them or a fix for them. The price is that there is one task, not hundreds, and one kind of program².
• The cost is counted, not timed. A count of machine instructions repeats exactly from run to run (the stand checks this on the starting interpreter), so one measurement is enough and no noise filter is needed. The price is that it is not speed: two interpreters with the same count can run at different speeds, and real time is not measured yet.
• Correctness is decided by tests the agent never sees, written by a separate session from the specification alone, and checked again afterwards with random programs. The price is that the hidden set keeps its value only while it stays closed.

One more feature is shared with SWE-Opt: each result is compared with the code the agent started from, not with an expert's patch, so it has no ceiling, and a better agent can simply go further. The price is that the number alone does not show how far a result is from what is achievable. A reference point could be estimated, for example by writing each workload directly in C and counting its machine instructions, but we have not done this yet. SWE-Opt also keeps its performance tests from the agent.

What is not new: an agent changing code under tests and being scored on a speed-up, aggregating per-workload ratios into one mean, and asking how results grow with budget. GSO (in the table below) already compares more steps with more parallel attempts. In our bench, budget is counted differently: as dollars spent so far at official prices, with checkpoints along a single run. Within this shared frame, our bench still differs in the three ways listed above: code no model has seen, a cost that does not need a timer, and a correctness check the agent cannot see.

The closest benchmarks we found all take their tasks from real repositories and pull requests. We searched widely, but there may be others we missed:

The score. On each of the 8 hidden workloads, the machine instructions of the agent's interpreter are divided by those of the starting interpreter. The score is the inverse of the geometric mean of these ratios³: 5.02 means about five times fewer machine instructions. Only workloads with correct output enter the mean, because a workload that crashes early looks cheap.

A failed gate gives 0. If any hidden test or workload gives a wrong output or exit code, or the source breaks a rule, the score is 0. Beside the 0 we still report how many items failed and the same mean over the workloads that ran correctly, because "4.2 with two failures in edge cases" and "nothing works" are different outcomes. That number only describes what broke; it is not a result.

Two more numbers are recorded and do not enter the score: simulated cache misses, and the ratio of the score on the open workloads to the score on the hidden ones, which shows whether an agent fitted what it could see.

Spending. The runs use a subscription, so there is no bill. Spending is counted as tokens times the official price of each model, with input, output, cache reads and cache writes each at its own price. We compute this from the token counts ourselves, because the dollar figures the CLI reports are wrong for two of the four models⁴. Prices, in dollars per million tokens:

Haiku 5.5 has a higher tier for long prompts (input 0.50, output 2.50), applied per request.

Effort and attempts. Claude Code passes an effort setting to the model (--effort low, medium or high), which governs how much the model reasons and so how many tokens it spends. Each model was run at all three settings. Each pair of model and effort has three independent attempts, every one starting from the same starting interpreter in a fresh container. A failure is called systematic for a model when the same hidden tests fail in at least two of its three attempts. Three attempts are few: they are enough to find a mechanism worth looking into, not to estimate how often it happens. A failure of the stand itself, such as Docker or the network, does not count as an attempt; the run is marked and repeated.

How a run goes. A run has two phases: the task prompt, and after the agent stops, the fixed "continue" prompt. Along the way the agent's interpreter is saved at fixed marks of spending ($0.02, $0.03, … $2, then doubling), and each saved copy is put through the gate out of the agent's sight. A run ends in one of two ways: the agent stops on its own in the second phase, or it reaches the cap. The cap is $10 at effort low and medium and $20 at effort high, or 10 million tokens of uncached input and output, whichever comes first.

The stand also has a rule that stops a run once the open workloads stop improving. In the series it fired once, on Sonnet 5.5 at effort medium, first attempt, and that run was resumed until the agent stopped on its own. So every result here comes from a run that ended by itself or at the cap. In the longer continued runs of section 5 the rule is switched off.

A stop is taken only between two steps of the agent, and any spending past the cap is recorded. The score of a run is the score of its last saved copy.

Four Claude models, each at three effort settings (low, medium, high), three attempts each: 36 runs, all on the same starting interpreter and the same images. The series ran in the order medium, high, low, between 3 and 4 November 2026. The cap was $10 for low and medium and $20 for high, so the high column is not directly comparable with the other two.

Score: how many times fewer machine instructions on the hidden workloads. 0 means the gate failed; in brackets, the same mean over the workloads that still ran correctly, for description only. Tokens: uncached input and output. Minutes: working time of the run, without the planned pause in one Opus 5.5 medium run. † Passed the gate, but random programs later found a real memory error in it (section 6).

What can be read from this, with three attempts per cell:

• Haiku 5.5 reaches 3.0–4.1 for under $1.10 and always stops on its own. Effort changes little.
• Sonnet 5.5 depends on effort more than any other model: 1.2–2.4 at low, 4.8–5.0 at high. It also always stops on its own, at high after about $4.40.
• Opus 5.5 stops on its own at low, at about $3 and 4.4–4.6. At medium and high it runs to the cap, and two of its three medium runs fail the gate.
• Fable 5.1 runs to the cap every time, at every effort, so its limit is not visible in this series. When the gate passes it scores 4.4–5.4, and like Opus 5.5 it fails two of three runs at medium.
• The gate passed 12 of 12 runs at low, 7 of 12 at medium and 9 of 12 at high.

The order of the models changes with effort. At low, Fable 5.1 and Opus 5.5 lead and Sonnet 5.5 is last. At high, Sonnet 5.5 comes within about half a point of Fable 5.1 and of Opus 5.5's second-best run, for about a fifth of their spend. With three attempts per cell and different caps, none of these differences has been tested statistically.

Five runs that reached the cap and had failed the gate were later continued with no cap: the same session, model and effort, the same "continue" prompt, and nothing said about the hidden results. Four ran until the agent stopped on its own, at $22 to $64. Opus 5.5 at effort high, attempt 2, was stopped at $40. None of the four medium runs passed the gate at the end:

So the agents kept making the interpreter cheaper, but in the medium runs they did not find their own errors. The one run that ended up correct had broken only at the very end of the series, with an edit the cap cut off before it was checked. Eleven more continuations are in the queue, and they will be reported in a later note.

Read together, the two tables suggest two things.

• Opus 5.5 and Fable 5.1 can go further than Sonnet 5.5, at a much higher price. Their best results that passed the gate are 8.24 (Opus 5.5, with the memory error of section 6), 5.97 (Opus 5.5 at $40), 5.46 (Opus 5.5 at $20) and 5.37 (Fable 5.1 at $20), against 5.02 for Sonnet 5.5 at $4.38. Leaving 8.24 aside, that is a gain of about 7 to 19 percent for 4.5 to 9 times the money. Their broken versions go further still, up to 8.05, which suggests that the extra room is real but harder to take without breaking something.
• An error that survives to the cap is not found later. In all four medium runs that failed the gate, the agent went on for another $12 to $54 and kept improving the score, but the original failure stayed. Why is open: the agents' own tests may never reach these cases, or the agents may stop checking what they already consider done. Telling these apart needs the traces read. This has not been done yet; it will be done in one of the next notes.

Both readings rest on few runs: four continued failures from two models, and one or two passing results per cell at the top.

Every run of the series, and the five continued runs (dashed ring). Colour is the model, shape is the effort; a filled mark passed the gate, a cross failed it. "Without gate" puts a failed run at its descriptive value, as a hollow mark. "Paths" draws each run through its checkpoints; the continued part is dashed. The dotted line at 1 is the starting interpreter. Minutes are working time, without pauses between a stop and a resume. Tokens are budget tokens, uncached input and output: a rough proxy for how long the model worked, not a measure of speed.

One row per model and effort: the attempt with the highest score among those that passed the gate, from the series only. "$ per million tokens" divides the run's whole spend, cache reads and writes included, by its budget tokens (uncached input and output), so it shows how expensive each budget token was for that model. † As in the table above.

Of the 36 runs, 8 failed the gate: one of Haiku 5.5, one of Sonnet 5.5, three of Opus 5.5 and three of Fable 5.1, none at effort low. One more passed the gate with a real defect. Each failure was traced to its cause, all but one down to the line of code. They fall into five kinds:

Three things stand out.

• Most failures come from the same kind of change. The first three kinds are all rewrites or fusions of opcode sequences, which is also where most of the gain comes from (section 2). The agents broke the interpreter in the place where they sped it up.
• Some failures repeat. Two of three Opus 5.5 runs at medium have the float-zero defect, and two of three Fable 5.1 runs at medium have the operand-role defect. The float-zero defect is also the only failure shared by two models: Sonnet 5.5 made it too, in its own code. Each run wrote its own faulty code.
• Neither check proves that no defect is left. The hidden tests reach the float-zero defect in only one program shape: a copy of Sonnet 5.5's interpreter fixed at the one site they reach passes all of them, yet is still wrong at two other sites. The random programs hit the same defect in only one program of 10,000, and missed it in one of the three interpreters that have it. The memory error in the 8.24 was found only by them, at a rate of 43 in 10,000.

The one-character fix of the allocator keeps the 8.24 and passes both the gate and the random programs. By the rules written before the series, the score stays as measured, and it is shown here only with this flag.

Quern is one task. The method behind it is not tied to interpreters: code written for the bench, a cost that is counted rather than timed, and a correctness check the agent cannot see.

In general terms, every task of this kind has the same shape. There is a function F, given as a working implementation and treated as a black box. On a hidden set of inputs X, its correct outputs Y are computed in advance: this is the gate. A generator produces a much larger set of random inputs X′, and for each of them the correct output Y′ is cheap to obtain: this is fuzzing. The agent has to build an F′ that

• keeps the behaviour of F: it gives Y on X and Y′ on X′;
• and is cheaper than F by a cost that is fixed in advance and measured on workloads the agent does not see.

Any F that has these three ingredients, a reliable source of correct outputs, a generator of inputs, and a cost that can be counted, can become a task of this class. The three lines below are three ways to choose F.

The random inputs are there because the set of possible inputs has no end: a Quern program can be of any length, so any fixed set of tests leaves almost all of them unchecked. A generator lets the check reach much further than the tests do, but it still only samples, which is why section 6 found a defect that both checks nearly missed. Where equivalence can be proved for every input at once, by an equivalence checker for circuits or a proof in a system such as Lean, that proof can replace both checks and is stronger than either. For a program of the size of a Quern interpreter such a proof would be a large project of its own; for smaller functions it may be within reach.

The task can be widened along three lines. What follows are ideas and sketches that we expect to work; none of them has been built or tested by us.

Harder within the same class. The virtual machine itself can grow: more opcodes, and harder ones, such as closures, exceptions or garbage collection. This makes the task harder but not different; an agent that speeds up one interpreter will reach for the same tools on the next. The more useful step is to generate virtual machines from a description, a new one for each release of the bench. There would then be many tasks rather than one, and a solution to an old machine published somewhere would not help with a new one.

Other programs with the same counter. The machine-instruction count works for any program that runs on a CPU. Instead of an interpreter, the agent could get a small compiler for a new language and be asked to make it emit code that executes fewer instructions, which is a different kind of work: improving programs through the tool that generates them. It could also get computational routines written for the bench, such as a filter, a hash or a parser for a new format, or a data structure behind a fixed interface, with workloads given as sequences of calls.

Other costs and other machines. Some costs other than instructions are also counted, not timed: peak memory and the number of allocations, or the size of an output, for example a compressor for a new format that must restore its input exactly. Circuits are a further case: the cost of a new design can be counted as cells or area after synthesis, and correctness can be checked by formal equivalence, which is a proof rather than a set of tests and so would close the gap shown in section 6. GPU kernels are the obvious next target and already have a benchmark, KernelBench (Ouyang et al., 2025, arXiv 2502.10517), which measures time; counting the work of a GPU kernel instead of timing it would need a simulator, which is much slower.

Each new class has the same price as Quern: a specification, a reference implementation, hidden tests written without seeing any solution, and a generator of random inputs for fuzzing. When models write these, the specification itself has defects: writing the Quern reference turned up nine.

• Exploratory. The rules of each series were written down before it ran, but the series were planned one after another, medium first, and each plan was made after the previous results were seen.
• Small cells and unequal caps. Three attempts per model and effort. The cap was $10 at low and medium and $20 at high, so effort and budget are partly mixed. No difference between models has been tested statistically.
• A count, not time. The score counts machine instructions under cachegrind on aarch64 in a Linux container. Real running time was not measured, and it can rank two interpreters differently.
• One task. One virtual machine and one starting interpreter, written by Haiku 5.5. A more or less efficient starting interpreter would change every score.
• Written by models. The specification, the reference interpreter, the tests and the workloads were written by Sonnet 5.5, and nine places in the specification needed fixing. The hidden tests cover some behaviours in only one form (section 6), and fuzzing only samples. Neither proves that a passing interpreter is correct.
• Claude evaluated by a stand built with Claude. Only Claude models were run, and Claude models built the stand. A bias in favour of how Claude models work cannot be ruled out until other vendors' models have run.
• Not quite the usual Claude Code. The built-in file and shell tools were replaced by MCP tools running in a container, and the CLI was pinned to version 2.1.281. An agent in its usual setup may behave differently.
• Dollars are notional. All runs used a subscription. Dollars are tokens times official prices, computed by us; nothing was billed per token.
• Traces not read. The check for cheating in section 5 is automatic: rules, isolation, a search of the source and the open-to-hidden ratio. What the agents reasoned and why they missed their own errors has not been read.
• Continuations partial. Five of sixteen are done. One continuation, of Opus 5.5 high 1, was lost to a stand failure (the agent's files outgrew the size limit for saved copies), discarded without being looked at, and is being repeated from the start. The decision to repeat was made when attempt 1 was already known to be the best of its three, so the repeat will be reported apart from the series, where attempt 1 stays as it was.

Part 1 found that problems with a single checkable answer stop separating the strongest models once they solve them, and that the items left unsolved are too often broken. This part tried a task with no fixed top instead.

We proposed a synthetic benchmark with controlled parameters. Quern and its interpreter were written for this bench, so no model has seen them. The cost is a count of machine instructions, which repeats exactly from run to run. Correctness is decided by a gate of hidden tests and hidden workloads against the reference interpreter, and checked again by fuzzing with random programs. Spending is counted in dollars at official prices, so a run gives a curve of score against budget. Because both the correct behaviour and the cost are known exactly, such a task can show whether, and under what conditions, a model does hard work without introducing errors, which on benchmarks built from real code is hard to separate from defects and measurement noise in the benchmark itself.

The measurements are 36 runs: four models at three effort settings, three attempts each, with a cap of $10 at low and medium and $20 at high. Three attempts per cell are few, so every reading below is noisy and none has been tested statistically. We give them as impressions to check with more runs.

Haiku 5.5 barely responds to effort. It reaches 3.0–4.1, passes the gate in 8 of 9 runs and always stops on its own. Its spend grows with effort (a median of $0.49, $0.51 and $0.86) while its score grows little. The starting interpreter was written by Haiku 5.5 itself, which may shift its results in a direction we cannot tell.

Sonnet 5.5 depends on effort more than any other model: 1.2–2.4 at low, 4.8–5.0 at high, where it stops on its own after about $4.40. Its score seems to follow what it spends rather than the setting: the one low attempt that spent as much as the medium runs ($0.39) also scored close to them (2.36, against 2.64–2.92).

Opus 5.5 gives the best results in these runs. At low it stops on its own at about $3 with 4.4–4.6, close to the range Fable 5.1 reaches at any effort (4.4–5.4 when the gate passes) for a third or less of its spend. At medium and high it runs to the cap and goes further.

Fable 5.1 is the most expensive and runs to the cap at every effort, so its limit is not visible here. It spends long before it first changes the interpreter: the first saved change comes at $3.8 to $16 of spend, so in one high run the copy saved at $9 was still the starting interpreter. Its two passing results at high are close to each other (5.33 and 5.37), where Opus 5.5 spreads from 5.46 to 8.24; with two runs each this is only a hint.

Compared with Sonnet 5.5 at high, which scores 5.02 for $4.38, the best results of Opus 5.5 and Fable 5.1 that passed the gate are 5.37 to 5.97, for $20 to $40. That is a gain of about 7 to 19 percent for 4.5 to 9 times the money. The 8.24 of Opus 5.5 is left out here because of its memory error.

Going further costs correctness. Opus 5.5 and Fable 5.1 each failed the gate in 3 of 9 runs, against 1 of 9 for Haiku 5.5 and for Sonnet 5.5, and their broken versions reach up to 8.05. Most failures come from rewriting or fusing opcode sequences, which is also where most of the gain comes from. An error that survives to the cap was not found later: in all four continued runs that had failed, the agent spent another $12 to $54 and kept improving the score, but the original failure stayed. Whether these models would find such errors if asked directly to check their own work is an open question for the next runs.

Quern is one task, but the shape behind it is general. A function F is given as a working implementation; a hidden set of inputs X with correct outputs Y is the gate; a generator produces random inputs X′ whose correct outputs Y′ are cheap to obtain, and that is fuzzing; the agent builds an F′ that matches F on both and is cheaper by a counted cost on workloads it does not see. Within the same class, the virtual machine can grow harder, or virtual machines can be generated from a description, a new one for each release. With the same counter of machine instructions, the agent could get a small compiler for a new language, or routines and data structures written for the bench. Other costs and machines are possible too: peak memory, allocations, output size, or circuits whose cost is counted after synthesis and whose correctness is checked by formal equivalence, a proof rather than a set of tests. All of these are ideas; none has been built or tested by us.

The specification and the setup for running the bench are available on request: write to aziz.latypov@gmail.com.

Archived with a DOI: 10.5281/zenodo.24011976 (all versions: 10.5281/zenodo.24011975). Cite as: Latipov, A. (2026). What Is Hard for LLMs, Part 2: Optimization Instead of an Answer (Interim Results). Zenodo. https://doi.org/10.5281/zenodo.24011976

← notes · github.com/Alatip