Working note · Limits of learning

Repairing One Step in the Weights

A first test of "train the step, not the task" on Qwen3.5-4B: find the step that breaks long multiplication, find where the model does it, change 256 rows of one weight matrix, check on items the edit never saw.

In short Qwen3.5-4B multiplies a 5-digit number by a 4-digit one correctly in 32 of 40 tries. Every failure is in the final sum, and inside the sum the arithmetic is right: the model takes the wrong digit, usually the one a place to the left or right. One attention head, layer 23 head 0, does that lookup by content: each source digit carries a place label, and the head's query asks for a place. On errors the query is blurred between the right place and its neighbour. A rank-16 change to the 256 rows of that head's query weights, fitted on half of the items and judged once on the other half, cut wrong digits from 9.9% to 1.6% on the held-out items, with the other measured skills within the preregistered bounds. That rate counts only column lines with the right number of terms. Counting every line, wrong lines fell from 36% to 13% (61 → 23 of 171), and most lines still wrong drop a term. The step is much improved, not repaired. The gain reaches the answer where the model copies it from the lines: the answer's six lowest digits are right in 86 of 120 held-out cases instead of 60. The task is still mostly failed: even when every line is right, the final answer is right in only 7 of 16, because the model does not build its whole final answer from its own column lines.
Correction, 14 November 2026 Version 1 of this note (12 November) said the step was repaired, that the column lines were "nearly all right", and that 30 of 39 edited answers had all lines right, with 7 of those 30 ending in the right number. Those counts came from a parser that skips column lines with a dropped or extra term. Checking every line, 16 of 39 edited answers have all lines right (3 of 39 before the edit), and 7 of those 16 are correct. The preregistered result (9.9% → 1.6%) is unchanged, but it measures only part of the step. Sections 4 and 6 now give both counts, and section 6 adds the per-digit gain in the answer, which version 1 missed.
1

The idea being tested

In There Is No Free Lunch After All (8 September) I wrote down a claim and a corollary. A task is a chain of steps, and its success is limited by the weakest one. So the useful thing to improve is that step, not the task: "Train the step, not the task." The note planned to find the step by feeding the model a correct prefix up to step t and to fix it with step-targeted training data. It also warned that a model trained on its weakest step may become "brittle one step to the side," so the price has to be measured as well as the gain.

This note is the first measurement. Two things changed from the plan. The repair is a direct, local change to the weights rather than new training data. And the step is found in two stages: first by behaviour (which step fails), then inside the model (which component does that step and how it goes wrong).

The setup is deliberately small: Qwen3.5-4B in bf16 on a laptop, greedy decoding, thinking off, one process with no batching, up to 16,384 new tokens per answer. The main task is 40 products of a 5-digit and a 4-digit number, for example 56544 × 3166.

2

Finding the step

Four small experiments, each narrowing the failure.

experimentresultwhat it says
Arithmetic expressions, no tool vs a calculator tool52/60 vs 60/60failures are in long sums and digit assembly, never in one-digit facts
Each step alone (one-digit multiply, two- to four-number add, subtract)49/50the steps are fine in isolation
Multiplication from scratch; with the correct partial products already written32/40; 40/40all failures are in summing the partial products
Partial products given, sum forced "column by column"6/40column lines: 133 of 339 have a wrong digit, 0 of 339 wrong arithmetic

The last row is the useful one. Asked to add four numbers place by place, the model writes lines whose sums are always right and whose digits often are not. For 96804 × 4798 the partial products are 387,216,000, 67,762,800, 8,712,360 and 774,432, and the model writes "Hundred-thousands: 2 + 7 + 8 + 7 = 25". The hundred-thousands digit of 8,712,360 is 7; the 8 next to it is the millions digit. So the weak step is not adding. It is taking the digit at place k of a number written earlier in the context. Of 746 digits written this way, 89 are wrong; most often the model took the digit one place over in the same number, next most often the same place of another number.

The forced column format is worse than the model's own (6/40 vs 32/40) because it calls this weak step dozens of times per answer. That makes it a good place to study the step and a bad way to solve the task.

3

Where the model takes the digit

Items are split by number: even items for all the looking and fitting, odd items held out. Everything in this section is measured on the even half, by re-running the model on its own text (teacher forcing) and reading the position where it writes a digit.

  • One head does the lookup. Of the 128 attention heads in Qwen3.5-4B's eight full-attention layers, layer 23 head 0 puts the most weight on the source digit: 0.57 on right digits. On wrong ones it splits, 0.28 on the right digit and 0.29 on the one the model then writes.
  • It looks by content, not by distance. Splitting the head's score into its position part (the 64 rotary dimensions) and its content part shows the content part carries 96% of the margin. Each source digit carries a place label; the query asks for a place.
  • On errors the request is blurred. When the model takes the neighbouring place, the content margin between the right place and its neighbour falls from +1.85 to +0.17. The request is written right before layer 23, by the MLPs of layers 20–22; on these errors their contribution turns negative. The place labels on the source digits depend on another head, layer 15 head 1, acting at earlier tokens.
  • The head matters, but is not the only route. Replacing its output with a perfect lookup fixes 28 of 53 wrong digits; pointing it at the neighbour breaks 56 of 103 right ones. So a repair of this head alone could be expected to fix about half of the errors.
Writing the hundred-thousands digit of 8,712,360: the right digit is 7, the original model wrote 8, the edited one 7. Here the head already leaned to the right digit and the model still wrote the wrong one; after the edit the head looks at nothing else.
original model
80.08
, 
70.52
10.17
20.01
, 
30.00
60.00
00.00
after the edit
80.00
, 
71.00
10.00
20.00
, 
30.00
60.00
00.00
0.00.20.40.60.81.0right digitsn = 326original: right 0.52, neighbours 0.29edited: right 0.93, neighbours 0.05wrong digitsn = 37original: right 0.32, neighbours 0.31edited: right 0.81, neighbours 0.10
Figure 1. Top: attention of layer 23 head 0, at the position where the model writes the digit, over the digits of the number it should read from (held-out item m05, teacher-forced on the model's own text; numbers are shares of the head's total attention, the outlined digit is the right one). Bottom: mean attention on the right source digit (solid) and on its neighbours one or two places away (light), on all held-out digit cases, before and after the edit. "Wrong" means wrong in the original model.
4

The edit

Only the query of layer 23 head 0 changes: the 256 rows of q_proj that produce it, as W ← W + UVᵀ with rank 16. Keys, values, and everything else stay as they were.

The edit is fitted to an internal target, not to the output: the head should put its attention on the right source digit at the writing position. A second term keeps the head's attention unchanged on five unrelated texts, on 40 answers from the other experiments, and at random positions (a KL penalty), and a third keeps the edit small. Because nothing before layer 23 changes, the inputs to the head can be cached once from the original model and the fit runs on that cache, exactly, without a forward or backward pass through the model. The numbers were found by gradient descent (Adam, 400 steps); the rank and the weight of the protecting term were chosen on half of the even items, by a rule written down in advance, and the chosen setting was then refit on all of them.

Before any odd item was scored, the fitted edit's hash and four predictions were written into the preregistration.

prediction (held-out odd items)result
Teacher-forced: ≥ 9 wrong digits fixed, ≤ 3 right ones broken28 of 37 fixed, 0 of 326 brokenholds
Regenerated column sums: wrong-digit rate ≤ 6.5% (was 9.9%)1.6% (8 of 495); Fisher p = 4·10⁻⁸holds
Other skills: isolated steps ≥ 47/50, calculator ≥ 58/60, no-tool ≥ 50/60, multiplication ≥ 30/4047, 60, 53, 34holds
Drift on unrelated text no larger than random edits of the same sizeKL 0.00035 vs 0.00030–0.00042holds

The second prediction, as preregistered, counts digits only in column lines with the right number of terms. A line that drops a partial product's digit or adds one is skipped, and after the edit that is the more common error. Counting every line, 61 of 171 held-out lines were wrong before the edit and 23 of 171 after (Fisher p = 2·10⁻⁶); 14 of the 23 drop a term. So the edit fixes most wrong picks but leaves a third of the line errors, and those are of the kind the fit never targeted. Version 1 of this note did not say this.

0%4%8%12%16%held-out (odd) itemsoriginal: 36/363 = 9.9%edited: 8/495 = 1.6%fitting (even) itemsoriginal: 53/383 = 13.8%edited: 7/472 = 1.5%
Figure 2. Share of wrong digits in the column lines, generated from scratch with the forced column format, before and after the edit. Digits are extracted by the same rules from both runs; the edited model also writes fewer malformed lines (68 skipped instead of 121), so its count of digits is larger.

Inside the head the change is large: on held-out digits its attention on the right source goes from 0.52 to 0.93 where the model was already right, and from 0.32 to 0.81 where it was wrong (Figure 1). The fitted edit has a third of the norm of the rows it changes (3.24 against 9.96). It also beats the ceiling the oracle suggested: 76% of held-out wrong digits are fixed, not half. A likely reason is that the oracle replaced the head's output at one position, while the edit changes its query everywhere, including in the model's earlier column lines; this was not tested.

5

A second route: going around the step

Independently of the repair, I tried the other option: avoid the weak step. The model has two ways to add its partial products, column by column (many digit lookups) or pairwise, adding whole numbers two at a time (few). A direction in the residual stream at layer 18 switches the choice; adding it as a constant bias to that layer's MLP output, a weight change too, moved the model from 11 to 35 pairwise sums out of 40 and multiplication from 32 to 37 of 40.

The price was elsewhere. Plain two-number addition and subtraction fell from 20 of 20 to 13 of 20: the model began miscopying 9-digit operands and dropping the lowest digit group in its sums. Its drift on unrelated text was also above random edits of the same size (KL 0.024 against 0.016–0.021). The repair costs less than the bypass on every measure here.

6

What the repair did not fix

Answer accuracy on the forced column format went from 6 to 8 of 40. The column lines are much better but not all right: 22 of 40 answers still contain a wrong line, usually one with a dropped term. Exact match also hides a real gain. The six lowest digits of the final answer, which the model copies from its lines, are right in 164 of 240 cases instead of 122; on the held-out items, 86 of 120 instead of 60, with 30 digits gained and 4 lost. The higher digits were mostly right before and did not change (96 and 97 of 114). The final number is still mostly wrong, partly because of the lines left wrong and partly because the model does not assemble the answer from its lines: in 37 of 40 answers of the original run, the final answer disagrees with the model's own column digits. That is the next weak step, and it is a different one.

Why the task fails although the step mostly works: one answer has about nine column lines and calls the step about twenty times, and errors compound. Dziri et al. made this argument for transformers on multi-digit multiplication (Faith and Fate, 2023); the numbers here fit it.

line rightlines per answerall right, if independentall right, observedanswer correct
original60.1%~90.601⁹ ≈ 1%3 of 396 of 40
edited85.1%~90.851⁹ ≈ 23%16 of 398 of 40

Errors cluster in some items, so more answers come out clean than independence predicts. All 40 items, both halves; a line counts as right when its digits are the true column's, whatever their number. After the edit the chain of lookups survives in 16 of 39 answers, and only 7 of those 16 end with the right number: the task now fails at two links, the lookups still wrong and the assembly of the answer. (Version 1 gave 8 → 30 of 39 and 7 of 30; that count left out lines with a dropped or extra term.)

For the claim of the September note this is the expected picture: a task is a chain, and strengthening one link exposes the next. It also bears on the cases the note distinguished. The digit step was not missing from the model: it failed 10–14% of the time, not always, and a small change was enough to fix it. In the note's words, the edit moved mass onto a step the model already had; it did not create one. A step that is truly absent has not been tested here.

7

Checking the fast check

Teacher forcing (one forward pass over a reference answer) is hundreds of times faster than generating, so it is the obvious way to screen edits. I compared it with generation on 1,209 pairs of model and correct reference answer, for the original model and both edits.

  • When a model agrees with a correct reference at every token, it also solves the item when generating: 81 of 82.
  • "Any wrong digit along the original model's correct answer" flagged all 15 items the two edits lost in generation, but also flagged many items they kept. For the bypass, which changes the method, the check is useless; for the repair it works as a filter before regenerating.
  • A one-pass score does not reproduce token-by-token decoding exactly: near-ties (top two tokens within 0.5 logit) flip from numerical differences alone. All four items the repair lost in generation were such knife-edge items for the original model too. Differences of one or two items between runs should be read with that in mind.
8

Related work

Locating a component and then changing it is an old idea in interpretability. Two recent pieces are closest to what is done here, and both differ in ways that matter.

Weight-space programming (Takehi, Teoh, Zhang, Khattab, MIT CSAIL, blog post, 2 November 2026). A general-purpose LLM reads another network's weights and writes new ones, by definition without an optimizer. The headline edit subtracts one direction, found by the editing model, from 64 output columns of two MLPs in Qwen3.5-0.8B and raises MATH-500 from 7.9% to 48.1%; the diagnosed problem is thinking loops. An independent replication (OpenWSP) reports that turning thinking off alone gives 46.8%, so the gain is mostly loop removal.

Attention Behavior Fine-Tuning (Cho et al., BlackboxNLP 2025, arXiv 2505.14233). Also trains on where heads look instead of on the output: in in-context classification, induction heads are rewarded for attending to the correct label token and penalised for the others. Gradients go to the query and key weights of all heads in all layers.

weight-space programmingABFTthis note
how the failure is foundediting LLM traces a behaviour (looping) to a directiona generic induction-head filterbehavioural step isolation, then one head, then its query side
what changesoutput columns of two MLPsquery and key of all heads, all layers256 query rows of one head, rank 16
how the numbers are foundwritten by the LLM, no optimizergradient through the whole modelgradient on a cached, exact local computation
targetremoving a directionattention on label tokensattention on the source digit, with attention elsewhere held fixed
side effects checkednot reportedaccuracy on other classification setsfour other skills, random-edit control, preregistered bounds

By the MIT definition the edit here is not weight-space programming, since its numbers come from an optimizer. The target, though, is nearly linear in the query, so a closed-form version without any optimizer looks possible; that would make the two directly comparable. Not done yet.

The tooling started from Lyra's neuron explorer for Qwen3.5-4B, which is built around MLP neurons and steering. This work needed attention maps, activation patching and scores over hundreds of cases, so those were added in a small library over the same model and hook points.

9

Limits

  • One model, one task format, 40 items (19 held-out items with 363 digit cases), greedy decoding, thinking off.
  • The repair targets one step of one task; whether the same edit helps the digit lookup in other formats is open (two items lost in isolated steps were digit errors in other formats, both knife-edge).
  • The rank and protection weight were chosen on validation data with only 27 wrong digits, and validation broke 5 of 181 right digits; the held-out test broke none, which may be luck as much as design.
  • Code and data are not public yet.

Archived with a DOI: 10.5281/zenodo.24163236. Cite as: Latipov, A. (2026). Repairing One Step in the Weights. Zenodo. https://doi.org/10.5281/zenodo.24163236