betaFeedback

WorksComputer Science

Just sort it! A note on the LP behind rank-one anti-loop edits

Abstract

Weight-space programming removes the thinking loops of a small reasoning model by a rank-one edit W' = W + u v^T of an MLP output matrix, where u is a loop direction and v is the solution of a sparse Linear Program (LP) over the neuron activations. We investigate whether the LP is needed. We replace it by a sort: we take the 8 neurons whose activation has the smallest coefficient of variation, and give them equal shares of the push that the LP solution realizes. On Qwen3.5-0.8B the sorted edit scores 67.0% on 1000 problems (MATH-500 and GSM8K) against 67.4% for the LP edit and 9.5% for the unedited model; the paired difference is -0.4 points, with 95% confidence interval [-2.9, +2.1], inside a non-inferiority margin of 5 points declared before the runs. On three further sampling seeds the difference is -0.6 points, interval [-2.0, +0.8], and -0.8 points, interval [-2.3, +0.6], when the dose is set without the LP. The sort does not solve the LP, but the neurons it shares with the LP solution carry most of its mean push. The outcome is that, on this model, the edit acts as a nearly constant bias along u.

The LP

In AHA [1] the edit is applied to the down_proj matrix of two layers of Qwen3.5-0.8B [3]. Vector uu comes from matched pairs of looping and productive continuations of the same prefix. Vector vv comes from the LP of [2]: let hth_t be the activation vector at the input of down_proj on token tt; then

min⁡∑j∣vj∣+C∑tsts.t.vTht≤−δ+stfor the looping tokens,∣vTht∣≤ε+stfor the productive tokens,∣vj∣≤10,s≥0,\begin{aligned} \min\quad&\sum_j |v_j| + C\sum_t s_t \\ \text{s.t.}\quad&v^{\mathsf{T}}h_t \le-\delta+ s_t && \text{for the looping tokens,} \\ &|v^{\mathsf{T}}h_t| \le\varepsilon+ s_t && \text{for the productive tokens,} \\ &|v_j| \le10,\quad s \ge0, \end{aligned}

followed by a restriction to the 8 neurons with largest ∣vj∣std⁡(hj)|v_j|\operatorname{std}(h_j) and by a second solve.

The LPs are not small, and they are dense. In our run of AHA on Qwen3.5-0.8B, the LP of each layer has 12 000 constraint rows (4000 looping and 8000 productive tokens, the latter with a two-sided constraint), 3584 variables vjv_j (the width of the MLP) plus one slack per row, and a dense constraint matrix with about 43 million nonzeros. It is solved twice per layer, on all neurons and then on the 8 selected ones, for each of the 20 layers on which AHA solves it. Note that the optimum of the first solve is very sparse: between 6 and 189 nonzero vjv_j out of 3584. With the GPU implementation of the primal-dual hybrid gradient method (PDHG) shipped with AHA, the search of a layer, essentially the two solves, takes 3 to 11 seconds, about 2.2 minutes over the 20 layers, on one NVIDIA RTX PRO 6000 Blackwell. By contrast, the sort of Section 2 needs two moments per neuron, i.e., one pass over the activations of the fit tokens.

At first glance, this is a separation model: push the looping tokens, leave the productive ones alone. However, the solutions do not separate. On our fit rows the LP edit pushes the looping tokens by −0.79-0.79 and the productive ones by −0.70-0.70 at layer 15 (−0.89-0.89 and −0.83-0.83 at layer 19), with a coefficient of variation of the push of 0.210.21 (0.120.12). The public sweep of [2] tells the same story: multiplying δ\delta by 2 or by 8 leaves the accuracy unchanged, whereas ε=0.25\varepsilon= 0.25 destroys it. Hence the LP synthesizes a bias along uu, by using neurons that are almost always active.

The sort

If a bias is all one needs, the best neurons are those whose activation is as constant as possible. For each neuron jj we compute the mean mjm_j and the standard deviation σj\sigma_j of hjh_j on the fit rows, with the class weights of the LP. We call sort_k8 the edit defined by the k=8k = 8 neurons with smallest σj/∣mj∣\sigma_j/|m_j| and by the weights vj:=c/(kmj)v_j := c/(k m_j), where the dose cc is the mean push of the LP solution on the same rows. Note that the dose is borrowed from the LP on purpose, so that the two edits differ in the neurons and not in the dose; the window of useful doses is known to be narrow [2]. As a control we also consider knap_k8, that takes the 8 neurons with largest ∣mj∣|m_j| (the continuous knapsack rule, variance ignored). No solve is involved: two moments per neuron and a sort.

Results

We ran the unmodified AHA pipeline (commit e6e94ff5) on Qwen3.5-0.8B with layers 15 and 19 fixed, on a single NVIDIA RTX PRO 6000 Blackwell, with one sample per problem. The protocol, the arms and the verdict rule were committed before any number was produced.

armMATH-500GSM8K testpooleddiff95% CI
base6.612.49.5−57.9-57.9[−61.0,−54.8][-61.0, -54.8]
lp61.473.467.4
sort_k859.874.267.0−0.4-0.4[−2.9,+2.1][-2.9, +2.1]
knap_k857.475.066.2−1.2-1.2[−3.7,+1.3][-3.7, +1.3]

Table 1. Accuracy in %, seed 0. “Diff” is the paired difference with respect to lp on the 1000 problems; the 95% confidence interval comes from the discordant pairs.

Table 1 reports the outcome on MATH-500 [4] and on the first 500 problems of the GSM8K test set [5]. sort_k8 and lp disagree on 166 problems, 81 won by the sort and 85 by the LP. The knapsack-like rule knap_k8 is also within the margin, and the difference between the two sorts is not significant (+0.8+0.8 points for sort_k8, interval [−1.6,+3.2][-1.6, +3.2]). Hence, on the task, we cannot tell the coefficient of variation from the mean as a sorting key; the structural measures of Table 2 do separate them.

As to the structure, the sort is not the LP optimum: its hinge value is about 5 times larger, and at layer 19 it satisfies the loop constraint on 76% of the rows against 91%. Note, however, that the neurons shared by the two supports carry 95% (layer 15) and 88% (layer 19) of the mean push ∑jvjmj\sum_j v_jm_j of the LP solution. The other LP neurons contribute little to the mean push, although some of them have a large weight (neuron 77 at layer 19 has ∣vj∣=0.53|v_j| = 0.53): they shape the hinge term, not the dose. Interestingly, this correction does not show up in the accuracy.

Sampling seeds

After seeing the outcome of seed 0 we declared a second protocol, again before the runs: the same edits, rebuilt from the stored uu and vv, evaluated with three new sampling seeds. Note that the seeds measure the sampling noise on fixed edits; they do not replicate the pipeline that produces uu and the LP solution. The second protocol adds one arm, sortd_k8, with the neurons and the shares of sort_k8 but with dose c:=−δc := -\delta, where δ\delta is computed by AHA from uu before any LP is solved. Hence sortd_k8 takes nothing from the LP solution. For every problem we average the correctness over the three seeds, and the standard error is computed over the 1000 problems.

layerarmloopprodCVhithingecommon
15lp−0.789-0.789−0.701-0.7010.2120.2120.9910.9910.0050.005
15sort_k8−0.768-0.768−0.729-0.7290.1480.1480.9740.9740.0240.0243
15knap_k8−0.950-0.950−0.659-0.6590.4850.4850.8450.8450.5230.5234
19lp−0.890-0.890−0.834-0.8340.1230.1230.9100.9100.0620.062
19sort_k8−0.882-0.882−0.864-0.8640.1590.1590.7630.7630.3190.3195
19knap_k8−1.062-1.062−0.763-0.7630.3840.3840.8100.8100.7240.7244

Table 2. The three vectors vv on the fit rows. “Loop” and “prod” are the mean pushes v⊤hv^{\top}h on the two classes, “CV” is the coefficient of variation of the push, “hit” is the fraction of looping rows with v⊤h≤−δv^{\top}h \le-\delta, “hinge” is the mean LP slack per row, “common” counts the neurons shared with the LP support (out of 8).

Table 3 confirms the outcome of seed 0, with an interval about half as wide: both sort_k8 and sortd_k8, whose dose does not come from the LP, are well inside the margin of 5 points. The two sorting keys remain indistinguishable: sort_k8 minus knap_k8 is +0.3 points, interval [−1.2-1.2, +1.8+1.8]. In a comparison not declared in the protocol, sortd_k8 splits by benchmark: on MATH-500 it is 2.1 points below lp (interval [−4.2-4.2, 0.00.0]) and 2.3 points below sort_k8 (interval [−4.3-4.3, −0.3-0.3]), whereas on GSM8K it is above both. A smaller dose seems to cost on the harder problems.

armMATH-500GSM8K testpooleddiff95% CI
lp59.974.367.1
sort_k860.272.766.5−0.6-0.6[−2.0-2.0, +0.8+0.8]
sortd_k857.974.766.3−0.8-0.8[−2.3-2.3, +0.6+0.6]
knap_k858.573.966.2−0.9-0.9[−2.3-2.3, +0.5+0.5]

Table 3. Seeds 1, 2 and 3. Accuracy in %, averaged over the seeds. “Diff” is the paired difference with respect to lp, with the 95% confidence interval clustered by problem.

We did not test a constant shift along uu at the same dose. On a development set of 128 problems, the sweep of [2] reports 57.8% for such a steering against 62.5% for the LP edit, a gap within the noise of 128 problems but in the direction of the LP; hence “nearly constant” is the right word, and a steering arm is the natural next control.

Conclusions

We have shown that, on Qwen3.5-0.8B, a sort of the neurons by coefficient of variation matches the LP of weight-space programming over three sampling seeds (difference −0.6-0.6 points, interval [−2.0-2.0, +0.8+0.8]), without any solver. The dose, a single scalar per layer, can be taken from the threshold δ\delta that AHA computes before the LP. Admittedly, we tested one model and one run of the pipeline that produces uu. In our view the informative part of the AHA edit is the direction uu, not the LP that implements it.

Role of AI

This note was produced with an AI assistant, and we prefer to state the division of labor plainly. The idea is the author’s: the LP of [2] looked like a knapsack in disguise, hence its solution might be replaced by a sort. The author also supervised the work, decided what to run and what to publish, and takes responsibility for the content. Everything else was done by Claude (Anthropic; models Fable 5.2 and Opus 5.5, used within Claude Code): reading the material of AHA and of [2], the choice of the sorting key and of the dose, the two protocols, the code, the runs on the cluster, the statistical analysis, and the text of this note, including this section. The whole work took one day, 16 November 2026.

Data and code. The two pre-registered protocols, the code, the vectors uu and vv of all arms and the per-problem results are available, with this note, at https://doi.org/10.5281/zenodo.24286137.

References

  1. [1]A. Khamis, R. Tutunov, H. Bou Ammar. Who needs an LLM to program an LLM? Blog post, November 2026, https://haithamb.github.io/aha/; code https://github.com/Eppy-EpHard/AHA.
  2. [2]N. Barry. Can a linear program do weight-space programming? Blog post, https://nathan.rs/posts/weight-space-lp/; code https://github.com/nathanrs/weight-space-lp.
  3. [3]Qwen Team. Qwen3.5-0.8B, model card, https://huggingface.co/Qwen/Qwen3.5-0.8B.
  4. [4]H. Lightman et al. Let’s verify step by step. ICLR 2024.arxiv.org/abs/2305.20050
  5. [5]K. Cobbe et al. Training verifiers to solve math word problems. arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168

Paper details

vlnc:W_z3skh65c
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents