WorksComputer Science
The Frontier Must Grow - Self-proposed, interpreter-labelled training roughly doubles a 4B model's accuracy on a fresh exam as its verified frontier grows
Abstract
A model can learn from problems it poses itself when an executable oracle supplies the answers. Qwen3-4B proposed Lua 5.1 programs; those it predicted wrong at least once in four samples joined a pool labelled by the interpreter, each round adding twice as many programs as the last (300, 900, 2100 in total), and each round trained a fresh adapter from the base on the pool. On the original exam the procedure met its registered criterion, an item-level test of each adapter against its predecessor. On a 2340-item exam drawn afterwards, accuracy rose from 8.3% to 16.7%; that follow-up's registered criteria were not met because round 1 gained nothing, while rounds 2 and 3 added 5.0 and 2.9 points on average, both intervals above zero. One fit on the same amount of base-proposed frontier data reached 14.6%, so the rounds' 2.1-point advantage is inconclusive. Three earlier registered series met no per-lineage criterion, and the same loop showed no gain on repository tasks or for a 27B agent in those configurations.
Introduction
A model that poses its own problems can learn from them if something outside it supplies the correct answers. Here that is an executable oracle: Qwen3-4B writes Lua 5.1 programs about the language manual, and the interpreter says what each prints. This is environment-supervised learning: the model is trained on the interpreter’s targets for problems it gets wrong, including problems it never solves, not reinforced on its own successes, and no stronger model or human answer takes part. Executable feedback for self-improvement is not new; [15][CT24] our contribution is a registered procedure that worked, the evidence that it did and what limits it, and the failure analysis of the series that did not.
The procedure, S4, is pool-growing self-training (Figure 1, detailed in Figure 2). In each round the current champion proposes programs and predicts each four times; those it gets wrong at least once join a pool labelled by the interpreter, and a fresh adapter is trained from the base on the whole pool; each round adds twice as many programs as the last (300, 900 and 2100 in total).[RS25,RS26] The rounds share no parameters; the state that persists is the dataset.

Figure 1. S4 at a glance. Left: one round; rounds share no parameters, only the growing pool. Right: share of a freshly drawn 2340-item exam answered (large marks: means of four lineages; small marks: single lineages), against the interpreter-labelled records trained on: S4’s rounds (300, 900, 2100 cumulative records) and one fit from the base on frontier data the base proposed (900, 2100). The rounds’ 2.1-point lead at 2100 records is inconclusive under the registered analysis (§ 3.4 and § 3.5).

Figure 2. One generation of S4, the series that met its criterion, read clockwise from the champion. The champion proposes programs about the manual and the interpreter labels them; a program joins the frontier when the champion predicts it wrong in at least one of 4 sampled attempts (300, 600 and 1200 new programs at generations 1 to 3); the candidate is a fresh LoRA trained from the base on the union of every generation’s frontier; the learner’s gate decides whether it becomes the next champion, and the examiner, not the gate, decides the registered outcome. Dotted arrows are what each stage refuses.
Outcome of S4 (four lineages, three rounds; § 3, § 3.4 and § 3.5). Fresh exam (2340 items, 78 templates, drawn after S4 ran, not selected by the base): answered; rounds 2 and 3 add 5.0 and 2.9 points on average, both intervals above zero; round 1 adds nothing, so the registered criteria are not met.[FU04,FU12] Matched data (one fit on 2100 base-proposed frontier programs): 14.6%; the rounds’ advantage of 2.1 points is inconclusive under the registered analysis.[FU07] Original exam (889 items the base missed at selection; adapter level): the registered criterion is met in 4 of 4 lineages; generation 3 holds in 1 or 2 of 4 under the clustered intervals.[RS29,RS38]
What it shows. Accuracy grows with the verified frontier: on an exam drawn after the fact, three rounds roughly doubled the share the base answered, and base-proposed frontier data scaled the same way from 900 to 2100 records (Figure 6).[FU04,FU07] Whether the rounds help beyond the data they accumulate is not shown. We separate three senses of recursion: S4 has a recursive structure, since each round’s data is proposed and selected by the previous round’s model; a benefit of that recursion over matched data is not demonstrated; and an improving rate of improvement is not demonstrated (§ 3.5). The growing frontier is what the runs that improved share; a fixed-size but better-selected frontier, more optimisation on fixed data and other replay schedules were not tested. The first round’s gain on the original exam was largely the exam’s selection on the base’s failures (§ 3.4). The earlier series show what S4 needed (§ 4); the same loop on repository tasks and on a 27B agent showed no gain, in those configurations (§ 5).

Figure 6. Accuracy on the fresh exam against the amount of interpreter-labelled training data. Blue: S4’s generations, each a fresh adapter from the base on 300, 900 and 2100 cumulative records; red: the matched-data controls, one fit from the base on 900 or 2100 frontier programs the base proposed. Large marks are group means, small marks single lineages. Accuracy grows with the verified frontier in both; the first 300 records gain nothing on F.
Contributions. (i) A registered multi-round, environment-supervised training procedure on a self-proposed frontier, examined on a fresh exam and against matched data. (ii) A learner/examiner separation specified fully enough to re-implement, with a reference implementation that reproduces the published decisions. (iii) A registry of every test of the four series and why most did not meet their criterion. (iv) A claims ledger tying every statement to pinned code or a raw record (§ 8).
Method
Terms. A series is a registered experiment of four lineages, each a chain of three generations from the base that differs from its siblings only in its seed; the champion is the model a generation’s material comes from; a step compares a generation with the one before. The registrations call the series RSI1 to RSI4 (“recursive self-improvement”); we call them S1 to S4, because what they test is multi-round self-training on self-proposed, oracle-labelled material; whether an improved model is a better improver is what the matched-data control tested (§ 3.5). [RS01]
The learner. The task family is predicting what a Lua 5.1 program prints. The learner is the loop of Appendix § 24 reduced to this one kind: the policy, Qwen3-4B with thinking off, writes programs about sections of the Lua 5.1 manual, the pinned Lua 5.1.5 interpreter runs each program and its output becomes the reference, and the policy is trained on its own programs with the interpreter’s output as the target. From S2 on every model call and every training run goes through Brain’s API: one brain serve per lineage serves the proposals, the predictions and the dev and retention exams, and trains each candidate as a Brain training job. [RS09,RS10]
A lineage is a chain of generations from the base that differs from its siblings only in its seed; each series runs four lineages of three generations. Inside the loop a candidate is accepted by a dev gate on 400 units (100 generator programs and 300 programs the base proposed, held out from training) with a retention set of 300 units; neither holds an exam item. [RS09] The generator programs come from a separate seeded generator of six levels (a sum, a product, a short string transformation, a loop, a three-item table sort, an arithmetic expression) that shares no code or templates with the exam’s generator, although two of its levels call functions for which the exam has templates (string.upper, string.rep, the length operator, table.sort).[RS51] The fresh exam of § 3.4 comes from the examiner’s own generator, and none of its items matched a program any S4 adapter was trained on or handed over (0 excluded at every step).[FU15]
The exam. The exam is 889 Lua 5.1 programs written by code from 25 standard-library templates over randomised and edge-case arguments, the items the base never answered in four sampled attempts under a neutral request at selection (at S4’s serving tier it answers 12 of them); the reference of each is what the interpreter prints (twice, equal).[RS03] Because it was selected as what the base could not do, a count of items right on it measures progress on that hard subset, not a model’s accuracy on Lua programs in general. It was selected once, in S1, and every series uses the same file (its SHA-256 begins 60ac3ae44651 in every verdict).[RS20,RS31] No model writes or answers an exam item to build it, and no run of the learner reads it. The reuse has a cost: S1 graded this exam item by item with its own scripts, and each later design was chosen after the aggregate results of the series before it on the same exam, so across the series the exam is a holdout reused adaptively, a setting in which a reused holdout can be overfitted. [11][CT22]
Each registration fixed its criterion before its run, but what the earlier series showed on the reused exam informed each later design, so the registrations are not independent of it. S4’s champions were therefore re-examined on a freshly drawn exam (§ 3.4).[FU01]
The examiner. From S2 on the exam is held by an examiner, a separate program operated beside the learner by the same operators; it is independent of the learner, not of the authors. The learner’s only contact with it is a one-way hand-off (Figure 3).[RS04] For each lineage and generation the learner writes a directory with the list of every program the candidate’s chain trained on and, last, a candidate record (base model, base digest, adapter path and digest, serving flags, commits). The examiner then

Figure 3. The learner and the examiner. The only channel is one-way: per lineage and generation the learner writes the list of every program the candidate’s chain trained on (with its held-out dev and retention programs) and then the candidate record (base, adapter and their digests). The examiner holds the exam, drops items that match a listed program (exactly, or after removing comments and whitespace), serves every model itself on one pinned build, grades by the direct-completion protocol, applies the registered per-step rule and signs aggregates only, which are published with the learner’s records after the series. The dotted path does not exist.
serves the base and every candidate itself, on one pinned Brain serving binary (SHA-256 d11e9134..., a different Brain build from the learner’s commit 796871647) and, in S4, at one KV-cache tier (f32), and marks a verdict invalid when the served base is not the declared checkpoint (identified from the checkpoint’s own files) or the adapter does not have the declared digest; that build’s source is not in Brain’s published history, so the reference examiner of § 8 serves on the learner’s Brain instead;[RS06,RS32]
excludes from each step’s comparison every exam item that equals, exactly or after removing comments and whitespace, any program on either compared model’s hand-off list (the programs its chain trained on and, beside them, its held-out dev and retention programs), and reports how many it excluded; it also reports a near-duplicate count whose similarity measure the verdicts do not define, and a program of the same template with other arguments is not excluded;[RS05]
asks each item once under the direct-completion protocol: the program and its question, followed by the sentence “Answer with exactly what it prints, in one fenced code block.” (the request S1 registered), as the only user message, no system prompt, one greedy completion with thinking off and, in S4, at most 2048 tokens; the answer is the last fenced block of the reply, or the whole reply if it has none, compared with the reference token by token per line with whitespace runs collapsed and blank lines ignored; a reply the token limit cut is graded as written;[RS32,RS36]
pairs the candidate with its predecessor item by item (a win: the candidate is right and the predecessor wrong; a loss: the reverse), computes the exact one-sided sign test over the discordant items, and decides the registered rule: a step holds when wins exceed losses at , a lineage succeeds when all three of its steps hold, and the series meets its criterion when at least three of the four lineages succeed;[RS02]
returns, per step, a verdict with aggregates only (items right, wins, losses, , whether the step holds, exclusion and overlap counts, integrity flags) and, over the series, a plan verdict; each is signed with the examiner’s HMAC-SHA256 key, so the learner, which does not hold the key, cannot alter or forge one; the key protects against the learner, not against the authors. Each verdict also carries the paired gain in the fraction of items right with a 95% interval clustered over the exam’s 25 templates (§ 3.2). No item text and no item-level grade leaves the examiner.[RS04,RS31,RS38]
The items themselves are not secret: the exam file was released with S1’s records, so the separation rests on the learner’s code never opening it and on the exclusion of overlapping programs, not on secrecy.[RS03]
What is compared. Every comparison is within a lineage and between consecutive generations, on the items no compared model trained on, so a model’s gain is measured against the model it was trained from. The registered decision is per lineage. The four lineages share the exam’s items, so a sign test pooled over lineages treats correlated pairs as independent; the pooled figures are reported as a description, not as a test. The examiner’s base-against-base null replicate, run before S4, answered 12 of the 889 items on both sides with 0 discordant items and 0 failed requests: the protocol is deterministic at the serving build and tier, so a discordant item is a difference between the models.[RS07]
Three levels of claim. The item-level sign test supports a claim about two adapters: this adapter answers more of these items than its predecessor. The template-clustered interval supports a claim about programs of these templates. Only replication over seeds supports a claim about the procedure, and with four lineages that is a count, not a test: 4 of 4 has a one-sided sign-test of 0.0625, and the generation-3 champions answered 219.0 items on average with a standard deviation of 20.4 between seeds.[RS53] Each result below is stated at its level.
Results
S4: retraining from the base on a frontier that grows each round
S4 changed what S3’s records pointed at (§ 4), and with it the schedule (Figure 2). Retraining from the base: generation ’s candidate is a fresh LoRA trained from the base on the union of every generation’s frontier, not a continuation of generation ’s adapter; only the data of generation is new, proposed and predicted by generation ’s champion. Frontier selection: a program enters when at least one of the champion’s 4 sampled predictions (temperature 1.0) is wrong, so the frontier also holds programs it usually gets right: one predicted right with probability 0.9 per sample enters with probability 0.34.[RS25,RS35] Schedule: 300, 600 and 1200 new programs per generation, two passes over the union, no separate replay, so the generations trained on 300, 900 and 2100 records for 75, 225 and 525 steps.[RS25,RS26] These changes were made together and no ablation separates them. Before any verdict, the examiner’s null replicate (base against base) found no discordant item, the base was served at the configuration under which it answers 12 exam items (Appendix § 9), and the examiner’s build was changed so that it identifies the declared base by the checkpoint’s content.[RS07,RS27,RS06]
| Lineage | Items right of 889: base | g1 | g2 | g3 | Gen. 1 vs base: W/L | Gen. 2 vs gen. 1: W/L | Gen. 3 vs gen. 2: W/L | All hold | |||
| r0 | 0 | 58 | 122 | 123 | 58/0 | 78/14 | 29/28 | 0.5 | no | ||
| r1 | 0 | 72 | 101 | 91 | 72/0 | 41/12 | 18/28 | 0.948 | no | ||
| r2 | 0 | 67 | 86 | 121 | 67/0 | 38/19 | 0.00817 | 48/13 | yes | ||
| r3 | 0 | 91 | 118 | 142 | 91/0 | 42/15 | 0.00023 | 47/23 | 0.00279 | yes | |
| pooled | 2 of 4 lineages succeed | 288/0 | – | 199/60 | – | 142/92 | – | not met | |||
| r0 | 12 | 92 | 138 | 192 | 82/2 | 69/23 | 93/39 | yes | |||
| r1 | 12 | 131 | 181 | 225 | 121/2 | 79/29 | 99/55 | 0.00024 | yes | ||
| r2 | 12 | 118† | 169† | 241† | 109/3 | 79/28 | 95/23 | yes | |||
| r3 | 12 | 121 | 187 | 218 | 110/1 | 93/27 | 67/36 | 0.00146 | yes | ||
| pooled | 4 of 4 lineages succeed | 422/8 | – | 320/107 | – | 354/153 | – | met |
Table 1. S3 and S4 as the examiner decided them: items right of 889 and each step’s wins/losses with the exact one-sided sign test (a step holds at with wins above losses; the criterion is 3 of 4 lineages holding at every step). Pooled rows sum the lineage rows (they equal the plan verdict’s) and are descriptive, without a test, because the lineages share the exam’s items.[RS30] †r2 is compared on the 888 items left after one overlapping item was excluded; at generation 3 it answered 242 of the 889 before the exclusion. S4 trained on 300, 900 and 2100 cumulative records at generations 1 to 3. S3’s base was served under a different configuration from S4’s (Appendix § 9), so compare steps, not levels.
Outcome (adapter level). All twelve S4 steps held (Table 1). From 12 items at the base the lineages reached 192, 225, 241 and 218 at generation 3; the weakest step, r3’s third, won 67 items and lost 36 ().[RS28] The plan verdict reads met, 4 of 4 lineages where the rule needed 3.[RS29] One exam item equalled a program on lineage r2’s hand-off list from generation 1, a base proposal held out in its dev set, not a training program; the examiner excluded it from r2’s comparisons and raised its overlap flag on r2’s three verdicts; no other overlap or flag was raised, and no request failed.[RS31,RS40,RS50]
Template level: the clustered intervals
The sign test counts the 889 items as independent, but they come from 25 templates. The examiner also reports, for every step, the mean per-item paired difference with a cluster-robust standard error over templates, with , and a two-sided Student- 95% interval with degrees of freedom (Figure 4). Every S4 step at generations 1 and 2 excludes zero, and generation 3 only r2’s interval does (0.0811, from 0.0267 to 0.1355); r0’s, r1’s and r3’s include zero. The matching one-sided 95% bounds are −0.0054, −0.0071, 0.0360 and 0.0003 for r0 to r3. At the template level, then, the third generation improves in one lineage (two-sided) or two (one-sided) of four, and a 3-of-4 rule would not be met; in S3 no lineage clears all three steps.[RS38] The average generation-3 gain over the lineages is 5.7 points.[RS53] On the fresh exam, where the examiner computed the lineage-averaged estimate with an interval over templates and lineages, the third round adds 0.029 (0.011 to 0.050) and the second 0.050 (0.026 to 0.079).[FU12]

Figure 4. Every step’s paired gain with the examiner’s 95% interval clustered over the exam’s 25 program templates, read from the signed verdicts. Filled points are steps whose registered item-level sign test holds, open points steps whose test does not. Every first and second S4 step excludes zero; of the third steps only r2’s does, and r0’s, r1’s and r3’s reach below zero, so the third generation’s gain is not robust to the clustering of items by template in three lineages.
What the learner’s records show
Step 1 is inflated by selection. The exam was selected from items the base got wrong, so at step 1 the base can lose at most the 12 items it answers, and lost 2, 2, 3 and 1; steps 2 and 3 are not so constrained. [RS28] On the fresh exam, whose items were not selected by the base’s answers, step 1 holds in no lineage (§ 3.4). [FU03]
Churn. Items right are not cumulative mastery. At step 3 the new adapter lost 28%, 30%, 14% and 19% of the items its predecessor answered (r0 to r3) while winning 12.4%, 14.0%, 13.2% and 9.5% of those it missed, although it was trained on a superset of its predecessor’s data from the same base. [RS48] Retraining the same data under another training seed shows how much of this is noise: on the fresh exam such retrains split 87/57, 70/56, 48/56 and 35/55 items against the originals while their mean accuracy agreed (§ 3.5). [FU08]
Dev accuracy and frontier yield. On each lineage’s own 400-program dev set, which was not selected by failure, accuracy rose from 0.4025, 0.40, 0.435 and 0.545 at the base to 0.67, 0.6525, 0.65 and 0.725 at generation 3 (Figure 5).[RS46] The champions kept proposing programs they could not predict: the share of predicted proposals on the frontier fell from 84–93% at generation 1 to 61–71% at generation 2 and 47–54% at generation 3, so even the third champion missed about half of its own proposals.[RS47]

Figure 5. S4 accuracy on each lineage’s own 400-program dev set (100 generator programs and 300 base proposals, not selected by failure), the absolute effect size the learner’s records give.
The dev gate filtered nothing. As run, the learner’s gate advanced every candidate with a positive net dev gain; significance at its spent alpha (0.025, 0.0083 and 0.0042 at generations 1 to 3) was recorded, not required. Three of S4’s twelve accepted steps (r1’s third, r3’s second and third) and six of S3’s twelve were not significant at that alpha, and in S3 two accepted steps did not hold on the exam. At r1’s third step the gate saw 25 dev wins against 19 losses () while the exam showed 99 against 55.[RS49,RS24,RS33] TRUST replaces such a greedy gate with an online multiple-testing procedure that controls the rate of accepted inferior updates; [12][CT15] a gate of that kind exists in the learner as an option but no series used it.[RS34]
The fresh exam
Both follow-ups were registered after S4’s outcome and amended once before any of their verdicts (prereg/rsi4-followups.md).[RS41,RS52] re-examined the base and the twelve S4 champions on an exam the examiner drew afresh: 2340 programs from 78 templates, never published and not selected by the base’s answers; the base answers 194 of them (8.3%), and the examiner’s base-against-base replicate was exact.[FU01] Neither registered criterion is met: step 1 holds in no lineage, by the sign test or by the one-sided clustered bound; steps 2 and 3 hold in every lineage by the sign test, and by the clustered bound step 2 in four and step 3 in three.[FU02,FU03] Averaged over the lineages, the share answered was 8.3% at the base, 8.7% after generation 1, 13.8% after generation 2 and 16.7% after generation 3, a gain of 0.084 over the base (95% interval over templates and lineages 0.046 to 0.124); the first round’s gain, 0.004 (−0.016 to 0.023), is indistinguishable from zero.[FU04] The first round’s apparent gain on the original exam was therefore largely selection: on the items the base answered fell by 0.371 at generation 1 while those it missed rose by 0.038, the signature of regression to the mean, and by generation 3 the base-wrong items had gained 0.119.[FU05] The similarity audit puts the median exam item at a token-set Jaccard of 0.54 to its nearest training program (0.67 at the 90th percentile); one item has exactly the token set of a training program without matching it textually, so it was not excluded.[FU06]
Matched data: is the improvement recursive?
trained one adapter from the base, in each of four new lineages, on 2100 frontier programs the base proposed and selected, the same amount of data as generation 3. On it answered 14.6% against generation 3’s 16.7%: a difference of 0.021 whose registered interval, resampling templates and lineages jointly, runs from −0.002 to 0.046, so the registered outcome is inconclusive; an interval over templates alone (0.002 to 0.040), which ignores seed variation, is reported only as a secondary reading.[FU07] The base-proposed data scale: 900 records gave 12.4% and 2100 gave 14.6% (difference 0.022, interval 0.004 to 0.043), and generation 3 exceeds the 900-record control by 0.043 (Figure 6).[FU07] Retraining each generation-3 training set with another seed gave 16.9%, equivalent to generation 3 within the registered margin (difference 0.002).[FU07] That seed-null arm also decides how to read the registered item-level M1: all four generation-3 champions beat their paired controls (129/81, 122/94, 119/80 and 136/56), but identical-data retrains already split 87/57, 70/56, 48/56 and 35/55 against the originals, so item-level comparisons of single adapters measure seed noise as much as treatment and are not evidence here, as the registration’s amendment anticipated.[FU08] One identical-data retrain even passes the item-level step rule against its original (87/57, ), so per-lineage item sign tests, including the original exam’s 4 of 4, are facts about pairs of adapters, not evidence that the procedure gains.[FU15] M is matched in data, not in compute: S4 spent 825 training steps over three rounds against 525 for one fit, 57% more before the cost of proposing and predicting.[FU09]
What the base can do by reasoning. S4 improves direct prediction without thinking, from 8.3% to 16.7% of F (Table 2). The same base with thinking on answers at least 47.3% (1107 items, with 372 replies cut by the 8192-token budget), Qwen3.8-27B-FP8 answers 39.3% zero-shot without thinking, and the base’s majority over four samples answers 8.1%.[FU13] The gain is therefore in direct prediction, not beyond what the base reaches by reasoning, which is why verified, compact reasoning is the natural target for the next oracle-supervised step. In the direct protocol, replies cut by the 2048-token limit are rare: on the original exam the reference re-grade counts 21, 15, 1 and 14 of 889 for the generation-1 adapters and at most 4 later.[FU14]
| Model | Training records | Thinking | F answered (%) |
| Qwen3-4B base | – | off | 8.3 |
| S4 generation 1 | 300, own frontier | off | 8.7 |
| S4 generation 2 | 900, own frontier | off | 13.8 |
| S4 generation 3 | 2100, own frontier | off | 16.7 |
| same data, new seed | 2100, own frontier | off | 16.9 |
| M-900 (one fit) | 900, base frontier | off | 12.4 |
| M (one fit) | 2100, base frontier | off | 14.6 |
| base, majority of 4 samples | – | off | 8.1 |
| base, thinking on (8192 tokens) | – | on | ≥47.3 |
| Qwen3.8-27B-FP8, zero-shot | – | off | 39.3 |
Table 2. Every model examined on the fresh exam F (2340 items, direct-completion protocol unless noted; S4 and control rows are means of four lineages). The last three rows are the examiner’s descriptive baselines, with no criterion; the thinking-on figure is a lower bound, because 372 replies ran out of their 8192-token budget while thinking.
What this says about recursion. S4 has a recursive structure: each round’s training data is proposed and selected by the model the previous round produced; resetting the weights does not remove it, because the state that persists is the dataset. A benefit of that recursion is not
| Test | What it trained on | Registered criterion | Outcome |
| S1 A | own tasks, closed-book loop | gen. 3 beats gen. 2 | not met; gains were protocol adaptation[RS11,RS42] |
| S1 B | A without starving proposals | gen. 2 and 3 each improve | not met[RS42] |
| S1 H | own programs, interpreter labels | pooled steps, no replicate falls | met (pooled)[RS12] |
| S1 HP | H in Python | as H | not met at gen. 3[RS13] |
| S1 HR | HP with replay and pooled retention | as H | met (pooled), each step’s gain from one replicate[RS14] |
| S1 R | own verified reasoning | monotone gain on unseen families | not met[RS15] |
| S1 P, P2 | policy gradient, two update scales | as R | not met[RS15] |
| S1 HL | H’s recipe, four lineages | 2 of 4 lineages | not evaluated (disk quota)[RS16] |
| S1 HL rerun | as HL | 2 of 4 lineages | not met, 1 of 4[RS17] |
| S2 | on-policy policy gradient | 3 of 4 lineages | not met, 0 of 4[RS19] |
| S3 | own frontier, continued adapter | 3 of 4 lineages | not met, 2 of 4[RS22] |
| S4 | own frontier, retrained from the base | 3 of 4 lineages | met, 4 of 4 (clustered: 1 or 2 of 4)[RS29,RS38] |
caption: Every registered test of the four series, in order (P and P2 are two tests). One per-lineage criterion was met, S4’s; the pooled criteria H and HR met cannot tell one improving model from gains spread over replicates.
:::
shown: generation 3 beats data the base proposed and selected by a point estimate of 2.1 points, with a registered interval that includes zero. An improving efficiency of improvement is not shown either. The comparison tests the whole adaptive acquisition, in which the champion both proposes and selects; which of the two matters would need a proposer by selector design.
The frontier is a failure filter. A program enters when at least one of four sampled predictions is wrong, so a program predicted right with probability 0.99 per sample enters with probability 3.9%, one at 0.90 with 34.4%, one at 0.50 with 93.8%; four identical wrong answers qualify too.[FU10] The filter selects difficulty, not learnability, and keeps programs the champion never solves, which is right here because the interpreter supplies the correct output: S4 is learning from the environment’s targets, not reinforcement of the model’s own successes.
What did not work before S4
Table 3 lists every registered test; Appendix § 9 gives each with its numbers. S1 (one registration with dated arms, graded by its own scripts before the examiner existed) found that gains on the loop’s own probes were mostly adaptation to the probing protocol, built the exam, and showed large first steps on it with self-proposed programs labelled by the interpreter; third steps did not hold within a lineage, and arms that replaced the labels with rewards on the policy’s own attempts (self-imitation of reasoning, policy gradient) did not move the exam.[RS11,RS12,RS15,RS17] S2, on-policy policy gradient, moved the exam by 2 to 4 items per lineage at generation 1 and no step held.[RS19] S3 trained a continued adapter on the champion’s greedy mispredictions: all four lineages improved at generations 1 and 2, but at generation 3 one held flat and one fell, and over the four lineages the mispredicted share of proposals fell from 2127 of 2400 to 860 of 1942.[RS22,RS23] S4 therefore retrained from the base on a growing frontier. Because that design search used one exam, S4 was re-examined on a fresh one (§ 3.4).
The same loop elsewhere: no gain shown
The full loop (Appendix § 10) was also tested where it was built for. On repository tasks, an 8B student trained on a 27B teacher’s verified trajectories was not shown better than its base by the registered confirmatory query: it solved 23.0 of 46 tasks on average over three seeds against 20.0, 18 families won and 12 lost (); a same-seed contrast showed that supervising every action, rather than only file edits, does harm (Appendix § 20).[QUAL1] Two promoted generations of a 27B model trained on its own agent experience did not beat their base on one measurement (Appendix § 23).[G2703,G2704] An exploratory, unregistered run in which a 4B model learned from a raw manual gained on its own probes, with no detected effect of which model wrote its tasks (Appendix § 24).[SL23] These negative results hold for those configurations: small corpora, records truncated to the training block, harness failures, and an exam the 27B model already solved nine times in ten; they do not show that supervised training memorises and reinforcement learning generalises, or the reverse. The supervision finding, that training on every agent action does worse than training on the decisive edits, is secondary but held at the registered seeds.
Related work
Self-training on self-proposed problems. The closest prior design is Haluptzok et al.: a model synthesises programming problems with solutions, an interpreter filters them for correctness, and the model is fine-tuned on the result. [15][CT24] STaR and ReST-EM also restart from the base each iteration on accumulated, filtered self-generated data, but they filter the model’s outputs against known answers; [28, 38][CT02,CT16] here the model generates the inputs and the oracle supplies the outputs. Self-Instruct generates inputs too, but labels them with the same model; [30][CT01] ReST iterates grow and improve steps on filtered samples. [14] Noisy Student iterates generations on a growing pool, with a teacher’s labels. [32][CT26] Expert iteration amplifies the policy by search and imitates the result; [2][CT19] an interpreter is ground truth, not an amplified policy, so only the generate–filter–retrain skeleton is shared. In self-play designs a proposer is trained to set learnable tasks, as in asymmetric self-play and in Absolute Zero, where a code executor validates the proposed tasks; [29] [42][CT28,CT18] our proposer is not trained to propose, only filtered. Other self-improvement work changes a model’s own context or scaffold rather than its training data: STOP improves the improver scaffold itself, [39][CT25] Learning to Self-Evolve trains a model to improve its context at test time, [7][CT31] and hyperagents let a meta agent modify the agent; [40][CT32] SEAL has a model write its own fine-tuning data and directives. [43][CT33] Our matched-data control does not show that the recursion itself helps here. On reinforcement learning with verifiable rewards, Yue et al. find gains at small bounded by the base model at large , and new patterns from distillation; [36][CT34] our own RL arms are too narrow to add to that question.
Selection and curricula. Selecting what to label is active learning, here by query synthesis with the model’s own disagreement as the acquisition function; ordering by difficulty is curriculum learning; [3, 26][CT29] learned curricula over problem categories are a recent variant. [6][CT23]
Predicting execution. Predicting what a program prints is CRUXEval’s output-prediction task; [13][CT17] models have been trained to predict execution since Learning to Execute, and more recently with execution traces. [10, 21, 37][CT27] CodeRL trains a generator with a critic of functional correctness from unit tests; [20][CT20] training repeatedly on a model’s own generated content can make the tails of the original distribution disappear, [27][CT21] which our series counter only by labelling every program with the interpreter.
Evaluation. A greedy gate that accepts any higher validation score lets inferior updates through across rounds; TRUST controls their rate by online multiple testing. [12][CT15] Reusing a holdout adaptively can overfit it, [11][CT22] and fresh test sets test replication; [25] evaluations need clustered standard errors and seed-level variance. [1, 4, 22][CT30] Related work for the system components is in Appendix § 26.
Limitations
What the result is about. One task family (predicting what Lua 5.1 programs print), one base model, three rounds of LoRA training on interpreter labels, and no tools and no reasoning: thinking is off everywhere and the training records are answers only. S4 improves direct prediction without thinking; nothing is claimed about other tasks, models, longer series or reasoning, where the base with thinking on already answers at least 47.3% of the fresh exam against S4’s 16.7%.[FU13]
What the statistics support. Four lineages are encouraging, not a reliability estimate. “Every generation improves” is the registrations’ strict criterion, not the measure of useful learning; the learning curves of Figure 6 are. On the fresh exam the first round gains nothing and the registered 3-of-4 criteria are not met; the third round holds in three lineages by the clustered bound. The advantage of the rounds over matched base-proposed data is inconclusive, and M is matched in data, not in compute.[FU03,FU07,FU09] The exam that guided the design was reused; the fresh exam was drawn afterwards, but its templates include those of the old exam.[CT22] The overlap exclusion catches exact and normalised matches only, and the similarity audit shows the fresh exam lies close to the training programs (median token-set Jaccard 0.54).[FU06] The examiner is a separate program run by the same operators, and its symmetric signature protects the verdicts against the learner, not against the authors. The learner’s dev gate advanced on any net gain and filtered nothing.[RS49]
Not continual learning. Each round retrains from the base, a consolidation baseline rather than a solution for an unbounded stream: an adapter loses 14 to 30% of the items its predecessor answered, and a retention test of what each round acquired is not implemented. The seed-null retrains show that this churn is mostly training noise at the level of items while lineage means are stable.[RS48,FU07,FU08]
The rest of the paper. The limits of the system specification and of the coding, 27B and documentation experiments are in Appendix § 27.
Open directions. The questions the evidence leaves are richer oracle supervision (execution traces, compact verified reasoning), a proposer-by-selector design to separate who proposes from who selects, compute-matched baselines, a learnability rather than a failure curriculum, and a retention matrix over acquired capabilities.
Reproducibility
What is needed. Brain, Sven and Splinter, cloned from their remotes, and the evidence dataset swedishembedded/the-frontier-must-grow-evidence. The dataset holds only what this paper cites: the 889-item exam, the examiner’s signed step and plan verdicts and the rule it applied, the learner’s round records, every candidate record with its hand-off list of programs, the twelve S4 champion adapters, S1’s analyses and per-model grades, and the records of the documentation runs and the 27B generations, one archive per series. reproduce/dataset.json pins the dataset’s revision and the SHA-256 of every file and archive member, and data/measurements.json names each cited record by its path in the unpacked dataset and its digest. Every commit named below is contained in a branch of its repository’s remote; make verify fails on a pin that is not.
Three tiers. The targets are run in the paper’s directory (make -C with the paper’s directory); downloads, builds and outputs land under target/papers, and EVIDENCE_DIR, REPRO, BRAIN_SRC, SVEN_SRC, DEVICE and LOCK move them, name the clones and the card (make reproduce lists the tiers).
| Target | What it reproduces | Hardware and time |
| reproduce/verify | Downloads the dataset at its pinned revision (or uses a local copy), checks every file, archive member and adapter digest, unpacks it and runs make verify on it: the structure checks at the pinned commits, and every measured value re-extracted from its record and compared with data/measurements.json. | CPU; a 1.6 GB download; under a minute on our host. |
| reproduce/examine | Builds Brain at 796871647, fetches Qwen/Qwen3-4B at its pinned revision (every file checked), serves the base and each lineage’s three adapters with the paper’s flags (--kv-cache f32, --max-batch 64, adapters by digest, the base identified by the checkpoint’s content digest), re-grades all 889 items with the reference examiner and writes each step beside the published verdict. | One GPU (the paper’s runs used a GH200); time not yet measured. |
| reproduce/rerun | Builds the S4 lineage driver at Splinter 86f13672a against Sven fc99a2f1 and Brain 796871647, fetches the Lua 5.1 manual (and cuts the three sections the proposer reads, checked against their digests), builds Lua 5.1.5 from its official tarball (SHA-256 pinned), runs the four S4 lineages with the launch record’s settings and seeds, then grades the new champions as reproduce/examine does. | One GH200; about 12 h, since S4’s lineages took 2.4 to 3.4 h each on a card of their own.[RS26] |
Table 1 (PDF p. 13).
The reference examiner. reproduce/examiner.py implements the examination protocol of § 2 from that specification: the neutral request (the item’s instruction and the sentence “Answer with exactly what it prints, in one fenced code block.”), no system prompt, thinking off, one greedy completion of at most 2048 tokens, the last complete fenced block or else the whole reply as the answer, a reply cut at the limit graded as written, the comparison token by token per line with whitespace runs collapsed and blank lines ignored, the exclusion of every item equal to a program on either compared model’s hand-off list (exactly, or once comments and whitespace are removed), the exact one-sided sign test, the paired gain with its template-clustered Student- interval ( degrees of freedom) and the registered per-lineage plan rule. It checks the served base against the declared base digest and each adapter file against its declared digest, and serves one lineage at a time, because adapters of different lineages carry the same card ids. Its unit tests run in make verify. It serves on the learner’s Brain 796871647: the build the published verdicts were served on (binary SHA-256 d11e9134...) was made from a Brain source that is not in Brain’s published history. Run on that build, the reference examiner agrees with the published verdicts on all 12 S4 step decisions, with identical items right, losses and exclusions at every step; the only differences are the base answering 13 items instead of 12 on lineage r1’s server, and r1’s first step counting 120 wins instead of 121.[FU11]
What a reader cannot reproduce. (1) The fresh exam F of the registered follow-ups and its generator are not released; only its signed aggregate verdicts are published. (2) The published verdicts were produced by a different implementation of the examiner than the reference one, on a different Brain build, so reproduce/examine re-measures the result rather than replaying the published grades; a difference within serving nondeterminism is expected, and the comparison reports it. (3) The verdicts are signed with a symmetric HMAC key, so a reader cannot verify a signature; the signature guards against the learner, not against the authors. (4) reproduce/rerun is a statistical replication, not a bit-identical one: proposals and frontier predictions are sampled at temperature 1.0 and GPU kernels are not deterministic across hosts, so the reproduced lineages are new draws of the same procedure. (5) The raw logs of the training-substrate qualification and throughput probes, a trajectory probe, and the coding-task pilots and probes were not kept; data/measurements.json records their values, digests and commands, marked "kept": false, and make verify reports them as not kept.
Verification. make verify first runs the specifications of the pin rule, of the record parsers and of the reference examiner, then re-runs every structure check against the commits in data/commits.json, then scripts/build_data.py --check, which resolves each measured and computed claim to its value, requires the claim text to contain the value, requires every ledger row to be cited in the text and every citation to be in the ledger, and checks that the pooled steps computed from the examiner’s lineage verdicts equal its plan verdicts; with EVIDENCE=DIR it also re-extracts every value from the records in DIR. The ledger holds 381 claims, 240 of them structure claims backed by 494 checks. The sibling repositories are given by --sven and --brain. make check includes this verification as check/papers. The bibliography entries were checked against the archive’s abstract pages on 2026-11-15.
Draft and final. The paper’s status is recorded in status.txt; when it reads final, build_data.py --check also fails while any registered-but-unmeasured or planned claim remains, so a final version holds only structure facts, measurements, computations from measurements and citations. This version is final: the registered follow-ups have reported, and their verdicts are in the evidence dataset (records/followups.tar.xz).[FU02,FU07]
Registrations. All registrations are in the paper’s prereg/ directory: the coding-task protocol in coding-tasks.md, the acquisition experiments in acquisition.md, S1 in recursive-improvement.md, and S2 to S4 each in its own file (rsi4.md for S4) with a launch record (rsi4-launch.md) that fixes builds, binary digests, settings and seeds before the first generation and carries the series’ dated deviations; the outcome of each series is appended to its registration.
Build. From the repository root the paper is built by
make papers/splinter-brain-sven-flywheel/pdf
into target/papers; every table, plot and number of the series S1–S4 and of the coding-task pilots is generated from data/ into the build directory. The same build writes target/papers/splinter-brain-sven-flywheel-arxiv.tar.gz, the source archive for arXiv: the main file, the sections, the generated tables, macros and plot data at the paths the source names, the bibliography and its .bbl, with comments removed. The build compiles the archive in an empty directory with pdflatex and bibtex and with latexmk, and fails unless both give the main PDF’s page count with no error or undefined reference; the archive is uploaded as is.
The earlier series in detail
S1: self-taught generations, judged first pooled and then per lineage
S1 is one registration with dated arms, each added before it ran; its exams were selected and graded by the arm’s own registered analysis script, without a separate examiner.[RS01] Arms A and B ran the closed-book loop of Appendix § 24 for three generations and were judged on the loop’s own probes; re-graded under a neutral request, most of the gain they measured was adaptation to the probing protocol and did not transfer.[RS11] Arm H therefore built the exam described above and measured every generation on it greedily under the neutral request. Its three replicates, trained only on tasks their own models wrote, answered 38, 31 and 34 of the 889 items at generation 1, 60, 63 and 49 at generation 2 and 67, 83 and 59 at generation 3 (base 0); pooled over the replicates the steps were 103/0, 95/26 and 103/66 wins/losses ( at the third) with no replicate falling, which met the registered pooled criterion.[RS12] The same design in Python (arm HP, 820 items, base 2) met its first two steps (49/7 and 83/8) and not the third (59/50, ), where one replicate fell from 69 to 60.[RS13] Arm HR added replay across generations and a pooled retention bound and met the pooled criterion (66/4, 16/1, 25/3), but each step’s gain came from a single replicate, so it is not evidence that one lineage improved twice.[RS14] Three arms changed the update instead: self-imitation of the model’s own verified reasoning (R) and policy gradient with negative rewards at two update scales (P, P2). After three iterations they answered 21, 4 and 12 of the 889 items, and on six families none of them trained on their pass@1 went from 0.41 to 0.4479, 0.4408 to 0.4242 and 0.4419 to 0.4408: none met its criterion.[RS15]
Because a pooled test cannot tell staggered single gains from one model improving repeatedly, arm HL repeated arm H’s recipe in four fresh lineages under a per-lineage criterion: a lineage succeeds when all three of its steps hold, and the arm when two of four do. Its first run ended on the project disk’s inode quota before any lineage finished and was not evaluated.[RS16] The rerun met the criterion in 1 of 4 lineages: r3 rose at every step (16/0; 14/5, ; 33/12, ), r2 rose twice and fell at generation 3 (27/36), r0’s later steps were not significant (35/25, ; 37/28, ), and r1 accepted no round after generation 1. Pooled, the third step was 97/76 ().[RS17] S1 thus showed large first steps on items the base never answered and third steps that did not hold within a lineage.
S2: on-policy policy gradient
S2 asked whether replacing the update with on-policy policy gradient under powered gates makes the later steps hold. Each generation trained on the current policy’s own graded attempts at programs it solves sometimes ( attempts, group-standardised advantages, clip edges 0.2 and 0.28, learning rate for 24 steps, LoRA rank 16), and the exam moved to the independent examiner.[RS10,RS18] At generation 1 the four lineages won 4, 2, 3 and 3 items against the base and lost none ( from 0.0625 to 0.25): no step held, so no lineage could succeed, and the examiner’s plan verdict reads not met.[RS19] The examiner’s overlap audit found that the policy’s own proposals had reproduced 1, 1 and 2 exam programs exactly in three lineages; S2 had no exclusion and those items count as scored.[RS19] Generation 2 could not be served, because every trained candidate carried the same adapter card id; the criterion was already decided, so it was not rerun after the fix.[RS08] The recipe raised the learner’s own dev set in its pilot but moved the exam by two to four items.

Figure 7. Items right on the held-out exam per generation and lineage, as the examiner measured them; the exam holds items the base did not answer when it was selected, so these are counts on that hard subset, not an accuracy on Lua programs. In S3 two lineages stop rising at generation 3 (r0 flat, r1 down); in S4 every lineage rises at every generation. The base level differs between the panels because the base was served at a different KV tier (Table 1).
S3: training on the interpreter’s answers to the policy’s own programs
S3 returned to arm H’s source of information, training on the interpreter’s output for programs the policy itself proposed, and replaced what had failed in S1: acceptance on a dev set powered for an effect of , replay of earlier generations, proposals diversified per manual section, and the whole learner behind Brain’s API. A generation proposes up to 600 programs, predicts each greedily and trains a LoRA continued from the champion (rank 16, learning rate , 40 steps) on the mispredicted programs, with the correctly predicted ones and earlier generations’ records each drawn at a share of ; the examiner now excludes overlapping items.[RS09,RS05]
Every lineage improved at generations 1 and 2; at generation 3 r2 and r3 improved again, r0 held flat (29/28, ) and r1 fell (18/28) (Table 1). Two of four lineages succeeded and the plan verdict reads not met; pooled, the steps were 288/0, 199/60 and 142/92, and no exam item matched a training program.[RS21,RS22] The learner’s records show the signal running out: over the four lineages the champions mispredicted 2127 of 2400 proposals at generation 1, 1125 of 2228 at generation 2 and 860 of 1942 at generation 3, so later generations trained mostly on what their predecessor already knew, while each also continued the previous adapter.[RS23] The learner’s dev gate accepted all 12 steps, and the examiner found 2 of them not to hold: r0’s and r1’s third steps, the two smallest dev margins (13/8 and 7/5).[RS24]
Serving configuration. The examiner served S4’s models with an f32 KV cache, at which the base answers 12 exam items; S3’s stored baseline was served at the build’s default tier, at which it answered none, so S3’s and S4’s levels are not comparable, only their steps are.[RS27]
The learning loop
Three projects divide the work, and the division is enforced by a layering file and a gate rather than by convention: Sven runs agents and keeps the record of what an agent did, Brain computes (inference, training, checkpoints, adapters, serving), and Splinter decides what is learned, which experience qualifies as training data, what to train and whether to release it. [S05,S01,S02] Figure 8 shows one turn. A source is captured with its provenance; a model proposes tasks and code admits or refuses each for a named reason; an agent attempts them in a confined copy of the tree; verifiers that the policy cannot reach keep what they confirm and an exposure audit withholds attempts that touched hidden material; verified experience is projected into training records with whole held-out groups; Brain trains an adapter or every weight from the current champion; and a release gate admits the candidate only if four checks on material it was not trained on pass (improvement by a paired sign test, retention of earlier releases, an anchor of general tasks, and agreement of the served model with the trained one). Releases are immutable, an alias moves, and rollback moves it back. [A03,V02,L08,G01,X02,X06] Every stage is specified, with its ledger, in Appendix § 11; this section is what the experiments below need.

Figure 8. The self-learning loop, read clockwise from the source. Every stage keeps what passes and refuses the rest with a named reason (dotted arrows; Appendices § 12 to § 15 and § 17 give the full lists). Splinter owns tasks, verification, datasets, the gate and releases; Sven runs the attempts; Brain trains and serves. The dashed arrow is the next turn: attempts run on the policy the release alias points at.
In the experiments of § 3 the loop is reduced to one task kind whose answers an interpreter fixes, the verifier is the interpreter, and the release gate is replaced by the learner’s own dev gate and, for the registered decision, by a separate examiner program. In Appendix § 20 the full loop runs on repository tasks.
Division of labour
The flywheel is a composition of three projects with one rule of ownership. Sven runs agents (tools, sessions, delegation) and keeps the authoritative record of what an agent did. Brain computes: inference, training algorithms, checkpoints, adapters and serving. Splinter decides what is learned, which experience qualifies as training data, what to train and whether to release the result. [S05] Splinter’s repository states that neither Sven nor Brain depends on it, so each can demonstrate its own capabilities without it and the released artifact is an ordinary Brain adapter. [S05,X05]
The boundary is enforced by a machine-readable layering file rather than convention. Splinter’s crates are placed in ten tiers from a dependency-free foundation to standalone samples; a crate may depend only on a strictly lower tier. [S01] Only the model adapter crate may depend on Brain, only the agent adapter and the model adapter may depend on Sven, and the command line and every sample depend on the SDK and nothing else of Splinter’s.[S02,S03,S04] A model is never named by a stage: stages ask for one of seven roles (policy, teacher, generator, planner, judge, critic, router) and configuration decides which checkpoint plays it.[S06] Experience is immutable; verdicts, rewards, step labels and dataset membership are facts derived about it, never written into it.[S07]
Source intake
A source is content addressed (a blake3 digest over its canonical origin and parts) and has one of four origins: a document, a repository, a command, or a recorded session.[I01] A document is one UTF-8 text file of at most 1 MiB; larger or non-text files are refused.[I02] A directory is captured file by file, ignoring .git, target and node_modules, and every file left out is recorded with a reason.[I03] A command is run without a shell, with a 30 s timeout, a 1 MiB output cap and an environment limited to PATH, HOME and LANG, so its real output becomes a source with provenance.[I04] A repository revision is archived as a snapshot of its tracked files and a dirty tree is refused.[I05] A recorded agent session is an ATIF trajectory of at most 64 MiB; unknown ATIF keys and sessions without a user step are refused; secrets are removed before storage and only the fact of the removal is kept.[I06] On the command line, source add takes a file, a directory, or cmd: followed by a command.[I09]
Text is cut into sections that are byte ranges: Markdown at headings outside code fences, other text at blank lines. Evidence points at a section by content digest and byte offsets, so a task’s grounding can be resolved back to exact source bytes.[I07] A generator model is shown three sections per request.[I08]
Tasks and admission
Who decides. Models propose; code admits.[A03] The built-in catalogue holds model-writable kinds (recall, taught, arithmetic, format, advise, converse, explain, predict, construct, debug, counterexample, transform, classify, retrieve, multi-turn, combine); repository-change tasks are made by code and never proposed by a model.[A01] One kind, denoise, is built by code from the source itself: a window of 12 words with one word dropped and two neighbours swapped, graded by exact match, so it needs no generator model.[A02]
The funnel and its rejection reasons. A model-written task is refused for one of twenty-one reasons: malformed, no_reply, over_count, unknown_section, quote_not_found, too_few_sections, unresolved, missing_material, material_not_shown, not_self_contained, wrong_form, ungrounded, not_quoted, checks_failed, test_not_admitted, duplicate, near_duplicate, reference_changed, no_subject, contradiction, invalid.[A04] Three deserve explanation. Grounding: a text answer is admitted only if at least 80% of the reference’s content words and every number in it occur in the cited evidence.[A05] Subject: a task must name what it is about, so that someone who never saw the source gets exactly one answer, and is self-contained only if it does not quote a dropped privileged item of eight or more words.[A06] Contradiction: two tasks on one subject whose grounded references disagree are both refused.[A07] The batch driver gives up on a kind after 24 proposals if fewer than a tenth were admitted, and sends the generator a correction after three refusals for one reason, at most twice.[A08] Repository tasks have their own vocabulary of refusals (red or flaky baseline, equivalent mutant, tests not separable from the change, not restorable, already held and others).[A09]
The task report counts, per kind, tasks admitted and rejected, rejections by reason, every rejection with its detail, and each model call with its pre-admission candidate count, so the funnel is read from the report rather than reconstructed.[A10]
Verification and leakage control
Rollouts. Sven runs the attempt. A closed-book solve builds an engine with no tools;[R01] a code task gives the solver exactly one tool, run_code;[R02] a repository episode uses Sven’s coding toolset with automatic approval inside a fresh copy of the snapshot.[R03] Every request runs under Splinter’s single system prompt,[R04] and a solve refuses to run in an environment other than the one its task records.[R05]
Experience and verdicts. An experience is content addressed by the digest of its canonical JSON.[R06] Verdicts, step labels and relations are annotations appended beside it.[R07] Verdicts have a strength ordered judged, consistency, formal, executable, constraint.[R09] The decision rule ignores abstentions, uses only verdicts of the strongest rank present, and returns no decision if they conflict;[V08] the reward is derived from the decision and never stored.[R08] Tasks are graded by one of thirteen verifier kinds.[V01] A verifier sees the output and privileged material but never the trajectory.[V02] Examples: a stated reference may be matched by an answer at most four times its length plus 60 characters;[V05] an executable check passes only if a completion line derived from its digest is seen, so a solution that exits early cannot pass;[V04] a generated test oracle must kill every one of at most 32 mutants;[V06] a repository verifier overlays the hidden checks on a fresh copy and fails a patch that touches a protected path.[V07] A judge model may grade only when calibrated to a precision of at least 0.9 and never grades its own model’s output.[V03]
Leakage. Four mechanisms separate what is trained on from what is measured. (i) Held-out records are whole groups, a tenth of the samples grown to at least eight units and at most a quarter;[L01] groups are formed by merging texts that share at least eight distinct runs of eight words, ignoring runs held by more than six texts.[L02] (ii) Before grading, every gate task that matches a prompt the candidate trained on, including replay and rehearsal, is excluded and counted as leaked.[L04] (iii) For recorded sessions, sealed probes refuse any training record that contains a probe’s question or shares an eight-word run with it.[L05] (iv) For repositories, exam families are reserved by stable hash before any task is generated, and a family needs at least 120 words;[L07] the exposure audit fails an attempt whose tool calls name a hidden path or a forbidden string, or whose outputs repeat a 12-token shingle of hidden content;[L06] exposed attempts and harness failures are withheld and counted in no denominator.[L08]
Datasets and corpus expansion
Projection. Datasets are projected from verified experience by named views and six objectives (SFT, classification, DPO, contrastive, reward, continued pre-training).[D01] SFT is written as Brain’s generic-messages-v2 and DPO as generic-preference-v1.[D02] A manifest stored beside the file names the views, experiences, tasks and sources every record came from.[D03] The student’s turn is the instruction alone: privileged teacher context is stripped by default;[D04] only the assistant action is supervised, system and user turns carry no loss;[D05] compacted or pruned trajectories are excluded rather than repaired.[D06] The training set is capped per concept (0.25), task kind (0.5) and verification strength (0.75), after deduplication by the same near-duplicate rule as admission,[D07,C03,C04] which compares 3-word shingles and refuses a Jaccard overlap of at least 0.8.[L03] Replay sampling is deterministic in the content digest and the seed.[D09] A monitoring split of whole families (at least three, default share 0.1) supports early stopping.[D08]
Packing. The step view can also write a trajectory as one record instead of one record per action: the conversation up to its last admitted action, with every admitted action supervised and every other message as context. It supervises exactly what the per-action records supervise together, with the context rendered once instead of once per action, and within a step each supervised token weighs the same in both forms.[D10] What differs is the unit a step counts: a step of eight packed records spans eight whole trajectories rather than eight actions, so the same step count moves the adapter on far more tokens. The choice of form is reported with every run that uses it (Appendix § 22).
Expansion beyond the first pass. The corpus grows by five mechanisms, each with its own gate. (i) Variants: a model-written task may be given up to three differently worded questions that keep its reference, evidence and verifiers; a variant that changes the reference is rejected, and variants are only measured, never trained on.[C01,C02] (ii) Teacher solves: a task the policy never solves is solved once more by a teacher shown the source sections it is grounded in, and the answer is kept only if the task’s verifiers pass it.[C08] (iii) Retention queue: when a retention check fails, the forgotten concepts are queued and their sections receive new tasks on the next run.[C05] (iv) Rehearsal: half code-built form tasks and half general requests the base writes, eight per request, answered greedily by the base; requests matching the anchor are excluded and repeats refused.[C07] (v) Claims from sessions: per claim, a model writes twelve paraphrases of which four are held out for stopping, plus four statement, two reverse and two implication forms;[C10] a claim of kind procedure is stored and never trained on.[C11] Mastery of a concept is its pass rate over the last 32 decided attempts and the five weakest are listed.[C06] Exam and development families are reserved before generation (50 each by default, 8 tasks per exam family, 4 per development family).[C09] We found no mechanism that generates new tasks from failed attempts other than the retention queue and the teacher solve.
Claims intake. A claim is a correction, a fact or a procedure;[K01] extraction keeps only claims that each of two passes produced.[K01] A claim is admitted by code only when every quote is a verbatim substring of the person’s own message at the cited step,[K02] and every number, quoted term and name in its statement occurs in the cited words;[K03] eleven refusal codes are defined;[K04] a ruling is admitted (superseding earlier claims), reinforced, forgotten or refused.[K05]
Training
Splinter chooses the objective and the data; Brain runs the optimisation. LoRA supervised fine-tuning is Brain’s ChatFineTune and DPO is its PreferenceFineTune; both are called from the model adapter crate.[T01] The defaults are a learning rate of , rank 8, alpha twice the rank and no weight decay;[T02] the step budget is three passes over the records clamped to between 40 and 2000 steps.[T03] A candidate continues the champion: its training starts from the parent release’s adapter,[T06] mixes in a replay of each earlier release’s records (a quarter of each earlier release’s records, entering a quarter of training draws),[T04] and mixes in a rehearsal of the base’s own answers (a quarter of draws, half of it code-graded form tasks).[T05] A monitored run keeps the best evaluation’s adapter.[T07]
Consolidation. A policy that keeps learning would grow its adapter, or must retrain from scratch on every claim. Splinter therefore consolidates a generation into the weights in one of two modes: a full fine-tune of every weight of the generation’s base, or merging the adapter into the base.[T08] The full mode defaults to a learning rate of for six epochs with a retention replay share of a quarter;[T09] a generation is consolidated automatically after more than 2000 trained records.[T10] Both are carried out by Brain (ChatFullFineTune, merge_adapter).[T11]
Release gate, releases, rollback and lineage
The gate. A candidate is released only if four checks all pass, each over material it was not trained on.[G01] A check that was not measured fails with the reason not measured.[G02]
Improvement: a one-sided exact paired sign test at over discordant clusters, a family of items counting once; the candidate must also have solved at least as many of the suite’s assigned tasks as its champion, an attempt withheld as exposed or as a harness failure counting as unsolved for the arm that produced it.[G03] Retention: accuracy on every earlier release’s held-out suite may not drop by more than 0.05.[G04] Anchor: accuracy on the frozen suite of general tasks may not drop by more than 0.02; the report carries a 95% bootstrap interval over 4000 resamples.[G05] Serve: plain brain serve must report the candidate’s adapter digest and agree with the in-process answers on at least 75% of up to eight re-answered held-out tasks.[G06] Repository exams use the same four checks, with the exam families as the improvement suite and agreement judged on the graded verdicts.[G07] The bootstrap seed is drawn when not given and recorded;[G08] a failed retention check queues the forgotten concepts;[G09] and only a candidate trained from the current champion can replace it.[G10]
Releases. A release manifest (format splinter-release-v4) records the artifact, parent, candidate, datasets, replay, rehearsal, training settings, the whole gate report, metrics, terms, provenance and generation.[X01] A release is written once and an existing one is refused;[X02] unrestricted distribution is refused unless the terms of everything it was made from permit it.[X03] The default policy is the alias policy:default; moving an alias is a compare-and-set from the release the gate measured against.[X04] After every move the serving manifest <state>/serving/<alias>.json is rewritten, so brain serve --adapter-manifest follows the release without a restart.[X05] splinter rollback moves the alias to the release the current one was trained from and refuses when there is none.[X06] splinter lineage walks any artifact up to its source bytes and down to everything derived from it, computed from the stores on every call.[X07]
Training a 27B model on one card
Training a 27-billion-parameter model on one 95 GiB card needs the frozen base in a reduced-precision tier, activation recomputation, a loss that never materialises the full logits, an attention kernel that supports the model’s head dimension, and a loader that never holds the checkpoint in host memory. This section states what Brain’s main tree contains for each, what its qualification runs measured for bf16 LoRA and NF4 QLoRA, and what a later throughput measurement of the same architecture shows. What a trained 27B adapter changes in what the model does is measured in Appendix § 22.
What the tree contains
The Splinter trainer’s seam is the Qwen3-family trainer.[B01] In that trainer the frozen base can be held in bf16,[B02] each layer is recomputed in the backward pass from its saved residual,[B03] and the logits are sized in row chunks with a chunked weighted cross-entropy.[B04] Brain has a causal grouped-query flash-attention forward and backward supporting head dimensions up to 256, with finite-difference gradient checks at head dimensions 40, 128 and 256.[B05,B10] For the 27B model’s crate it has a streaming FP8 source that dequantises in 128-row bands and a base-dtype enumeration with F32, bf16 and NF4 members,[B06] and an NF4 weight type with a 16-entry lookup table.[B07] The numerical gates are make test/gradcheck (finite-difference checks of the Qwen3.5 architecture including LoRA) and make test/parity, which runs the checks on the CPU backend.[B08,B09]
Qualification runs
Brain’s qualification at commit 258eaba9 ran the 27B architecture on one 95 GiB card; Table 4 lists the runs. Probes use a randomly initialised model of the real width and depth (64 layers), recomputation, rank-16 LoRA and one step, so their loss is not a quality number. With an NF4 base, 8192 tokens fit in 35947 MiB at 20 tokens/s;[M11] with a bf16 base, 8192 tokens took 67011 MiB and 16384 tokens 77879 MiB, both at 19 tokens/s.[M12,M13] Through the command line the real FP8 checkpoint trained one step with an NF4 base and with a bf16 base with a host peak of 6924 and 6970 MiB respectively[M14,M15] and, with an NF4 base, 40 steps drove the loss on eight training examples from 2.7806 to 0.0002.[M16] That is a statement about fitting the training examples, not about generalisation: the validation examples were two of the training examples. The command line’s default batch ran the device out of memory at 1024-token rows.[M17]
| Run | Base | Tokens | Peak MiB | Seconds | Outcome |
| probe, 64 layers, random init | NF4 | 8192 | 35947 | 404.74 | step done |
| probe, 64 layers, random init | bf16 | 8192 | 67011 | 435.47 | step done |
| probe, 64 layers, random init | bf16 | 16384 | 77879 | 873.64 | step done |
| 27B, 1 step | NF4 | 1024 | 27989 | 460.4 | finished |
| 27B, 1 step | bf16 | 1024 | 59053 | 321.1 | finished |
| 27B, 40 steps | NF4 | 256 | 24559 | 936.8 | finished |
| 27B, default batch | NF4 | 32768 | 92571 | 322.8 | out of memory |
Table 4. Qualification runs of the 27B architecture on one card (Brain’s qualification logs, copied unmodified apart from path names; their values, digests and commands are in data/measurements.json).
Throughput as measured on a later Brain revision
The qualification rates above are those of the commit it ran at. A later probe of the same architecture (random weights, 64 layers, rank 16, NF4 base, 8192 tokens, recomputation) on the idle card measured 622 tokens/s at steps two and three, a step of 13.18 s and a peak of 44725 MiB.[M38] Brain’s own record of the real checkpoint at 8192 tokens states 635 tokens/s with an NF4 base, and 640 tokens/s with a bf16 base at 79.8 GiB;[B13,B14] we did not rerun those two, and the same record states that 1000 tokens/s was not reached.[B15] At the measured probe rate one million trained tokens take 0.45 h of card time, computed as .[C12] We state the measurement and not a target; a rate is only valid for the revision, the card and the sequence length it was taken at.
What this does not show
That a trained 27B adapter changes what the model generates. The qualification logs predate a generation path for such an adapter; Brain has since added one (the sample-adapter verb of its qwen35 command)[B12] and keeps its own record of a run, which we did not rerun or verify here. No run beyond 16384 tokens and no NF4 run at 16384 tokens on the qualification commit exists.
What the loop needs in order to work
A turn of the loop adds a capability only if its stages cannot be satisfied by an artefact of the setup. This section states the requirements we found necessary, the mechanism that implements each, and which are not yet implemented. Each names its commit or file in the ledger, which checks that the commit is in the history of the pinned revision of its repository (§ 8).
Held-out units must be independent. Records carry the group of the source text they are grounded in, and a held-out measurement needs whole groups. A dataset that is one source text cannot be divided, so the split refuses it with a typed error naming the cause instead of dividing the group and failing the leakage check afterwards; a free record follows the fixed side of its group, monitoring with no whole group to set aside is skipped with the reason recorded, and a source that is one independent text is refused at intake before any rollout is paid for.[FX01,FX05] A source of independent units is therefore one file per unit.
Admission must see through markup. The subject of a task is compared with the instruction, the source and the variants through one function that ignores code quotes, quotation marks, emphasis and a call’s empty parentheses, so that “the q key” contains the subject “q key”.[FX02] A number written with a thousands separator is the same number in the stated-reference check.[FX06]
The serve check must start the real server. The gate’s serve check runs brain serve --adapter. A candidate’s card id must follow Brain’s adapter grammar (base, owner, name, tag), and the check must pass the bank option that covers every linear Splinter’s LoRA corrects; the test double of the server refuses an adapter offered without it, because a double that accepts any flags hides both.[FX03,FX04]
The anchor bound must count net lost answers on a large enough suite. A rate bound of 0.02 on a 40-question anchor suite allows no lost answer, since one net loss is 0.025. The gate can decide the anchor on answers lost net of answers gained (an option; the default rate bound is unchanged), and a 200-question anchor suite built by code and a reviewed list exists.[FX07]
Retention must test acquired capabilities. The retention suite of each earlier release is built from that release’s held-out split, and the exams of workspace families are added to it.[OP01] For a domain of facts the held-out split holds facts the release never taught, so a passed retention check is not evidence that taught facts were kept. A retention exam drawn from what each release demonstrably acquired is not implemented; the registered coding-task retention check (Appendix § 21) uses the exam families of every earlier round for this reason.
The verifier must prove that the tests ran. The workspace verifier passes a patch only if every fail-to-pass and pass-to-pass command exits with status 0.[OP02] That does not prove that the intended tests ran, so a suite that exits zero without running them would pass. At the current head a command that carries a test expectation passes only when the runner’s own report shows every expected test run and passed, with readers for python3 -m unittest and cargo test; a command without an expectation is still judged by its exit status.[OP05] The coding-task experiments register conformance measurements for the verifier before they count (Appendix § 21).
The gate must carry canaries. A gate with an empty canary set skips that check. At the revision the paper’s architecture claims are pinned to, the release gate called from practice was built without canary tasks;[OP03] at the current head it is handed up to 20 canaries made from the exam the run reserved, so a release from practice carries that evidence.[OP04]
Trajectory and training requirements of repository tasks
Training an agent on its own repository trajectories adds requirements to the stack. Each is located in source; the ledger names the file, commit or test.
Loss masking of tool conversations (Brain; implemented). Under the Qwen3 template an empty reasoning block is added to the last assistant turn only, so non-final assistant turns
of a multi-step conversation failed the trainer’s boundary check. Tool-calling conversations now train per assistant turn in both thinking modes.[AU01]
Tool definitions lost between recording and training (implemented). The agent records the tool definitions of the run, but the trajectory projections dropped them; an ATIF trajectory’s tool definitions now become the chat sample’s tools, and Splinter’s dataset check covers tools.[AU02]
Trajectory fidelity in the agent SDK (Sven; implemented). The exported trajectory marked no context compaction and kept the uncompacted history, listed every registered tool rather than those offered, and turned unparseable tool-call arguments into an empty object. Compactions are now marked and the compacted history kept, the offered tools are exported, and the raw text of non-JSON arguments is kept.[AU03]
The record is the whole conversation (Sven; implemented). After a context compaction the export was rebuilt from the compacted history, so a run that compacted lost the steps the compaction replaced and the model requests recorded on them. In the withdrawn 27B run the five episodes stopped at the 60-call cap counted 61 or 62 tool calls while their trajectories held four to 42. The agent now keeps an append-only record beside the history the model reads, a child run’s store keeps a transcript the same way, and a compaction is marked where it fired. The projection into training conversations still refuses a trajectory that holds a compaction marker, so such episodes can be analysed and are not trained on.[TJ03,TJ04]
Sampling identity on every request (Sven and Brain; implemented). A run’s sampling (temperature, top-p, top-k, seed) and its request for the sampled tokens reach every model request of the run and of the children it starts. Brain’s chat endpoint returns the prompt token ids, the sampled token ids, the log-probability of each sampled token under the distribution it was decoded from, and the seed it used, the adapter, the base-checkpoint digest, the backend and the engine build; a step of the trajectory records them, and records each request’s own prompt and tokens when a step covers several. A seed the server drew above was once dropped from the reply and refused when sent back.[TJ05,TJ06,TJ07]
Crash-robust capture (Sven; implemented). The trajectory can be written one record at a time as the run happens, with a flush policy and size bounds; a journal that was never closed reads back marked incomplete, a test kills a real second process inside a model request and reads the journal back, and Splinter’s projection refuses a trajectory marked incomplete.[TJ08,TJ09]
Children in the parent’s record (Sven; implemented). A harness run’s trajectory embeds every finished child as a trajectory with its tool actions and the account of each of its model requests, sampling identity included.[TJ10]
Release probes were closed-book (implemented). A coding agent is now gated on the exam its run reserved, with report types for workspace suites.[AU04]
Contamination paths (implemented). The semantic-memory tool is withheld from the worker, and a context tool that could open any readable path is jailed.[AU05]
Silent waits (Sven; implemented). A model server that goes quiet is bounded and the wait is reported, with a stalling mock provider to test it.[AU06]
Tool-definition mismatches (Sven; implemented). Definitions disagreed with the implementation and were tightened.[AU07]
Serving several adapters (Brain; implemented in the qwen3 server). An adapter bank lets one batch serve the base and many LoRA adapters.[AU08]
Containment (implemented for commands run on the host; a container remains the stronger option). A solver’s forbidden operations are prevented by the kernel, not detected afterwards by the exposure audit. A command of the solver (its shell, and the checks, and a code call of the run_code tool[CN13]) runs under Landlock with read access to the system and write access to its own tree and a private directory only, so the withheld tests,[CN01] a path traversal or a symlink out of the tree,[CN02] the host’s credential files,[CN03] and a sibling candidate’s
tree[CN06] are refused with a permission error, also when reached through sh -c, an interpreter or a compiled helper the candidate brought along.[CN07] The command’s environment is built from an allowlist from which names that look like secrets are dropped even when the allowlist names them;[CN04] Landlock’s ptrace check already refuses the files of another process’s /proc directory that need ptrace access (its environment among them),[CN05] and the command runs in a private user and pid namespace with a /proc of its own, in which no other process of the host exists at all;[CN14] the network is denied (Landlock’s TCP rules and a socket filter that also covers UDP), so a command cannot reach a model server on the loopback;[CN08] and a process budget stops a fork bomb when the operator sets one.[CN09] A kernel without Landlock ABI 4 refuses to run the command rather than run it open. The agent’s own file and search tools are confined to the attempt’s tree by the agent framework’s project root, which its delegated children share.[CN10] Only an experience that an independent verdict passed at executable strength is training evidence.[CN11] The step-level view of the training data also admits an action a labeller marked good from an experience nobody verified; that is the default, --control outcome-verified turns it off, and the dataset manifest records which was used.[CN15] A path outside the tree that a command names is reported by what came of it (the sandbox refused it, it did not exist, or something came back), so the audit separates a harmless mention from an access; all three still fail it.[CN12] Not covered: the process budget and network opt-ins change the environment record, so they are off or unset by default to keep the identity of tasks generated earlier, and a kernel that cannot create a user namespace refuses to run a command unless the operator allows the host’s /proc.
After the first generation: teaching, acquiring, choosing, retaining
Four further requirements follow from asking the loop to run past one repository and one round. Each is a property of the stack, shown by a specification that runs on scripted agents and a test double of the trainer; none is a measurement of a trained model. The experiments that would be are registered in prereg/acquisition.md (ACQ1, INT1, GEN1)[PL03] and were not run (§ 7).
A task nobody solves needs a teacher that is not trusted. A repository task has no passage to show a teacher, so practice used to drop every task the policy never solved: the frontier keeps only what is solved sometimes. A teacher is now named for those tasks. It is either the task’s own reference change (the inverse of the seeded fault, which the task keeps and the student is never shown) replayed as the actions of an agent through Sven’s tools, or a stronger model solving the task as an agent. Either produces an experience that is graded by the swarm workspace verifier and the same exposure audit as the policy’s attempts: a demonstration that does not make the hidden checks pass is not kept, whoever proposed it.[TC01,TC02,TC03] A replayed demonstration names the file without searching for it, so it teaches the edit once the place is known and not the localisation; that is a stated limit.[TC04]
The probes must exist before the training and decide nothing in it. A run that turns documentation into skills measures each skill on probes written from the skill’s own sections and checked to exercise it. The probes are split: the dev half measures the baseline, each round’s gain and what a round cost the skills measured before, and the final half is touched only by the closing assessment, which compares the policy the run began with and the one it ends with. No training task repeats a probe, and no teacher answers one.[AQ01,AQ02] A candidate that learns nothing is rejected, counted against its skill, and leaves the policy as it was.[AQ03] What a planner model proposes is kept only as far as the text bears it out: a quote that is not in a section the skill cites, a symbol that is not in them, a prerequisite that is no skill or closes a cycle are refused with the reason.[AQ04]
An intervention is chosen by what it was measured to do, net of what it costs. The choice among more samples, another episode protocol, a trained candidate and no change rests on the lower confidence bound of the paired gain over families, priced against what a success is worth, and a mechanism that is not available is reported as not measured, never as a gain of zero.[IV02] On four synthetic clusters whose outcomes are drawn from a known world, the choice differs by cluster and its realised value exceeds every strategy that applies one intervention to all four.[IV01]
Retention is measured on every generation, not inferred. A generation of the coding policy measures the champion on every protected task before it trains and the candidate and the champion on the same tasks at the same settings; the full matrix has the base as generation 0 and every generation on every protected task, so backward and forward transfer have measured cells.[GN01,GN02] A candidate that loses a protected family beyond a limit, or does not gain on the new exam, is rejected before the gate is asked, the alias stays, and the rejection is recorded with the numbers it rested on.[GN03] Replay is a policy rather than a constant: none, uniform, balanced over capabilities, weighted by the champion’s failures, or a rebuild from the base on everything accumulated.[GN04]
A language’s version is part of the environment. The Lua families and the lua runtime ran on whatever lua the host’s alternatives named. The runtime now refuses an interpreter whose version line is not Lua 5.1, and the interpreter is built from its checksummed release tarball.[LU01]
Repository tasks with a stronger teacher: the confirmatory query
Question and design. Can verified engineering-agent experience improve completion of unseen repository tasks?[PR09] The registered test (Appendix § 21, registered before any run) trains an 8B student on the verified trajectories a 27B teacher produced on development families and compares it with the untrained 8B on a frozen qualification set of 46 task families, each task solved at three registered decoding seeds, by a one-sided exact sign test over families at .[PR15,PR04]
Outcome: not shown. The candidate registered before the query, the decisive-action adapter of training seed 13, solved 23.0 of the 46 tasks on average over the three seeds against 20.0 for its base, winning 18 families, losing 12 and tying 16 (): the candidate is not demonstrated better than its base.[QUAL1] It had been chosen on a six-seed development read in which it solved 24.5 against the base’s 19.67 (), so part of that margin was selection.[PROTO06] A secondary contrast did separate: against its all-action control, trained on the same corpus with the same recipe, seed and binary but supervising every action instead of only file edits and writes, it won 26 families and lost 11 (). The control itself solved 16.33, below the base, so the contrast shows that supervising every action does harm rather than that supervising decisive actions helps.[QUAL1]
What else ran. Before the query, pilots of the headline and of a 27B model trained on its own trajectories showed no held-out improvement on a 23-task subset, and the base solved twice at two protocol seeds differed from itself by as much as either adapter did from it.[M47,M52,M53,M56,M57] Appendix § 21 holds the protocol, the pilots, the development probes and the run-to-run variation they exposed.
The coding-task experiment: protocol
The question that matters to a team, once the loop of Appendices § 11 and § 13 to § 15 exists, is: given a git repository, can a local agentic model be tuned to work competently in it, repeatedly and with further repositories, and does it get better at what it has met rather than only at the newest? This section states the protocol that asks it. It was registered before any run (prereg/coding-tasks.md; the changes made since are listed in Appendix § 28); everything in it is a registered rule or a statement about the code; the coding-task measurements are reported in Appendix § 22.[PR00]
Tasks. A task is a snapshot of a repository tree at a revision, an instruction stating a symptom, and privileged material the solver never receives: the hidden checks (fail-to-pass and pass-to-pass commands) and the paths that carry them.[PR01] A candidate task is admitted only if the snapshot’s selected tests pass twice, a change makes a previously passing test fail reproducibly, and the reverse change restores green (Figure 9); a red or flaky baseline stops the run.[PR01,PS01] Splinter’s repository tasks are made by code and never proposed by a model, and have their own refusal vocabulary.[A01,A09] The failing tests are moved out of the tree the solver receives, and a repair passes when the hidden files are overlaid fresh on the patched tree and the fail-to-pass and pass-to-pass commands pass; a patch that touches a protected path fails.[V07]

Figure 9. Admission of a mutation-seeded repository task, left to right. Every check is run on a fresh copy of the snapshot, every refusal is counted by its reason, and an admitted task’s failing tests are moved out of the tree the solver receives.
Solving, recording and trajectory data. The solver is a Sven coding worker with file, search and shell tools, a tool-call budget and a repetition stop, started in a fresh copy of the snapshot. One harness profile is pinned for every solve of every arm and for data collection alike: at most two follow-up turns, at most sixty tool calls and a repetition stop at three; a follow-up is given when the visible checks stay red and also when a turn ends with the tree unchanged, asking the worker to make the change by calling the edit tools.[R03,PS01] The recorded unit is the full trajectory; the verdict is produced outside the agent. Only attempts whose patch passes and whose exposure audit is clean are eligible for training, and for the headline experiment the attempts are those of a stronger teacher model, not of the student (Appendix § 22);[PR15][L06,L08] each assistant action becomes a training record conditioned on the prior conversation and on the tool definitions the agent was offered, with system and user turns carrying no loss, and compacted trajectories are excluded rather than repaired.[D04,D05,D06,AU02]
Validity and leakage. Four registered rules keep the exam honest. (i) Families are assigned to partitions by a stable hash before any rollout, and a family of the qualification partition is never solved for data purposes.[PR02,L07] (ii) Hidden files are absent from the solver’s tree and are materialised only in the verifier’s.[V07] (iii) An exposure audit fails an attempt whose tool calls name a hidden path or whose outputs repeat a shingle of hidden content, and exposed attempts and harness failures are withheld and counted in no denominator of the paired test; in the release gate’s improvement check they count as tasks the arm did not solve, so a candidate cannot remove a loss from the test.[L06,L08,PR06] (iv) The isolation level of every run is recorded; with none a host command is still confined by the kernel (Appendix § 19.§ 1) and the audit is a second line, and a task whose soundness needs a container is not admitted.[PR06,AU09,CN01] Whether a repository is in the base model’s pretraining data is not known; the base arm on the same exam bounds that effect without removing it.
Units, arms and the paired exam. The unit of analysis is the family, the file a fault was seeded in for the authored repositories and the subsystem for real ones.[PR02] Tasks within a family are not independent, so no test treats them as such. The base arm is the unmodified checkpoint in a pinned harness profile; the adapter arm is the same checkpoint with the adapter attached; system prompt, tool schemas, budgets, decoding (the model family’s recommended sampling, each task solved under three registered seeds and pass@1 averaged over them, thinking off as the primary variant and a 1024-token thinking budget as the secondary one) and environment image are identical.[PR03] A family is won by the adapter when its resolved fraction is higher, lost when lower, and dropped when tied. The primary decision is a one-sided exact sign test over the non-tied qualification families at , on the adapter served through Brain and driven by Sven; a result that is not significant is not shown, never equivalence.[PR04,G03]
What the test can decide. Table 5 and Figure 10 give the wins needed and the power at assumed win probabilities: with fewer than five non-tied families no decision is possible, 12 wins are needed at fifteen non-tied families and 18 at twenty-five, where the power at a win share of 0.8 is 0.89.[CP01,CP02] The registration therefore asks for about twenty-five non-tied qualification families, registered as 46 families (28 Python, 18 Lua) before any run, with the base-versus-teacher discordance on development families reported as the sizing input, and labels a smaller comparison a pilot without changing the rule.[PR04]
| Non-tied families | Wins needed for | Smallest (all wins) | Power if | 0.8 | 0.9 |
| 4 | none | 0.062 | 0.00 | 0.00 | 0.00 |
| 5 | 5 | 0.031 | 0.17 | 0.33 | 0.59 |
| 6 | 6 | 0.016 | 0.12 | 0.26 | 0.53 |
| 8 | 7 | 0.0039 | 0.26 | 0.50 | 0.81 |
| 10 | 9 | 0.00098 | 0.15 | 0.38 | 0.74 |
| 15 | 12 | 3.1e-05 | 0.30 | 0.65 | 0.94 |
| 20 | 15 | 9.5e-07 | 0.42 | 0.80 | 0.99 |
| 25 | 18 | 3e-08 | 0.51 | 0.89 | 1.00 |
| 30 | 20 | 9.3e-10 | 0.73 | 0.97 | 1.00 |
Table 5. The exact one-sided sign test at : wins required and power if each non-tied family is a win with the stated probability (a design calculation computed by scripts/build_data.py, not a measurement).

Figure 10. Power of the exact one-sided sign test () against the number of non-tied families. The sawtooth is the discreteness of the test.
Retention across rounds. After each round , the exam families of every earlier round are solved again by the round- adapter and by its predecessor. Retention holds only if the one-sided 95% lower bound of the change in resolve rate (family-cluster bootstrap) exceeds absolute; a failed check refuses the candidate.[PR05] The gate of Appendix § 17 already holds each earlier release to the exam of its own run for repository runs;[G07] the gate’s suite does not yet test acquired capabilities in every domain (Appendix § 19), which is why the registered check is stricter than the gate’s.
Controls and stop rules. A gain must survive three controls, each a separate training run on the same budget: training on trajectories regardless of verdict (does verification matter), an adapter trained on a different repository (did it learn the tool protocol rather than the repository), and no replay of earlier rounds; for teacher-distilled data these are teacher trajectories regardless of verdict and verified teacher trajectories of a different repository.[PR07,PR15] Registered stop rules include dropping a task kind whose verifier false-pass rate exceeds 5% and reporting a development gain without a qualification gain as memorisation.[PR08]
The code that exists. splinter practice implements the stages of a single-repository round from a revision to a gated release.[PS01] Eight authored repository families ship with the repository (Table 10); a family’s tests must pass five times identically and its faults be caught before it is catalogued.[PS02] Two verbs apply the partitions and the paired comparison of this section to the base and the served adapter. splinter tasks pick SET... --family-share F --seed S divides the union of task sets into a picked set and the rest by a stable hash of the seed and the family, whole families at a time (--per-family N caps each family instead, for the one-task-per-family verdict sample). splinter experiences compare CANDIDATE BASELINE pairs the two arms’ experiences by task, counts a family once, and reports the exact one-sided sign test the release gate uses, the per-family wins, losses and ties, quality, cost and autonomy with family-bootstrap intervals, and the share of tasks on which the arms’ verdicts agree; exposed and harness-failed attempts are left out and counted.[PS09] Appendix § 25 gives the command for each step of a replication.[PL01]
| Family | Language | Tests | Admitted tasks |
| iniparse | python | 279 | 177 |
| ttlcache | python | 257 | 159 |
| exprcalc | python | 256 | 186 |
| linediff | python | 278 | 152 |
| mdrender | python | 270 | 148 |
| caldate | lua | 250 | 132 |
| graphkit | lua | 262 | 155 |
| tokenbucket | lua | 296 | 135 |
Table 10. The eight authored repository families: language, tests, and tasks admitted by the fault-seeding funnel (values read from the family catalogue at the pinned revision and checked by the ledger).
The coding-task experiment: results
The question. Can verified engineering-agent experience produce a sustained improvement in completion of previously unseen repository tasks, under a controlled compute budget and without unacceptable loss of earlier capability?[PR09] The registered confirmatory query did not show it: on the 46 qualification families the candidate registered before the query solved 23.0 of the 46 tasks on average over three seeds against 20.0 for its base, winning 18 families, losing 12 and tying 16 (exact one-sided sign test , not significant at the registered alpha). The same-seed supervision contrast did separate: supervising only the decisive actions (file edits and writes) against supervising every action, with corpus, recipe, seed and binary fixed, won 26 families and lost 11 with 9 tied (), and the six-decoding-seed development comparison on D46 had given for the same candidate against its base.[QUAL1,PROTO6] The sections below report what was run, in order, with the pilots that shaped it.
What ran. The experiments are registered in prereg/coding-tasks.md, which holds the protocol of Appendix § 21. The headline’s student is trained on verified trajectories that the 27B teacher produced on development-partition families and is tested against the untrained 8B;[PR15] the 27B model trained on its own verified trajectories (CE8) is a separate comparison outside the headline.[PR16] Run, in this order: the teacher’s first pass over the eight development sets (CE1), a pilot of the headline (CE4), two CE8 training runs, CE8-T with its shuffled control and the G8 agreement check, development probes on D46, and finally the registered confirmatory query. Registered experiments that were not run are named in § 7. Table 6 lists the paired outcomes that have a row in data/results.csv; every other number is read from data/measurements.json through its claim.
| Exp. | Repository | Arm | Families | Wins | Losses | Status | |
| CE4 | eight repositories (pilot subset S1) | adapter against untrained 8B | 21 | 0 | 1 | 1.000 | PILOT |
| CE8 | eight repositories (pilot subset S1) | CE8-T against the 27B base | 13 | 3 | 4 | 0.773 | PILOT |
| CE8 | eight repositories (pilot subset S1) | CE8-T-S (shuffled control) against the 27B base | 16 | 2 | 6 | 0.965 | PILOT |
| CE8 | eight repositories (pilot subset S1) | CE8-T-u1b (decision-level) against the 27B base | 13 | 1 | 4 | 0.969 | PILOT |
Table 6. Paired outcomes over qualification-partition families (wins: the adapter resolves more of the family than the base; losses: fewer), with the exact one-sided sign test. Every cell is read from a finished run; none is an estimate.
Two findings of the first runs. In a pilot under an earlier, unbounded harness the untrained 8B model almost never edited a file, which is why the headline’s student trains on verified trajectories of the 27B teacher and why one bounded harness profile is pinned for every arm. The first teacher run was withdrawn because its shell had no filesystem confinement; of its 61 episodes, 29 were stopped by the repeated-answer bound and 5 by the tool-call cap (the 34 conclusions labelled cancelled), and the solve report now names the bound.[TJ01,TJ02] Every workspace command has since been confined by the kernel and a call naming a path outside the tree marks the episode exposed (Appendix § 19.§ 1).[CN01]
CE1: the teacher’s first pass, finished. The 27B teacher has solved the eight development sets once, greedily, confined and under the registered bounds; Table 7 lists every set, each row read from data/measurements.json and each backed by a ledger entry that quotes its counts.[M39,M40,M41,M42,M43,M44,M45,M46] “Stopped” counts the runs that ended at the harness’s own bounds (the cap on tool calls or the repetition stop, Appendix § 21), not infrastructure failures. The exposure audit is applied as registered: an attempt whose trajectory names a path outside its tree is excluded from training evidence and counted, whatever its patch did; the columns “Hidden pass” and “Passed clean” show how much of the evidence the rule removes in each set.
| Set | Tasks | Success | Stopped | Hidden pass | Audit removed | Passed clean | Failed |
| caldate p1 | 25 | 13 | 12 | 13 | 7 | 6 | 19 |
| exprcalc p1 | 56 | 13 | 43 | 10 | 7 | 3 | 53 |
| graphkit p1 | 51 | 27 | 24 | 24 | 13 | 11 | 40 |
| iniparse p1 | 61 | 21 | 40 | 18 | 7 | 11 | 50 |
| linediff p1 | 50 | 21 | 29 | 18 | 7 | 11 | 39 |
| mdrender p1 | 43 | 19 | 24 | 13 | 4 | 9 | 34 |
| tokenbucket p1 | 38 | 25 | 13 | 22 | 4 | 18 | 20 |
| ttlcache p1 | 46 | 31 | 15 | 34 | 3 | 31 | 14 |
Table 7. CE1, teacher pass 1 on development sets that have finished. Success and cancelled count solve conclusions, the other columns the verified outcome; exposed attempts are counted among the failed.
CE4, the first confirmatory pilot: not shown. The registered query could be run only on a subset (Appendix § 28), under the protocol in force before the solve protocol of Appendix § 21 (greedy decoding, one attempt per task): one task of each of the 23 families of the first qualification half, solved once by each arm with both served by Brain on the same build and tier. The adapter was trained on the verified, exposure-clean trajectories of the teacher’s first pass, projected one tool call per turn. Of the 23 tasks, 21 are paired; the untrained 8B resolves one and the adapter none, so the sign test has a single discordant family against the adapter and does not support the claim; the comparison is a pilot by the registered rule and is reported as not shown.[M47] It is a weak test: with the untrained model near zero there is almost no room for a gain, and the adapter’s attempts mostly end at the harness’s repetition stop. The step count (47) is not the registered rule’s (29 for the 114 training records the trainer reports), and the controls, the second qualification half, retention and the serve-agreement sample were not run.
CE8: trained, stopped by its guard, not evaluated. The registered CE8 training ran and stopped early at step 51: its training loss fell from 0.1476 at the first step to 0.0221 at the last, and its best monitoring loss, 0.0839 at step 27 against 0.1101 for the base, rose to 0.0867 by the last evaluation, so the registered guard calls it memorised.[M48] The one further run the registration allowed, on a one-pass budget, behaves the same way at a smaller scale: best monitoring loss 0.0829 at step 30, 0.0832 at its last evaluation, training loss 0.1476 to 0.0282, memorised by the same rule.[M49] Neither adapter is evaluated, and no claim about CE8 follows from either. The guard reads a single step’s training loss on two records and any rise at the last evaluation, so it fires on noise of this size; the registration keeps it as written, and the lower monitoring loss than the base’s is a training-side quantity, not evidence of a gain on any task.
CE8-T and its control: trained and evaluated on the pilot subset, no improvement. CE8-T trained on one pass over the 113 records the trainer reports (57 steps); its kept evaluation, step 48, has a monitoring loss of 0.1738 against 0.1799 for the base and a training loss of 0.1331 there, and the training record’s verdict is not memorised, so CE8-T is a candidate for evaluation.[M50] CE8-T-S, trained the same way on the same 113 records and 57 steps, keeps step 57 at 0.2315 against 0.2635 for the base on its own shuffled monitoring records, also not memorised.[M51] Both are training-side quantities: the control’s monitoring records are shuffled too, so the two losses are not comparable, and neither is evidence of a gain on any task.
G8: the served adapter agrees with the trained one. The registration puts G8 before any arm. On the held-out records of the CE8-T training (33 records in 5 whole families, 354 examples, 68407 supervised positions) every recorded assistant position is scored twice: by the in-process trainer’s forward (bf16 base with the adapter, no step) and by the evaluation server (Q8_0 base with the adapter in its bank), and the same pair is taken for the base. The mean absolute served-versus-in-process difference is 5.005e-02 for the base and 4.694e-02 with the adapter, 0.938 of the base’s, so condition (i) holds; top-1 agreement is 97.37% for the base and 97.76% with the adapter, a drop of -0.39 points, so (ii) holds; and the served adapter’s mean log-probability, -0.162778, is above the served base’s, -0.181931, so (iii) holds and the verdict is PASS.[M54] The served adapter is therefore the adapter that was trained, and the evaluation that follows is of the trained artifact.
The pilot evaluation: no held-out improvement. Both candidates were then solved on the pilot subset P23 (23 tasks, one per family of the first qualification half) under the solve protocol of Appendix § 21 with thinking off, one attempt per task, on one frozen build and one server process that offered the base and both adapters. CE8-T resolves 7 of the 23 tasks and the base 9; of the 13 paired families the adapter wins 3, loses 4 and ties 6, and the one-sided sign test gives .[M52] The shuffled control resolves 7 and, over 16 paired families, wins 2, loses 6 and ties 8 ().[M53] Neither comparison reaches the registered alpha, and the rule that labels an underpowered comparison a pilot applies (13 and 16 non-tied families against the 25 the rule requires): the outcome of both arms is reported as no held-out improvement, and no confirmatory query follows for CE8-T. The adapter does change behaviour on many of the same tasks, but not in one direction, and the trained adapter is not distinguishable from its shuffled-instruction control in size or direction.
CE8-T-u1b: decision-level supervision, the same outcome. The pilot’s result motivated a third candidate that changes one variable of the data rather than the recipe: CE8-T-u1b trains on the unpacked sft-step view of the same experiences, one record per admitted action with its own context, instead of one packed record per trajectory. The trainer reports 2490 records, and the run takes 1200 steps at a held-out monitoring share of 0.05; its kept evaluation, step 960, has a monitoring loss of 0.1676 against 0.1835 for the base and a training loss of 0.1359 there, and the guard calls it not memorised.[M55] Solved on the same pilot subset under the same protocol it resolves 6 of the 23 tasks where the base resolves 9; of the 13 paired families it wins 1, loses 4 and ties 8, and the one-sided sign test gives , with 6 attempts exposed against 7 for the base.[M56] Supervising decisions rather than whole trajectories does not rescue the comparison: the candidate is at or below the base and below its packed predecessor on this subset. Both candidates are development probes, not registered comparisons.
What these pilots can and cannot show. Two measurements bracket what the pilot numbers mean. First, the attempts differ in shape. The median episode is 32 steps for the base, 13 for CE8-T, 12 for its shuffled-instruction control and 16 for the decision-level candidate; the arms make 782, 474, 392 and 632 tool calls, and their solve conclusions report 11, 10, 9 and 3 successes, while the longest episodes still run to 58, 60, 61 and 52 steps.[M58] The control matches the trained adapter, so what fine-tuning on these records moves is when a turn ends, not what a trajectory solves; the decision-level candidate sits between the two, with more steps than the packed candidates and the most repetition stops. Second, the seed floor: solved twice on the same 23 tasks with two registered protocol seeds, the base itself pairs 13 families with 1 win, 3 losses and 9 ties, , which is statistically indistinguishable from either adapter’s comparison.[M57] A pilot of this size therefore shows no detectable difference, not a demonstrated loss; the registered confirmatory query on 46 families, reported below, is the measurement that could separate them. The training side points the same way: every trajectory longer than the 16384-token block was removed before training (130 of the 262 successful attempts of the eight development sets, and 1148 of 3837 for the decision-level view), so the records the adapters were trained on are the short half of the teacher’s successes, and their median record carries about 1650 supervised tokens.[M59]
Development probes on D46: the supervision ablation and recovery. On 2026-10-25, on two GH200 cards, two series of development probes were run on D46 (46 tasks, 46 families, one task per family, greedy decoding) with one rebuilt training binary (SHA-256 1d366e597bb504bcb22304ebbe07ef0165c442f50391340e74004475a120ab); they are reported as pilots and are not the registered confirmatory query, and their outcomes are not used as evidence for it. The first series holds the corpus (769 records, blake3:646692b2), the recipe (rank 16, alpha 32, , 120 steps, packed-bf16 base), the training seed and the binary fixed and changes only the supervision: every admitted action, or --only-actions edit\_file,write\_file (blake3:92252209). At seed 13 the decisive-action adapter verifies 30 of 46 tasks with 1 exposed, the all-action adapter 13 with 3 exposed; over 42 paired families the decisive arm wins 16 and loses 1 (exact one-sided sign test ), and the all-action arm is a regression below its own base (13 against 21, 4W-11L-20T, ).[D46A,D46B] At seed 1 the same contrast keeps its sign but is not significant (23 against 18, both with 4 exposed; 38 paired, 8W-4L-26T, ); the two seeds share the 46 families, so this is one strong same-seed demonstration plus a sign-consistent second seed, not two independent tests.[D46C] The behaviour is seed-stable: the decisive adapters of seeds 1, 7 and 13 edit a file in 35, 36 and 39 of their 46 attempts and end 1, 0 and 2 in the repetition stop, where the all-action adapter edits 22 and aborts 10 and the base edits 24 and aborts 13.[D46D] The same-seed design removes the rebuild confound that earlier comparisons carried: with corpus, recipe, seed and executable fixed, changing only the supervision changes the outcome.
The second series recovers the all-action adapter win1, damaged to 12/46, on the base’s own verified D46 successes (blake3:df000e8f, 14 records) at . Six steps (303 s of training) verify 23 of 46 with 2 exposed against the base’s 21 (36 paired, 4W-6L-26T, ); 18 steps verify 20 with 5 exposed (34 paired, 6W-8L-20T, , measured 2026-10-24); 36 steps (815 s) verify 28 with 7 exposed (31 paired, 10W-5L-16T, ); and the undamaged control, the decisive seed-1 adapter trained for 18 steps on the same corpus, verifies 24 with 10 exposed (30 paired, 7W-2L-21T, ).[RCVA,RCVB,RCVC,RCVD] The first six steps – about five minutes – therefore do most of the restoration, and longer continuation on the benchmark’s own families buys a few more tasks while raising the exposed count: the corpus is a labelled rehearsal of the scored families, not a transfer measurement and not free gain. The doses are one seed and share their start, corpus and families, so they are not independent tests, and no comparison reaches the registered alpha.
Six decoding seeds on D46. The development benchmark was re-solved under the protocol’s own sampling at six registered decoding seeds (the three of the earlier pilots and three drawn for this run) by the base, the decisive-action adapter of training seed 13 and its all-action control, on two GPUs. Tasks solved of 46, averaged over the six seeds: base 19.67, decisive 24.5, all-actions 19.83. Per family, decisive wins 22, loses 11 and ties 13 against the base (exact one-sided sign test ), 22W-11L-13T against the all-action control (), and the control 17W-15L-14T against the base (): the supervision separates from both controls, while the two controls do not separate from each other.[PROTO6] This is a development probe on D46 (46 families, not the qualification set) and no candidate is registered. It is the powered comparison that justifies registering the candidate and running the registered query, not a result of that query.
The registered confirmatory query: not shown. The candidate registered before the run - the decisive-action adapter of training seed 13 - was solved against its base and its all-action control on the frozen qualification set (46 families) at the protocol’s three registered decoding seeds and recommended sampling. Averaged over the three seeds the candidate solved 23.0 of the 46 tasks, its base 20.0 and the control 16.33, with exposed attempts on 10, 35 and 22 of the 138 assigned respectively. Per family the candidate won 18, lost 12 and tied 16 against its base: the exact one-sided sign test reads with a family-bootstrap 95 % interval of [−0.065,+0.188] for the mean-fraction difference. At the registered alpha of 0.05 this is not shown: the candidate is not demonstrated better than its base on the qualification set, and the paper does not report equivalence. The same-seed supervision contrast does separate: against the all-action control - same corpus, recipe, seed and binary, differing only in the supervision - the candidate wins 26 families and loses 11 (, mean fraction +0.145, interval [+0.007,+0.275]), while the control against the base is 16W-11L-19T (). That contrast is a secondary analysis, not the registered decision. The development read on D46 had reached 22W-11L-13T at ; it did not replicate at the registered level, and the registration’s own rule for a development gain with no qualification gain (sign test , reported as memorisation) is not met either, since - the registered result is a null whose direction matches the development read, next to a causal supervision effect that the control isolates.[QUAL1]
| Probe | Arm | Families | Wins | Losses | Status | |
| D46-SS | decisive actions against all actions (seed 13) | 42 | 16 | 1 | 0.000137 | PILOT |
| D46-SS | decisive actions against all actions (seed 1) | 38 | 8 | 4 | 0.194 | PILOT |
| D46-SS | all actions against the base (seed 13) | 35 | 4 | 11 | 0.982 | PILOT |
| D46-RCV | 6 steps from win1 against the base | 36 | 4 | 6 | 0.828 | PILOT |
| D46-RCV | 18 steps from win1 against the base | 34 | 6 | 8 | 0.788 | PILOT |
| D46-RCV | 36 steps from win1 against the base | 31 | 10 | 5 | 0.151 | PILOT |
| D46-RCV | 18 steps from the undamaged decisive seed-1 adapter (control) | 30 | 7 | 2 | 0.0898 | PILOT |
| D46-GEN2 | no rehearsal against the base | 32 | 8 | 4 | 0.194 | PILOT |
| D46-GEN2 | rehearsal against the base | 34 | 4 | 7 | 0.887 | PILOT |
| D46-GEN2 | rehearsal against no rehearsal | 37 | 3 | 13 | 0.998 | PILOT |
| S1-GEN2 | no rehearsal against the base | 13 | 2 | 2 | 0.688 | PILOT |
| S1-GEN2 | rehearsal against the base | 15 | 1 | 0 | 0.5 | PILOT |
Table 8. Development probes on D46 (pilot, not the registered confirmatory query): paired families among the 46, wins and losses from the adapter’s side, and the exact one-sided sign test computed from them. A row is a probe, not a confirmatory result.
Generation 2: rehearsal protected nothing at this budget. The decisive seed-1 champion (D46 23/46, P23 16/23) continued on P23’s verified successes (25 records) for 120 steps at , seed 1; the arms differ only by --rehearsal of the decisive corpus at 0.25. On D46 the no-rehearsal arm verifies 28 of 46 with 7 exposed (32 paired, 8W-4L-20T, ) and the rehearsal arm 19 with 5 exposed (34 paired, 4W-7L-23T, ); the direct same-start contrast pairs 37 families with 3 rehearsal wins and 13 losses ( with rehearsal as the candidate, 13W-3L the other way), so rehearsal cost nine tasks relative to the same update without it.[G2A,G2B,G2C] On P23 they verify 12 (5 exposed) and 11 (1 exposed) against the base’s 8 (13 paired, 2W-2L, ; 15 paired, 1W-0L, ), so acquisition over the champion’s 16/23 is not demonstrated.[G2D] At this budget the new data did not make the model forget D46 (the no-rehearsal arm added five tasks over its start), so there was nothing for rehearsal to protect and it cost tasks instead. These are one-seed greedy runs on the development benchmark, not the registered query, and the run-to-run variation in the limitations applies to them.[REP1]
How outcomes are read. The primary decision is the one-sided exact sign test of Appendix § 21; a comparison with fewer than 25 non-tied families is labelled a pilot and the rule is not loosened, and a development gain with no qualification gain (sign test ) is reported as memorisation.[PR04,P08]
Several generations of a 27B model on its own experience
The repository experiment trains a smaller model on a larger one’s trajectories. This section asks the question of § 3 on agent workloads at full scale: does Qwen3.8-27B, trained only on its own verified attempts, become a better agent generation after generation, on tasks it never trained on? No stronger model takes part, and the base is the only starting point.
Workloads
Four kinds of task are attempted by the same model in every generation, each with a training partition the generation practises on and a held-out partition it is examined on; the partitions are whole families and are fixed before the base attempts anything.
Repository faults (Appendix § 21): one seeded fault per task in the eight authored repositories; the exam is the reserved files of six repositories and every file of the two sealed ones (mdrender, graphkit), so the exam measures transfer to repositories never trained on.
Several faults at once: splinter tasks combine joins three admitted single-fault tasks of one snapshot whose faults lie in different files into one task (the union of their checks and reference changes, one merged report); it is admitted only if no member’s reference alone repairs it, and a combined task with any reserved file is an exam task.
Embedded firmware: defects in small firmware services for Cortex-M and nRF5340 boards, built with the vendor toolchain and graded by running the image in an emulator against hidden behavioural checks.
Mechanical design: a part drawing to a manufacturable solid, graded by geometry against a reference.
One generation
Each generation starts from the champion (the base, then the last promoted adapter) and runs four steps; every step records its result before the next starts.
Explore. The champion attempts every training task times ( for the repository workloads) under the model family’s recommended sampling; a separate verifier grades each attempt and an exposure audit fails any attempt that reached hidden material.
Learn. The graded attempts are admitted as experience (held-out families are refused by the exam seal) and become on-policy training records, one per model call. Each candidate continues from the champion:
GRPO: an attempt’s advantage is its reward (1 verified, 0 not) less its task group’s mean over the group’s standard deviation; the clipped surrogate with a KL to the base trains the adapter (clip 0.2, ).
GRPO with privileged self-distillation: the same, and every task whose attempts all failed (a group with no reward signal) is distilled instead: the same base model is shown what the verifier knows (the failing checks and their output, the reference change where the task has one) and each token the student wrote gets the advantage , with a cap of 40 on its sum over a sequence, through the same clipped step.
Examine. Every candidate and the champion attempt every held-out task under the same agent build, serving build, benchmark revision and load per card.
Decide. Each candidate is paired task by task with the champion on every workload; the candidate with the largest gain over all paired tasks replaces the champion when that gain is positive and no workload’s mean falls by more than 0.2. The decision, the paired counts and the sign test are recorded whether or not the champion changes.
What the harness must not decide
A comparison of agents is only a comparison of models if the software around them fails neither arm. Two failures of that kind were found while running the loop and are part of the protocol now. First, a wall-clock limit per trial makes the server’s load part of the outcome: with about a hundred agents on one card, per-stream decoding slowed until correct trajectories were cut off, and the base’s held-out repository pass rate read far below its rate on the same exam at a fifth of the load. The loop therefore gives a model examined on a share of the cards that share of a workload’s trials. Second, a serving engine that ran out of device memory for one prefill round failed every request of that round; it now returns the sequences to its queue and narrows admission instead. An exam in which any trial was ended by the server, the environment or the harness decides nothing.
Results
Generation 0. The base explored the training tasks of single faults, firmware and mechanical design, and two candidates were trained on its attempts; the generation was decided on three workloads with the deviations of Appendix § 28. Against the pooled exams the self-distilling candidate won more held-out tasks than it lost and replaced the base.[G2701] The GRPO candidate gained on both repository workloads but lost on firmware, so the rule preferred the self-distilling one even before the firmware loss is weighed.[G2702] Nearly all of the exam was already solved by the base: the gain is made on the few held-out tasks it solved only sometimes, and most of what first looked like failures of the base were trials the serving engine or the per-trial limit ended (Appendix § 23.§ 3).
Generation 1. Exploring every training task again teaches little: most single-fault training tasks were solved by every attempt of generation 0, and a group whose attempts all earn the same reward has no advantage to learn from. Generation 1 therefore practises a frontier: the training tasks the base did not always solve and the training tasks with several faults it had never attempted, with the promoted champion. Its GRPO candidate was promoted on a gain that is not significant.[G2704]
One measurement of all three. The generation-0 decision compared the candidates with a base exam taken on an older serving build, part of which the serving engine had ended. When the base sat the same exam on generation 1’s stack, the generation-0 champion no longer beat it,[G2703] and after two generations no gain over the base is measured.[G2704] The exams are near their ceiling for this model: the base already solves about nine held-out tasks in ten,[G2703] so the few tasks it solves only sometimes carry the whole comparison, and two promotions on small positive means were not enough to move it. The generations did change the model, in one direction: on the tasks both solved, the generation-1 champion needed about a tenth fewer completion tokens per solved task than the base on every workload.[G2705] That was found after the data and over several comparisons, so it is exploratory; it was registered as a co-primary outcome for later generations of these workloads,[G2706] and no later run of this paper measured it, because the series that followed moved to a task with headroom for a small model. Self-improvement in pass rate across generations is therefore not shown by these runs; what they establish is the loop itself, run end to end without a stronger model, and the conditions an exam must meet before its decision means anything (Appendix § 23.§ 3).
A small model learning from raw documentation (exploratory)
This section closes the whole loop on a smaller scale: a 4B model is given raw documentation and an objective, writes its own tasks, has their answers fixed by an interpreter, is trained on every weight and, once trained, writes the next round’s tasks itself; in a third arm a 27B model writes the tasks instead. These runs were not registered and are exploratory: their design changed between runs, and their probes were written by the base model and graded under the loop’s own protocol. A neutral re-grading of such probes in S1’s arms A and B found most of the gain to be adaptation to that protocol (Appendix § 9.§ 1),[RS11] so the gains below are gains on the loop’s own probes, not established Lua competence. They motivated the series S1–S4.
Setup
Source and objective. The source is three chapters of the Lua 5.1 reference manual (string manipulation and patterns, table manipulation, mathematical functions) as the HTML the project publishes, in its own Latin-1 encoding; the objective is the sentence “Become an exceptional Lua 5.1 programmer”. A local HTML page is read in the charset it declares and converted to the
Markdown the sectioner reads, exactly as a fetched page is.[SL01] With --structural-graph the skill graph is read from the sections, one skill per function or topic, without asking a planner model.[SL02]
Tasks and their answers. Every task is of the kind predict-lua: a short program and the question what it prints. The model writes the program and the question; admission sets the program below the question in a fenced block, so it is shown once and exactly as it is run,[SL03] and the answer is what the pinned Lua 5.1.5 interpreter prints: the program must run to completion, print a bounded, non-empty output and print the same again, and that output becomes the reference whatever the model predicted.[SL04] An answer is graded by the formal verifier, which takes an output given in a fenced block as the answer and treats two different blocks as a hedge.[SL05]
The loop (Figure 11). Probes are written once, before any training, by the model as it is, and split per skill into dev probes (the baseline and every round’s decision) and final probes (the closing assessment only); a skill is asked for probes again while it has fewer than both halves need.[SL06] Each round picks the skill expected to benefit most, has the incumbent write training tasks for it and attempt each four times, keeps the tasks it solved sometimes and the verified answers to them, trains every weight of the incumbent on them with a replay of the earlier accepted rounds’ records, and accepts the candidate only if the target skill improved on its dev probes and no other skill lost more than the retention bound. With --self-generate the incumbent writes every round’s tasks: the base in round one, the last accepted checkpoint after.[SL07] The control arm is the same run without it: the base writes every round’s tasks.[SL08] The run records this section cites (every round’s stage and outcome per arm and replicate, and the arms comparison) are in the evidence dataset the-frontier-must-grow-evidence (§ 8).

Figure 11. The documentation loop of Appendix § 24 (exploratory), read from the raw manual. Probes are written once by the untrained model and split per skill: the dev half decides every round, the final half is touched only by the closing assessment. Each round a task writer (the incumbent itself, the untrained base, or a 27B model in the third arm) writes programs, the interpreter fixes their answers, and every weight of the 4B incumbent is trained on the attempts it solved sometimes; dotted arrows are what is refused.
What full fine-tuning costs on one card
Brain’s full fine-tune holds the weights and gradients of a Qwen3 model in fp32 on the card and the AdamW moments and master weights in host memory; samples/text/chat-full-finetune runs it on any base in the model store.[SL09] Table 9 gives what eight steps cost on one GH200 at a 631-token block.[SL10]
| Model (optimizer state) | Card (GiB) | Host (GiB) | s / 8 steps | Loss |
| Qwen3-0.6B (fp32) | 7.8 | 10.9 | 62 | 1.38 → 0.91 |
| Qwen3-1.7B (fp32) | 15.9 | 32.6 | 128 | 1.56 → 0.81 |
| Qwen3-4B (fp32) | 33.6 | 75.2 | 251 | 1.76 → 0.91 |
| Qwen3-8B (fp32) | 65.6 | 152.7 | 488 | 1.12 → 0.47 |
| Qwen3-8B (int8 moments) | 65.7 | 107.5 | 459 | 1.30 → 0.83 |
Table 9. Full fine-tuning on one GH200: the run’s peak on the card beyond what it held before, the process’s peak resident host memory, wall time and training loss before the first and after the last of eight steps (80 chat records written from the Lua manual). The 8B fits the card; in fp32 its host state is 153 GiB, beyond what a 205 GiB job can spare beside a served 27B model, and int8 moments save 45 GiB of it.
What a small model needs from the loop
Each of the following stopped the loop on the Lua chapters before a single round could train; each was fixed in the code the paper describes, and each is a requirement any loop driven by a model of this size has to meet.
The task must show what is run. Asked to put its program in the material and again verbatim in the instruction, the 4B model wrapped it in other code (material = [[print(\dots)]]), a program that prints nothing, and admission, which checked only that the material occurs in the instruction, accepted it. The model now writes the question alone and admission sets the material below it.[SL03]
The answer must come from the runtime. With the model’s own prediction as the reference, a task was refused whenever the model mispredicted its program, and most skills were left with too few probes. The reference is now what the interpreter prints.[SL04]
The verifier must read the answer the model gives. The model answers a predict task with a sentence above a fenced block. The verifier took candidates only from emphasis and numbers, and most failing baseline probes held the exact output in a fence: the baseline measured the answer’s format, and training would have taught it.[SL05]
One request is not enough. A 4B generator admits two or three tasks per request; probes and training tasks are asked for again, shown what was already written, while a skill is short.[SL06,SL11]
Generation must be compared by what it shows. Every predict task asks the same question of a different program; compared by its whole instruction, every pair was a near duplicate and nearly every training task was refused as repeating a probe. A task that shows material is now compared by the material.[SL12]
Writing tasks is not executing a procedure. The generator sampled at the agent’s temperature (0.2), which keeps a small model on task while it acts but has it write the same programs whenever it is asked again; it now samples at 1.2 with a top-k of 20.[SL13]
The environment can teach. A task the policy never solves was taught by the policy shown the task’s passage, and a model cannot run a program; with --reference-teacher the runtime’s output is said as the teacher’s reply and graded like any answer.[SL14]
The exam is not material. A round’s request showed the generator its skill’s probes as tasks not to repeat; the 4B model copied them, and most training tasks of a round were exact probes. The generator is now never shown the exam.[SL15]
An answer must be one the language defines. A seeded math.random program prints the same on every run and passed every check, but what it prints is the C library generator’s draw. A candidate fully fine-tuned on such answers lifted its target skill from 0 to 4 of 4 dev probes while the other skills fell from 35 to 20 passed probes (string.byte(s, 1) answered 97 for every string); the retention bound rejected it.[SL18] Programs that read a generator, the clock, the environment, input or the collector are now refused.[SL16]
A probe must be answered greedily. Probes were answered on the model’s own sampling (0.2, a seed per request), so the baseline and every round’s decision were draws. They are now answered greedily, and the same base on the same exam scores the same in both arms, 34 of 56 dev probes.[SL17,SL19]
The loop learns at a small step. At a learning rate of (six steps, one verified answer per task, retention bound 0.5) rounds are accepted: each arm accepted 5 rounds, and on the final probes, which no round trains on, the self-generate arm went from 26 to 35 of 53 passes (7 skills up, 1 down) and the fixed-generator arm from 26 to 40 of 53 (8 up, 0 down).[SL20] The control did better than the arm whose trained model writes the next tasks, so these runs show gains on the loop’s own final probes, not that self-generated tasks help.
A second replicate. The same experiment on a fresh exam, written by the base model, with the full fine-tune of a later brain revision, accepted 3 rounds in each arm. On the final probes the self-generate arm went from 20 to 30 of 46 in 13 rounds (7 skills up, 0 down) and the fixed-generator arm from 20 to 29 of 46 in 8 rounds (7 up, 2 down).[SL21] Both arms learn in both replicates; which generator does better changes sign between them, so on this evidence the loop gains on its probes from tasks its own model writes, and whether the trained model’s tasks are better than its base’s is not settled. A third replicate on the second exam, which changes only the runs’ random draws, agrees: the self-generate arm went from 20 to 33 of 46 (2 of 10 rounds accepted, 8 skills up, 0 down) and the fixed-generator arm from 20 to 32 (3 of 9 accepted, 7 up, 1 down).[SL22] All six arms of the three replicates ended above their baseline on probes no round trained on; the self-generate arm finished ahead in two replicates and behind in one.
Who writes the tasks: no difference detected. If tasks written by a more capable model taught more, a 4B solver fed tasks from a 27B writer would gain more than one fed tasks from itself. We ran a third arm in which the 27B, served by brain on the second card, writes every round’s tasks while the 4B still solves, is verified by the interpreter and is fully fine-tuned, and repeated all three arms on the second exam until each had at least six replicates; runs whose training was refused for want of device memory are excluded, not counted as zero. The 7 self-generate, 9 fixed-generator and 6 teacher-generator replicates gained on average 10.86, 10.89 and 11.17 passes of 46 on the final probes. Exact permutation tests over replicates find no difference (p = 1.0 between self-generate and fixed-generator, p = 0.89 between self-generate and teacher-generator, p = 0.87 between fixed-generator and teacher-generator), and per skill the teacher-generator and fixed-generator arms are each better on 5 skills with 3 tied.[SL23] With 6 to 9 replicates per arm these are weak tests: the differences of mean gains are −0.03 (95% interval −1.99 to 1.92) between self-generate and fixed-generator, −0.31 (−2.69 to 2.07) between self-generate and teacher-generator and −0.28 (−2.22 to 1.66) between fixed-generator and teacher-generator, and only differences of about 2.7 to 3.3 passes of 46 would have been detected with power 0.8.[RS44] No effect of who writes the tasks was detected; an effect smaller than that is not excluded. The tasks here are short programs whose output the interpreter fixes, so a stronger writer has little room to write a better task.
Replicating the result on another repository
A result that only holds for the repositories it was developed on is a demonstration, not a method. This section is the recipe for running the loop on a repository of one’s own, step by step, each with the command in the pinned revision that does it and whose command line the ledger checks; there is no separate benchmark command.[PL01] It claims no outcome; the replication of the headline on a second repository was registered (CE7) and was not run (§ 7).
Requirements on the repository
splinter practice learns a codebase from its own tests: a git work tree or bare repository, a revision git resolves to a commit, and a test suite that runs offline and is green and stable on a clean copy of that revision. Splinter recognises a Python project with a tests directory (run as python3 -m unittest) and a Cargo project (run with cargo test --offline); any other suite is named with --test-command, and the files tasks may change with repeated --file.[PS01,PS04]
The suite is run twice on a clean copy first, and a red or flaky baseline stops the run before any model is loaded.[PS01] Isolation is recorded as none unless a container client and a digest-pinned image are configured, although a host command is confined by the kernel to its tree without a network; a repository whose tests need the network or a process-level boundary for soundness is not suitable on a host without a container.[AU09,CN08]
Task-family construction
The protocol’s unit is the family (Appendix § 21). Table 10 shows what the construction yields on the eight authored repositories that ship with Splinter: of several hundred seeded faults per repository, those that do not build or that no test notices are dropped, near-duplicates and truncated outputs are refused, and the rest are admitted.[PS02] For a new repository the steps are, in order:
Check the environment:
splinter doctorconfirms the state root is writable with room to spare and the policy’s weights fit the card.[PS05]
Generate and count the admitted tasks without a model: run
splinter practice REPO --rev REV --budget 1s --no-release --max-tasks N --mutants-per-file M --test-command ...; the[tasks]line states the admitted count and the refusals by reason. This is the funnel of Table 10, and it needs no model and no GPU.[PS03,PS06]
Reserve the exam:
--exam-familiessets families aside by a stable hash of their names before any attempt, and--exam-seedis drawn, recorded and refused if changed on a later run.[PS06]
Declare partitions and generations:
splinter experiment checkvalidates a plan whose sealed families are never trained on; a three-way split with a qualification partition queried once is the registration’s requirement, andsplinter tasks pickdraws the development holdout and the qualification partition from the task sets by seed and family.[PS07,PL01]
Measure the untrained agent (the teacher’s pass is reported in Appendix § 22): the base attempts the development and qualification partitions with attempts per task fixed from a throughput probe, and
splinter experiencescompare CANDIDATE BASELINE pairs any
two such experience sets by task and reports the family-clustered sign test and per-family counts.[PL01]
Train and gate (one round per run):
splinter practice REPOsolves the non-exam tasks as an agent, keeps the attempts the hidden checks confirm, builds the action-level SFT dataset, trains a LoRA candidate from the champion with earlier releases replayed, and runs the release gate over the exam the run reserved;--bf16-base,--steps,--rankand--max-blockbound the training to the card.[PS01,PS06]
Serve and roll back: a release moves the alias and rewrites the serving manifest,
brain serve --adapter-manifestfollows it, andsplinter rollbackrepoints it.[X05,X06,PS08]
What it costs
Task generation costs CPU time only: no model is loaded and no GPU is used.[PS03] Training cost on a 27B model is the token count over the measured rate; at the 622 tokens/s of the probe of Appendix § 18.§ 3 one million trained tokens take 0.45 h of card time.[M38,C12] The remaining terms of the cost of one verified outcome, wall time and tokens per attempt and attempts per resolved task, depend on the repository and the model and are not reported here; they are measured by solving a few families with the untrained model before card time is committed to training.
What to report
For a replication to count it must report, for the new repository, the same quantities as Appendix § 22: the admission funnel by reason, the baseline resolve rate with its interval, the paired sign-test counts on the qualification partition, the retention check, the isolation level and the three repository commits. Counts that disagree with ours are as informative as ones that agree.
Related work for the system
Parameter-efficient training. LoRA freezes the base and trains low-rank adapters; [16][CT03] QLoRA does so over a base quantised to 4 bits; [9][CT04] DPO optimises preferences without a separate reward model. [24][CT05] The flywheel uses LoRA and DPO as objectives it selects, and delegates their computation to Brain (Appendix § 16). Retrieval-aware fine-tuning. RAFT trains a model on domain documents with distractor passages; [41][CT06] Splinter has an optional passage-mixing share of the same family (Appendix § 15), which we do not evaluate here. Forgetting. Catastrophic forgetting is the loss of earlier capability when a network is trained on new data; [19][CT07] Splinter’s response is empirical: it measures retention against every earlier release at release time (Appendix § 17). Memory and compute for training. Recomputing activations trades compute for memory in the backward pass; [5][CT08] fused or chunked cross-entropy avoids materialising the logits of a large vocabulary; [31][CT09] FlashAttention computes exact attention without materialising the score matrix; [8] [CT10] Gated DeltaNet is a gated linear-attention layer based on the delta rule and is the recurrent layer type of the 27B model Brain implements (three Gated DeltaNet layers to one attention layer). [35] [CT11,B11] Section J reports which of these Brain contains. Repository tasks and agents. SWE-bench poses resolving real GitHub issues as the task, graded by the repository’s tests; [18] [CT12] SWE-agent shows that the interface between the model and the repository changes what a model achieves, so the interaction, not only the final patch, is what is recorded and trained on. [33] [CT13] SWE-Gym, R2E-Gym and SWE-smith build executable training environments, by collecting instances, by procedural generation from commits and by modifying code so that tests fail. [17, 23, 34] [CT14] We take the mutation-and-validate idea and the commit-based idea as task producers (Appendix § 21). The difference in aim is that the target here is one repository’s own tasks, reserved before rollouts and evaluated paired against the base under an identical harness, with retention on earlier repositories and a replication on a second one registered (and not run, § 7); we make no claim about priority.
Limitations of the system and of the other experiments
What the structure claims show. They show that definitions, constants, tests and messages exist at the pinned commits; they do not show that the code behaves accordingly. Behaviour is established only by the measurements of Appendices § 18 and § 22 to § 24 and § 3. The qualification of the 27B training substrate used random weights for the long sequences and a handful of examples for the real checkpoint, and no sequence longer than 16384 tokens was run. The requirements of Appendix § 19 include one that is not implemented: the retention check does not test what each release demonstrably acquired, so for a domain of facts a passed retention check is not evidence that taught facts were kept. The workspace verifier’s reading of the runner’s report and the practice gate’s canaries exist at the current head, not at the revision the architecture claims are pinned to. [OP04,OP05] The corpus expansion mechanisms (variants, retention queue, rehearsal, claims) are described from the code and none was evaluated for the value it adds.
Single greedy runs and run-to-run variation. The development probes of Appendix § 22 (the supervision ablation, the recovery dose-response and the generation-2 rehearsal ablation) are one run per arm at greedy decoding. The same adapter evaluated three times on the same benchmark decoded 30, 31 and 26 of 46 tasks, and the three reports’ verdicts differ pairwise on 9, 12 and 11 of the 46 tasks; a single greedy arm therefore carries about 2.5 tasks of standard deviation, so the paired p-values of single-run comparisons overstate their precision. [REP1] The variation is two different things, and the decode batch is only one of them. [REP2] A single-row decode is reproducible: five sequential identical task-derived requests, sent greedily with the requested seed echoed by the server, returned identical completion token ids and bitwise identical per-token log-probabilities, and a never-seen prompt sent once cold and then twice warm was identical both times. A row’s arithmetic does depend on its co-batch: six or eight identical requests sent at once return two distinct sequences, because the int8 projections run a block that is bit-identical below five rows and switch to an int8 tensor-core block from five, while the fused decode attention sizes its split-key count from the batch. But one request in flight does not make a whole episode reproducible either: the same adapter on the same eight tasks, greedy at concurrency one, passed three and then five of eight with two verdicts flipped. The registered multi-seed protocol is therefore the instrument for agent episodes. Training is not bit-reproducible either: two runs of one recipe with the same binary, corpus, start and seed produced two distinct adapters whose losses agree to the fourth digit while their gradient norms and weights do not. [REP3] A same-seed comparison therefore means the same data, recipe and binary, not a bit-identical run. The series’ examiner protocol is greedy at one request per item on one build and tier, and its base-against-base null replicate found no discordant item (§ 2).
The coding-task experiment. The registered confirmatory query does not show the claim (Appendix § 22), and the same-seed supervision contrast that does separate is a secondary analysis, not the registered decision. Registered experiments that were not run: the ten-stage training and deployment experiment for the 27B model (CE6), the replication of the unchanged procedure on a second independent repository (CE7), and the acquisition, choice-of-improvement and generations experiments of prereg/acquisition.md (ACQ1, INT1, GEN1). The paper therefore reports no replication of the headline, no result of the registered retention check or of the three controls of Appendix § 21, and no result of ACQ1 (the documentation runs of Appendix § 24 were not run under that registration). The memorisation guard of CE8 as registered reads one step’s training loss and any rise at the last evaluation, so it can stop a run on noise; it was kept unchanged.[M48,M49] Both pilots of the adapter are inside the same variation the base shows against itself on a second protocol seed.[M57] The registration fixes one model size for the first training result (Qwen3-8B), trained on a stronger teacher’s verified trajectories because the untrained 8B produced too few edits to supply its own, so a gain there would be distillation from the teacher, not self-improvement;[PR15,PR16] one agent harness; and process isolation none on the node (a solver’s commands are confined by the kernel to their tree and have no network, but the node is not a container; Appendix § 19.§ 1).[AU09,CN01,CN08] Mutation-seeded tasks test whether a model can undo a synthetic fault, not whether it can debug a historical one. A family is the file a fault was seeded in for the authored repositories, so clusters are files, not subsystems.[PR02] The base model may have seen public repositories in pretraining; the paired base arm bounds that effect, it does not remove it.
The 27B generations. Two generations were run, not the three planned, on three of four workloads, on exams the base already solved about nine times in ten, so the few tasks it solved only sometimes carried every comparison.[G2703] The token-efficiency finding was made after the data over several comparisons and is exploratory.[G2705]
Verification and grounding. Lexical grounding of a task in its source (Appendix § 13) establishes provenance, not entailment: a task whose reference shares the source’s words can still be wrong about it. The verifier kinds check different properties, and no ranking among them makes a stronger verdict override a failure of another kind. Aggregate gates (Appendix § 17) can hide the loss of a rare skill inside an unchanged total. The engineering defects found and fixed along the way (Appendices § 19 and § 24.§ 3) are part of the result: each would have produced a plausible but wrong measurement.
Deviations from the registration
The protocol of Appendix § 21 is the registration in prereg/coding-tasks.md, fixed before any coding-task run. Table 11 to Table 13 list the dated changes made in this paper after the registration; “Before” names the earliest outcome a change preceded. The self-improvement series (§ 3) record their own deviations, dated, in their launch records and outcome entries, and Table 14 lists them.
| Date | Change | Before |
| 10-21 | CE8-T’s memorisation guard judges the kept checkpoint: CE8-T’s training memorised if its monitoring loss is not below the base’s, or its windowed training loss is below half its monitoring loss. Defined after the original guard stopped both CE8 runs on a last-evaluation rise of 0.0003 to 0.003; Under this rule both CE8 runs would pass (kept checkpoints at steps 27 and 30, monitoring loss below the base’s, training loss above half of it); CE8 stays not evaluated, because the guard in force for it fired. From here on the protocol changes in this paper, not in the registration file. | CE8-T’s training |
| 10-21 | CE8 being closed, CE8-T gets its own shuffled negative control (CE8-T-S, from the CE8-T experiences) and the one-pass budget the second CE8 run used; its step count follows the trainer’s own record count, read from a one-step sizing run. | CE8-T’s training |
| 10-22 | G8 for the 27B compares served and in-process teacher-forced log-probabilities on held-out records, against the base’s own served-vs-in-process gap, instead of agent verdicts: the 27B has no in-process agent path (splinter’s local provider runs qwen3 only), found when the served half of the verdict check had run (9 of 23 verified) and before any comparison of the two halves existed. | G8 and any evaluation arm |
| 10-22 | CE8-T and CE8-T-S are solved on the pilot subset P23 under amendment 20 (thinking off) before G8 is measured, which the registration places before any arm; both comparisons have fewer than 25 non-tied families, are labelled pilots by the rule, and no confirmatory query follows from them. | any CE8-T evaluation |
Table 11. Changes to the protocol made in this paper, in the order they were made.
| Date | Change | Before |
| 10-26 | The registration’s amendment history (the dated amendments appended to prereg/coding-tasks.md before the runs they could affect) is kept in the repository’s history of that file and is no longer restated in this paper; the protocol of Appendix § 21 is the registration as it stands, and the changes made in this paper are the rows above. | the paper’s final form |
| 11-06 | Generation 0 of the 27B loop (Appendix § 23) is decided without the mechanical-design exam (its trials of up to three hours could not finish in the lease at the scaled load) and on the pooled exam trials: a disk-quota failure and one overloaded re-run ended part of the original exam jobs, so each candidate is paired with the champion over every trial of the exam and its re-runs that the agent itself ended, leaving out trials ended by an infrastructure exception, a model-server error or a step timeout (the original candidate exams ran overloaded). | generation 0’s decision |
| 11-07 | Generation 1 explores a frontier pool, the training tasks generation 0 did not always solve and the never-explored training tasks with several faults, with the promoted champion; its exploration is stopped at a fixed time (04:30) to leave the lease room for a second decision, which gave each pool task 3 or 4 attempts instead of the planned 8, and two of the four planned training variants are trained. | generation 1’s training |
| 11-07 | No third generation was run (it could not finish in the lease). Instead the base sat the held-out exams on generation 1’s stack, the repository exam with several faults completed by a second job on the idle cards, so that the base and both champions are compared on one measurement. | the comparison of the base with both champions |
Table 13. Table 13: Changes made in this paper, continued (the last three rows concern the 27B generations of Appendix § 23).
| Series | Change | Before |
| S1 HL | The first run of arm HL ended on the project disk’s inode quota and was not evaluated; the registered replication was rerun on a cleared disk with the same scripts and binaries. [RS16,RS37] | arm HL’s evaluation |
| S2 | The policy-gradient objective scores the behaviour policy as the start of the arm, because the records carry no sampler log-probabilities at the launch build; each arm samples fresh from the policy it starts from. Generation 2 was not run (the shared adapter card id); the criterion was already decided by generation 1. [RS08,RS37] | generation 1 |
| S3 | Lineage r2 was relaunched before its first round with the request cap of its held-out proposals raised to 1400 (its seed reached 415 of 450 programs in 700 requests); r3’s generation 3 was resumed from its journals after a transient connection close, with the same build and settings. [RS37] | r2’s first round; r3’s version |
| S4 | The examiner’s build was changed so that it identifies a declared base by the checkpoint’s content under either digest rule, and it served on another card; exam, protocol, serving build, tier, candidates and criterion unchanged. [RS06,RS37] | any verdict |
Table 14. Table 14: Dated deviations of the self-improvement series, as their launch records and outcome entries state them.
Glossary
base
The untrained checkpoint a series or comparison starts from (Qwen3-4B in the series S1–S4).
candidate, champion
A candidate is a newly trained model; the champion is the model the next round starts from. A candidate replaces the champion when the learner’s gate accepts it.
| Date | Change | Before |
| 10-25 | The release gate’s improvement check also requires the candidate to have solved at least as many of the suite’s assigned tasks as its champion; an attempt withheld as exposed or as a harness failure counts as a task that arm did not solve, so a regression can no longer leave the paired test’s denominator. The measured shape that motivated it: a base at 21/46 (9 exposed) against a candidate at 15/46 (19 exposed) read as 5W-3L-15T under the paired test alone. | any release after it |
| 10-25 | Development probes on D46 (46 tasks, one per family, greedy) measure the supervision ablation and the recovery dose-response. They are labelled pilots, are not the registered confirmatory query, and their outcomes are not used as evidence for it; the registration, its partitions and its denominators are unchanged. [D46A,D46C,RCVA,RCVC] | the D46 probes |
| 10-25 | The evaluation-variance explanation is corrected. The two serving probes that appeared to reproduce run-to-run nondeterminism hashed the whole response payload, which carries a random tool-call id and a random chat id, so their distinct-completion counts were artifacts. Remeasured on the completion’s token ids and log-probabilities, a batch-1 greedy decode is bit-reproducible, a cache hit equals a cold prefill, and the co-batch does change a row (the int8 projections switch to a tensor-core block from five rows and the decode attention sizes its split count from the batch) - but one request in flight does not make a whole episode reproducible, so a second, unidentified source remains. The three-run greedy spread this paragraph explains (30, 31 and 26 of 46) is unchanged. [REP1,REP2] | the REP1 paragraph and any reading of a single greedy arm |
| 10-25 | The confirmatory candidate is registered before its query: the decisive-action adapter of training seed 13 (corpus blake3:92252209, rank 16, alpha 32, , 120 steps, packed-bf16 base, the rebuilt training binary), chosen after the six-decoding-seed development read on D46 in which it won 22 families, lost 11 and tied 13 against the base and the same against its all-action control ( each) while the two controls did not separate. The query solves the frozen qualification set (blake3:f7fe502f, 46 families) with the candidate, its base and the all-action control at the three registered decoding seeds and the protocol’s sampling; the decision is the registered one-sided exact sign test at , and a result that is not significant is reported as not shown. [PROTO6] | any candidate query |
Table 12. Changes made in this paper, continued.
lineage, generation, step A lineage is one chain of generations from the base, differing from its siblings only in its seed; generation is trained from material generation ’s champion produced; a step compares generation with generation (generation 1 with the base).
frontier, frontier union In S4, the programs a champion predicts wrong in at least one of 4 sampled attempts; the union is every accepted generation’s frontier so far.
held-out exam The 889 code-written Lua items of the series S1–S4: held by the examiner, never read by the learner, items overlapping a trained program excluded. Its file is published, so held-out means kept from the learner, not secret.
examiner, hand-off, verdict The examiner is a separate program that holds the exam and grades candidates; the hand-off is the one-way directory through which the learner gives it each candidate and its trained programs; a verdict is its signed aggregate result for one step, and the plan verdict applies the registered rule over a series.
dev set, retention set, probes The learner’s own measurement sets, built from its generator and the base’s proposals (S series) or written by the base from the manual (probes, Appendix § 24); never exam items.
family The unit a sign test counts once: in the coding experiments the file a fault was seeded in; on the S-series exam the program template an item was generated from (25 templates).
exposed An agent attempt whose tool calls named a hidden path or whose output repeated hidden content; it is withheld from training and from the paired test.
D46, P23 Development benchmarks of the coding experiments: D46 is 46 development tasks, one per family; P23 is 23 tasks, one per family of the first qualification half.
CE1, CE4, CE8, CE8-T, CE8-T-S, CE8-T-u1b Coding experiments: the teacher’s first pass (CE1), the pilot of the headline (CE4), the 27B model on its own trajectories (CE8), its variant with its own guard (CE8-T), the shuffled-instruction control (CE8-T-S) and the decision-level variant (CE8-T-u1b).
pass@1 (coding) In the coding results, tasks solved out of 46, averaged over decoding seeds.
References
- [1]Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, et al. Deep reinforcement learning at the edge of the statistical precipice, 2021. arXiv:2108.13264.arxiv.org/abs/2108.13264
- [2]Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search, 2017. arXiv:1705.08439.arxiv.org/abs/1705.08439
- [3]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, 2009. doi:10.1145/1553374.1553380.DOI
- [4]Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, et al. Accounting for variance in machine learning benchmarks, 2021. arXiv:2103.03098.arxiv.org/abs/2103.03098
- [5]Tianqi Chen, Bing Xu, Chiyuan Zhang, et al. Training deep nets with sublinear memory cost, 2016. arXiv:1604.06174.arxiv.org/abs/1604.06174
- [6]Xiaoyin Chen, Jiarui Lu, Minsu Kim, et al. Self-evolving curriculum for LLM reasoning, 2025. arXiv:2505.14970.arxiv.org/abs/2505.14970
- [7]Xiaoyin Chen, Canwen Xu, Yite Wang, et al. Learning to self-evolve, 2026. arXiv:2603.18620.arxiv.org/abs/2603.18620
- [8]Tri Dao, Daniel Y. Fu, Stefano Ermon, et al. FlashAttention: Fast and memory-efficient exact attention with IO-awareness, 2022. arXiv:2205.14135.arxiv.org/abs/2205.14135
- [9]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, et al. QLoRA: Efficient finetuning of quantized LLMs, 2023. arXiv:2305.14314.arxiv.org/abs/2305.14314
- [10]Yangruibo Ding, Jinjun Peng, Marcus J. Min, et al. SemCoder: Training code language models with comprehensive semantics reasoning, 2024. arXiv:2406.01006.arxiv.org/abs/2406.01006
- [11]Cynthia Dwork, Vitaly Feldman, Moritz Hardt, et al. Generalization in adaptive data analysis and holdout reuse, 2015. arXiv:1506.02629.arxiv.org/abs/1506.02629
- [12]Zihe Gao, Zhengling Qi, and Yifan Cui. Trustworthy recursive self-improvement with statistical guarantees, 2026. arXiv:2611.10787.arxiv.org/abs/2611.10787
- [13]Alex Gu, Baptiste Rozière, Hugh Leather, et al. CRUXEval: A benchmark for code reasoning, understanding and execution, 2024. arXiv:2401.03065.arxiv.org/abs/2401.03065
- [14]Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, et al. Reinforced self-training (ReST) for language modeling, 2023. arXiv:2308.08998.arxiv.org/abs/2308.08998
- [15]Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach themselves to program better, 2022. arXiv:2207.14502.arxiv.org/abs/2207.14502
- [16]Edward J. Hu, Yelong Shen, Phillip Wallis, et al. LoRA: Low-rank adaptation of large language models, 2021. arXiv:2106.09685.arxiv.org/abs/2106.09685
- [17]Naman Jain, Jaskirat Singh, Manish Shetty, et al. R2E-Gym: Procedural environments and hybrid verifiers for scaling open-weights SWE agents, 2025. arXiv:2504.07164.arxiv.org/abs/2504.07164
- [18]Carlos E. Jimenez, John Yang, Alexander Wettig, et al. SWE-bench: Can language models resolve real-world GitHub issues?, 2023. arXiv:2310.06770.arxiv.org/abs/2310.06770
- [19]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, et al. Overcoming catastrophic forgetting in neural networks, 2016. arXiv:1612.00796.DOI
- [20]Hung Le, Yue Wang, Akhilesh Deepak Gotmare, et al. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning, 2022. arXiv:2207.01780.arxiv.org/abs/2207.01780
- [21]Chenxiao Liu, Shuai Lu, Weizhu Chen, et al. Code execution with pre-trained language models, 2023. arXiv:2305.05383.arxiv.org/abs/2305.05383
- [22]Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations, 2024. arXiv:2411.00640.arxiv.org/abs/2411.00640
- [23]Jiayi Pan, Xingyao Wang, Graham Neubig, et al. Training software engineering agents and verifiers with SWE-Gym, 2024. arXiv:2412.21139.arxiv.org/abs/2412.21139
- [24]Rafael Rafailov, Archit Sharma, Eric Mitchell, et al. Direct preference optimization: Your language model is secretly a reward model, 2023. arXiv:2305.18290.arxiv.org/abs/2305.18290
- [25]Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, et al. Do ImageNet classifiers generalize to ImageNet?, 2019. arXiv:1902.10811.arxiv.org/abs/1902.10811
- [26]Burr Settles. Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Springer, 2012. doi:10.1007/978-3-031-01560-1.DOI
- [27]Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, et al. The curse of recursion: Training on generated data makes models forget, 2023. arXiv:2305.17493.arxiv.org/abs/2305.17493
- [28]Avi Singh, John D. Co-Reyes, Rishabh Agarwal, et al. Beyond human data: Scaling self-training for problem-solving with language models, 2023. arXiv:2312.06585.arxiv.org/abs/2312.06585
- [29]Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, et al. Intrinsic motivation and automatic curricula via asymmetric self-play, 2017. arXiv:1703.05407.arxiv.org/abs/1703.05407
- [30]Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, et al. Self-instruct: Aligning language models with self-generated instructions, 2022. arXiv:2212.10560.arxiv.org/abs/2212.10560
- [31]Erik Wijmans, Brody Huval, Alexander Hertzberg, et al. Cut your losses in large-vocabulary language models, 2024. arXiv:2411.09009.arxiv.org/abs/2411.09009
- [32]Qizhe Xie, Minh-Thang Luong, Eduard Hovy, et al. Self-training with noisy student improves ImageNet classification, 2019. arXiv:1911.04252.arxiv.org/abs/1911.04252
- [33]John Yang, Carlos E. Jimenez, Alexander Wettig, et al. SWE-agent: Agent-computer interfaces enable automated software engineering, 2024. arXiv:2405.15793.arxiv.org/abs/2405.15793
- [34]John Yang, Kilian Lieret, Carlos E. Jimenez, et al. SWE-smith: Scaling data for software engineering agents, 2025. arXiv:2504.21798.arxiv.org/abs/2504.21798
- [35]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule, 2024. arXiv:2412.06464.arxiv.org/abs/2412.06464
- [36]Yang Yue, Zhiqi Chen, Rui Lu, et al. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?, 2025. arXiv:2504.13837.arxiv.org/abs/2504.13837
- [37]Wojciech Zaremba and Ilya Sutskever. Learning to execute, 2014. arXiv:1410.4615.arxiv.org/abs/1410.4615
- [38]Eric Zelikman, Yuhuai Wu, Jesse Mu, et al. STaR: Bootstrapping reasoning with reasoning, 2022. arXiv:2203.14465.arxiv.org/abs/2203.14465
- [39]Eric Zelikman, Eliana Lorch, Lester Mackey, et al. Self-taught optimizer (STOP): Recursively self-improving code generation, 2023. arXiv:2310.02304.arxiv.org/abs/2310.02304
- [40]Jenny Zhang, Bingchen Zhao, Wannan Yang, et al. Hyperagents, 2026. arXiv:2603.19461.arxiv.org/abs/2603.19461
- [41]Tianjun Zhang, Shishir G. Patil, Naman Jain, et al. RAFT: Adapting language model to domain specific RAG, 2024. arXiv:2403.10131.arxiv.org/abs/2403.10131