betaFeedback

WorksComputer Science

Rules, Traces and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2

Abstract

We study which training and calibration data help three open models solve ARC-AGI-2 tasks when the solver runs offline on four L4 GPUs for at most 12 hours. We compare data offline on public tasks and report 189 runs on the 120 hidden semi-private tasks. For the NVARC Qwen3-4B grid model, we built a corpus of 12,000 executable rules in a typed language and 95,993 tasks; a rule is admitted only if its behaviour on a fixed probe set differs from that of every admitted rule, and a task only if its pairs refute a shortlist of nearest competing rules and defeat 12 shortcut solvers. Continued fine-tuning on 1,920 real task records with revised four-GPU scheduling raised the semi-private score from 28.96 (mean of the released model) to 33.06 (mean of five runs, best 34.86); no checkpoint trained with synthetic tasks, even all 95,993 of them, improved on the real-task model. For the Qwen3.8-27B reasoning model, we fine-tuned on 1,036 of its own reasoning traces with correct final answers and compared sets that add traces of other models and synthetic tasks. With an inference procedure built around the model (a thinking cap enforced in the serving engine, a Python program checked on the demonstrations after each answer, a long first attempt per input), the score rose from 26.81 to 44.03, the level the ARC Prize Foundation measured for the base model without a time limit (42.36); most of the gain came from the inference procedure. For the Qwen3.8-Flash-Next mixture-of-experts model (125B parameters, 6B active), GPU memory limited concurrency. Pruning from 512 to 384 experts per layer by router-weighted activation saliency grew the key-value cache from 0.55M to 1.39M tokens; a 296-expert list adapted with ARC routing statistics, keeping experts for program tokens, plus an FP8 indexer cache, reached 1.96M. With a tool that checks programs, the model scored 56.67, our best result.

Introduction

ARC-AGI-2 asks a solver to infer a grid transformation from a few demonstration pairs and apply it to new inputs [1, 2]. Its hidden tasks are scored on a fixed computer: the solver runs without internet access on four NVIDIA L4 GPUs for at most twelve hours, over 240 hidden tasks [2]. Under this budget, the best private-set score reported for 2025 was 24.03%, by NVARC [3]. A solver in this setting must be small or fast. This paper is about the data that a model learns from, or is calibrated on, before the run, and about how much of the measured gains such data explain next to changes in serving and scheduling.

We worked with three open models that fit the budget: the NVARC Qwen3-4B grid model, which outputs the answer grid and is trained again on each task at test time [4, 5]; the dense Qwen3.8-27B reasoning model [6], which thinks for tens of thousands of tokens before it answers; and the Qwen3.8-Flash-Next reasoning model [7], a mixture of experts with 6B of its 125B parameters active per token. For each model we compared training or calibration data, offline on public tasks and, for the main configurations, on the 120 hidden semi-private tasks (189 runs in total). Our contributions are:

  • A corpus of 12,000 signature-distinct rules with identifiability checks (§ 3). Each rule is a typed program admitted only if its behaviour signature (on 220 probe grids and 352 typed slots) differs from that of every other rule, and each of the 95,993 tasks only if, when it was generated, its pairs refuted a shortlist of nearest competing rules and defeated 12 shortcut solvers. The corpus, with a set of 1,000 tasks that covers every grid size from 1 × 1 to 30 × 30, supplied the synthetic data of the experiments below, including 2,231 synthetic tasks in the fine-tuning experiments of Qwen3.8-27B (§ 5.5).

  • Real and synthetic training data for a grid model (§ 4). We compared continued fine-tuning of NVARC 4B on 1,920 real tasks with two FORGE50 recipes, a shape-balanced curriculum and checkpoints trained on the rule corpus (Table 18). Fine-tuning on the real tasks raised the mean semi-private

Table 18. Data experiments across the three models. Each row compares a training set (NVARC 4B, Qwen3.8-27B) or the data that choose the experts to keep (Qwen3.8-Flash-Next) with a control. The two sides share the evaluation; unless a row says “same code” or “same configuration”, or uses the dev panel, they can also differ in training length, initialisation or runtime, so most rows compare configurations rather than isolate the data. SP: semi-private score, mean ± standard deviation over draws, with the number of draws in parentheses. Offline: test inputs solved of the 172 public evaluation inputs by one attempt at the given cap, in the 1-request two-slot harness unless marked 2-request. Dev: correct attempts of 246 on the dev panel at a 96K cap (82 inputs × three seeds). The answer-only control was measured by the ARC Prize Foundation under its own protocol.

Training or calibration dataMetricResultControl
NVARC 4B grid model (Table 6); control: the released model unless stated
1,920 real tasks, continued fine-tuning then 25% FORGE50 episodesSP31.72 ± 1.52 (22)28.96 ± 0.66 (4)
86% FORGE50 episodes, 12% real and 2% hand-made tasksSP28.47 (1)28.96 ± 0.66 (4)
1,920 real and 95,993 corpus tasksSP28.61 (1)28.96 ± 0.66 (4)
1,920 real and 5,580 synthetic tasksSP28.06 (1)28.96 ± 0.66 (4)
shape-balanced curriculum of 36,000 tasksSP26.88 ± 0.49 (2)28.96 ± 0.66 (4)
FORGE50 continuation in place of ckpt-34000 on one of four GPUs, same codeSP32.64 ± 1.91 (3)ckpt-34000 on four GPUs: 33.06 ± 1.57 (5)
Qwen3.8-27B reasoning model (Table 9 and Table 12); control as stated
answer-only, 37,201 examples, thinking offSP2.08 (1)base, thinking off: 1.5%
RFT v1: 1,036 verified self-tracesoffline, 64K, 2-request63base: 57
RFT v2: +777 traces, 755 with program targetsoffline, 32K49RFT v1: 51
RFT v3: +hard samplingoffline, 32K42RFT v1: 51
short16k: traces of at most 16K tokensoffline, 32K, 2-request34RFT v1: 37
v3b: 1,000 synthetic explanations in the thinkingoffline, 32K, 2-request9RFT v1: 37
ft64k: 1,584 synthetic explanations as answer textoffline, 32K43RFT v1: 51
FINAL: +452 traces of other modelsSP, same configuration29.03 (1)RFT v1: 44.03 (1)
Qwen3.8-Flash-Next expert lists (Table 15); control as stated
REAP-384: saliency on coding trafficdev87unpruned: 78
ARC-288: routing frequency on ARC training tasksdev58REAP-288: 66
s288p: REAP-288 core, least-used experts exchanged by ARC routingdev80REAP-288: 66
c8: s288p +8 experts per layer for program tokensprograms that do not compile, matched replay5.5% (9/165)s288p: 37.6% (62/165)

score from 28.96 to 31.72, and running the fine-tuned model on all four GPUs with revised scheduling raised it to 33.06 (best 34.86); no checkpoint trained with synthetic tasks, alone or next to the real-task model, raised the mean.

  • Answer-verified self-traces for a reasoning model (Section 5). We trained Qwen3.8-27B on 1,036 self-generated traces whose answers match the known outputs, selected by closure, loop and give-up filters, and compared this set with later sets that add second-pass or shorter traces, rule explanations, program targets, harder samples, traces of other models and synthetic tasks (Table 9). Around the fine-tuned model we built an inference procedure: the thinking cap is enforced inside the serving engine, a Python program written after each answer is checked on the demonstrations, and the long first attempt of every input is started before any second attempt. The score rose from 26.81 to 44.03, the level that the ARC Prize Foundation measured for the base model without a time limit; most of the gain came from the inference procedure, and offline validation of the deployed schedule matched the semi-private score.

SetCompositionRowsEval rowsThinking tokens
Answer-onlysynthetic + official grids, thinking off37,2011250
RFT v1base xhigh self-traces1,036 (1,033)136 (61)17.08M / 12.2K / 38.0K
main2RFT v1 + second-pass traces at 88K1,109 (1,103)15219.66M / 13.0K / 41.2K
short16ktraces of ≤16K tokens61744.07M / 5.7K / 13.4K
v3b769 own traces + 1,284 explanations (1,000 synthetic) in the thinking channel2,053025.1M
ft64k1,039 own traces + 1,602 explanations (1,584 synthetic) as answer text2,641–20.2M + 17.9M
RFT v2RFT v1 + 777 RFT v1 traces, 755 with programs1,813 (1,810)15226.08M / 10.7K / 31.0K
RFT v3RFT v2 + hard sampling, rare-hit rule1,972 (1,969)192 (62)29.76M / 12.1K / 30.5K
E3cleaned RFT v1 + 315 boundary × view traces1,337 (not trained)19723.33M / 13.6K / 38.9K
FINALE3 + 452 teacher (28 synthetic) + 35 self-traces, H included1,824 (1,801)414 (106)36.59M / 17.2K / 41.8K

Table 9. Fine-tuning sets for Qwen3.8-27B. Rows are training examples (rows actually trained in parentheses), eval rows are rows from public evaluation tasks (tasks in parentheses), and thinking tokens are total / median / 90th percentile in Qwen tokens. All sets except the answer-only one contain only traces whose final grid equals the label.

  • Calibration data for expert pruning (§ 6). We pruned Qwen3.8-Flash-Next from 512 to 384 experts per layer by router-weighted activation saliency [8] and then to 296, exchanging the least-used experts by routing statistics from ARC training tasks and keeping experts for program tokens. The coding-calibrated saliency lists and the ARC routing-frequency lists selected substantially different experts, and a list chosen by frequency alone lost accuracy. Pruning, an FP8 indexer cache and a larger cache reservation grew the key-value cache 3.6 times, and Qwen3.8-Flash-Next reached 56.67, our best score.

  • An analysis of the fixed budget (§ 8): tokens per input constrain the score, answers forced at the token cap are rarely right, and a program checked on the demonstrations gives a second answer of high precision.

  • Unsuccessful and incomplete experiments (Sections A and B). We report the hyper-parameters and the variants that did not help with their measured results, including 16 expert lists and checkpoints of Qwen3.8-Flash-Next that we rejected (Table 21).

Table 21: Expert lists and smaller checkpoints of Qwen3.8-Flash-Next that we rejected. “Dev” is correct attempts on the dev panel at a 96K cap (246 with three seeds unless noted); “production mix” is the task-weighted score of thinking and programs as used on the L4 GPUs, and “thinking” the score of the thinking alone, both as differences in points against c8. The adopted lists were REAP-384, then s288p, then c8 (Table 15).

List (experts)ConstructionMeasured resultWhy rejected
REAP-448 (448)public REAP listdev 26 of 82 (one seed) against 30 unpruned; CUDA-graph capture out of memory at a 4.0 GiB cache pinfrees less memory, not more accurate
REAP-288 (288)public REAP listdev 66 against 78 unpruned and 87 REAP-384accuracy
ARC-288 (288)top 288 by ARC routing frequency FeF_edev 58, lowest of all lists (23 gained, 43 lost against unpruned, p=0.019p = 0.019); 85% natural completions at 62.6K thinking tokensaccuracy
CORE-320 (320)REAP-288 + 32 experts by FeF_edev 84; fresh panel +2.64 points against REAP-384, lower bound −4.46failed the adoption rule
CORE-300 (300)REAP-288 + 12 experts by FeF_edev 77accuracy
c336o (336)CORE-320 + the 16 highest-FeF_e experts that REAP-384 dropsdev 56 against 60 for REAP-384 (two seeds, 164); +4.3% throughputspeed gain below 5%
s288p v2 (288)s288p with exchanges chosen from the traces of lost tasksno missing expert was used heavily on the 9 lost tasksno target to fix; never run
c8−24 (272)c8 minus its 24 lowest-FeF_e experts outside the code experts and concentrated specialiststhinking +0.83; production mix −2.16accuracy
c8−40 (256)as above, 40 removedthinking +0.90 over 344 pairs; production mix −3.02 (probability of a gain 0.05); cache +14%accuracy
c8−72 (224)as above, 72 removedcache +25%; L4 probe: 17 of 32 first attempts right, s288p 21 (p=0.39p = 0.39)accuracy
SAL-256 (256)c8 cut by REAP saliency measured on ARC tracesthinking −3.61; production mix −1.15accuracy
FREQ-224 (224)c8 cut by FeF_e with soft protectionthinking −6.25 (probability of a gain 0.04)accuracy
c8−40 protected (256)variant of c8−40 with protected specialists and code expertsagent configuration −17.8 points task-weighted, −26.8 on single-input tasks; median first-turn thinking 13,441 against 27,540 tokensstops thinking early
e384 (384)third-party list by routing share on ARC agent transcriptsnot runcalibrated on 60 of the 120 public evaluation tasks
REAM-288 (288)third-party build with merged expertsnot runkeeps the same experts as REAP-288 in all 48 layers
INT4 attention (512)round-to-nearest INT4 for attention and Gated DeltaNet weightsteacher-forced log-likelihood −0.64 nats per token; 29.9% of tokens off by more than 1 natbreaks the model

Table 21.

All scores are semi-private scores (SP) of single twelve-hour runs unless noted; repeated runs of identical NVARC 4B configurations vary with a standard deviation of 1.94 points (§ 7), and we have too few repeats to estimate the variation of the other two models.

Background

ARC-AGI-2 and its evaluation protocol

An ARC task [1] consists of a few demonstration pairs of coloured grids (up to 30×3030 \times30 cells, ten colours) and one or more test inputs; the solver must produce the exact output grid of each test input. ARC-AGI-2 [2] provides 1,000 training tasks and 120 public evaluation tasks with 172 test inputs, and keeps 240 further tasks hidden: 120 semi-private and 120 private tasks. A solver may give two attempts per test input, and the score is the mean over tasks of the fraction of test inputs answered correctly. The hidden tasks are scored by running the solver without internet access on four NVIDIA L4 GPUs (24 GB each) for at most 12 hours [2]. One run covers the 120 semi-private and the 120 private tasks together [2]; at the 1.43 test inputs per task of the public evaluation set this is about 340 inputs. Served on the four GPUs with an FP8 key-value cache and multi-token prediction, Qwen3.8-27B generates about 490 tokens per second, or about 21M tokens in twelve hours and 62K tokens per input, so a long first attempt fits for every input but a second attempt of the same length does not. We write K and M for thousands and millions of tokens; thinking caps are round thousands (a 56K cap is 56,000 tokens). We report the score on the 120 semi-private tasks, the semi-private score (SP); private scores were not available at the time of writing.

Synthetic tasks

The ARC training set is small, and the highest-scoring systems of 2024 and 2025 trained on generated tasks [3, 9]. re-arc [10] provides procedural generators for the ARC-AGI-1 training tasks, which yield unlimited new instances of known rules. BARC [11] asked an LLM to write new generator programs from seed programs and used them to train both a program writer (induction) and a model that outputs grids directly (transduction). NVARC [4] produced synthetic puzzles at a larger scale and trained its grid model on 3.26M augmented samples. Our generator (§ 3) differs in that each rule is a typed program whose behaviour on a fixed set of probe grids must differ from that of every other rule, and every task is replayed by an exact oracle.

Grid models and test-time training

The ARChitects [5] fine-tuned an LLM to output grids as text, trained it again on each test task’s demonstrations at test time, sampled candidates by depth-first search under a probability cut-off, and ranked them by their likelihood under several symmetry views [12]; this system reached 53.5 points on the private set of ARC-AGI-1 [9], and their 2025 system replaced the autoregressive model with a masked-diffusion model [13]. Akyürek et al. [14] showed that test-time training (TTT) on augmented leave-one-out tasks improves few-shot accuracy on ARC. NVARC follows the ARChitects’ design with a Qwen3 [15] grid model of at most 4B parameters, trained on synthetic puzzles together with RE-ARC, ConceptARC [16] and MINI-ARC, and per-task LoRA TTT with rank 256. It reported 27.64% on hidden ARC-AGI-2 tasks in 2025, and the released model, built on Qwen3-4B-Thinking2507, later reached 29.72%. We call this released model NVARC 4B (qwen3_4b_grids15_sft139). Small recursive networks trained from scratch on ARC data, HRM [17] and TRM [18], take a different route; TRM reaches 7.8% on the ARC-AGI-2 public evaluation set with 7M parameters.

Programs for ARC

Code-trained language models write programs from natural-language specifications [19]. Greenblatt Greenblatt [20] sampled many Python programs per task from GPT-4o and kept those that reproduce the demonstrations. BARC’s induction models write programs directly, and SOAR [21] fine-tunes the program writer on its own search traces. A program that reproduces all demonstrations can still be wrong on the test input, so precision on the test matters as much as the rate of demonstration-perfect programs.

Reasoning models and rejection-sampling fine-tuning

Reasoning models produce a long chain of thought before the answer [22]. Training a model on its own correct solutions, selected by checking the final answer, is the idea behind STaR [23], rejection-sampling fine-tuning [24] and ReST [25]. On ARC the check is exact: a trace is kept only if its final grid equals the known output. Qwen3.8-27B [6, 26] is a dense 27B model with 64 layers, of which 48 use Gated DeltaNet linear attention [27] and 16 use full attention, and a multi-token-prediction head [28, 29] that we use for speculative decoding [30, 31]. Its chat template exposes a reasoning-effort setting. The ARC Prize Foundation measured the base model on the semi-private set without the twelve-hour budget: 42.4% at xhigh, for $0.45 per task, and 22.8% at low [32]. We serve it with vLLM [33] and quantize weights to FP8 or to NVFP4, a 4-bit floating-point format with FP8 scales per block of 16 values [34, 35], related to the microscaling formats of Rouhani et al. [36].

Mixture-of-experts models and expert pruning

A sparse mixture-of-experts layer replaces the feedforward block by many experts and a router that sends each token to a few of them [37]; Switch Transformers route each token to one expert and scale the number of experts at a fixed compute per token [38], and DeepSeek-MoE adds shared experts that every token uses [39]. Compute then grows with the active parameters, while memory grows with all of them. Expert pruning removes experts after training. Lu et al. [40] choose the experts to keep by the reconstruction loss of the layer output on calibration data, and Li et al. [41] merge experts guided by routing statistics. REAP [8] scores each expert by the gate value times the norm of its output, averaged over the tokens routed to it, and reports that pruning by this score preserves generative quality better than merging at the same compression. Qwen3.8-Flash-Next [7] is a mixture-of-experts reasoning model with Gated DeltaNet layers, 512 routed experts per layer and 6B of its 125B parameters active per token (§ 6.1).

A corpus of 12,000 executable rules

Rules as typed programs

A rule in our corpus (“law” in the generator code) is a program in a typed domain-specific language, stored as a JSON abstract syntax tree and executed by an exact NumPy oracle. The first version of the language (v1, 2,000 rules) has three stages and one parameter: a selector that picks one connected component by a metric and an objective (25 metrics, minimum or maximum), a mask operator applied to that component (38 operators, 48 with their amounts), a renderer that draws the result (19 renderers, 27 with their separator widths), and a connectivity of 4 or 8. This gives 50 × 48 × 27 × 2 = 129,600 distinct programs. The second version (v2, 10,000 rules) has seven families. One, the mask pipeline, extends v1 with one or two post operations on the mask; it spans 88.4M syntactic programs over both connectivities, of which production enumerated connectivity 4 only. The other six have their own typed stages, for example two selected objects combined by an aligned boolean relation, two framed panels combined by a 4-bit truth table, the repair of one defect in a symmetric template, rays and floods between role-coloured walls and sources, and counting a feature over 3 to 8 objects followed by packing tiles along an axis (Table 1). Figure 1 shows one program and the oracle branch that executes it.

A rule of the mask-pipeline family and the branch of the oracle that executes it

Figure 1. A rule of the mask-pipeline family and the branch of the oracle that executes it (condensed from the generator source).

FamilyStagesProgramsCandidatesAdmittedAttemptedValid tasksReleased
mask pipelineselect → mask → 1–2 post ops → render88.4M24,00017,1326,9753,206 (46%)2,400
two objectstwo objects → align → boolean relation104,04014,4006,5123,2572,484 (76%)1,800
multiple objectsgroup selector → aggregate43,05012,8003,6842,2592,110 (93%)1,600
panel booleantwo panels → align → truth table50,4009,6004,0761,5381,538 (100%)1,200
template symmetryrepair one defect of a symmetry12,9609,6002,3111,7091,500 (88%)1,200
path, ray, floodrays, shortest paths, floods9,3608,0002,1021,6131,307 (81%)1,000
count and packrank 3–8 objects → pack tiles49,15211,2001,2851,2841,021 (80%)800
total v289,60037,10218,63513,16610,000

Table 1. Admission of the 10,000 v2 rules by family. “Admitted” rules passed the behavioural identity gate; “attempted” rules entered task generation, which stopped at 1.25 times the quota; the yield is relative to the attempted rules.

Input grids are not written per rule. One scene generator per family proposes inputs for a given program, the oracle labels them, and gates decide whether the six resulting pairs identify the program. Every input carries a 10-cell palette key in its border that names the roles of six colours (frame, background, object ink, result, auxiliary, separator), and a new palette is drawn for every pair, so a rule never depends on absolute colours. The release manifest records that no language model wrote, selected, labelled or verified any rule; rules are points of the grammar chosen by a deterministic search. Building the corpus took 55.9 hours from the plan to the generation report on 14 CPU workers, and an independent audit took a further 26 hours on 8 workers after a first sealing attempt failed; no GPU was used.

Admission: behavioural uniqueness and identifiability

Algorithm 1 gives the construction of the v2 rules and Figure 2 the number of rules that passed each gate. Two programs can differ syntactically and compute the same function, so a rule is admitted only if its behaviour is new. The behaviour vector has 572 slots: 220 shared probe grids, 96 panel-boolean slots and 256 template-symmetry slots. Each rule runs only its own segment (the 220 grids, or the 96 or 256 typed inputs of its family) and records in each slot the output or the type of error; the other slots hold a fixed marker of its family, so rules of different segments always differ. A candidate is admitted if it executes on at least 16 probes, its vector is new both as is and in its canonical form under the eight dihedral conjugations x↦g−1(p(gx))x \mapsto g^{-1}(p(gx)), and its Hamming distance to every admitted rule is at least 3 in both forms. A difference of signature is a difference of function when two rules give different valid outputs on a shared input; a difference only in error types or family markers is not, so we call the admitted rules signature-distinct. Of 89,600 candidates, 37,102 passed (41.4%): 29,423 were dihedral-behaviour duplicates, 16,957 had fewer than 16 valid probes, 5,705 were raw-behaviour duplicates and 413 were near-collisions. No two candidates had the same syntax tree, so duplicates are a matter of behaviour, not syntax.

Admission funnel of the v2 rules

Figure 2. The admission funnel of the v2 rules (Algorithm 1). Task generation stopped once each family had 1.25 times its quota of valid tasks, so only 18,635 of the admitted rules were attempted. The release removed aliases, kept each family’s quota and was audited independently; the export added up to seven tasks per rule from new salts.

A task is accepted for an admitted rule only if its six pairs (five demonstrations and one test) separate the rule from its nearest competitors. The oracle must replay all six pairs exactly; every typed stage must change the output on at least two pairs; each of up to 8 shortlisted near rules and every rule that differs in one syntax node must be wrong on at least two of the six pairs, so that among these competitors any held-out pair is determined by the other five; and a battery of 12 shortcut solvers must solve none of them. The solvers are constant output, identity, shape and colour look-up tables, single colour swaps, dihedral colour maps, cropping to the non-background region, fixed sub-grids, row selection, column selection, scaling and tiling, and copying a demonstration output. Task generation stopped once each family had 1.25 times its quota of valid tasks; of the 18,635 admitted rules that were attempted, 13,166 (70.7%) produced a valid task. Before release we removed 199 mask-pipeline rules that reduce to a v1 rule when their post stages are dropped, 41 rules whose renderer is an exact alias of another, and one rule with a repeated input, and kept the first rules of each family up to its quota. Algorithm 2 gives the task gate as the generator runs it. Its nearest rules are a shortlist built from hashed projections of the raw and dihedral behaviour vectors and ranked by Hamming distance, not the result of an exact search over all admitted rules.

The 2,000 v1 rules were built earlier with stricter competitors (all 2,500 programs of the candidate pool) but a near-collision check only within each selector domain; two pairs of v1 rules lie at dihedral Hamming distance 2 and were kept as immutable. An independent auditor replayed every gate on the 10,000 v2 rules, treated the v1 rules as fixed, and found 12,000 distinct identities, 72,000 distinct canonical pairs and no task, pair or input that matches, under dihedral and colour canonicalisation, any of 19,721 reference tasks (145,053 pairs) from the ARC-AGI data files published by the ARC Prize Foundation from 2024 to 2026 and three earlier synthetic sets.

Tasks, variants and other generators

For training we generated up to seven further tasks per rule from new random salts, which gives 95,993 tasks (one rule yielded only its base task) with 575,958 labelled pairs. Each further v2 task passed the task gate of the base task when it was generated (exact replay, every stage material on at least two pairs, up to 8 shortlisted near rules and all one-node variants wrong on at least two pairs, no shortcut solver), and the further v1 tasks passed the v1 gate (materiality, all programs of the 2,500-program pool wrong on at least two pairs, shortcuts). At export every label was replayed by the oracle, and duplicate tasks, pairs or inputs shared with the ARC-AGI-2 public evaluation set and leakage across the rule split were rejected; the export did not rerun the salt search or the materiality, competitor and shortcut gates, and the 95,993-task extension was not independently re-audited or sealed. Four canonical inputs repeat within one split. We held out 5% of the rules of each family with all their tasks (600 rules, 4,800 tasks). Table 2 compares the tasks with ARC-AGI-2. The synthetic inputs are larger (median 750 cells against 361 for the public evaluation set), always contain six colours, and the output has the shape of the input in 10.2% of pairs against 69.8% in the public evaluation set. In the compact token format of NVARC 4B, the median task takes 5,749 tokens and the longest 11,076.

SetTasksInput cells (median)Same shapeColours/input (mean)Tokens (median / max)
v1 rules15,20081021.8%65,994 / 11,076
mask pipeline18,24081022.6%66,175 / 11,076
two objects13,6738100%65,882 / 6,707
multiple objects12,1608100%65,790 / 6,684
panel boolean9,1203080%62,647 / 3,020
template symmetry9,1202100%62,077 / 3,537
path, ray, flood7,60064424.3%66,486 / 9,014
count and pack6,0805760%64,832 / 6,020
all synthetic91,19375010.2%65,749 / 11,076
ARC-AGI-2 training1,00010065.9%3.901,130 / 9,330
ARC-AGI-2 public eval.12036169.8%5.493,068 / 9,702

Table 2. Statistics of the synthetic tasks (95,993-task export, training split) and of ARC-AGI-2. Cells and the share of pairs whose output has the input’s shape are per pair; tokens are per task in the compact format of NVARC 4B (worst dihedral view).

We also used two other generators. A shape-balanced generator samples 72 short programs so that every query-input size from 1 × 1 to 30 × 30 appears exactly 30 or 40 times (27,000 and 36,000 tasks). Before the typed corpus, a catalogue of 591 mechanisms in rule-specific Python code, which we call FORGE50, produced a 12,000-task release and a 50,000-task corpus split by program signature; earlier rule-specific generators also produced 25 families of basic transformations and 120 declarative program signatures.

Algorithm 1 Construction of the 10,000 v2 rules and of the 95,993-task export.

Input: grammars G1,…,G7\mathcal{G}_1,\ldots,\mathcal{G}_7; quotas qfq_f with ∑fqf=10,000\sum_f q_f = 10{,}000; candidate multipliers mfm_f; probe bank PP (220 grids, 352 typed slots); the 2,000 v1 rules Lv1L_{v1}; reference tasks RR; seed σ\sigma

  1. Cf←C_f \leftarrow first mfqfm_fq_f programs of a hash permutation of Gf\mathcal{G}_f, for each family ff

  2. for each candidate pp do

  3. v(p)←[p(x)]x∈Pv(p) \leftarrow[p(x)]_{x\in P}; v8(p)←min⁡g∈D8[g−1p(g x)]x∈Pv_8(p) \leftarrow\min_{g\in D_8}[g^{-1}p(g\,x)]_{x\in P}

  4. admit pp iff ≥16\ge16 valid probes, new vv and v8v_8, and Hamming distance ≥3\ge3 to every admitted rule

  5. end for

  6. for admitted rules pp and salts ss, in bounded waves, until each family has 1.25qf1.25q_f valid tasks do

  7. propose inputs with the family scene generator (fresh palette per pair, 10-cell palette key); label with pp

  8. choose 6 pairs with distinct inputs, outputs, target shapes and structures, ≥2\ge2 output shapes, ≥4\ge4 palettes

  9. accept iff exact replay ∧\land every stage material on ≥2\ge2 pairs ∧\land each of ≤8\le8 shortlisted near rules and one-node variants wrong on ≥2\ge2 pairs ∧\land no shortcut solver succeeds

  10. end for

  11. drop v1 aliases and renderer aliases; require canonical task, pair and input uniqueness against Lv1L_{v1} and RR; keep qfq_f rules per family

  12. independent audit of every gate; seal the release of 12,000 rules

  13. per rule, up to 7 further tasks from new salts through the same task gate; at export, replay, duplicate and leakage checks only, with no new audit or seal (95,993 tasks); hold out 5% of the rules of each family

Representative rules and the models trained on them

Table 3 gives one rule of each corpus family and of the two families of the shape bank, together with the grid algorithms that its oracle runs, and Figure 3 shows two pairs of the base task of each corpus rule. The descriptions paraphrase the explanations that we computed from the verified programs (see § 5.5); the identifiers name the rules in the release. In every corpus rule, colours are roles read from the palette key (object, auxiliary, result, background), so the same rule draws with different colours in every pair. The algorithms are those of the language: connected components in 4- or 8-connectivity with a selector metric, morphological operations on masks (dilation, halos, hulls, hole filling, endpoints and branch points), alignment and boolean combination of two or more masks, truth tables over framed panels, symmetry orbits, breadth-first distances for rays, paths and floods, and counting with binary codes and packing. The shape bank adds whole-grid colour statistics and orderings of the cells.

Figure 3. The first two of the six pairs of the base task of each corpus rule in Table 3, drawn at one cell size per panel. Each input carries the 10-cell palette key on one border, and every pair draws a new palette, so the same rule paints with different colours from pair to pair.

Family (rule)RuleAlgorithms
v1 select–mask–render (u2k_0053)Select the object with the smallest anti-diagonal span, shift it one cell up, and draw the object above its shifted copy with a one-cell separator.components, selector metric, shift, panel rendering
mask pipeline (u12k_03977)Select the object with the smallest anti-diagonal span, take the cells that one step of dilation adds around it, keep those in exactly one of this halo and the object’s bounding box, keep the cells with exactly one occupied neighbour, and crop.dilation, symmetric difference, endpoint detection
two objects (u12k_11778)Take the object with the largest vertical imbalance and the topmost object, align their centres, and keep the cells of the second that the first does not cover.two selectors, centre alignment, set difference
multiple objects (u12k_05969)Take the three topmost objects, align their upper-right corners, and keep the cells that all three cover.top-kk selection, alignment, per-cell consensus
panel boolean (u12k_07934)Read the two framed panels, flip both top to bottom, align their lower-right corners, and mark the cells occupied in both.frame detection, reflection, 4-bit truth table
template symmetry (u12k_09489)Inside a frame, find the one cell whose change from the object to the auxiliary colour makes a cross-shaped region symmetric under a top-to-bottom reflection, repair it, and output its symmetry orbit.symmetry orbits, unique single-cell repair
path, ray, flood (u12k_08636)With object cells as walls and auxiliary cells as sources, flood from the source component that reaches furthest right to distance 2 without crossing walls, and output the halo of the flooded region.breadth-first distance, bounded flood, dilation
count and pack (u12k_02537)Measure the width of every object, order the objects by decreasing perimeter, replace each by a vertical binary code of its width, and pack the codes from left to right.counting, ranking, binary encoding, packing
spatial logic (bank)Select the cells of the rarest colour other than the background, replace them by the outline of their bounding box, add the 180-degree rotation of that outline, and paint it where the input is background.colour frequencies, bounding box, rotation, union, occupancy gate
shape sequence (bank)Read the cells from the outside in (first, last, second, second-last, …), append the first value again if fewer than 30 were read, and output the (n+1)×(n+1)(n+1) \times(n+1) matrix whose cell (r,c)(r,c) is foreground exactly when values rr and cc differ.cell ordering, pairwise relation matrix

Table 3. One rule of each family. Families above the line are those of the 12,000-rule corpus (v1 and the seven v2 families); the last two are the families of the 72-program shape bank. “Object” means a connected component of the object colour.

Both model lines were trained on these tasks. For NVARC 4B, checkpoints trained on the 1,920 real tasks plus all 95,993 corpus tasks in two views (191,986 synthetic examples) scored 28.61 alone and 27.64 to 32.64 next to the real-task model (§ 4.3 and § 4.4); the shape-balanced curriculum of 36,000 tasks was drawn from the same 72 bank programs. For Qwen3.8–27B, 2,231 synthetic tasks from both sources, among them one task for every grid size from 1×11 \times1 to 30×3030 \times30, entered the fine-tuning data as explanations or teacher traces (§ 5.5).

What is new relative to earlier generators

Table 4 contrasts the corpus with the generators used by earlier ARC systems. re-arc [10] writes one generator per ARC-AGI-1 training task, so it adds new instances of 400 known rules but no new rules. BARС [11] and NVARC [4] ask a language model to write new generators or puzAlgorithm 2 Task gate for one admitted rule pp (the generator’s task worker).

GeneratorSource of rulesChecks on rules and tasksDistinct rulesIdentifiable
re-arcone hand-written generator per ARC-AGI-1 training taskexamples checked by a per-task verifierby designnot reported
BARCgenerators written by a language model from seed programsexecution of the generated codenot reportednot reported
NVARCpuzzles and code written by a language modelexecution of the codenot reportednot reported
Ourspoints of a typed grammar, deterministic searchsignatures on 220 grid probes and 352 typed slots, raw and dihedral, Hamming distance ≥3\ge3 (two v1 pairs at 2); up to 8 shortlisted near rules and one-node variants refuted on ≥2\ge2 pairs; 12 shortcut solvers fail; exact oracle replaysignaturesfinite set

Table 4. Synthetic task generators for ARC. “Distinct rules” means that rules are checked to behave differently; “identifiable” means that each task is checked to separate its rule from a set of competing rules. “Not reported” means that we found no such check in the description of the generator.

Input: rule pp of family ff; its hard neighbours N(p)N(p), at most 8 admitted rules of the same domain (and, for the mask pipeline, v1 rules), shortlisted by hashed projections of the raw and dihedral behaviour vectors and ranked by min⁡(draw,dD8)\min(d_{\mathrm{raw}}, d_{D_8}) Hamming distance; salts s0,…,s1s_0,\ldots,s_1; scene generator of ff; the 12 shortcut solvers SS

  1. C←{q∈N(p)∪OneNode⁡(p):canon⁡(q)≠canon⁡(p)}C \leftarrow\{q \in N(p) \cup\operatorname{OneNode}(p) : \operatorname{canon}(q) \ne\operatorname{canon}(p)\} ▷ one-node: programs that differ from pp in one syntax node

  2. for s=s0,…,s1s=s_0,\ldots,s_1 do

  3. (xk,yk)k=16←scene generator⁡(p,s,N(p))(x_k,y_k)_{k=1}^{6} \leftarrow\operatorname{scene\ generator}(p,s,N(p)) ▷ fresh palette per pair, distinct inputs, outputs and target shapes, ≥2\ge2 output shapes

  4. skip ss if p(xk)≠ykp(x_k) \ne y_k for some kk ▷ exact oracle replay

  5. skip ss if some typed stage of pp changes the output on fewer than 2 of the 6 inputs

  6. skip ss if min⁡q∈C#{k:q(xk)≠yk or q fails on xk}<2\min_{q\in C}\#\{k:q(x_k)\ne y_k\text{ or }q\text{ fails on }x_k\}<2

  7. skip ss if a solver in SS solves the test pair from the five demonstrations

  8. return the six pairs with their evidence

  9. end for

  10. return no task for pp from these salts

zle descriptions, which adds rules; their descriptions do not report a check that two generators compute different functions or that a task’s pairs determine its rule. Our corpus makes both properties explicit tests, within stated limits. A rule enters only if its signature on a fixed probe bank differs from that of every admitted rule in at least three of its 572 slots (outputs or error types), also after canonicalisation under the eight dihedral conjugations; two pairs of v1 rules at dihedral distance 2 are the exception. The 12,000 rules therefore have 12,000 distinct behaviour signatures, which is stronger than 12,000 distinct programs but weaker than a proof of 12,000 distinct functions. A task enters only if its six pairs refute up to 8 shortlisted near rules and every one-node variant of its program on at least two pairs each, every typed stage changes the output on at least two pairs, and none of 12 shortcut solvers succeeds. Rules are points of a typed grammar chosen by a deterministic search, every label is replayed by an exact oracle, and no language model wrote, selected or labelled any rule. Identifiability is relative to this finite set of competitors, not to every possible transformation. Because the tests are behavioural, they could be applied to other program languages with an executable semantics.

Grid distribution.

Figure 4 shows the grid sizes of all 547,158 synthetic pairs of the training split next to ARC-AGI-2. Synthetic inputs are large: the median longer side is 30 cells (quartiles 25 and 30), 89.5% of inputs have a side of at least 20 and none has both sides of at most 10, against a median of 20 for the evaluation set (58.9% at least 20, 9.4% at most 10) and 12 for the training set. The outputs are smaller, with a median longer side of 15 cells (quartiles 10 and 19), between the training set (10) and the evaluation set (19). An output has a median of 19% of the cells of its input, while in ARC-AGI-2 the median output has the input’s size. The scene generators use 105 distinct input shapes, against 434 in the training set and 169 in the evaluation set.

Figure 4. Grid sizes of the synthetic corpus (547,158 pairs of the 95,993-task export, training split) and of ARC-AGI-2 (4,308 training and 530 evaluation pairs). Left and middle: the longer side of input and output grids. Right: the ratio of output to input cells on a log⁡2\log_2 scale; 0 means that the output has as many cells as the input.

Difficulty.

The gates make every task hard to solve by a shortcut: the 12 shortcut solvers solve no task by

construction, whereas they solved 15.7% of the tasks of an earlier rule-specific generator of ours. The tasks are also hard for the models we trained. Without test-time training, the released NVARC 4B solved 2 of 30 held-out synthetic tasks, and 6 with it; on 18 public evaluation inputs it solved 2 and 3. Qwen3.8-27B models fine-tuned for ARC solved 1 of the 30 synthetic tasks at a 32K thinking cap and 3 to 5 at 64K, while single attempts of the same models at 64K solve 52 to 57 of the 172 public evaluation inputs (30% to 33%). A median synthetic task also takes 5,749 tokens in the format of NVARC 4B, against 3,068 for an evaluation task, because its inputs are larger and it has six pairs.

How the rules differ from ARC-AGI-2

Vocabulary. A v1 rule reads “select one object, transform its mask, draw the result”. The 25 selector metrics measure a connected component: area, bounding-box area, width, height, perimeter, density, number and area of holes, the four extreme coordinates, the centre row and column, border contacts and touched border sides, the longest row and column runs, endpoints, branch points, the diagonal and anti-diagonal spans, vertical and horizontal balance, and the number of symmetries. The 38 mask operators are the identity, bounding-box fills, outlines, complements and corners, object boundaries and interiors in 4- and 8-connectivity, hole fills, hole masks and hole boundaries, row and column spans, convex and orthogonal hulls, the seven non-identity dihedral transforms, dilation, erosion and halos, shifts, endpoints, branch points and the centroid. The 19 renderers extract the result, overlay it on the input, isolate it, place it beside the source object in a horizontal or vertical panel, or draw it inside a frame. The v2 families add relations between objects (a boolean relation between two aligned masks, or per-cell support counts over a group of objects chosen by a metric predicate), truth tables over two framed panels, the repair of one defect against a symmetry or a period, three grid algorithms (ray casting, the union of all shortest paths, and bounded flooding between walls and sources), and counting a feature over 3 to 8 objects followed by packing one tile per object along an axis.

Grids and pairs. Table 5 compares the pairs of the training split with ARC-AGI-2, and Figure 5 shows their distributions. Three differences follow from the design. First, no synthetic output is larger than its input and 83.5% are smaller, because the renderers draw a selected object, a panel or a packed strip; the median output has 19% of the input’s cells, whereas the median ARC-AGI-2 output has the input’s size, and 8.6% of training and 3.4% of evaluation outputs are larger, most of the training ones by integer factors. Cropping is also rare: 0.1% of the smaller synthetic outputs are a sub-grid of the input, against 23.7% in the training set, because the shortcut battery rejects tasks that cropping solves. Second, the v2 pair selection requires at least two output shapes, so only 0.2% of synthetic tasks (173, all v1) keep one output shape across their pairs, against 41.3% of training and 20.0% of evaluation tasks. Third, colours carry no fixed meaning. A fresh palette is drawn for every pair, so no synthetic task keeps the same most frequent colour across its inputs, against 74.5% of training and 46.7% of evaluation tasks, and black is the most frequent colour of 10.0% of synthetic inputs against 66.8% and 34.5%. Synthetic outputs never contain a colour absent from their input, which 30.2% of training and 12.1% of evaluation outputs do. Where the output keeps the input’s shape, synthetic rules change fewer cells (median 7.9% against 16.0% and 12.3%). The inputs are about as crowded as evaluation inputs, with a median of 12 single-colour components outside the most frequent colour, part of them from the palette key, against 6 in training and 12 in evaluation inputs.

Figure 5. Pairs of the training split of the synthetic export (91,193 tasks, 547,158 pairs) against ARC-AGI-2 training (1,000 tasks) and public evaluation (120 tasks). (a) Shape of the output relative to the input, by family; numbers are percentages of pairs. (b) Tokens per task in the compact format of NVARC 4B (worst dihedral view) for the synthetic rows of (a), with whiskers at 1.5 times the interquartile range and outliers not drawn; diamonds are the ARC-AGI-2 medians of Table 2. (c) Share of changed cells in pairs whose output has the input’s shape. (d) Single-colour 4-connected components per input outside its most frequent colour, palette key included, clipped at 59. (e) Distinct colours per input.

StatisticSynthetic (91,193 tasks)ARC-AGI-2 training (1,000 tasks)ARC-AGI-2 eval. (120 tasks)
Pairs per task (mean)64.314.42
Output has the input’s shape10.2%65.9%69.8%
Output smaller (neither side longer)83.5%24.9%23.4%
Output larger (neither side shorter)0.0%8.6%3.4%
Other shape changes6.3%0.7%3.4%
Output cells over input cells (median)0.191.001.00
Smaller outputs that are a sub-grid of the input0.1%23.7%0.8%
Tasks with one output shape for all pairs0.2%41.3%20.0%
Changed cells in same-shape pairs (median)7.9%16.0%12.3%
Outputs using only colours of their input100%69.8%87.9%
Colours per input (mean)6.003.905.49
Tasks whose inputs share the most frequent colour0%74.5%46.7%
Inputs whose most frequent colour is black10.0%66.8%34.5%
Components per input (median)12612

Table 5. Transformation statistics of the synthetic tasks (95,993-task export, training split) and of ARC-AGI-2. Shares are of pairs unless the row says tasks; a sub-grid is a contiguous block of the input. Components are 4-connected single-colour regions of all colours except the input’s most frequent one; the synthetic count includes the cells of the palette key.

The design rules of ARC-AGI-2. Chollett et al. [2] describe four properties that ARC-AGI-2 tasks were built to require: several rules that act at once and interact, several steps where each depends on the previous one, rules whose application depends on a cue in the grid, and symbols whose meaning is defined inside the task. Our rules cover the first two only in a fixed form. A rule composes typed stages in one order (select, transform, draw, or the stages of its family), and a task applies exactly one rule. Stages run in a fixed order with no conditional or recursive composition; some stages iterate inside a fixed algorithm (paths, rays and floods, hole filling, component labelling), and the input-dependent choices are the object that a selector picks, the source of a path, the ranking of a count pack and the unique repair that the template family finds by search. The palette key is a symbol definition, but always of the same six roles, in the same 10-cell strip at the border in one of four orientations; a model can learn to read it once, while a legend in an ARC-AGI-2 task has a new form in every task. Two ingredients of real tasks have no counterpart in the vocabulary: colours with a meaning of their own, because every rule refers to colour roles, and outputs larger than the input, as in tiling, scaling or drawing beyond an object.

Limitations of the corpus

The gates guarantee that each task separates its rule from a finite shortlist of near rules of the corpus and from the one-node variants of its program, not that a person, or a model trained on real ARC tasks, can infer it; we ran no human or model inferability test. The format differs from ARC-AGI-2 in ways that a model can learn as shortcuts: the palette key appears in every input, every input has six colours, and most outputs are smaller than their input (§ 3.8). Only 4,400 of the 12,000 rules (the v1 rules and the mask-pipeline family) have a rule-specific English description; the other 7,600 carry a placeholder. Of the 600 held-out rules, 457 reappear in one derived training set and, by our notes, 498 in a standalone program set, although no pair does. § 4 and 5 report the consequences: the grid model fitted synthetic tasks when trained on them, but no model trained with them was better on real tasks, alone or next to the real-task model, and the best NVARC 4B configuration uses no checkpoint trained on the corpus (§ 4.4).

Real and synthetic training data for the NVARC grid model

The base pipeline

NVARC 4B is a Qwen3-4B model with its vocabulary cut to 16 tokens, which include the ten colours, a newline, role markers, padding and end of sequence; it was fully fine-tuned by its authors on 3.26M samples from 104,539 puzzles, about 90% of them synthetic [4]. The released pipeline solves each task in five steps, which we kept (Algorithm 3). First, a fresh LoRA adapter (rank 256, α=32\alpha= 32, rank-stabilised, on the seven projection types plus the embedding and output layers) is trained for one epoch on 128 augmented copies of the task’s demonstrations (8 dihedral transforms ×\times 16 colour permutations, demonstrations shuffled), with learning rate 5×10−55 \times10^{-5}. Second, every test input is rendered in 16 views (8 dihedral transforms ×\times 2 colour permutations). Third, a depth-first search over the 12 legal tokens keeps every branch whose cumulative probability stays above 0.2. Fourth, each distinct candidate, mapped back to the original frame, is scored by its negative log-likelihood (NLL) under 8 fresh views. Fifth, the score of a candidate is its number of

search hits minus its mean rescoring NLL, and the top two are the two attempts. On four L4 GPUs, one worker per GPU, test-time training takes 193 to 870 seconds per task and a whole task 517 to 1,371 seconds. Four runs with the base model, three of the released pipeline unchanged and one in our runtime with a 12,288-token context, scored 28.47, 29.86, 29.03 and 28.47 on the semi-private set (mean 28.96, standard deviation 0.66); the three unchanged runs alone average 29.12. For every run of this section we read the checkpoints from the source code or the log of the version that produced the score; a fifth run that we had first counted as a base run loads a checkpoint of our own fine-tuning and is counted in § 4.2.

Continued fine-tuning on real tasks

Before any per-task training, we continued to fine-tune NVARC 4B on all real tasks we could label: the 800 ARC-AGI-1 training and evaluation tasks, the 1,000 ARC-AGI-2 training tasks and the 120 ARC-AGI-2 public evaluation tasks with their test outputs appended as further demonstrations. This gives 1,920 source records with 6,427 demonstration pairs (3,852 distinct), rendered in 16 views per record (30,720 examples). The records cover 1,147 task identifiers: 767 ARC-AGI-1 tasks reappear among the ARC-AGI-2 training tasks, mostly with the same demonstrations in another order, and 6 among the evaluation tasks with one more demonstration, so these tasks carry twice the weight of the others. We trained a rank-256 LoRA adapter with the same targets as test-time training, learning rate 5×10−55 \times10^{-5} with a cosine schedule over two planned epochs (61,440 steps), batch 1 and an 8,192-token context, on one L4. The run stopped after 10 hours at step 34,273 (1.11 epochs); the checkpoint at step 34,000 is the model we call ckpt-34000. Training on the public evaluation tasks makes local evaluation on them meaningless for this model and all its descendants,

but it does not touch the hidden semi-private tasks. Twenty-two hidden runs used ckpt-34000, or a checkpoint of the same training run saved 2,000 or 4,000 steps earlier, as the only model on all four GPUs, and averaged 31.72 on the semi-private set (standard deviation 1.52, Table 6). The one run of ckpt-34000 in the otherwise unchanged released pipeline scored 31.53, and the best of the 22 scored 34.44. A half-epoch checkpoint of an earlier one-epoch run on the same records scored 29.03.

SystemDrawsMean ± SDBestΔ\Delta
Base model (three runs of the released pipeline, one in our runtime)428.96 ± 0.6629.86–
frozen base sampled without test-time training or search, Qwen3.8-27B selects110.4210.42−18.54
ckpt-34000 or a checkpoint of the same run, continued on 1,920 real tasks2231.72 ± 1.5234.44+2.77
+ 25% FORGE50 synthetic episodes (one model)129.3129.31+0.35
FORGE50 mixture with 86% synthetic episodes (one model)128.4728.47−0.49
Base + 1,920 real + 95,993 rule-corpus tasks (one model)128.6128.61−0.35
Base + 1,920 real + 5,580 synthetic tasks†128.0628.06−0.90
Shape-balanced curriculum (36,000 tasks)†226.88 ± 0.4927.22−2.08
Ensembles of ckpt-34000 and other checkpoints3630.62 ± 2.0834.31+1.66
with a checkpoint trained on 1,920 real + 95,993 rule-corpus tasks530.53 ± 1.9332.64+1.57
ckpt-34000 on all four GPUs, + runtime changes533.06 ± 1.5734.86+4.10
FORGE50 continuation on the fourth GPU, same code332.64 ± 1.9134.31+3.69

Table 6. Semi-private scores of the NVARC 4B line (ARC-AGI-2, 120 semi-private tasks, two attempts). Draws are separate runs; the pooled standard deviation of repeated runs of identical configurations is 1.94 points (27 draws of 9 configurations). Checkpoint lists come from the source code or log of the version that produced each score. Δ\Delta is the difference of means to the base model. Rows marked † ran in the same runtime as the row of ckpt-34000 with runtime changes, against which their differences are −5.8 to −6.5 (shape-balanced) and −5.0 (real and synthetic).

Synthetic tasks

We added synthetic tasks from several sources. Continuing ckpt-34000 with 25% of its episodes drawn from the FORGE50 generator (a catalogue of rule-specific Python generators, separate from the typed corpus of § 3; learning rate 10−510^{-5}) gave a model that scored 29.31 alone, a single draw 2.4 points below the ckpt-34000 mean. A second FORGE50 recipe, a weighted mixture of 86% FORGE50 episodes, 12% real ARC tasks from three pools including the public evaluation set (5%) and 2% hand-made tasks, scored 28.47. A curriculum of 36,000 tasks from the shape-balanced generator plus 1,090 real tasks, started from an adapter trained on a 6,000-task mix of real and rule-corpus tasks, ran for 47.8 hours on one RTX 3090 Ti. It raised accuracy on new tasks from the 72 programs of its own generator from 5.8% to 61.7% (14 to 148 of 240) and changed accuracy on 30 real evaluation tasks from 9 to 8 of 47 inputs; on the semi-private set it scored 27.22 at step 5,500 and 26.53 at the end. Training from the base model on 1,920 real tasks, 3,600 shape-balanced tasks and 1,980 rule-corpus tasks for 120,000 steps scored 28.06. We also fine-tuned checkpoints from the base model on the real tasks together with all 95,993 tasks of the rule corpus in two views each (191,986 synthetic examples; rank 256, one epoch, on an RTX 5090). One of them, run alone on all four GPUs, scored 28.61, 3.1 points below the ckpt-34000 mean; next Algorithm 3 Fine-tuning and test-time pipeline for NVARC 4B. Lines 1–4 run once offline; lines 5–12 run per task on each of the four GPUs.

Input: base model M0M_0; offline corpus DD; hidden test tasks T\mathcal{T}; deadline 12 h

1: D←D \leftarrow concatenation of ARC-AGI-1 train+eval, ARC-AGI-2 train and ARC-AGI-2 public eval (1,920 records; test pairs appended as demonstrations) 2: for each task, 16 views: 8 dihedral transforms ×\times 2 colour bijections, demonstrations shuffled 3: train rank-256 LoRA (+ embedding and output layers), cross-entropy on every output grid, learning rate 5×10−55 \times10^{-5} cosine, batch 1, 8,192 tokens ⇒M1\Rightarrow M_1 4: optionally continue M1M_1 on real plus synthetic tasks ⇒M2\Rightarrow M_2; place a checkpoint on each GPU 5: for each task τ\tau popped from a shared queue ordered by estimated cost do 6: reset the LoRA adapter; build 128 augmented sequences of τ\tau’s demonstrations; train one epoch 7: for each test input xx of τ\tau do 8: render 16 views of xx; depth-first search keeping branches with cumulative probability ≥0.2\ge0.2 9: for each distinct de-augmented grid yy: s(y)←s(y) \leftarrow mean NLL of yy under 8 fresh views 10: score⁡(y)←hits⁡(y)−s(y)\operatorname{score}(y) \leftarrow\operatorname{hits}(y) - s(y); attempts ←\leftarrow two highest scores 11: end for 12: end for 13: write the answer file; unanswered inputs get [[0]]

to ckpt-34000 on other GPUs, these checkpoints scored 27.64 to 32.64 in five runs (mean 30.53, § 4.4).

About 25 shorter controlled runs on an RTX 5090 or an RTX 3090 Ti tested data mixtures and losses: 1,000 to 97,113 tasks from the rule corpus and real data, plain cross-entropy, a likelihood-threshold “gate” loss matched to the 0.2 search cut-off, a hard-tail weighting, a KL anchor to the base model’s correct tokens, colour-consistency regularisation, and one to eight views per task. On 15 public evaluation tasks (18 inputs, 16 views, greedy decoding, no test-time training), they solved 1 to 3 inputs against 2 for the base model. In the same setting, per-task test-time training alone solved 3, and on 30 held-out synthetic tasks it solved 6 against 2 (bootstrap 95% interval for the difference +3.3 to +26.7 points).

Runtime changes and checkpoint ensembles

Our best NVARC 4B configuration (best draw 34.86) keeps the algorithm of Algorithm 3 and the model ckpt-34000, and changes how the four GPUs and the twelve hours are used. We recovered the executed source of this configuration together with its log and per-GPU task statistics:

  • every GPU loads ckpt-34000, with a context length of 13,312, 12,288, 10,240 and 8,192 tokens on GPUs 0 to 3 instead of 8,192 on all four, so the GPUs differ only in how many demonstrations fit into the prompt;

  • tasks are taken from one shared queue in order of estimated cost, cheapest first, and each task gets a time limit between 1,200 and 3,000 seconds computed from the remaining budget and queue, instead of a fixed 1,200 seconds;

  • the global deadline is 12 hours plus 10 minutes instead of 12 hours minus 10 minutes;

  • search scores are kept in FP32 instead of BF16, and the rescoring views use a deterministic seed per task;

  • a worker that fails on three consecutive tasks exits and the others take its tasks.

Each worker solves whole tasks, and there is no vote across GPUs. This configuration therefore contains no checkpoint trained on our synthetic data; its gain over the base model comes from continued fine-tuning on real tasks and from the runtime changes.

Five runs of this configuration scored 33.61, 34.86, 33.06, 33.19 and 30.56 (mean 33.06, standard deviation 1.57), 4.10 points above the base model (Welch t=5.3t = 5.3) and 1.33 points above the mean of ckpt-34000 in earlier runtimes (t=1.7t = 1.7), a difference within the run-to-run variation. In the same code, a configuration that ran the FORGE50 continuation in place of ckpt-34000 on the fourth GPU (with a 13,312-token context) scored 34.31, 33.06 and 30.56 (mean 32.64), 0.41 points lower. In 36 runs whose checkpoint lists we read from the source or log of each version, ensembles of ckpt-34000 with other checkpoints averaged 30.62; the five of them that included a checkpoint trained on the real tasks plus the whole rule corpus averaged 30.53. No model trained with synthetic tasks that we ran alone scored above the real-task model (29.31, 28.61, 28.47, 28.06, 27.22 and 26.53 against a mean of 31.72), and placing a synthetic-trained checkpoint next to ckpt-34000 did not raise the mean score.

What did not work. Sampling the frozen base model without test-time training or search, with Qwen3.8-27B choosing two of the candidates, scored 10.42. Within test-time training, a learning rate of 10−510^{-5} instead of 5×10−55 \times10^{-5} scored 25.69, 160 sequences for two epochs with a 0.19 search cut-off scored 20.00, and a reduced profile (64 training sequences, 8 inference views, 4 rescoring views,

ModelInitRealSyntheticViewsRankStepsHardware, time
ckpt-34000base1,920–1625634,0001 × L4, 10.0 h
FORGE50 continuationckpt-340001,920FORGE50, 25% of episodes1625645,000rented GPU
mixed6000base1,0904,910 rule corpus2641,500RTX 3090 Ti
shape37090mixed60001,09036,000 shape-balanced2649,273RTX 3090 Ti, 47.8 h
original7500base1,9203,600 shape + 1,980 rule16256120,000RTX 5090
all-trainbase1,12095,993 rule corpus125612,140RTX 5090
real + corpusbase1,92095,993 rule corpus22561 epochRTX 5090

Table 7. Offline fine-tuning runs of NVARC 4B. Real tasks come from ARC-AGI-1 and ARC-AGI-2; synthetic tasks come from FORGE50, the shape-balanced generator and the 12,000-rule corpus of § 3. All runs use rank-stabilised LoRA. Steps are optimizer steps of one task (rank 256) or of four tasks (rank 64).

720 seconds per task) scored 12.50 in a run that left part of the inputs unanswered. Merging the adapter of ckpt-34000 into BF16 weights and running it at longer contexts scored 28.61 and 28.19, against 31.1 to 31.5 for the adapter loaded at 8,192 tokens, although merge, context and code changed together. A short 4-task check run, which we used to check every configuration before a hidden run, did not predict the semi-private score: over 49 runs the Pearson correlation between its score and the SP score was 0.14. We first describe two fixes that the base model needed (§ 5.1), then the trace pipeline (§ 5.2), the training recipe (§ 5.3), the later data variants (§ 5.4) and the inference procedure (§ 5.6). § 5.7 compares the models on the public evaluation set without a time limit and, where they were run there, on the semi-private set.

Fine-tuning Qwen3.8-27B on its own answer-verified reasoning traces

This section describes the main line of work: rejection-sampling fine-tuning (RFT) of the dense Qwen3.8-27B reasoning model [6] on traces that it generated itself and whose answers match the known outputs, and the inference procedure that we built around the fine-tuned model. The line took the semi-private score of Qwen3.8-27B from 26.81 to 44.03, the level that the ARC Prize Foundation measured for the base model without a time limit (42.36).

Evaluation sets. Unless noted, offline numbers in this section are on the ARC-AGI-2 public evaluation set, 120 tasks with 172 test inputs. We report the number of exactly correct test inputs and, where noted, the task score (the mean over tasks of the fraction of solved inputs, times 100). Because the RFT training data include traces for some evaluation tasks, we also use three subsets: HH, 30 randomly chosen tasks (45 inputs) that no model of the RFT line except FINAL was trained on; TT, 61 tasks (90 inputs) whose correct base-model traces are in the RFT v1 data; and NN, 29 tasks (37 inputs) that no model had solved at the time the subset was defined. HH is the only random subset that is disjoint from the training data of the RFT line before FINAL; we used it repeatedly during development, so it is a development subset rather than an untouched test set. Unless stated otherwise, decoding uses T=0.6T = 0.6, top-p=0.95p = 0.95, top-k=20k = 20 and the xhigh reasoning effort of the chat template.

Two fixes to the base model

End-of-turn token inside the thinking block. Qwen3.8-27B samples the end-of-turn token (id 248046) in the middle of its reasoning, typically while it enumerates grid rows. The serving engine then returns an empty answer. At low effort on a ten-task positive-control set (parity10, 13 inputs that the evaluation of the ARC Prize Foundation reports as solved), 11 of 26 requests ended this way, and on two tasks every attempt did. At the stopping step the token had probability between 5% and 62%. The failure persisted with a BF16 output head, a BF16 key-value cache, a different attention backend and two other checkpoints; all of these runs used quantized weights, so we did not test unquantized ones, but nothing points to the quantization. We split each attempt into two requests on token ids: a thinking request in which the end-of-turn and end-of-text tokens carry a logit bias of −100 and which stops at </think>, followed by an answer request under a regular-expression grammar for grids. This raised task-averaged two-attempt accuracy on parity10 from 36.7% to 66.7% (5 to 8 of 13 inputs), and valid grids from 15 to 26 of 26. Because the serving engine rejects logit biases when speculative decoding is enabled, the production checkpoints instead have the two rows of the output head set to zero; every fine-tuned export below carries this patch.

Effort and forced completions. With the fix in place, the reasoning effort decides most of the accuracy. On the semi-private set, low effort scored 9.86 and xhigh with a 64K-token thinking cap scored 26.81 (twice, in two draws of the same configuration). Offline, xhigh with two 64K attempts at T=1.0T = 1.0 solved 84 of 172 inputs (task score 49.86), but 168 of the 344 attempts reached the cap. We call an attempt a natural completion when the model ends its thinking before the cap, and a forced completion when the engine closes the thinking at the token cap. Forced completions are rarely right: 14 of 168 (8%), against 42% to 86% for natural completions in each length bucket up to 64K (Table 8 and Figure 6). At low effort, accuracy collapses with length, from 72% for attempts below 16K tokens to 3% above 48K; at xhigh it declines more slowly, and length is confounded with task difficulty. Under the twelve-hour limit the problem is therefore how to spend tokens: long caps leave inputs unanswered (uniform 88K caps scored 18.89), and short caps turn natural completions into forced ones.

Run≤16K16–24K24–32K32–48K48–56K56–64Kforced at cap
Base, T=1.0T = 1.0, 344 attempts2/412/1421/2740/6624/4110/2414/168
Base, T=0.6T = 0.6, 172 attempts2/29/129/1225/365/114/163/83
RFT v1, T=0.6T = 0.6, 172 attempts2/212/147/1119/337/189/167/78

Table 8. Accuracy of single attempts by thinking length on the public evaluation set (172 inputs per run, xhigh, 64K cap, 2-request harness: thinking, then a separate answer request). Columns are thinking lengths of natural completions; the last column counts forced completions at the cap. Entries are correct/attempts.

Figure 6. Single attempts of Qwen3.8-27B on the public evaluation set at a 64K cap (the runs of Table 8). (a) Share correct among natural completions, by thinking length, and among forced completions at the cap (crosses). (b) Attempts in each bucket (light) and correct attempts (dark).

Self-generated answer-verified traces

Answer-only training does not transfer. Our first fine-tune taught the model to emit the answer grid with thinking disabled: 37,201 examples (33,915 shape-balanced synthetic tasks, 2,085 tasks from the 95,993-task corpus of § 3, 1,076 official training inputs and 125 public evaluation inputs), LoRA rank 32, learning rate 2×10−52 \times10^{-5}, 3,149 steps in about 21 hours. It scored 2.08 on the semi-private set, close to the thinking-off score that the ARC Prize Foundation reports for the base model (1.5% on the same set; [32]). A continuation on all 1,147 labelled official tasks in 16 symmetry and colour views solved 0 of 9 inputs of five tasks that were in its own training data. We concluded that the targets must keep the reasoning and that the prompt during training must be the inference prompt, token for token.

Rejection-sampled traces (RFT v1). We then sampled the base model on every labelled training input. The generator was Qwen3.8-27B-FP8 with the end-of-turn patch, three-token multi-token prediction (MTP) and an FP8 key-value cache, served by vLLM [33] on one RTX PRO 6000 (96 GB). The prompt was the prompt of the ARC Prize Foundation’s benchmarking harness, with JSON grids, rendered by the chat template at xhigh. Each input received two attempts (one in a 100-task pilot) with a 48,000-token thinking cap; at the cap the sentence “Considering the limited time by the user, I have to give the solution based on the thinking directly now.” and </think> were appended, and the answer was decoded under the grid grammar. Algorithm 4 gives the procedure. The base model solved 959 of the 1,076 ARC-AGI-2 training inputs (89.1%; 1,000 tasks). To these we added 170 correct traces for public evaluation tasks from earlier base-model runs at 64K and 88K caps (T=0.9T = 0.9 to 1.0), excluding the 30 H tasks.

A trace was kept only if its grid equals the label, it is a natural completion, it contains no chat tags and no give-up phrase (“best guess”, “I’m stuck”) in its last 6,000 characters, it is at most 220K characters long, no line of 30 or more characters repeats six times or more, at least 60% of its lines are unique, and its zlib compression ratio is at least 0.18 (a loop detector). Per input we kept the shortest surviving trace for training tasks and the two shortest for evaluation tasks. The result, sft_main1, has 1,036 rows (900 from 846 training tasks and 136 from 61 evaluation tasks) and 17.08M thinking tokens; the median trace has 12,159 thinking tokens, the 90th percentile 37,962, and the evaluation rows have a median of 38,481. A correct trace of at most 16K tokens exists for 619 of 935 training inputs but for only 4 of 113 evaluation inputs. After RFT v1 was trained, an LLM agent read all 1,144 unique traces of this set and of the next round end to end and removed 16 (1.4%): seven contained contradictions, five drifted off the task, three dithered and one asserted an unjustified rule. The cleaned set (1,022 rows) is the base of every later training set.

Training recipe

We train a LoRA adapter [42] of rank 32 and α=32\alpha= 32 (233.5M parameters) on twelve projection types of the hybrid architecture: query, key, value and output of the 16 full-attention layers, the five input and output projections of the 48 Gated-DeltaNet layers [27], and the gate, up and down projections of all 64 MLP blocks. The base stays frozen in BF16. We use AdamW with learning rate 10−410^{-4}, cosine decay after a 5% warm-up, gradient clipping at 1.0, one epoch, micro-batch 1 without packing (packing would leak the recurrent state of the linear-attention layers across examples) and gradient accumulation 8. The loss is the token mean of cross-entropy over supervised tokens; the prompt is masked and asserted to be token-identical to the inference prompt. Examples up to 81,920 tokens fit on one 96 GB GPU with re-entrant gradient checkpointing, activation offload to host memory, and MLPs chunked over 8,192 positions. RFT v1 trained 1,033 rows (three exceeded the length limit) in 130 optimizer steps and 7.06 hours; the loss stayed near 0.20 nats per token. After merging, we re-quantized the model to NVFP4 with the recipe of the public unsloth checkpoint: NVFP4 weights with group size 16 on the MLPs of layers 0 to 55; FP8 weights and activations on attention, the query-key-value, gate and output projections of the Gated-DeltaNet layers, the output head and the last eight MLPs; BF16 for the remaining small projections and norms; and FP8 key-value scales, calibrated on 256 training-task samples. With the same seeds at a 32K cap, the BF16 and NVFP4 models solved 50 and 51 of 172 inputs (grid or verified program); parity at 64K was not established.

Data variants after RFT v1

Table 9 lists the training sets. Each later set changed one aspect of the data. More of the same (main2) added second-pass traces at an 88K cap on the 127 inputs partly unsolved tasks. Short traces (short16k) kept only traces of at most 16K thinking tokens. Explanations put verified rule explanations in the thinking channel (v3b: 284 explanations of official tasks written by an LLM and checked against the labels, plus 1,000 template-generated explanations of synthetic tasks) or in the ordinary answer channel (ft64k: 1,602 explanations, 1,584 of them of synthetic tasks; § 5.5). Program targets (RFT v2) added 777 traces generated by RFT v1 with the production request; 755 of them carry a Python program target: the trace’s own verified program, a program the model wrote after its own frozen correct thinking (832 of 929 inputs verified), or a compact program from a library of 1,115 programs that an LLM agent wrote for official tasks and that we checked in a sandbox. Hard sampling (RFT v3) drew up to eight attempts on unsolved inputs, dropped lucky hits on small or rarely solved inputs, and capped new rows at two per input and per task. Boundary × views (E3) sampled the 300 training inputs that RFT v1 solves at most half the time and the 90 T inputs, each in two views (a D8 element and a colour permutation that keeps black fixed, applied to every grid of the task), at a 48K cap; this set was folded into FINAL and never trained on its own. Teacher traces (FINAL) added 452 traces from external models that return their

Algorithm 4 Rejection-sampled trace data (RFT v1; bracketed lines are the extensions of later sets).

Input: labelled inputs PP (ARC-AGI-2 training, public evaluation minus holdout HH); generator GG (RFT v1: base Qwen3.8-27B; later sets: RFT v1); thinking cap CC; attempts AA

1: for each input x∈Px \in P do 2: [E3: for each boundary input (solve rate ≤1/2\le1/2) and each view v=v = D8 element ×\times colour permutation fixing 0] 3: for a=1,…,Aa = 1, \ldots, A do ▷ RFT v1: A=2A = 2, C=48C = 48K; RFT v3: A≤8A \le8, stop after two correct 4: seed ←\leftarrow SHA256(task, test index, aa, tag) 5: t←G(prompt⁡(x);T=0.6,p=0.95,k=20t \leftarrow G(\operatorname{prompt}(x); T=0.6, p=0.95, k=20, stop at </think>, cap C)C) 6: if tt reached the cap then append the forced-close sentence and </think> 7: end if 8: y←G(prompt⁡(x)∥t;y \leftarrow G(\operatorname{prompt}(x)\mathbin\Vert t; grid grammar, ≤8,192\le8{,}192 tokens) 9: [RFT v2+: program ←G(…∥\leftarrow G(\ldots\mathbin\Vert program cue); run it on all train pairs and the test input] 10: end for 11: end for 12: keep (t,y)(t,y) iff y=y = label ∧\land natural completion ∧\land task ∉H∧\notin H \land no give-up phrase ∧\land no repeated lines ∧\land zlib ratio ≥0.18\ge0.18 13: [RFT v2+: last grid in tt equals yy, ∣t∣≤32\lvert t\rvert\le32K (E3: 48K); RFT v3+: drop lucky hits on small or rarely solved inputs unless the trace’s own program verifies or two attempts were correct] 14: per input keep the kk shortest (k=1k = 1 training, k=2k = 2 evaluation) 15: [after RFT v1: an LLM agent reads every trace of the first two rounds; 16 of 1,144 removed] 16: supervise t∥t\mathbin\Vert</think>∥y\mathbin\Vert y [ ∥\mathbin\Vertprogram]∥\mathbin\Vert<|im_end|> with the prompt masked

reasoning text: dots3 (165 rows), DeepSeek V4.1-Flash (152), two free models of a coding command-line tool (121) and others (14).

Two findings about teacher traces shaped FINAL. First, supplying the known test output as a hint (“never mention it”) leaks into the reasoning: dots3 solved 8 of 16 previously unsolved inputs this way, and all 8 correct traces mention the hint. A regular-expression filter for such references also fires on 9% to 21% of unhinted correct traces, so we stopped hinting; one hinted trace passed the filter and is in FINAL. Second, a leave-one-out view gives the teacher the answer legitimately: the real test pairs become extra demonstrations and one training pair, chosen by seed, becomes the query. In a pilot the teacher solved 8 of 12 such queries and all 8 passed the filters. The FINAL set (1,824 rows, 36.59M thinking tokens) combined the cleaned RFT v1 data, the E3 rows, the 452 teacher rows and 35 further self-traces, and for the first time included the H tasks (74 rows), so the public evaluation set could no longer validate it.

Before FINAL, we measured how well RFT v1 fits its own data (E1). Comparing the base model and RFT v1, the per-token negative log-likelihood of the thinking fell by 0.047 nats on training-family rows (0.230 to 0.184) but only by 0.020 on evaluation-family rows (0.294 to 0.274); the difference of the two reductions is 0.027 (bootstrap 95% interval 0.024 to 0.030). RFT v1 therefore fits the long evaluation-level traces less than the short training-level ones. A run with a higher learning rate that was prepared to address this (E2) was cancelled in favour of a single final run.

Synthetic tasks in the fine-tuning data

Synthetic tasks entered three of the training sets of Qwen3.8-27B (Table 10): 2,231 distinct tasks in total, 1,435 from the 95,993-task corpus of § 3 and 796 from the 72-program bank of the shape-balanced generator. The largest addition, synthetic1000, was built to cover every grid size. It has 1,000 tasks with one test input each, and every ordered size of the test input from 1 × 1 to 30 × 30 (900 sizes) occurs in at least one task. 205 of its tasks keep the five demonstrations and the test input of a corpus task (205 rules from all eight corpus families), and 795 are new tasks of 68 of the 72 bank programs, which come in two families: spatial logic (a cell selector, a relation, a symmetry and an occupancy gate; 684 tasks) and shape sequence (an order on the cells followed by a relation, a mask or a layout; 111 tasks). The generating programs replayed all 4,410 pairs, no task equals a task of the existing fine-tuning data after canonicalisation, and the set takes 8.85M tokens in the training format (at most 22,022 per task). Every task carries an explanation of its rule that is computed from the verified application of the program, not written by a model; Table 3 gives one rule of each family in the words of these explanations.

SetChannel of the synthetic rowsSyntheticCorpusBankOther rowsResult
v3bthinking (explanations)1,000205795284 official explanations, 769 self-traces9*
ft64kanswer text (explanations)1,5841,41217218 official explanations, 1,039 self-traces43
FINALthinking (teacher traces)2810181,796 traces of official tasks44
All2,2311,435796

Table 10. Synthetic tasks in the fine-tuning data of Qwen3.8-27B. “Corpus” tasks come from the 95,993-task export of the rule corpus, “bank” tasks from the 72-program shape bank. The result is inputs solved of 172 by one 32K attempt, against 51 for RFT v1 in the same harness; *2-request harness, against 37 for RFT v1.

We used the synthetic tasks as supervised explanations and not as self-traces, because the model rarely solved them: in the trace generation of RFT v2, RFT v1 solved 5 of 45 inputs of synthetic1000 and none of 28 corpus inputs, all 28 of which reached the 32K cap. v3b put the 1,000 explanations of synthetic1000 in the thinking channel; ft64k put 1,584 explanations of synthetic tasks (1,248 from v2 rules, 164 from v1 rules and 172 from the bank) in the ordinary answer channel, next to 1,039 self-traces in the thinking channel; and FINAL contains 28 teacher traces of synthetic tasks. No set with synthetic tasks was better than RFT v1 on the public evaluation set (Table 12), and the deployed RFT v1 contains none. A held-out set of 30 synthetic tasks, three from each of the ten families, shows how far the models are from the corpus: RFT v3 solved 1 at a 32K cap and 5 at 64K, and FINAL 1 and 3.

ModelHarnessMetric32K48K64K
Base FP8, T=0.6T = 0.62-requestgrid––10 / 57
Base FP8, T=1.0T = 1.0 (two attempts)2-requestgrid––13–17 / 59–64
RFT v12-requestgrid8 / 37–13 / 63
v3b2-requestgrid2 / 9––
RFT v11-requestgrid8 / 409 / 5014 / 54
RFT v11-requesttwo-slot10 / 5111 / 5615 / 62
RFT v21-requesttwo-slot12 / 499 / 54–
RFT v31-requesttwo-slot7 / 4212 / 5611 / 63
ft64k1-requesttwo-slot– / 43––
FINAL1-requesttwo-slot11 / 44–12 / 60

Table 12. Single-attempt accuracy of Qwen3.8-27B models on the public evaluation set at fixed thinking caps (inputs solved, of 45 for the subset H and of 172 overall). The 2-request harness thinks and then answers in a separate request (FP8 weights for the base model, BF16 for fine-tuned models); the 1-request harness is the production request of Algorithm 5 (NVFP4 weights). “Grid” counts direct answers; “two-slot” also counts inputs whose train-perfect program output is right. FINAL was trained on H tasks.

Inference procedure

The configuration that reached our best Qwen3.8-27B score (44.03) is given in Algorithm 5. It runs RFT v1 in NVFP4 on four L4 GPUs with tensor parallelism 4, an FP8 key-value cache, three-token MTP, a server limit of 36 sequences and at most 32 client requests in flight. Compared with the first RFT v1 run on the semi-private set (28.19), it also switched from FP8 to NVFP4 weights, added two decode speed-ups (batch-sharded sampling, +5.5%, and local-argmax draft reduction, +3.9%) and added repair passes, and it made three changes to how attempts are made:

  1. One request per attempt with the thinking cap enforced inside the engine. We patched the engine’s thinking budget so that at the cap it emits the forced-close sentence and </think>, and the grid grammar applies only after </think>. Natural completions already ended in one request; for attempts that reach the cap, this removes the second request and its reprefill. In a 40-task A/B on the four L4 GPUs with the same seeds and a 32K cap, completed attempts per hour rose from 31.6 to 43.7 (+38%) and the prefill tokens on the server fell from 3.50M to 0.79M; both arms solved 20 of 59 inputs (33.9% input-weighted), although the arm without the change completed only 91 of its 118 attempts within the three-hour test.

  1. A program in the same request. After the grid, a grammar-forced cue asks for the transformation as a Python function transform(grid); the program is executed on all training pairs, and a program that reproduces them all (“train-perfect”) ranks before the direct grid.

  1. A budget schedule. Attempt 1 with a 56K cap is started for every input before any second attempt is started; the queue does not wait for first attempts to finish, so second attempts start while up to 31 first attempts are still running. Attempt 2 uses a 32K cap on a view with reordered demonstrations; each request’s cap shrinks so that it can finish before the deadline at the measured rate of 15 tokens per second per stream.

Programs after the answer. Qwen3.8-27B does not write programs on its own: none of its 344 xhigh attempts on the public evaluation set contains Python. The cue after the grid asks for “The transformation above, written as a self-contained Python function” and opens a code block with def transform(grid):; the model writes the body under a grammar that ends at the closing fence, and the program runs in an isolated interpreter (10 s CPU, 1 GiB memory) on every training input and on the test input. Such a program checks the answer against the demonstrations and avoids copying errors in long grids. On the saved base-model attempts (64K cap, two attempts), 98 of 344 programs were train-perfect and 76 of those were right on the test input (77.6%); using the program as the second answer raised the task score from 38.47 (attempt 1 alone) to 48.47, and ranking programs first over both attempts gave 54.03. A program in a second request lowered the semi-private score from 26.81 to 21.39, because each request re-read about 55K tokens of context, about 1.8 of the 12 hours. In the same request, with RFT v1 at a 32K cap, 37 of 172 programs were trainperfect and 27 of those were right (73%); ranking them first rescued 9 inputs and lost none (41 to 50 inputs, task score 25.69 to 31.53) for 4.5% of the generated tokens, and consecutive versions of the configuration without and with programs scored 33.61 and 37.22 on the semi-private set. A repair turn that shows the model the cell-level differences of its best program on each training pair solved 8 more inputs and lost 1 at a 16K budget, but a replay of the deployed queue indicates that repairs did not run on the hidden set.

Validation without the time limit and on the semi-private set

Table 11 puts the offline results of the models next to their semi-private scores where these exist. Offline runs are on the 172 public evaluation inputs on one RTX PRO 6000 (Colab), with no limit on time; runs on the semi-private set are on four L4 GPUs for twelve hours. Without the time limit, the base model reached task scores of 49.86 with two 64K attempts, 53.33 with an 88K and a 64K attempt, and 54.03 with a program after each attempt, and RFT v1 solved 99 of the 172 inputs (57.6% of inputs, task score 59.6) in at least one of its attempts (8 to 22 per input, 13.4 on average). On the semi-private set, the same token budgets did not fit: two 88K attempts scored 18.89 and a program in a second request 21.39, because many hidden inputs received no answer. Once the schedule fitted the twelve hours, validation and semi-private scores agreed. For RFT v1 with a 64K and a 32K attempt, the offline task score was 44.5 on all tasks and 42.5 on the subset H (mean over four pairings, see Table 11), and the run on the semi-private set with a 56K and a 32K attempt scored 44.03. These results suggest that throughput and token allocation constrain the deployed score. They do not isolate how much of the difference between the 54 points reached offline without a time limit and the 44 points on the semi-private set the token budget causes: the two sets differ in tasks and in training exposure, the offline runs used another GPU, engine and quantisation, and the offline 64K and 32K combination differs from the deployed 56K and 32K schedule. The inference procedure of Section 5.6 is our means of spending the budget.

ModelOffline settingInputsTaskHSemi-private settingSP
Basetwo 64K attempts, T=1.0T = 1.08449.86–two 64K attempts26.81
Base88K and 64K attempts8953.33–two 88K attempts18.89
Basetwo 64K attempts, program after each–54.03–program in a second request21.39
RFT v1one 32K attempt, grid or program5130.6910two 32K attempts, programs37.22
RFT v1one 48K attempt, grid or program5633.061148K and 32K attempts37.08
RFT v1one 64K attempt, grid or program6239.0315––
RFT v164K and 32K attemptsa^{a}71.544.516.056K and 32K attempts44.03
RFT v1any of 8 to 22 attempts9959.621––
RFT v2one 32K attempt, grid or program4930.691248K on half of the tasksc^{c}22.50
RFT v3one 64K attempt, grid or program6338.8911––
FINALd^{d}one 64K attempt, grid or program6035.1412as for 44.0329.03

Table 11. Qwen3.8-27B models offline without a time limit (public evaluation set, 172 inputs, Colab) and on the semi-private set (4×L4, 12 hours). Offline rows give inputs solved and the task score; “grid or program” counts an input as solved if the direct grid or a train-perfect program output is right. H is the development subset of 45 inputs. a^{a}Mean over four pairings of one 64K attempt with four separate 32K draws (74, 73, 71 and 68 inputs), each pair ranked by the production rule; the 32K draws differ in weights (NVFP4 or BF16), GPU and seed, and none uses the reordered demonstrations of the deployed second attempt. c^{c}Half of the tasks answered by design. d^{d}Trained on H and on all evaluation tasks, so its offline numbers are in-sample.

Table 12 compares the models at fixed thinking caps with the same seeds where possible. At a 64K cap and a single attempt in the 2-request harness, RFT v1 solved 63 of 172 inputs and the base model 57 at T=0.6T = 0.6; RFT v1 won 29 inputs and lost 23 (exact sign test p=0.49p = 0.49). The base model’s single attempts at T=1.0T = 1.0 solved 59 and 64 inputs, so the difference is inside the base model’s own variation. On the 90 T inputs, whose correct traces were in the training data, both solved 46. Mean thinking length at 64K did not change (51.2K and 51.8K tokens). RFT v1 therefore did not measurably change single-attempt accuracy at a matched budget. In the production harness at 32K, no later training set was better than RFT v1: RFT v2 49 and RFT v3 42 inputs against 51, ft64k 43 and FINAL 44 (main2, short16k and v3b were only run in the 2-request harness, where main2 equalled the base model at 64K and short16k solved 34 against 37 at 32K). On the subset H, RFT v2 solved 12 against 10 for RFT v1 at 32K and 9 against 11 at 48K, so no variant was better on both budgets. With the 2-request harness, v3b solved 9 inputs against 37 for RFT v1; 149 of its 172 attempts begin with “Solution summary.” and none of them is correct. This suggests that explanations written with knowledge of the answer teach the model to state a rule without exploring, although v3b also differed in trace count, maximum length and batch size.

The semi-private set agrees with the one matched model comparison we made there. FINAL and RFT v1, run with identical code and container image, scored 29.03 and 44.03. With a per-draw standard deviation of 1.94 points, measured on repeated NVARC 4B runs on the semi-private set (2.7 for the difference of two draws), a 15-point drop is far larger than the run-to-run variation. We did not test why. Two candidate causes are that a quarter of FINAL’s rows come from other models, whose reasoning style differs from the native one, and that FINAL completes its thinking later: in the production harness at 64K, 20 of its natural completions fall between 56K and 64K tokens against 9 for RFT v1, so a 56K cap forces more completions.

Pruning the experts of Qwen3.8-Flash-Next to fit four L4 GPUs

The third line of work used Qwen3.8-Flash-Next [7], a sparse mixture-of-experts reasoning model from the same Algorithm 5 Inference and answer selection for Qwen3.8-27B on four L4 GPUs (configuration of the 44.03 run).

Input: tasks sorted by prompt length; time limit L=12L = 12 h; hard stop L−7L - 7 min; rate r=15r = 15 tokens/s per stream

  1. queue levels: L0 attempt 1 for every input (plain view, cap 56K); L1 attempt 2 (demonstrations shuffled, cap 32K); L2 repair turn with per-pair program feedback (24K, then 16K unless a program is now train-perfect), queued after both attempts only if no program is train-perfect and a plain-view program reproduces at least one demonstration, started only at least 30 min before the hard stop; L3 one extra attempt, only if no program is train-perfect and the two grids differ, with at least 35 min left

  2. while queue not empty and time <L−15< L - 15 min do

  3. pop the lowest level; cap ←min⁡(cap,r(hard stop−now−240 s)−3000)\leftarrow\min(\text{cap}, r(\text{hard stop} - \text{now} - 240\text{ s}) - 3000); skip if cap <4000< 4000

  4. one request: thinking →\to forced completion at cap →\to grid (grammar) →\to def transform(grid): body

  5. run the program on all training pairs and the test input in a sandbox (10 s CPU, 1 GiB)

  6. rewrite the answer file

  7. end while

  8. per input: candidates ←\leftarrow train-perfect program outputs ∪\cup direct grids

  9. order: candidates with ≥2\ge2 votes [as in [43]], then program outputs by votes, then grids by votes

  10. attempt_1, attempt_2 ←\leftarrow first two distinct candidates (fill with [[0]])

family as Qwen3.8-27B. We did not fine-tune it. Offline it was as accurate per attempt as Qwen3.8-27B and gained more from long thinking, but on four L4 GPUs it first generated about a third of the tokens per second of Qwen3.8-27B, because its weights left little memory for the key-value cache. This section describes how we ran it on the evaluation hardware (four L4 GPUs), how we chose the experts to remove, and what the removal changed in accuracy, memory and tokens. The model has 125B parameters, of which 6B are active per token, plus a 51B-parameter n-gram embedding table that is kept off the accelerator [7]; by our count from the configuration, the routed experts of the 48 main layers hold 120.8B parameters.

Model and serving

Qwen3.8-Flash-Next has 48 layers with hidden size 2,560: 36 Gated DeltaNet layers [27] and, in every fourth layer, full attention with a sparse-attention index (compression ratio 4, budget 2,048 positions). Every layer has 512 routed experts with intermediate size 640, of which the router selects 10 per token, and one gated shared expert as in Dai et al. [39]. A multi-token-prediction layer, itself a mixture of 512 experts, serves as the draft model. The model has 125B parameters, of which 6B are active per token, plus a 51B-parameter n-gram embedding table that is kept off the accelerator [7]; by our count from the configuration, the routed experts of the 48 main layers hold 120.8B parameters.

We served a community W4A16 checkpoint: routed experts in symmetric INT4 with groups of 128, the Gated DeltaNet projections in per-channel INT8, the n-gram table in FP8 and the remaining weights in BF16. It occupies 115.2 GiB, of which the n-gram table (47.7 GiB) stays in host memory. The serving engine is Merlin, a community fork of vLLM [33] for Ampere and Ada GPUs, installed from prebuilt wheels: version 2.5.2 (based on vLLM 0.30) at first and version 3.0 (based on vLLM 0.32) later. It splits the model into two pipeline stages of 25 and 23 layers, each with tensor parallelism 2 and expert parallelism, and stores the key-value cache in FP8 with calibrated static scales; against a BF16 cache, the FP8 cache changed the teacher-forced log-likelihood by 0.0002 to 0.0004 nats per token in each of five length bins. Speculative decoding uses the multi-token-prediction layer with four draft tokens and an INT8 output head for the drafts.

Difficulties on four L4 GPUs

Memory. An L4 exposes 22.03 GiB to PyTorch. With all 512 experts, the weights on each GPU took 16.9 to 17.3 GiB, which left about 2.2 GiB per GPU for activations, CUDA graphs and the key-value cache. Table 13 lists the layouts we ran with the unpruned model. Stock vLLM with tensor parallelism 4 and expert parallelism fitted 94,001 cache tokens, enough for four concurrent requests, and generated 164 tokens per second; the server was ready 68 minutes after the run started. Tensor parallelism 4 or a four-stage pipeline in Merlin fitted 96,743 to 255,273 tokens at 136 to 168 tokens per second. Two pipeline stages with tensor parallelism 2 and an FP8 cache fitted 420,026 tokens and reached about 250 tokens per second, and Merlin 3 with an INT8 draft head 547,542 tokens and 342 tokens per second. Even then the engine ran only 13 to 19 requests at a time, since one attempt that thinks for 96K tokens occupies about a fifth of a 547,542-token cache.

Engine, layout and cacheCache tokensRunningTokens/s
stock vLLM 0.30, TP4, BF16 cache94,0014164 (whole run)
Merlin 2.5.2, TP4, BF16 cache96,743–137 (steady)
Merlin 2.5.2, TP4, FP8 cache170,393–136–168 (steady)
Merlin 2.5.2, PP4, FP8 cache255,273–136 (steady)
Merlin 2.5.2, TP2 × PP2, FP8 cache420,02612.6249–256 (steady)
+ four draft tokens, no prefix caching, 96K context466,15116.7301 (first 150 windows)
Merlin 3.0, TP2 × PP2, FP8 cache, INT8 draft head547,54218.9342 (first 150 windows)

Table 13. Serving the unpruned Qwen3.8-Flash-Next (512 experts per layer) on four L4 GPUs. TP and PP are the tensor- and pipeline-parallel degrees; all layouts use expert parallelism. Throughput is total generated tokens per second, measured as noted: over the whole run, as the steady per-minute rate after the first five minutes, or over the first 150 ten-second windows of a probe; rows with different definitions are not directly comparable.

Code paths and checkpoints. Stock vLLM rejected the quantised n-gram embedding of the first W4A16 checkpoint we tried, and Merlin’s host-memory offload of the n-gram table crashed with two other published checkpoints because the offload process initialised CUDA; one checkpoint booted on every layout, and all runs below on the evaluation hardware use it. Merlin’s custom all-reduce failed on the PCIe-connected L4s and was disabled.

Speculative decoding. The indexer’s cache ring must divide the 4,096-token attention block, so only 1 to 4 or 9 to 12 draft tokens are valid; six draft tokens did not boot, and SGLang [44] refuses more than four. We used four; three were 1.2% slower on the 384-expert model. SGLang itself, without pipeline parallelism, fitted about 434,000 cache tokens and generated 221 to 255 tokens per second in the probes where Merlin generated 639 to 657.

Graph capture and start-up. Capturing CUDA graphs ran out of memory on the last pipeline stage for several cache sizes, so every configuration needed its own largest cache size that boots. Merlin 3 was ready 635 to 1,033 seconds after the run started, including a 7.6 GB copy of its Python environment and, when the system pruned experts at start-up, 109 to 287 seconds to drop them.

Long requests and the time limit. At 9 to 11 tokens per second per stream, a 96K attempt runs for close to three hours. Our runner had a 7,200-second read timeout with three retries, so every attempt still running two hours after it started was aborted and sent again from the beginning; this affected the first two Merlin runs (25.00 and 32.64). With the read timeout removed, a three-hour probe completed 63 attempts without error, 8 of them 7,382 to 8,541 seconds after they started. A run with an internal limit of 12 hours and 15 minutes was stopped for exceeding the time limit and returned no score, and later runs stop at 11 hours and 50 minutes.

Accuracy before pruning

Offline comparisons ran on one RTX PRO 6000 with stock vLLM, a BF16 cache, effort xhigh, T=0.6T = 0.6, top-pp 0.95 and top-kk 20, on a checkpoint with the same INT4 experts but BF16 attention and Gated DeltaNet weights. We used two panels of public evaluation inputs: the dev panel, 82 inputs of 59 tasks (the 45 inputs of the subset H and 37 inputs never solved by the Qwen3.8-27B line), and the fresh panel, 90 inputs of the other 61 tasks. Table 14 compares the unpruned model with Qwen3.8-27B. On the full public evaluation set with two 64K attempts, the two models tied (task scores 49.31 and 49.86, 52 fully solved tasks each). On the dev panel at 48K, Qwen3.8-Flash-Next tied RFT v1: 29 of 164 attempts each, with mean thinking lengths of 44.5K and 44.3K tokens. At 96K, three seeds of Qwen3.8-Flash-Next solved 30, 23 and 25 of 82 inputs (mean 26.0) against 13 for one seed of RFT v1; a bootstrap over the 59 tasks gives a difference of +13.0 inputs (95% interval +4.7 to +21.3). Only the best seed, 30, used the same request seeds as the RFT v1 run; in this selected pair, 23 inputs were solved only by Qwen3.8-Flash-Next and 6 only by RFT v1 (sign test p=0.0023p = 0.0023, not adjusted for the choice of seed). From 48K to 96K the solves of Qwen3.8-Flash-Next in that seed rose from 16 to 30, while those of RFT v1 stayed at 13.

Set and settingMetricQwen3.8-Flash-NextQwen3.8-27B
Public evaluation, 172 inputs, two 64K attemptstask score49.3149.86
inputs solved8084
completion tokens per attempt55,05753,756
Dev panel, 48K, two seedsattempts correct (of 164)2929
Dev panel, 96K, one seedinputs solved (of 82)3013
thinking tokens per attempt68,54171,592

Table 14. The unpruned Qwen3.8-Flash-Next against Qwen3.8-27B offline. The full-evaluation run of Qwen3.8-Flash-Next used INT8 attention and Gated DeltaNet weights, 12 concurrent requests and a BF16 cache; the base Qwen3.8-27B used FP8 weights, an FP8 cache and 32 concurrent requests. Dev-panel runs compare against RFT v1 in NVFP4.

On the semi-private set, the first two runs used stock vLLM at effort low with 40K caps and scored 8.61 and 6.11. With Merlin 3, all 512 experts and caps of 96K and 48K, the score was 25.00, with the timeout fault above. Our simulator of the twelve-hour run replays offline attempts at the measured throughput; its forecast for the later 53.33 run, made before the score was known, was 53.0 to 54.5. For the unpruned model at these caps it leaves 160 of about 343 hidden test inputs without any answer. Accuracy per attempt was therefore not the constraint; tokens were.

Choosing the experts to remove

Expert pruning removes whole experts with their router rows, so the model keeps its architecture and needs no retraining. Criteria proposed for it include routing frequency, the reconstruction loss of the layer output [40] and the merging of experts with similar routing [41]. REAP [8] scores expert jj of a layer by its router-weighted activation norm over the tokens Xj\mathcal{X}_j routed to it,

Sj=1∣Xj∣∑x∈Xjgj(x)∥fj(x)∥2,(1)S_j = \frac{1}{\lvert\mathcal{X}_j \rvert} \sum_{x \in\mathcal{X}_j} g_j(x) \lVert f_j(x) \rVert_2, \tag*{(1)}

where gj(x)g_j(x) is the gate value and fj(x)f_j(x) the expert output, and removes the experts with the lowest scores. We removed the same number of experts from every layer, because the model configuration has one expert count for all layers. No saliency list covers the multi-token-prediction layer, so there we kept the experts with the largest router-row norm.

Lists. We did not compute REAP scores ourselves. For 448, 384 and 288 experts per layer we used the lists of three public REAP builds of the same model, recovered by matching their router rows exactly against the original checkpoint. Their author reports calibrating the 384 and 288 builds on about 686K tokens of agentic-coding traffic, and the 288 list is a subset of the 384 list in all 48 layers. To adapt the lists to ARC, we recorded routing on 60 single-input ARC-AGI-2 training tasks: the unpruned model thought for up to 96K tokens (53 tasks solved), and a prefill replay of the 60 traces recorded 864M routing decisions over 1.80M tokens. For each expert ee we computed an ARC routing frequency FeF_e, its share of the routing decisions, averaged over tasks within eight length bins (prompt, six 16K bins of thinking, answer) and then weighted towards long thinking, with bin weights from 0.05 for the prompt to 0.19 for 48K to 64K tokens. A second capture recorded routing on 188,607 program tokens of 236 programs written for 59 training tasks. No evaluation task entered either capture.

Variants. Besides the three public lists, we built lists from these statistics. ARC-288 keeps the 288 experts with the highest FeF_e in each layer. CORE-320 and CORE-300 keep REAP-288 and add the 32 or 12 experts with the highest FeF_e among the 96 that REAP-384 keeps beyond it. s288p exchanges, in each layer, up to 16 experts of REAP-288 that are nearly unused on ARC for the next REAP-384 experts by FeF_e, and never removes an expert that serves one task or length bin heavily; 753 experts were exchanged, 15.7 per layer on average. c8 adds to s288p, in each layer, 8 experts of REAP-384 that program tokens prefer, which gives 296 experts per layer; 6.2 of the 8 were chosen for code on average and the rest by FeF_e. Algorithm 6 gives the two constructions.

Evaluation. Lists were compared offline on the dev panel at a 96K cap with three seeds (246 input and seed

Algorithm 6 Construction of the s288p and c8 expert lists for one layer.

Input: REAP lists K288⊂K384K_{288} \subset K_{384}; counts Ct,e,bC_{t,e,b} of routing decisions of the unpruned model on 60 training traces (trace tt, expert ee, length bin bb) with Nt,bN_{t,b} tokens; bin weights wbw_b; routing on the program tokens of 236 programs for 59 training tasks

  1. Fe←∑bwb⋅mean⁡t:Nt,b>0Ct,e,b/(10Nt,b)F_e \gets\sum_b w_b \cdot\operatorname{mean}_{t:N_{t,b}>0} C_{t,e,b}/(10N_{t,b}) ▷ 10 experts per token

  2. ce←∑t,bCt,e,bc_e \gets\sum_{t,b} C_{t,e,b}; u←mean⁡u \gets\operatorname{mean} of cec_e over the 512 experts

  3. specialist⁡(e)←\operatorname{specialist}(e) \gets some (t,b)(t,b) has Nt,b≥256N_{t,b} \ge256 and Ct,e,b>0.05Nt,bC_{t,e,b} > 0.05N_{t,b}

  4. weak⁡←\operatorname{weak} \gets up to 16 experts of K288K_{288} with ce≤0.06uc_e \le0.06u and not specialist, smallest cec_e first

  5. Ks288p←(K288∖weak⁡)∪K_{\mathrm{s288p}} \gets(K_{288} \setminus\operatorname{weak}) \cup the ∣weak⁡∣|\operatorname{weak}| experts of K384∖K288K_{384} \setminus K_{288} with the highest FeF_e

  6. pool⁡←K384∖Ks288p\operatorname{pool} \gets K_{384} \setminus K_{\mathrm{s288p}} (96 experts)

  7. code⁡←\operatorname{code} \gets experts of pool that are among the top program-token experts in >90%> 90\% of 40 task-level half samples, pass a within-task permutation test of the maximum tt statistic at level 0.05, and receive at least 1.5 times the uniform share of program-token decisions; keep at most 8, by program-token share

  8. Kc8←Ks288p∪code⁡∪K_{\mathrm{c8}} \gets K_{\mathrm{s288p}} \cup\operatorname{code} \cup the 8−∣code⁡∣8 - |\operatorname{code}| experts of pool⁡∖code⁡\operatorname{pool} \setminus\operatorname{code} with the highest FeF_e

  9. drop every expert outside the list with its router row; renumber the kept experts in their original order

pairs) and the same request seeds in every arm, and the last candidates also on the fresh panel (270 pairs). Before testing s288p and CORE-320, we fixed the rule for adopting a list with fewer experts than REAP-384: the lower end of a one-sided 97.5% task-bootstrap interval for the difference in task-weighted accuracy above −1.0 point, and at least 5% higher measured throughput. Speed and cache size were measured on the evaluation hardware with Merlin 3, the INT8 draft head and four draft tokens, on 60 single-input public evaluation tasks for 6,000 seconds; we report throughput over the first 150 ten-second windows, while the cache fills, and over the full probe. The lists were built from training tasks only, but we chose among them by accuracy on the two public evaluation panels, so the panel scores of the adopted lists are optimistic.

Results after pruning

Table 15 lists the variants and Figure 7 plots them. Removing a quarter of the experts with REAP-384 lost no accuracy that we could measure: 87 of 246 attempts against 78 for the unpruned model, at least as many on each seed (34, 26 and 27 against 30, 23 and 25; 31 attempts solved only after pruning and 22 only before, p=0.27p = 0.27), with similar rates of natural completion (70.7% for REAP-384 and 70.3% for the unpruned model) and similar thinking lengths (68.5K and 69.3K tokens), pooled over the three seeds. The 246 attempts are 82 inputs of 59 tasks crossed with three seeds and are not independent, so the sign test over attempts is descriptive. The weights on each GPU fell from about 17.0 to 13.3 and 13.7 GiB, the cache grew from 547,542 to 1,391,054 tokens, and throughput over the first 150 windows rose from 342 to 521 tokens per second (+52%).

Figure 7. Expert lists of Qwen3.8-Flash-Next. (a) Correct attempts of 246 on the dev panel at a 96K cap by experts per layer and by the criterion that built the list (REAP-448, measured on one seed only, is left out); the dashed line is the unpruned model. (b) Key-value cache tokens and throughput over the first 150 windows of the L4 probe (Table 15); open circles are lists served with the FP8 indexer cache. (c) Jaccard overlap of the kept experts, averaged over the 48 layers. (d) What c8 does with each of the 512 experts of each layer: experts outside REAP-384, experts of REAP-384 that c8 drops, the REAP-288 core that c8 keeps, the 753 REAP-288 experts that s288p exchanged out and the experts it took in their place, and the 8 experts per layer that c8 adds.

ListExperts per layerCheckpoint (GiB)Weights/GPU (GiB)Pin (GiB)Cache tokensTokens/s (150 / full)Dev
unpruned512115.2216.9 / 17.02.3547,542342 / –78
REAP-448448–15.1 / 15.5–––26{sup}a`
REAP-384384100.3813.3 / 13.75.81,391,054521 / 43387
CORE-32032092.96–6.71,605,632561 / 46384
CORE-300300–––––77
c829690.17– / 11.27.71,957,095{sup}c`––
s288p28889.2510.5 / 11.07.61,823,909566 / 49280
REAP-288288––7.61,823,909589 / 50066
ARC-288288–––––58
c8−4025685.549.4 / 10.18.82,234,566{sup}c`––
c8−72224–8.5 / 9.29.62,438,044{sup}c`––

Table 15. Expert lists for Qwen3.8-Flash-Next. Checkpoint size and weights per GPU (pipeline stage 0 / stage 1) are from the serving logs; the cache pin is the memory reserved for the key-value cache, set to the largest value that booted. Throughput is total tokens per second on the 60-task L4 probe over the first 150 windows and over the full probe. “Dev” counts correct attempts of 246 on the dev panel at 96K (three seeds). {sup}aOne seed, of 82. {sup}bTwo seeds, of 164. {sup}c`With an FP8 indexer cache, which by the cache formula of the serving logs holds about 6% more tokens at equal pin; the measured gain of 11% for s288p also includes a larger pin. c8−40 and c8−72 remove from c8 the experts with the lowest FeF_e that are neither code experts nor specialists.

Deeper cuts by a single criterion lost accuracy. REAP-288 solved 66 attempts and the frequency list ARC-288 58, the lowest of all lists (23 attempts gained and 43 lost against the unpruned model, p=0.019p = 0.019). The damaged lists complete their thinking earlier and think less: 85% of the attempts of ARC-288 were natural completions, with 62.6K thinking tokens on average, against 70.3% and 69.3K for the unpruned model (all pooled over three seeds). The coding-calibrated REAP lists and the ARC-frequency lists select substantially different experts; the comparison changes the criterion and the calibration domain together, so it is not a test of independence. Each REAP list drops about the share of ARC routing decisions that a random cut of the same size would drop (10.8%, 24.9% and 42.3% for 448, 384 and 288 experts, against 12.5%, 25% and 43.8%), the ARC-frequency top 384 shares 287 experts per layer with REAP-384 where chance gives 288, and among the 296 experts of c8 the rank correlation between REAP saliency on ARC traces and FeF_e is 0.17 to 0.62 by layer. The lists themselves split the same way (Figure 7c): ARC-288 has a Jaccard overlap of 0.40 to 0.56 with every other list, while the lists built on the REAP-288 core overlap by 0.90 to 0.97. Keeping the REAP-288 core and changing only

its least-used experts was what worked: s288p solved 80 attempts on the dev panel, and on the fresh panel it scored 58.38% task-weighted against 52.00% for REAP-384 (+6.38 points, lower 97.5% bound −0.73), at 13.6% higher full-probe throughput (492 against 433 tokens per second). It passed the adoption rule; CORE-320 did not (lower bound −4.46). Cutting c8 further to 256 or 224 experts added 14% and 25% to the cache, but the production mix lost 2.16 and 3.02 points at 272 and 256 experts, and a 256-expert list with every code expert and specialist protected lost 17.8 points in the agent configuration of § 6.7, where its median first-turn thinking fell from 27,540 to 13,441 tokens. We kept c8.

Two screens that avoid generation did not replace these tests. The teacher-forced excess negative log-likelihood against the unpruned model on 49 reference traces ranked the public lists as expected (3.9, 17.4 and 68.5 milli-nats per thinking token for 448, 384 and 288 experts) but gave ARC-288 20.3, close to REAP-384, so its pre-registered check failed. The mean probability of the closing tag, a measure of pressure to stop thinking taken in 16K-token windows of the reference traces, separated them: 3.07×10−43.07 \times10^{-4} for ARC-288 against 1.68×10−41.68 \times10^{-4} for REAP-384 and 1.26×10−41.26 \times10^{-4} for the unpruned model.

Pruning also changed program writing, which the generation tests on grids did not show. In unmatched runs on the evaluation hardware, 91 of 248 programs written by s288p did not compile, mostly through indentation errors, against 1 of 144 for REAP-384; s288p had removed 31.2% of the routing decisions of program tokens, against 18.0% for REAP-384. In a matched replay on Colab, in which both lists continued the same 165 frozen thinking prefixes of s288p, the eight code experts of c8 reduced the share of programs that do not compile from 37.6% (62 of 165) to 5.5% (9 of 165) and raised the share of train-perfect programs from 16.4% to 29.1%. The replay measures programs, not accuracy on the hidden set, and its rates are not comparable with those of the L4 runs (REAP-384 failed to compile 7.3% of its replay programs). Against s288p, the production use of programs gained 3.88 points (standard error 2.09), and over 516 dev and fresh pairs thinking alone lost 1.34 points (standard error 2.44).

The main observed benefit was the additional cache capacity. At 56 running requests, decode cycles were similar: in matched windows (no prompt tokens, exactly 56 running requests) a cycle took about 258.5, 260.6 and 263.9 milliseconds with 224, 256 and 288 experts, and minutes 10 to 30 of the probes gave 590 to 604 tokens per second for all three lists. In these probes fewer experts did not make a step noticeably faster; they freed memory, and more requests fit in it.

Tokens before and after pruning

Table 16 summarises the token budget before and after pruning. From stock vLLM with all experts to the production configuration (c8 with an FP8 indexer cache), the cache grew from 94,001 to 1,957,095 tokens, a factor of 20.8, and the number of running requests from 4 to 56. Within Merlin 3, pruning to 384 experts multiplied the cache by 2.5 (547,542 to 1,391,054 tokens), and the 296 experts of c8 with an FP8 indexer cache and a larger pin by 3.6 (1,957,095 tokens). Thinking per attempt changed little: pooled over three seeds at a 96K cap, the unpruned model thought for 69.3K tokens on average and REAP-384 for 68.5K. In production-scale previews on the 120 public evaluation tasks, with the twelve-hour schedule, the pruned model generated 17.7M thinking tokens in 39,911 seconds with direct answers and 15.6M in 37,230 seconds with the tool agent. Counting all generated tokens (thinking, answers and programs), the server’s ten-second windows after start-up averaged 495 and 469 tokens per second, and 77.8% of the windows of the agent run were limited by the cache. Even after pruning, a twelve-hour run is limited by tokens. At caps of 96K and 48K, the simulator leaves 46 of about 343 hidden inputs without an answer and gives almost no input a second attempt; the answer file of the 44.17 run had the size that this forecast predicts. Most of the remaining tokens go to forced completions. Of 1,032 attempts of s288p at a 64K cap, 38.7% reached the cap and ended wrong, and they used 50.3% of all tokens; natural completions that were right were 36.9% of attempts and used 26.2% of tokens. Of the correct natural completions, 44.6% came within 32K tokens and 89.0% within 56K, so shorter caps lose answers. On the evaluation hardware, 5 of 46 answers forced at the cap were right. The simulator, calibrated on the 53.33 run, values throughput at about 0.2 points per percent and cache size at about 0.13 points per percent near the production configuration.

Quantity512 expertsPruned
Cache tokens, stock vLLM / Merlin 394,001 / 547,5421,957,095 (c8)
Running requests on the probe4 / 18.956
Tokens per second, first 150 windows (Merlin 3)342521 (REAP-384), 566 (s288p)
Tokens per second, full 6,000 s probe–433 (REAP-384), 492 (s288p)
Thinking tokens per attempt, offline, 96K cap69.3K68.5K (REAP-384)
Thinking tokens per twelve-hour preview (120 tasks)–17.7M direct, 15.6M agent
Inputs without any answer, 96K/48K caps (simulated)16046 (s288p)
Inputs without any answer, 64K/48K caps (simulated)–2 (s288p)

Table 16. Tokens before and after expert pruning. Rows not marked as offline or simulated are measured on four L4 GPUs; “simulated” rows come from our run simulator for about 343 hidden test inputs. Offline thinking lengths are on the dev panel at a 96K cap, pooled over three seeds.

Inference changes on the semi-private set

Table 17 lists every run of Qwen3.8-Flash-Next on the semi-private set. Ten scored runs moved the semi-private score from 25.00 to 56.67, but most steps changed several factors and only the 54.44 configuration was scored twice (54.44 and 55.28), so the steps are not separately resolved. Pruning to 384 experts and then to s288p, with the timeout fault fixed in the second step, took the score from 25.00 to 32.64 and 44.17. Four inference changes, each tested offline first, entered the 53.33 run together with c8: a prompt that states the grid sizes and writes each grid row on its own line (+5.23 points task-weighted over 516 pairs, probability of a positive effect 0.979), answer tokens sampled at temperature 0.3 after the thinking (the same grid accuracy; 29.6% of programs train-perfect against 22.1% at 0.6), the transfer of a train-perfect program found for one test input to the task’s other inputs, and caps of 96K for single-input and 64K for multi-input tasks with second attempts started early. Transferred outputs replace the program outputs of an input that has no train-perfect program of its own and then vote with its grids; when the input has its own, they do not vote and rank after its grids. Second attempts are queued when their first attempts finish, ordered by a confidence class and then by thinking length, with every multi-input task after every single-input task; if the first attempt of a single-input task completed naturally with a grid in fewer than 40,000 thinking tokens and produced no train-perfect program, its second attempt is placed ahead of all unstarted first attempts, and the coverage guard that could defer such a promotion was disabled. The last step replaced the first attempt by a tool-using agent (Algorithm 7): the model may call a function that runs its candidate program on the demonstrations and returns the per-pair errors, at most twice, within a total of 96K tokens for single-input and 64K for multi-input tasks. Each call is run by two checkers in parallel: the tool checker (check_transform), whose per-pair report is returned to the model, and the production checker (the runner’s check_program), whose result also ranks the candidates of the answer file. The loop ends when both find the program train-perfect with the same valid test output, after the second check, or when the remaining budget cannot hold another turn; a final turn then follows unless the second turn has already written the answer. In a 172-input preview of this configuration, 87 attempts were accepted at the first check and 25 at the second, 53 used both checks without acceptance, 7 had no room for a second check, none answered in the second turn, and the two checkers never disagreed in 250 checks. On 71 single-input evaluation inputs in two draws, the agent solved 60.6% against 57.0% for a direct 96K attempt; in twelve-hour previews on the public evaluation set (input-weighted) it led direct answers by 56.98 to 52.91 after 6.25 hours and by 60.47 to 59.88 at the end. Unlike the pilot with Qwen3.8-27B (Section B), this agent is limited to checking a program it has already written, and its first turn thinks for up to 80K tokens. Batch-sharded sampling (+6.4% throughput) and a draft-sampling option of the engine (+2.1%) were added in the same run. A second draw of this configuration scored 55.28, and the same configuration with a 75K agent budget for single-input tasks (caps of 59,000 and 8,000 tokens instead of 80,000 and 8,000) scored 56.67; the two differ by 1.39 points, within the run-to-run variation.

ExpertsConfiguration (change from the previous row)SP
512stock vLLM, effort low, 40K8.61
512same, attempts of 24K and 40K6.11
512Merlin 3, xhigh, 96K / 48K, client timeout of 7,200 s25.00
384REAP-38432.64
288s288p, timeout removed44.17
28864K / 48K, second attempts ordered by confidence, FP8 indexer cache, T=1.0T = 1.040.97
296c8, row-wise prompt, answer temperature 0.3, program transfer, 96K 1-input / 64K multi, early second attempts, T=0.6T = 0.653.33
296first attempt on three inputs together, second attempt revises earlier answers52.50
296as the first run with the row-wise prompt, with two grids per answer, second attempt on disagreement, vote, batch-sharded sampling, 12 h 15 min49.86
296as the first run with the row-wise prompt, with batch-sharded sampling and draft option, 12 h 15 minno score
296tool agent for attempt 1, batch-sharded sampling, draft option, 11 h 50 min54.44
296same configuration, second draw55.28
29675K agent budget for 1-input tasks (caps 59,000 / 8,000)56.67

Table 17. All runs of Qwen3.8-Flash-Next on the semi-private set, in the order they were made. Caps are thinking caps for attempts 1 and 2. Runs from the third row on use Merlin 3, four draft tokens and T=0.6T = 0.6 unless noted; “1-input” and “multi” are tasks with one or with several test inputs.

On the same hardware and under the same limit, the best Qwen3.8-Flash-Next run exceeds the best Qwen3.8-27B run by 12.64 points (56.67 against 44.03), more than twice the resolution of single draws. The comparison is between systems, not models: the two runs differ in the inference procedure as well as in the model.

Results

Measurement. Unless noted, every number in this section is a semi-private score on the 120 hidden ARC-AGI-2 tasks. The published description of the protocol says that the score averages over all test outputs. The scores returned for our runs follow a mean over tasks instead: each of the 89 distinct scores among them is a multiple of 100/720, which is what averaging per-task fractions over 120 tasks with one, two or three test inputs gives, while a mean over NN test inputs gives multiples of 100/NN, and no NN from 100 to 799 other than 720 fits all of them. We therefore use the task-weighted mean offline as well; one fully solved task is worth 0.83 points. Each run is one stochastic twelve-hour run. Nine NVARC 4B configurations were run two to five times without change (27 draws); their pooled within-configuration standard deviation is 1.94 points (95% interval 1.46 to 2.87). This estimate holds for the NVARC 4B pipeline. For Qwen3.8-27B and Qwen3.8-Flash-Next we have two repeated pairs (two

Algorithm 7 First attempt of the 54.44 run: the Qwen3.8-Flash-Next tool agent for one test input. The 56.67 run gives single-input tasks 75K tokens in total, with caps of 59,000 and 8,000.

Input: task with demonstrations DD and test input xx; budget BB of generated tokens and thinking caps (c1,c2)(c_1,c_2): B=96KB = 96\mathrm{K}, (80K,8K)(80\mathrm{K},8\mathrm{K}) for tasks with one test input and B=64KB = 64\mathrm{K}, (48K,8K)(48\mathrm{K},8\mathrm{K}) otherwise; final thinking cf=4Kc_f = 4\mathrm{K}; at most 2 checks

  1. context ←\leftarrow the row-wise prompt of DD and xx with the grid sizes, plus the tool check_transform

  2. n←0n \leftarrow0; c←c1c \leftarrow c_1; answered ←\leftarrow false

  3. while n<2n < 2 and the budget leaves at least 1,000 thinking tokens after reserving the call, the tool response and the final turn do

  4. think for at most cc tokens, then emit a check_transform call; from the second turn on, the final grid and program are also allowed

  5. if the turn wrote the final answer then answered ←\leftarrow true; break

  6. end if

  7. n←n+1n \leftarrow n + 1 (a call that cannot be parsed also counts and is not run); run the transform on DD and xx with the tool checker (check_transform) and, in parallel, the production checker (the runner’s check_program)

  8. append at most 2,500 tokens of feedback from the tool checker: the result of every pair, the expected and produced grids of up to two failing pairs, up to 12 wrong cells per pair, and the test output if it has at most 225 cells; add a note if only the tool checker finds the program train-perfect

  9. if both checkers find the program train-perfect with the same valid test output then break

  10. end if

  11. c←c2c \leftarrow c_2

  12. end while

  13. if not answered then

  14. final turn: think for at most cfc_f tokens (or close at once if no room is left), then write the grid and a program under the grid grammar at T=0.3T = 0.3

  15. end if

  16. attempt 1 ←\leftarrow the final grid; checked programs that the production checker finds train-perfect, in check order, and then the final program if it passes, join the program candidates; other programs that ran with a valid test output join the weak pool

  17. on an infrastructure error that left neither a grid nor a checked program, run the direct attempt instead

identical Qwen3.8-27B configurations that both scored 26.81, and two draws of the 54.44 Qwen3.8-Flash-Next configuration, 54.44 and 55.28), too few to estimate their variation. Where we use 1.94 points for these models below, it is an assumption: under it, two configurations measured once each must differ by more than about 5.4 points to be told apart, and the best of five draws of one configuration exceeds its mean by about 2.3 points on average. We therefore report means over repeated draws where they exist, and treat single-draw differences of the other two models as descriptive.

Data experiments. Table 18 collects the comparisons of training and calibration data from § 4 to § 6. For the grid model, the model continued on real tasks is the best single model, and the best system runs it on all four GPUs; no checkpoint trained with synthetic tasks, alone or next to it, raised the mean score. For the reasoning model, no later training set beat RFT v1 at the same cap, and at a matched 64K cap RFT v1 itself differs from the base model by 6 of 172 inputs (p=0.49p = 0.49). For the expert lists, the calibration data mattered at a fixed number of experts: at 288 experts, the list chosen by ARC routing frequency alone reached 58 correct dev attempts, the saliency list calibrated on coding traffic 66, and the

saliency core with its least-used experts exchanged by ARC routing 80.

Runs by family. We made 189 runs on the semi-private set. Table 19 groups all of them. The 136 runs of the NVARC 4B pipeline with our checkpoints averaged 29.57; among configurations with repeated draws, the best averaged 33.06 over five draws (one single-draw configuration scored 33.89). The Qwen3.8-27B configurations without our fine-tuning averaged 22.66 and reached 26.81 at best; the fine-tuned Qwen3.8-27B line averaged 32.35 (34.44 without two runs that answered only half of the tasks by design) and reached 44.03. Runs that combined both models scored at most 40.28. The ten scored runs of the pruned and unpruned Qwen3.8-Flash-Next on the Merlin engine averaged 46.49 and reached 56.67.

FamilyRunsMeanSDMedianBest
NVARC 4B pipeline with the base checkpoint726.593.4628.4729.86
the released pipeline, unchanged329.120.7029.0329.86
NVARC 4B pipeline with our checkpoints13629.572.7729.4434.86
Qwen3.8-27B, without our fine-tuning622.663.5122.0926.81
Qwen3.8-27B, fine-tuned by us1132.356.9033.6144.03
NVARC 4B and Qwen3.8-27B in one run327.4115.3531.5340.28
Qwen3.8-Flash-Next on Merlin, 512 to 296 experts (§ 6)1046.4910.6951.1856.67
Runs with no score or a score of 06––––
Completed runs scoring below 1010––––

Table 19. All runs on the semi-private set, grouped by method. The family rows count runs that scored 10 or more; the last two rows count, across families, runs that returned no score or 0 (one of them a Qwen3.8-Flash-Next run that exceeded the time limit) and completed runs that scored below 10 (five NVARC 4B runs, three Qwen3.8-27B runs including the answer-only fine-tune at 2.08, and the two Qwen3.8-Flash-Next runs at effort low with stock vLLM, 8.61 and 6.11), so the family means are conditional on a score of at least 10. “Without our fine-tuning” includes one third-party fine-tune of the same base model.

What moved the Qwen3.8-27B score. Table 20 lists selected runs of the Qwen3.8-27B line in order, with the factor that changed. Two steps changed the model. Replacing the base model by RFT v1 also changed the cap from 64K to 32K and the temperature from 1.0 to 0.6, and the score rose from 26.81 to 28.19, within the variation of one draw; at a matched 64K cap the two models differ by 6 of 172 inputs offline (p=0.49p = 0.49,

ModelConfiguration (change from the previous row)SP
answer-only fine-tunethinking off, 8 dihedral views, vote2.08
basethinking at low effort, end-of-turn masked9.86
basexhigh, 64K cap on both attempts26.81
base+ program in a second request, T=0.6T = 0.6, reordered second attempts21.39
base88K cap on both attempts18.89
RFT v132K cap on both attempts, T=0.6T = 0.6, two requests per attempt28.19
RFT v1one request, cap enforced in the engine, two decode speed-ups33.61
RFT v1+ program in the same request37.22
RFT v1, NVFP4+ repair and extra passes36.25
RFT v1, NVFP4first attempt 48K, second 32K37.08
RFT v1, NVFP4first attempt 56K for every input first, second 32K on reordered demonstrations, deadline-aware caps44.03
FINAL, NVFP4same configuration as 44.03, model replaced29.03
RFT v1, NVFP4task-weighted value queue (its check run took 68 minutes to load the model; cause of the drop unresolved)38.19
RFT v1 + NVARC 4B4.5 h of NVARC 4B, then the 44.03 schedule, then a merge40.28

Table 20. Selected runs of the Qwen3.8-27B line and the factor that changed at each step. Runs from 26.81 onward use four L4 GPUs, tensor parallelism 4, MTP and an FP8 key-value cache. Rows below the rule are comparisons against the 44.03 run.

§ 5.7. Replacing RFT v1 by FINAL in an identical configuration lowered the score from 44.03 to 29.03. With RFT v1 held fixed, inference changes raised the score from 28.19 to 44.03 in five single-draw steps: one request per attempt with the cap enforced in the engine and two decode speed-ups (+5.42), the program in the same request (+3.61), NVFP4 with repair passes (−0.97), a 48K first attempt (+0.83), and the 56K/32K schedule with deadline-aware caps (+6.95). Under the variation assumed above, only the first and the last step reach the resolution of about 5.4 points on their own, the first by a small margin; their sum, 15.84 points, is about three times that resolution.

Comparison with a measurement without a time limit. The ARC Prize Foundation evaluated the base Qwen3.8-27B at xhigh on the same 120 semi-private tasks without the twelve-hour budget, and reports 42.36 (305 of 720 points) at $0.45 per task; at the listed price of $2.55 per million output tokens, this allows at most 175K output tokens per task [32]. Our best run scored 44.03 (317 of 720) on these tasks in twelve hours on four L4 GPUs, with at most 88K thinking tokens per test input over its two attempts. The difference is 1.67 points, or two fully solved tasks. A difference this small cannot be resolved by single runs, but that does not show the two systems to be equal, and our 44.03 is the best of our runs rather than a mean; we therefore say only that our best twelve-hour run scored at the level of the Foundation’s measurement without the budget. On the public evaluation set both measurements use the same base model. The Foundation’s per-task results mark 40 of the 120 tasks as passed at xhigh; the base model in our serving setup (FP8 weights, the end-of-turn fix, two attempts with 64K caps; § 5.1) answered every test input of 52 tasks (task score 49.86). We did not rerun their protocol, so we cannot attribute the difference of 12 tasks to any single change.

Discussion

What the data experiments show. For the grid model, real tasks were the most useful training data that we tested: continued fine-tuning on the 1,920 real tasks raised the mean from 28.96 to 31.72, and every model trained with synthetic tasks that we ran alone scored below it (29.31, 28.61, 28.47, 28.06, 27.22 and 26.53; Table 18). The rule corpus did not raise the score in our runs: a checkpoint trained with all 95,993 of its tasks scored 28.61 alone, and checkpoints trained on it did not raise the mean when they ran next to the real-task model. Its tasks differ from ARC-AGI-2 in grid size and in the absence of fixed colour meaning (§ 3.8), which may limit transfer to real tasks; we did not test this. For the reasoning model, the training data changed less than the inference procedure: at a matched 64K cap, RFT v1 and the base model differ by 6 of 172 inputs offline (p=0.49p = 0.49), and none of the later sets with synthetic tasks beat RFT v1 (Table 10). Fine-tuning on the model’s own answer-verified traces did not lengthen the thinking in our runs: mean thinking at a 64K cap was 51.2K tokens for RFT v1 and 51.8K for the base model, although the selection by correctness and length changes the distribution of the traces. A set with 452 traces from other models closed later (20 natural completions between 56K and 64K tokens against 9) and scored 29.03 in the configuration in which RFT v1 scored 44.03; FINAL also changed other parts of the data, so a cost of off-policy traces under a fixed cap remains a hypothesis that we did not test. For pruning, the calibration data decides which experts stay. At the tested concurrency the decode step of the L4 GPUs changed little with the number of experts (about 258 to 264 milliseconds per cycle at 56 requests for 224 to 288 experts), so the main observed benefit of removing experts was memory, and in our tests the experts chosen by their contribution to the output (saliency) worked better than those chosen by how often ARC tokens use them: the coding-calibrated saliency lists and the ARC routing-frequency lists selected substantially different experts, and the frequency list completed its thinking early and lost accuracy (§ 6.5). Routing statistics from ARC training tasks served to exchange the least-used experts of a saliency list, not to rank the experts on their own.

Why the inference changes work. Within the twelve-hour budget, the tokens that a reasoning model can spend on each hidden input constrain its score, and three measurements point this way. First, the same models score far higher without the limit: the base Qwen3.8-27B reaches task scores of 49.86 to 54.03 on the public evaluation set with two long attempts, but 18.89 to 26.81 on the semi-private set with similar caps, while the schedule that fits the twelve hours scores 44.5 offline and 44.03 on the semi-private set (Table 11). Second, naturally completed attempts were more accurate in these runs: attempts whose thinking was closed at the token cap were right in 4% to 9% of cases for Qwen3.8-27B and in 5 of 46 cases for Qwen3.8-Flash-Next, against 42% to 86% for natural completions (Table 8). Third, the unpruned Qwen3.8-Flash-Next at 342 tokens per second leaves an estimated 160 of about 343 hidden inputs without any answer at 96K caps, and the pruned model 46. The changes that helped turned the fixed budget into more natural completions: one request per attempt with the cap enforced in the engine (38% more completed attempts per hour in a 40-task test, with the same 20 of 59 inputs solved), a schedule that starts every first attempt before any second one, with deadline-aware caps (+6.95 points), and a larger key-value cache through pruning (cache ×2.5, throughput +52%). A program checked on the demonstrations adds a second answer channel of high precision for few tokens: 73% to 78% of train-perfect programs were right on the test input, at 4.5% of the generated tokens, and the score rose from 33.61 to 37.22 when programs were added.

What may carry over beyond ARC. Several components could be reused beyond ARC, given appropriate calibration data and a task-specific verifier; the rule language, the palette key, the shortcut solvers, the dihedral transforms and the output grammar are specific to grids, and we have not evaluated transfer to other domains. Enforcing a thinking budget inside the serving engine and ordering attempts by a deadline could serve a reasoning model that must answer many questions under a fixed compute budget. Pruning experts by saliency, adapted with routing on the target domain and with experts kept for an output format such as code, could serve a mixture-of-experts model on GPUs with too little memory. Rejection-sampling fine-tuning needs an exact checker of the final answer, and checking a program against the given examples needs a few-shot problem with executable programs. The admission tests of the rule corpus need a program language with an executable semantics and a finite set of probes and competitors.

What is new. Relative to public work, we contribute (i) a synthetic corpus whose rules are checked to have distinct behaviour signatures and whose tasks are checked against a finite set of competing rules, checks that the descriptions of re-arc, BARC and NVARC do not report (Table 4); (ii) ARC-adapted expert pruning that keeps the core of a saliency list, exchanges only its least-used experts by task routing, and protects experts for program tokens, together with the observation that coding-calibrated saliency lists and ARC routing-frequency lists select substantially different experts and that frequency alone performed worse in our tests; (iii) a single-request attempt in which the engine closes the thinking at the cap and a grammar constrains only the answer, with its measured effect on attempts per hour; and (iv) a comparison of offline validation without a time limit with the semi-private score, which suggests that throughput and token allocation constrain the deployed score, without isolating how much of the gap they cause. Answer checking of self-generated traces, expert pruning by saliency and checking programs on demonstrations build on prior work (§ 2).

Remaining headroom. Our best Qwen3.8-27B run scored at the level of the base model without a time limit, and the fine-tuned model solves 99 of the 172 public evaluation inputs (57.6% of inputs, task score 59.6) in at least one of 8 to 22 attempts, against 71.5 inputs (task score 44.5, a mean over four pairings of attempts) in a two-attempt schedule. Within this budget, further gains must come from more tokens per input or from more accuracy per token. The Qwen3.8-Flash-Next line shows that the first route is open on the same hardware: with the same kind of inference procedure, pruning and serving changes took a model of the same family from 25.00 to 56.67. The appendices list unsuccessful variants and their measured results.

Experimental record. The appendices give the hyper-parameters and data provenance (Section A), the run-to-run noise of repeated runs (Section C), and the unsuccessful and incomplete experiments (Section B), including 16 expert lists and checkpoints of Qwen3.8-Flash-Next that we rejected (Table 21); the synthetic training sets of Qwen3.8-27B that did not beat RFT v1 are in Table 10.

Conclusion

We compared training and calibration data for three open models on ARC-AGI-2 under a budget of twelve hours on four L4 GPUs: a 4B grid model, a 27B reasoning model and a mixture-of-experts reasoning model. For the grid model, we built a corpus of 12,000 rules with distinct behaviour signatures, with 95,993 tasks that were checked against a shortlist of competing rules; continued training on 1,920 real tasks, run on all four GPUs with revised scheduling, raised the semi-private score from 28.96 to 33.06 (mean of five runs, best 34.86), while no checkpoint trained with synthetic tasks, including one trained on all 95,993 corpus tasks, improved on the real-task model. For the 27B reasoning model, fine-tuning on 1,036 answer-verified self-traces together with an inference procedure built around the model (the thinking cap enforced in the engine, a program checked on the demonstrations, a long first attempt for every input) raised the score from 26.81 to 44.03, the level of the ARC Prize Foundation’s measurement of the base model without the time limit (42.36); offline validation of the deployed schedule agreed with the semi-private score, and the comparison suggests that throughput and token allocation constrain the deployed score. For the mixture-of-experts model, memory rather than compute limited the 4 L4 GPUs: removing a quarter of the experts by router-weighted activation saliency kept accuracy on our dev panel and grew the key-value cache 2.5 times, lists adapted with ARC routing from training tasks and with experts kept for program tokens reached 296 experts and, with an FP8 indexer cache, 3.6 times the cache, and the score rose from 25.00 to 56.67.

Our measurements have limits. All hidden-set scores are semi-private scores; the private scores of the same runs were not available to us. Each run is one draw with a standard deviation of about 1.9 points, measured on repeated NVARC 4B runs and assumed for the other two models, so single-factor comparisons below about 5.4 points are not resolved, and several of our steps changed more than one factor. The NVARC 4B base model, ckpt-34000 and its descendants, and every Qwen3.8-27B training set from RFT v1 on contain public evaluation tasks, so offline numbers on those tasks are in-sample for the affected models; the subset H is training-disjoint only for the Qwen3.8-27B line before FINAL, and we used it repeatedly during development, so it is a development subset rather than an untouched test set. Our best NVARC 4B configuration changed the context lengths and the scheduling together, so we cannot say how much each change contributed. Only one Qwen3.8-Flash-Next configuration has been scored twice (54.44 and 55.28), its accuracy gates ran on a different engine and checkpoint from those used for hidden evaluation, and its expert lists were chosen by accuracy on public evaluation panels. Section B lists unsuccessful variants and their measured results. A direction we did not try is to measure transfer to real tasks while a synthetic corpus is being built, and to admit rules by their effect on a held-out set of real tasks as well as by their distinctness.

Acknowledgements

We thank the authors of NVARC and of the Qwen models for releasing their weights and code, and the ARC Prize Foundation for ARC-AGI-2 and its evaluation of hidden tasks. Development runs used Google Colab, rented GPUs and machines with four L4 GPUs.

Hyper-parameters and setup

NVARC 4B test-time training. Fresh LoRA adapter per task, rank 256, α=32\alpha= 32, rank-stabilised, dropout 0, on the query, key, value, output, gate, up and down projections plus the embedding and output layers (trained in full). Data: 128 sequences (8 dihedral transforms ×\times 16 random colour permutations of all ten colours, demonstrations shuffled and cut from the front to fit the context). Loss on every output grid and the end-of-sequence token. One epoch, batch 1 (128 steps), learning rate 5×10−55 \times10^{-5}, 10% warm-up, cosine decay, gradient clipping 1.0, no weight decay. Inference: 16 views per test input in 4 batches of 4, depth-first search with a cumulative-probability cut-off of 0.2 and 540 seconds per batch, rescoring under 8 views (one colour permutation each), selection by search hits minus mean rescoring negative log-likelihood.

ckpt-34000. Initialised from NVARC 4B; 1,920 tasks (ARC-AGI-1 training and evaluation, ARC-AGI-2 training, ARC-AGI-2 public evaluation with test pairs appended), 16 views per task, 30,720 examples. Rank-256 rank-stabilised LoRA with α=32\alpha= 32 on the same targets, learning rate 5×10−55 \times10^{-5}, 5% warm-up, cosine decay over 61,440 planned steps, batch 1, AdamW, context 8,192, loss on all output turns, checkpoint every 2,000 steps. One L4, stopped at step 34,273 after 10.0 hours. Software: Unsloth 2025.9.7, transformers 4.55.4, PyTorch 2.8.0.

Qwen3.8-27B RFT v1. LoRA rank 32, α=32\alpha= 32, dropout 0 (233.5M parameters, 496 modules). AdamW (0.9, 0.999), no weight decay, learning rate 10−410^{-4}, 5% warm-up, cosine decay, gradient clipping 1.0, one epoch, micro-batch 1 without packing, gradient accumulation 8, maximum length 81,920 tokens. Loss: token mean of cross-entropy over the reasoning, the closing tag, the answer grid, the program (when present) and the end-of-turn token. Memory: re-entrant gradient checkpointing, activation offload to host memory, MLPs chunked over 8,192 positions, chunked cross-entropy over 2,048 positions; peak 85 GiB on one RTX PRO 6000 (96 GB). Software: transformers 5.17, PEFT 0.21.1, PyTorch 2.11. RFT v1: 1,033 rows, 130 steps, 7.06 hours. FINAL: 1,801 rows, 226 steps, 14.6 hours, maximum length 65,536.

Trace generation. vLLM 0.30, maximum model length 81,920, 40 sequences, 32 concurrent requests; T=0.6T = 0.6, top-p=0.95p = 0.95, top-k=20k = 20, shigh; thinking cap 48,000 tokens, answer at most 8,192 tokens under a grid grammar; seed derived from the task, test index, attempt and run tag. One generation pass over 490 training inputs took 5.2 hours and produced 19.4M completion tokens.

Qwen3.8-27B on four L4 GPUs (44.03 run). vLLM 0.30, tensor parallelism 4 on four L4 GPUs, NVFP4 weights (W4A16 Marlin kernels on L4), FP8 key-value cache, three-token MTP with the end-of-turn rows of the output head zeroed, batch-sharded sampling, maximum model length 98,304, 36 sequences, 32 concurrent client requests, 4,096 batched tokens. Caps: 56,000 thinking tokens for attempt 1 and 32,000 for attempt 2; deadline-aware caps at 15 tokens per second per stream; new requests stop 15 minutes and the run stops 7 minutes before the 12-hour limit.

Qwen3.8-Flash-Next on four L4 GPUs (54.44 run). Merlin 3.0 (vLLM 0.32.0, PyTorch 2.13, CUDA 13.0) with its profile for four 24 GB Ada GPUs: tensor parallelism 2 and pipeline parallelism 2 with expert parallelism, stages of 25 and 23 layers, custom all-reduce off. The uploaded REAP-384 checkpoint is pruned at start-up to the 296 experts of c8 in 49 mixture-of-experts layers (38,808 tensors dropped). FP8 key-value cache with static scales and an FP8 indexer cache, 7.7 GiB reserved per GPU (1,957,095 tokens); maximum model length 114,688; 2,048 batched tokens; no prefix caching; a server limit of 64 sequences and 56 concurrent client requests; four draft tokens with probabilistic draft sampling and an INT8 draft head; batch-sharded sampling; effort xhigh, T=0.6T = 0.6 for thinking, answer logits multiplied by 2 after the closing tag. Prompts state the grid sizes and give one row per line. Attempt 1 is the tool agent with at most two checks and a total of 96K tokens for single-input tasks (80K for the first turn, 8K for later turns) or 64K for multi-input tasks; attempt 2 has a 48K cap on reordered demonstrations and is queued when attempt 1 finishes, ordered by a confidence class and then by thinking length, with an early second attempt for single-input tasks as described in § 6.7 (coverage guard off); a 16K repair turn with per-pair program feedback is queued after attempt 1 only if no program is train-perfect and a plain-view program reproduces at least one demonstration, and starts at least 45 minutes before the hard stop; one extra 48K attempt runs only if no program is train-perfect, the two grids differ and at least 50 minutes remain. Deadline-aware caps assume 8 tokens per second per stream and also cut the total budget of the agent, and a watchdog stops the run 11 hours and 50 minutes after it starts. The 56.67 run differs only in the agent budget of single-input tasks: 75K tokens in total, with caps of 59,000 and 8,000.

Data provenance. The 30 H tasks of the public evaluation set were excluded from every training set of the RFT line before FINAL; the answer-only fine-tune held out a different set of 30 tasks, which shares 8 tasks with H. ckpt-34000 and all NVARC 4B checkpoints derived from it were trained on all 120 public evaluation tasks. No hidden task was available to us at any time; all semi-private numbers are on hidden tasks.

Unsuccessful and incomplete experiments

For completeness, we list each idea that did not help, with its measured result, and in Table 21 every expert list and smaller checkpoint of Qwen3.8-Flash-Next that we rejected. Together with the experiment tables of the main text, this is the full record of what we tried for the three lines.

Answer-only fine-tuning of Qwen3.8-27B. Training on 37,201 answer grids with thinking disabled gave 2.08 on the semi-private set. A continuation on all 1,147 labelled official tasks solved 0 of 9 inputs of tasks in its own training data (§ 5.2).

Explanations inside the thinking channel. Placing 1,284 verified explanations (284 written by an LLM for official tasks, 1,000 generated from templates for synthetic tasks) in the reasoning lowered accuracy at 32K from 37 to 9 of 172 inputs; most attempts opened with a solution summary (§ 5.7). Placing 1,602 explanations, 1,584 of them of synthetic tasks, in the answer channel (ft64k) gave 43 against 51 (two-slot, production harness; Table 10).

More or shorter self-traces. Adding 73 traces from an 88K-cap second pass (main2) gave 57 of 172 inputs at 64K, equal to the base model. Training only on traces of at most 16K tokens gave 34 against 37 at 32K and did not shorten thinking on hard tasks (83% of attempts still reached the cap).

Program targets in the fine-tuning data. RFT v2 (1,655 program targets) and RFT v3 (1,941, including 319 programs from other attempts) did not raise the rate of train-perfect programs (41 and 30 against 42 at 32K).

A larger final training set. FINAL, with 452 teacher traces, 315 symmetry-view traces and all public evaluation tasks, scored 29.03 against 44.03 for RFT v1 in the same configuration.

Fine-tuning a program writer. Qwen3-8B trained on 512 to 10,000 compiled programs never produced a program that passes the demonstrations of any of 15 public evaluation tasks (0 of 120 programs). With 512 and 3,000 targets that call the corpus library, 85 and all 120 of the programs failed while reading the palette key, which real tasks do not have; 10,000 standalone targets without the library gave more malformed and truncated programs than the base model.

Synthetic curricula for NVARC 4B. 36,000 shape-balanced tasks raised accuracy on new tasks of their generator from 5.8% to 61.7% but scored 26.53 on the semi-private set. Two FORGE50 recipes scored below ckpt-34000 alone (mean 31.72): continuing ckpt-34000 with 25% FORGE50 episodes next to all 1,920 real tasks in 16 views scored 29.31, and a mixture of 86% FORGE50 episodes with 12% real tasks and 2% hand-made tasks scored 28.47. A checkpoint trained from the base model on the 1,920 real tasks and all 95,993 rule-corpus tasks scored 28.61 alone.

A likelihood-threshold training loss. A gate loss matched to the 0.2 search cut-off solved 2 of 18 inputs, the same as the base model and the cross-entropy control.

Program retrieval. None of the 11,997 library programs for tasks outside the public evaluation set reproduces all demonstrations of any public evaluation task; three retrieved programs in the prompt gave 48 against 54 of 328 attempts.

Code agents. A tool-calling agent with three code executions per attempt at 32K solved 5 of 20 inputs against 7 and 4 for direct answers in two seeds; 57 of its 82 calls were exploration, and 13 of the 20 inputs failed under every protocol. Two runs of tool-integrated reasoning were invalid for different reasons: in the first (5 against 20 of 90) the model ended its thinking to emit code and no code was executed; in the second (21 against 20) the sandbox’s memory limit broke 135 of 246 calls. We count tool-integrated reasoning as untested.

Combining NVARC 4B and Qwen3.8-27B. Giving NVARC 4B the first 4.5 hours and Qwen3.8-27B the rest scored 40.28 against 44.03 for Qwen3.8-27B alone.

Allocation variants. Uniform 88K caps scored 18.89; a program in a separate request scored 21.39 (with other changes); a task-weighted value queue scored 38.19 (its check run needed 68 minutes to load the model); a soft wrap-up sentence at 32K inside a 45K cap scored 26.39.

Expert lists for Qwen3.8-Flash-Next. REAP-288 solved 66 and the ARC-frequency list ARC-288 58 of 246 dev attempts, against 87 for REAP-384; CORE-320 failed the fresh-panel rule (lower bound −4.46 points). Cutting c8 to 272 or 256 experts lost 2.16 and 3.02 points on the production mix; at 224 experts we have only an L4 probe (17 right first attempts of 32 against 21 for s288p with an FP8 indexer cache, p=0.39p = 0.39) and a thinking-only screen (−6.25 points), and a fully protected 256-expert list lost 17.8 points with the agent. Teacher-forced likelihood ranked ARC-288 close to REAP-384 and did not predict its loss. Table 21 lists all 16 rejected lists and checkpoints.

Serving Qwen3.8-Flash-Next. Stock vLLM fitted 94,001 cache tokens (four requests, 164 tokens per second); SGLang fitted about 434,000 and generated 221 to 255 tokens per second against 639 to 657 for Merlin; six draft tokens did not boot and three were 1.2% slower than four; prefix caching was slower; a larger indexer budget of 8,192 positions cost 5.5% of the cache for a gain of 2.0 points that did not reach our threshold.

Prompts and attempts for Qwen3.8-Flash-Next. One request for all test inputs of a task lost 7.95 points offline, two single-input tasks in one request 5.7, rendered grid images 4.52, and code-first prompts 1.41 and 9.04. Effort medium lowered the task-weighted score from 28.10 to 5.88, and T=1.0T = 1.0 closed fewer attempts by 64K (61 against 90 of 243) and solved fewer by 96K (65 against 78); its run scored 40.97. Showing the first-attempt grid and asking for a fix broke 8 right answers and fixed 1 at a 2K budget. On the semi-private set, a first attempt on three inputs together with a refining second attempt scored 52.50 and two grids per answer with voting 49.86, both below 53.33.

Run-to-run noise

ConfigurationDrawsMeanSDScores
ckpt-34000, single model532.052.5533.47, 34.44, 27.78, 32.36, 32.22
ckpt-34000 on four GPUs with runtime changes (§ 4.4)533.061.5733.61, 34.86, 33.06, 33.19, 30.56
ckpt-34000 and the FORGE50 continuation, two GPUs each331.302.7634.31, 30.69, 28.89
as the runtime-changes row, FORGE50 continuation on the fourth GPU332.641.9134.31, 33.06, 30.56
ckpt-34000, single model, another runtime331.390.7031.39, 32.08, 30.69
four further configurations2 each28.12–32.290.59–1.87

Table 22. Configurations run more than once without change (semi-private set). All are NVARC 4B pipelines. The pooled within-configuration standard deviation over the 27 draws is 1.94 (95% interval 1.46 to 2.87, 18 degrees of freedom).

Two runs of Qwen3.8-27B with identical code both scored 26.81. Two draws of the 54.44 Qwen3.8-Flash-Next configuration scored 54.44 and 55.28.

References

  1. [1]François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. URL https://arxiv.org/abs/1911.01547.
  2. [2]François Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. ARC-AGI-2: A new challenge for frontier AI reasoning systems. arXiv preprint arXiv:2505.11831, 2025. URL https://arxiv.org/abs/2505.11831.
  3. [3]François Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. ARC Prize 2025: Technical report. arXiv preprint arXiv:2601.10904, 2026. URL https://arxiv.org/abs/2601.10904.
  4. [4]Ivan Sorokin and Jean-François Puget. NVARC solution to ARC-AGI-2 2025. https://drive.google.com/file/d/1vkEluaaJTzaZiJL69TkZovJUkPSD H5Xc/view, November 2025. Code: https://github.com/1ytic/NVARC. Accessed 2026-11-02.
  5. [5]Daniel Franzen, Jan Disselhoff, and David Hartmann. The LLM Architect: Solving ARC-AGI is a matter of perspective. Technical report, ARC Prize 2024, https://da-fr.github.io/arc-prize-2024/the_architects.pdf, December 2024. Code: https://github.com/da-fr/arc-prize-2024. Accessed 2026-11-02.
  6. [6]Qwen Team. Qwen3.8-27B. Hugging Face model card, https://huggingface.co/Qwen/Qwen3.8-27B, 2026. Accessed 2026-11-02.
  7. [7]Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, et al. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026. URL https://arxiv.org/abs/2608.30320.
  8. [8]Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, and Vithursan Thangarasa. REAP the experts: Why pruning prevails for one-shot MoE compression. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. URL https://openreview.net/forum?id=ukGxWd2aDG.arxiv.org/abs/2510.13999
  9. [9]François Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. ARC Prize 2024: Technical report. arXiv preprint arXiv:2412.04604, 2024. URL https://arxiv.org/abs/2412.04604.
  10. [10]Michael Hodel. Addressing the abstraction and reasoning corpus via procedural example generation. arXiv preprint arXiv:2404.07353, 2024. URL https://arxiv.org/abs/2404.07353. Code: https://github.com/michaelhodel/re-arc.
  11. [11]Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, Wei-Long Zheng, Zenna Tavares, Yewen Pu, and Kevin Ellis. Combining induction and transduction for abstract reasoning. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=UmdotAAVDe.arxiv.org/abs/2411.02272
  12. [12]Daniel Franzen, Jan Disselhoff, and David Hartmann. Product of experts with LLMs: Boosting performance on ARC is a matter of perspective. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 17657–17671. PMLR, 2025. URL https://proceedings.mlr.press/v267/franzen25a.html.arxiv.org/abs/2505.07859
  13. [13]Daniel Franzen, Jan Disselhoff, and David Hartmann. The ARCitects technical report: ARC Prize 2025 solution summary. https://lambdalabsml.github.io/ARC2025_Solution_by_the_ARChitects/, 2025. Accessed 2026-11-02.
  14. [14]Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyotish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 942–963. PMLR, 2025. URL https://proceedings.mlr.press/v267/akyurek25a.html.arxiv.org/abs/2411.07279
  15. [15]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenghen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388.
  16. [16]Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The ConceptARC benchmark: Evaluating understanding and generalization in the ARC domain. Transactions on Machine Learning Research, August 2023. URL https://arxiv.org/abs/2305.07141.
  17. [17]Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734, 2025. URL https://arxiv.org/abs/2506.21734.
  18. [18]Alexia Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks. arXiv preprint arXiv:2510.04871, 2025. URL https://arxiv.org/abs/2510.04871.
  19. [19]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.org/abs/2107.03374.
  20. [20]Ryan Greenblatt. Getting 50% (SoTA) on ARC-AGI with GPT-4o. Redwood Research blog, https://blog.redwoodresearch.org/getting-50-sota-on-arc-agi-with-gpt, June 2024. Code: https://github.com/rgreenblatt/arc_draw_more_samples_pub. Accessed 2026-11-02.
  21. [21]Julien Pourcel, Cédric Colas, and Pierre-Yves Oudeyer. Self-improving language models for evolutionary program synthesis: A case study on ARC-AGI. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 49659–49688. PMLR, 2025. URL https://proceedings.mlr.press/v267/pourcel25a.html.arxiv.org/abs/2507.14172
  22. [22]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633–638, 2025. doi: 10.1038/s41586-025-09422-z. URL https://doi.org/10.1038/s41586-025-09422-z.
  23. [23]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, pages 15476–15488, 2022. doi: 10.5220/068431-1126. URL https://arxiv.org/abs/2203.14465.
  24. [24]Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023. URL https://arxiv.org/abs/2308.01825.
  25. [25]Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (ReST) for language modeling. arXiv preprint arXiv:2308.08998, 2023. URL https://arxiv.org/abs/2308.08998.
  26. [26]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork. Qwen blog, https://qwen.ai/blog?id=qwen3.8, August 2026. Accessed 2026-11-02.
  27. [27]Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=r8H7xhYPwz.arxiv.org/abs/2412.06464
  28. [28]Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozìere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 15706–15734. PMLR, 2024. URL https://proceedings.mlr.press/v235/gloeckle24a.html.arxiv.org/abs/2404.19737
  29. [29]DeepSeek-AI. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. URL https://arxiv.org/abs/2412.19437.
  30. [30]Yaniv Levithan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR, 2023. URL https://proceedings.mlr.press/v202/levithan23a.html.
  31. [31]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Leprial, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023. URL https://arxiv.org/abs/2302.01318.
  32. [32]ARC Prize Foundation. Qwen3.8-27B: ARC-AGI results. https://arcprize.org/results/alibaba-qwen3-8-27b, 2026. ARC Prize Verified. Accessed 2026-11-02.
  33. [33]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23), pages 611–626. ACM, 2023. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165.
  34. [34]NVIDIA, Felix Abecassis, Anjulie Agrusa, Dong Ahn, Jonah Alben, Stefania Alborghetti, et al. Pre-training large language models with NVFP4. arXiv preprint arXiv:2509.25149, 2025. URL https://arxiv.org/abs/2509.25149.
  35. [35]Eduardo Alvarez, Omri Almog, Eric Chung, Simon Layton, Dusan Stosic, Ronny Krashinsky, and Kyle Aubrey. Introducing NVFP4 for efficient and accurate low-precision inference. NVIDIA Technical Blog, https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/, June 2025. Accessed 2026-11-02.
  36. [36]Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023. URL https://arxiv.org/abs/2310.10537.
  37. [37]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations (ICLR), 2017.
  38. [38]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  39. [39]Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, 2024.
  40. [40]Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zheng, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6159–6172, 2024.arxiv.org/abs/2402.14800
  41. [41]Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoE with hints from its routing policy. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=eFWG9cy3WK.arxiv.org/abs/2310.01334
  42. [42]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
  43. [43]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=1PL1NIMMrw.arxiv.org/abs/2203.11171
  44. [44]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Koryvatis, Jon Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, pages 62557–62583, 2024. doi: 10.52202/079017-2000. URL https://arxiv.org/abs/2312.07104.

Paper details

vlnc:W_6hzpkfyt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents