Dory1.0-4B: A Looped Normalized-GPT

Introduction

We present Dory1.0-4B, the first open looped language model in the Nemotron Labs family. Scaling test-time compute has become a major driver of LLM capability, but this compute is mostly spent in token space: models generate longer reasoning traces, and latency, KV-cache footprint, and serving cost grow with every generated token. Looped (latent-recurrent) language models offer a complementary compute scaling axis [8, 32, 11, 38]. By repeatedly applying a shared block of layers, a looped model spends more computation per token in latent space without adding parameters. Dory1.0-4B brings this idea to a fully post-trained model built for reasoning and agentic tasks. We train Dory1.0-4B with a mixture of loop counts [16], so that a single checkpoint can be served at any loop count $k$ and trade inference compute for accuracy per request, with no retraining and no model swap.

Figure 1 summarizes the effect of looping. On the equal-weight average of 12 benchmarks (Table 2), Dory1.0-4B improves from 35.2 at one loop to 39.7 at two loops and 40.6 at four loops (Figure 1a). It outperforms larger models such as Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B, and comes within 1.7 points of Qwen3.6-35B-A3B, a model with roughly $9\times$ as many total parameters. The additional latent computation also makes Dory1.0-4B more token-efficient: across four agentic benchmarks, four loops reduce the number of generated tokens per attempt by $1.3$–$2.5\times$ relative to one loop, with the largest reductions on GDPval-AA v2 and Terminal-Bench v2.1 (Figure 1b and § 5.4), partially offsetting the higher compute per token.

Task	Dory1.0-4B	Dory1.0-4B	Dory1.0-4B	Qwen3.5 4B	Qwen3.5 9B	Gemma 4 12B	Gemma 4 26B-A4B	Qwen3.6 35B-A3B
	1 loop	2 loops	4 loops
Coding & Agentic
SciCode	27.3	34.94	35.11	25.82	28.21	37.89	40.32	37.24
SWE-Bench Verified	50.3	59.5	61.8	54.7	61.0	47.3	57.40	73.4†
Terminal-Bench v2.1	12.5	18.1	22.2	15	24	27.0	35	43
$\tau^{3}$-Banking	11.6	12.89	14.69	5.03	7.22	8.63	13.66	9.54
GDPval-AA v2 (Elo)	815	946	967	424	546	657.4	751.6	987
Reasoning
GPQA Diamond (no tools)	69.19	72.41	72.16	77.71	80.93	73.36	80.43	85.35
HLE (text-only, no tools)	8.25	10.89	10.43	11.31	15.43	13.67	17.79	19.18
Instruction Following
IFBench	74.29	77.94	79	53.83	59	73.44	77.89	57.44
Long Context
AA-LCR	63	71	73	64.7	68.4	58.9	64.6	68.5
GDP.pdf (text-only all-pass)	3.4	6.2	6.0	2.8	4.6	4.00	3.60	4.40
General Knowledge
AA-Omniscience (accuracy)	10.03	10.08	10.03	15.02	18.05	17.20	21.13	19.95
AA-Omniscience (non-hallucination)	76.42	79.74	79.38	20.18	17.39	21.84	16.36	65.25
Average (Equal weights)	35.17	39.67	40.60	28.53	32.21	32.59	36.73	42.30

Dory1.0-4B uses a normalized Transformer (nGPT) backbone (Loschilov et al., 2025) with gated

Correspondence to: {mfishman,simengs,kpuvvada}@nvidia.com

© 2026 NVIDIA. All rights reserved.

Architecture

grouped-query attention and a 3:1 interleaving of sliding-window and full-attention layers. Its 72 layers are partitioned into a 16-layer input block, a 40-layer shared recurrent block, and a 16-layer output block. The input and output blocks run once, while the recurrent block runs $k$ times, with the input block’s output re-injected before every loop.

Dory1.0-4B was pretrained on 6T tokens. The model was first pretrained with $k = 1$ on ∼4.5T tokens as a standard, non-looped model in NVFP4 ([23]), which yields an end-to-end speedup of 20% over BF16. We then introduced looping during a BF16 continuation over ∼1.5T tokens, sampling the loop count from $\{1, 2, 3, 4\}$ and training with truncated backpropagation through loops. Finally, we extended the context length to 256K tokens over an additional 300B tokens.

We post-trained Dory1.0-4B with an extended SFT stage followed by a short RLVR stage. SFT ran for nine epochs over packed 256K-token sequences containing both thinking and non-thinking responses, and retained the random loop sampling used in pretraining. RLVR then ran 200 steps of asynchronous GRPO ([33]) across 23 environments spanning agentic tasks, coding, reasoning, long-context question answering, instruction following, and safety. A key ingredient of RLVR is a difficulty-aware loop curriculum: rollout workers run at fixed loop counts, and prompts with lower profiled pass rates are preferentially routed to deeper loops. In an ablation, this curriculum yields its largest gains at four loops, for example raising Terminal-Bench v2.1 accuracy from 14.0% to 21.4%. At four loops, Dory1.0-4B outperforms Gemma 4 26B-A4B on SWE-Bench Verified (61.8% vs. 57.4%) and GDPval-AA v2 (967 vs. 752 Elo; Table 2).

With standard per-loop caching, the recurrent block’s KV cache grows linearly with $k$. Dory1.0-4B supports last-loop KV caching ([16]), in which each loop overwrites the previous loop’s cache entries, so attention KV memory stays at its one-loop size for any $k$. At four loops, last-loop caching changes benchmark scores by between −2.1 and +0.4 points relative to per-loop caching, with nearly unchanged output lengths.

Finally, Dory1.0-4B was converted to MXFP8 using post-training quantization, and to NVFP4 and Q5_K_M using quantization-aware distillation ([36]). All checkpoints are open-sourced under the OpenMDW-1.1 license at https://huggingface.co/collections/nvidia/nemotron-labs-dory.

Architecture

Dory1.0-4B is based on the normalized Transformer (Loshchilov et al., [20]), retaining its residual-update rule. Our full nGPT implementation additionally L2-normalizes the concatenated, gated attention outputs before the output projection and the SwiGLU activations before the MLP downprojection. We implement these normalizations with fused Triton kernels. We initialize the raw residual-rate, query–key scale, and logit-scale parameters at 0.0177, rather than the paper’s $1/\sqrt{d_{\mathrm{model}}}$, while retaining an initial effective residual rate of 0.05.

We augment this formulation with input-dependent sigmoid attention-output gates and grouped-query attention with 16 query heads and 4 KV groups. The stack contains 72 physical layers: 36 MLP-only layers interleaved with 27 sliding-window and 9 full-attention layers. Thus, the attention layers follow a 3:1 sliding-window/full-attention pattern. Figure 2 shows the overall architecture of Dory1.0-4B.

Pretraining

In this section, we provide details on the NVFP4 pretraining recipe, data, and hyperparameters used for Dory1.0-4B Base. We also share details on the long-context extension phase.

NVFP4 Training

We first pretrain a model using Megatron-LM and NVIDIA Transformer Engine’s NVFP4 block-scaling recipe ([23]). This mixed-precision recipe yields an end-to-end training speedup of approximately 20% over BF16 by using E2M1 4-bit operands for supported linear-layer matrix multiplications, while retaining BF16 model weights and FP32 optimizer master weights, Adam moments, and gradient accumulation and reduction. We do not exempt the first or last decoder layers from NVFP4. We use distributed Adam with $(\beta_1, \beta_2) = (0.9, 0.95)$, global gradient-norm clipping at 1.0, and no weight decay or learning-rate warmup.

We use the pretraining data of Nemotron 3 Ultra ([25]), following its two-phase curriculum (Feng et al., [9]): a phase-1 mixture that favors diversity, followed by a phase-2 mixture that favors quality. We train on approximately 3T tokens of phase-1 data at a sequence length of 8,192, then switch to the phase-2 mixture and increase the sequence length to 32,768. Reducing the global batch size from 768 to 192 sequences preserves 6,291,456 tokens per optimizer step. The learning-rate schedule spans both NVFP4 pretraining and the subsequent BF16 continuation: we hold the learning rate at $3.2 \times10^{-4}$ through 4.5T tokens, then apply minus-square-root decay toward $10^{-5}$ at 6T tokens. The NVFP4 stage ends at iteration 720,750, after approximately 4.535T tokens.

Training with Loops

We continue from the NVFP4 checkpoint in BF16 for approximately 1.465T additional tokens, reaching the 6T endpoint. This is a normal checkpoint restart: optimizer, scheduler, RNG, and data-position state are preserved, and the phase-2 data mixture remains unchanged. We retain the 32,768-token sequence length and global batch size of 192, using 192 data-parallel GPUs across 48 nodes, a microbatch size of one, and no tensor, pipeline, or context parallelism.

We partition the pretrained stack into a 16-layer input block, a shared 40-layer recurrent block, and a 16-layer output block. The input and output blocks run once, while the recurrent block runs $k$ times with all of its parameters, including nGPT residual rates, shared across invocations. The effective forward depth is $32 + 40k$, ranging from 72 to 192 physical-layer applications. Let $x$ denote the input block’s output, $F_\theta$ the shared recurrent block, and $N$ per-token L2 normalization. We initialize the recurrent state by copying $x$. Our loop-boundary transforms add this same $x$ to the current state before every invocation and replace the state with the recurrent block’s output. With normalization at both boundaries, the forward recurrence is

These transforms introduce no learned adapter parameters or outer residual addition; the final state $s_k$ is passed to the output block and language-model head.

We use a base loop-count distribution with probability 0.25 for each $k \in\{1, 2, 3, 4\}$. The draw is deterministic in the optimizer step and microbatch index and is shared across data-parallel replicas, rather than sampled independently on each replica. We additionally set a rank-level override fraction of 0.1: a rotating subset of replicas is forced to execute exactly one loop, while the remaining replicas use the common sampled depth (Ladhak et al., [16]). Integer rounding selects 19 of the 192 replicas (approximately 9.9%) for this override. The subset rotates with the step and microbatch index, providing additional one-loop training signal; consequently, the realized depth mixture is not uniform.

Truncated Backpropagation

We use truncated backpropagation through loops (TBPTL) with depth one. Earlier invocations run without gradient tracking; the final recurrent invocation and the input/output blocks remain trainable, with the input block receiving gradients through the final invocation’s residual input connection. We apply next-token cross-entropy only after the final sampled invocation, without intermediate-exit or early-loop auxiliary losses.

Long-Context Extension

Starting from the 6T recurrent checkpoint, we perform a long-context extension in BF16, increasing the sequence length from 32,768 to 262,144 tokens. An additional approximately 300B tokens brings the total training budget to approximately 6.3T tokens. We increase the full-attention RoPE base from 500,000 to 10,000,000, while retaining the sliding-window RoPE base of 10,000 and window size of 1,024. The recurrent architecture, input/output transforms, base loop-count distribution, and TBPTL depth remain unchanged. We continue the pretraining objective on a long-context data blend similar to Nemotron 3.5 Super ([24]).

Unlike the NVFP4-to-BF16 continuation, we load model weights only and reset the optimizer, scheduler, RNG, and iteration state. We train for 47,683 optimizer steps at a constant learning rate of $10^{-5}$ without warmup. Reducing the global batch size from 192 to 24 sequences preserves 6,291,456 tokens per step. We use 192 GPUs across 48 nodes with eight-way context parallelism, 24-way data parallelism, a microbatch size of one, and no tensor or pipeline parallelism. All-gather communication accommodates the sliding-window attention implementation. With 24 data-parallel replicas, the unchanged 0.1 override setting selects two replicas (approximately 8.3%) for one-loop execution after integer rounding.

Base Model Evaluations

We evaluate Doryl.0-4B Base after the long-context extension. The base model has seen approximately 6.3T training tokens. Table 1 reports ten benchmarks at fixed inference loop counts $k \in\{1, 2, 4, 6, 8\}$. Here, $k$ is the number of passes of the shared 40-layer recurrent block. All columns use the same model weights, with training-time rank-based loop overrides disabled. Training samples one to four loops; the six- and eight-loop evaluations measure inference-depth extrapolation.

Task	Metric	Dory1.0-4B Base
	Inference loops	1	2	4	6	8
General Knowledge
MMLU (Str)	5-shot EM	68.20	70.77	71.19	71.22	71.17
MMLU-Pro	5-shot CoT EM	51.52	55.06	56.09	55.72	55.86
Mathematical Reasoning
GSM8K	8-shot CoT EM	86.43	89.99	89.84	89.01	89.99
MATH-500	4-shot greedy EM	60.60	65.00	69.40	68.40	69.40
Code Generation
HumanEval	0-shot pass@1, $n = 64$	58.33	59.76	62.34	60.52	61.90
MBPP-Sanitized	3-shot pass@1, $n = 30$	57.09	61.50	60.89	59.13	61.73
Commonsense Reasoning
ARC-Challenge (Llama)	25-shot $acc_{\mathrm{norm}}$	87.63	89.08	88.74	89.85	88.91
HellaSwag	10-shot $acc_{\mathrm{norm}}$	78.52	79.53	79.76	79.75	79.91
WinoGrande	5-shot acc	74.11	76.24	77.43	77.35	77.82
Reading Comprehension
RACE-H†	0-shot acc	83.06	85.26	86.03	86.03	85.74

Evaluation implementation. We use NVIDIA’s fork of the LM Evaluation Harness² through the Megatron-LM completions server. The model receives raw base-model prompts without a chat template or system instruction. All evaluations use the full test or validation split.

1. General knowledge. MMLU (Hendrycks et al., [12]) uses the string-generation protocol: five demonstrations, greedy decoding capped at two tokens, and exact matching of the extracted answer letter. We report the question-weighted aggregate across 57 subjects and 14,042 test questions, not the subject-macro average or a likelihood-based MMLU score. MMLU-Pro (Wang et al., [35]) uses five-shot chain-of-thought prompting on 12,032 questions, greedy decoding capped at 1,024 tokens, and exact match on the extracted answer.

2. Mathematical reasoning. GSM8K (Cobbe et al., [7]) uses eight-shot chain-of-thought prompting over 1,319 test problems. MATH-500 (Hendrycks et al., [13]; Lightman et al., [18]) uses four fixed demonstrations over 500 test problems. Both use one greedy completion per problem, a 1,024-token generation limit, and exact match after answer extraction.

3. Code generation. HumanEval ([4]) uses zero-shot prompts for 164 problems and 64 completions per problem, with a 512-token generation limit. MBPP-Sanitized ([2]) uses three demonstrations for each of 257 test problems and 30 completions per problem, with a 768-token generation limit. Both sample at temperature 0.6 and top-$p = 0.95$. We apply the EvalPlus sanitizer ([19]), but execute the original HumanEval or sanitized-MBPP tests, not the additional HumanEval+ or MBPP+ tests. Pass@1 is the mean fraction of passing completions per problem, not the fraction of problems with at least one passing sample.

4. Commonsense reasoning. We evaluate ARC-Challenge ([6]) using the Llama-style 25-shot task on 1,172 test questions, HellaSwag ([37]) using ten-shot prompting on 10,042 validation examples, and WinoGrande ([31]) using five-shot prompting on 1,267 validation examples. These are multiple-choice likelihood evaluations. We report acc_norm for ARC-Challenge and HellaSwag, and acc for WinoGrande.

5. Reading comprehension. RACE-H ([17]) uses the high-school test split with 1,045 passage-level examples. Only the final question of each passage is scored, with the preceding questions and their gold answers included in the prompt and no additional demonstrations. Scores are therefore not directly comparable to all-question RACE results.

Post-Training

Our post-training pipeline (Figure 3) combines a heavy supervised fine-tuning (SFT) stage and a light 200-step reinforcement learning with verifiable rewards (RLVR) stage. Initialized from the long-context extended checkpoint, SFT adapts Dory1.0-4B to instruction following using mixed thinking and non-thinking responses. The RLVR stage begins from the 9-epoch SFT checkpoint and optimizes the model using task-specific rewards across multiple environments. Both stages train the shared recurrent block at multiple loop counts. While SFT maintains the variable-loop setup from pretraining and context extension, RLVR dynamically routes harder prompts (lower pass rates) to deeper loops with higher probability.

Supervised Fine-Tuning

Starting from the long-context extended checkpoint, we perform SFT in BF16 using Megatron-LM on packed sequences of 262,144 tokens with a global batch size of 128 and a microbatch size of one. We retain the optimizer configuration used during long-context extension: distributed Adam with $(\beta_1, \beta_2) = (0.9, 0.95)$, a constant learning rate of $10^{-5}$ without warmup, no weight decay, and gradient-norm clipping at 1.0. As in long-context extension, SFT uses eight-way context parallelism, with no tensor or pipeline parallelism. Training runs on 32 nodes with 128 GB300 GPUs and a one-loop override fraction of 0.1 on top of uniform loop-count distribution for $k \in\{1, 2, 3, 4\}$. We use TBPTL with depth one, as in pretraining. We select the SFT checkpoint at iteration 32,400 to initialize RLVR.

The SFT data blend is the same as the text blend for post-training Nemotron 3.5 Super,³ and packed offline to a maximum sequence length of 262,144 tokens. The blend contains both thinking and non-thinking responses. For thinking examples, the prompt supplies an open <think> tag, and the model generates a reasoning trace followed by </think> and the final response. For non-thinking examples, the prompt supplies an empty <think></think> block, and the model generates the response directly. The chat template delimits messages with <|im_start|> and <|im_end|> and exposes tool definitions in the system message. Tool calls use XML-style <tool_call> blocks while tool results are wrapped in <tool_response> blocks within user-formatted messages. We optimize token-level cross-entropy on assistant tokens, including the reasoning trace and closing </think> tag in thinking examples, while masking system, user, and tool tokens from the loss.

Extended SFT yields continued improvements across multiple benchmarks evaluated with one loop. Between iterations 3,600 and 32,400 (epochs one and nine), SciCode improves from 13.11% to 18.14%, Terminal-Bench v2.1 from 4.2% to 13.5%, and GPQA Diamond from 43.13% to 54.95%. Over the same interval, IFBench increases from 28.94% to 38.50%, MATH-500 from 60.2% to 75.2%, and AA-LCR from 29.5% to 37.3%. These results show that longer SFT continues to improve coding, agentic, reasoning, and instruction-following performance at a fixed one-loop inference budget.

RLVR

Environments and rewards. We train across 23 single-step environments covering coding, agentic tasks, reasoning, long-context question answering, instruction following, structured outputs, safety, and other environments. For LLM-based reward, we use Qwen3-235B-A22B-Instruct-2507⁴ for scoring the policy responses.

Reward profiling and data selection. Before RLVR training, we profile candidate prompts using the SFT checkpoint at iteration 32,400 with one-loop execution. For each prompt, eight responses are collected and graded with the corresponding environment’s reward function. We retain prompts whose mean reward is strictly between 0 and 1, excluding extremes of consistently zero reward and consistently full reward. Each prompt’s profiled pass rate (or mean profiling reward) is preserved for the difficulty-aware loop curriculum introduced later. We then re-balance the profiled data by oversampling targeted environments and filter prompts to fit the RLVR training context. The final RLVR data mixture comprises ∼47% agentic tasks, ∼27.5% reasoning and coding, ∼8.1% long-context question answering, ∼17.4% instruction following, safety, and other domains.

RLVR training setup. Starting from the SFT checkpoint, we train for 200 steps using asynchronous group relative policy optimization (GRPO) ([33]) in NeMo RL ([22]). Each step samples 384 prompts and generates 16 responses per prompt, yielding a global batch size of 6,144 rollouts. We train in BF16 with distributed Adam, using $(\beta_1,\beta_2)=(0.9,0.999)$, a constant learning rate of $4 \times10^{-6}$ without warmup, no weight decay, and gradient-norm clipping at 1.0. Rollout workers use temperature 1.0 and top-$p$ = 1.0 within the 65,536-token context window. We launch separate vLLM rollout workers each with a fixed loop count $k$ from $\{1,2,3,4\}$. Within each global batch, prompt quotas follow weights of 4:1:1:1 for $k=1,2,3,4$, respectively. Each prompt is assigned a loop count before rollout expansion, and all 16 rollouts in its GRPO group use that same count.

RLVR with difficulty-aware loop curriculum. The mean profiling reward of each prompt from the reward-profiling stage serves as a proxy for “difficulty.” In the RLVR stage, we adopt a difficulty-aware loop-routing curriculum, where “easier” prompts are routed with higher probability to rollout workers with fewer loops. Intuitively, two routing strategies are interpolated: (a) ordering prompts by difficulty so that harder prompts receive deeper execution, and (b) assigning prompts independently of their pass rates. Within each prompt batch, we compute the midpoint quantile $q_i$ of each prompt’s profiled score, assigning the same quantile to tied values. We interpolate this quantile with a uniform random draw $u_i$ to form a routing score,

Prompts with lower routing scores fill deeper-loop slots first. At $\alpha= 1$, routing follows difficulty order, with random tie-breaking; at $\alpha= 0$, it is a uniformly random assignment while obeying the specified prompt loop distribution (e.g., 4:1:1:1 in our case). We use $\alpha= 0.3$ to bias routing toward deeper execution for harder prompts while retaining substantial randomness. Figure 4 illustrates how the interpolation changes prompt assignments while preserving the loop quotas.

We compare RLVR with and without the difficulty-aware loop curriculum using an earlier SFT checkpoint. Both runs use the same 4:1:1:1 prompt loop distribution; the curriculum run uses $\alpha= 0.3$ (Figure 5). The loop curriculum improves all six benchmarks at both one and four loops, with the largest gains on agentic tasks at four loops: GDPval-AA v2 improves by 98 Elo, $\tau$-Banking by 4.9 points, and Terminal-Bench v2.1 by 7.4 points.

Evaluation

We evaluate Dory1.0-4B on benchmarks spanning agentic capabilities, reasoning and knowledge, instruction following, factual reliability, and long-context understanding. Our evaluation follows a recipe similar to that of Nemotron 3.5 Super and Lightning (NVIDIA, [24]). Evaluation runs are orchestrated through NeMo Evaluator.⁵ Most benchmarks use NeMo Gym;⁶ Terminal-Bench v2.1 and SWE-Bench Verified use the evaluator backend with Harbor⁷ sandboxes on AWS ECS. For benchmarks with small task sets, we evaluate multiple independent responses per task to reduce sampling variability and obtain more reliable score estimates.

Unless otherwise specified, all models in Table 2 are evaluated using an identical evaluation protocol for each benchmark, including the evaluation harness, task configuration, and scoring procedure. These benchmark scores may differ from those published by Artificial Analysis. The shared protocol makes the benchmark scores and averages comparable across models within our evaluation; externally reported results are explicitly marked.

Coding and agentic capabilities. We evaluate coding and agentic capabilities using SWE-Bench Verified ([15, 5]) for software engineering agents; Terminal-Bench v2.1 ([21]) for terminal-based task execution; GDPval-AA v2 ([27]) for economically valuable real-world workplace tasks; $\tau^{3}$-Banking ([3]) for conversational tool use in banking; and SciCode ([34]) for scientific coding. We report SWE-Bench Verified as pass@1 (the fraction of issues resolved by the generated patch), Terminal-Bench v2.1 as pass@1, GDPval-AA v2 in Elo, $\tau^{3}$-Banking as task success rate, and SciCode as pooled subtask accuracy.

Reasoning and general knowledge. We evaluate expert-level knowledge and reasoning using GPQA Diamond ([30]) and HLE ([28]), both without tools. We measure factual reliability using the judge accuracy and non-hallucination metrics of AA-Omniscience ([14]). The latter is one minus the proportion of incorrect answers among all non-correct responses, including partially correct and abstained answers.

Long-context understanding. We evaluate long-context reasoning using AA-LCR ([1]), which measures information retrieval, synthesis, and reasoning over long inputs, and report judge accuracy (the fraction of responses judged correct). We also evaluate GDP.pdf ([10]), which tests reasoning over long professional documents with task-specific grading criteria. As Dory1.0-4B is a text-based model, we use text-only document inputs and report the fraction of responses satisfying every grading criterion, averaged over five repeats per task.

Main results. Table 2 compares Dory1.0-4B at inference loop counts $k \in\{1, 2, 4\}$ with Qwen3.5-4B,⁸ Qwen3.5-9B,⁹ Gemma 4 12B,¹⁰ Gemma 4 26B-A4B,¹¹ and Qwen3.6-35B-A3B¹² across 12 benchmarks. Dory1.0-4B demonstrates improved agentic performance with deeper execution: GDPval-AA v2 increases from 815 to 967 Elo, SWE-Bench Verified from 50.3% to 61.8%, and Terminal-Bench v2.1 from 12.5% to 22.2% between one and four loops. However, gains from deeper execution are task-dependent: AA-Omniscience accuracy remains essentially unchanged across loop counts. At four loops, Dory1.0-4B outperforms all evaluated baselines on $\tau^{3}$-Banking, IFBench, and AA-LCR, including models with 26B and 35B total parameters. It also exceeds Qwen3.5-9B on SciCode (35.11% vs. 28.21%), while approaching Qwen3.6-35B-A3B on GDPval-AA v2 (967 vs. 987 Elo). These results suggest that increasing recurrent depth enables Dory1.0-4B to compete with larger models on several coding, agentic, and long-context tasks without adding parameters.

Token Efficiency

Additional loops not only improve accuracy on agentic tasks but also reduce the number of tokens the model generates to complete them. Table 3 decomposes the output tokens per attempt into the number of turns per attempt and the output tokens per turn. Going from one to four loops, output tokens per attempt fall by 1.3–2.5×, and both factors contribute, in proportions that depend on the task. On the long-horizon tasks, most of the savings come from completing the task in fewer turns: on GDPval-AA v2, turns per attempt drop from 116 to 49 while output tokens per turn change by only 5%, and on Terminal-Bench v2.1, turns drop by 39% and tokens per turn by 20%. On SWE-Bench Verified, the two factors contribute comparably, with 17% fewer turns and 11% fewer tokens per turn. On $\tau^3$-Banking, where the conversation length is largely set by the simulated user, the number of turns changes by only 7%, and most of the savings come from 18% shorter turns. Thus, with additional loops, Dory1.0-4B both takes fewer actions to complete a task and writes less per action. These savings offset much of the higher compute per token. Approximating generation compute as output tokens times layer applications per token (72, 112, and 192 at $k = 1, 2, 4$), two loops cost only 1.03–1.25× the compute of one loop across the four tasks, yet recover most of the accuracy gain of four loops: 86% of the GDPval-AA v2 Elo gain, 80% on SWE-Bench Verified, 58% on Terminal-Bench v2.1, and 42% on $\tau^3$-Banking. Four loops, by contrast, cost 1.07–2.04× the compute of one loop. The second loop therefore gives the largest accuracy gain per unit of added compute, making $k = 2$ a good default, with $k = 4$ for tasks where accuracy matters most.

Task	Turns per attempt			Output tokens per turn			Output tokens per attempt
	$k = 1$	$k = 2$	$k = 4$	$k = 1$	$k = 2$	$k = 4$	$k = 1$	$k = 2$	$k = 4$
GDPval-AA v2	116	76	49	1,832	1,836	1,740	212k	140k	85k
Terminal-Bench v2.1	134	100	82	815	773	655	109k	78k	54k
SWE-Bench Verified	88	76	73	384	349	341	33.6k	26.4k	24.8k
$\tau^3$-Banking	21.3	20.1	19.8	1,081	919	890	23.0k	18.5k	17.6k

Quantization

Overview

We study post-training quantization (PTQ) of Dory1.0-4B, followed by quantization-aware distillation (QAD) (Xin et al., [36]) where needed, to create efficient quantized versions of the trained BF16 checkpoint that preserve its accuracy. We evaluate three weight-only quantization formats: MXFP8, NVFP4, and Q5_K_M (GGUF). For each format, we first apply PTQ to all weight matrices except the token embeddings and LM head, then apply QAD if the quantized model falls short of the target accuracy. All learned vectors and scalars remain in high precision.

Quantization-Aware Distillation (QAD)

Post-Training Quantization (PTQ)

We apply PTQ using a custom implementation of NVIDIA ModelOpt ([26] (NVIDIA Corporation 2026)). For NVFP4, we use dynamic MSE scale selection: for each block, we select the scale that minimizes the MSE across all 126 positive E4M3 values. For the other formats we use standard maximum scaling. We find that MXFP8 achieves accuracy comparable to the BF16 baseline, while NVFP4 and Q5_K_M exhibit accuracy gaps that we aim to reduce through QAD.

Data blend. We reuse the original SFT mixture from § 5.1 and create a weighted subsample of 10B raw input tokens, with $\sim$3B stored assistant mask tokens.

QAD setup. The frozen teacher is the original BF16 checkpoint corresponding to the student. The student is the PTQ version and applies the selected weight quantizer in every forward pass. Both models receive the same SFT token sequence and the same recurrent-loop count.

With teacher and student distributions $p_T$ and $p_S$, respectively, we minimize full-vocabulary forward KL at temperature one:

Here $m_t$ marks valid assistant next-token targets. Prompt, system, user, document, padding, and invalid conversation-boundary targets do not contribute to the objective, although their context still affects assistant predictions.

During QAD, we unfreeze all parameters, including embeddings, the LM head and vectors. For NVFP4, global and block scales are recomputed from the current weights on each quantized forward. We follow the original SFT-style training regime including the loop sampler, truncated backpropagation through loops (TBPTL) with depth one and matrix-normalization after each optimizer update. We use AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, and a cosine schedule with linear warmup and a peak learning rate of $10^{-5}$. We use a global batch of 128 packed sequences, a maximum context of 256K tokens, eight-way context parallelism, and 16-way data parallelism. We run QAD on 128 NVIDIA GB300 GPUs for 12 hours for each of the candidates.

Results

Quantization results. Table 4 compares the BF16 baseline with MXFP8-PTQ, NVFP4-QAD, and Q5_K_M-QAD at inference loop counts $k \in\{1, 2, 4\}$. MXFP8-PTQ remains broadly comparable to BF16 without additional distillation. The QAD variants also retain competitive performance across the benchmark suite, largely preserving the BF16 model’s capabilities at lower weight precision, with some task-dependent accuracy gaps.

Task	BF16			MXFP8-PTQ			NVFP4-QAD			Q5_K_M-QAD
	1	2	4	1	2	4	1	2	4	1	2	4
Coding & Agentic
SciCode	27.30	34.94	35.11	27.78	32.68	33.98	26.78	33.81	34.85	25.48	33.51	34.29
Terminal-Bench v2.1	12.5	18.1	22.2	11.94	17.56	22.19	15.03	19.24	20.08	15.03	19.24	18.26
$\tau^{3}$-Banking	11.60	12.89	14.69	12.78	12.78	14.43	12.16	15.88	14.85	12.16	12.78	15.05
GDPval-AA v2 (Elo)	815	946	967	856	961	985	759	929	965	804.5	952.8	985.2
Reasoning
GPQA Diamond (no tools)	69.19	72.41	72.16	69.51	70.77	71.91	68.06	72.16	72.35	68.69	72.54	72.66
HLE (text-only, no tools)	8.25	10.89	10.43	9.18	10.43	10.98	8.67	9.92	10.66	9.5	10.38	11.91
Long Context
AA-LCR	63.06	71.31	72.94	65.69	71	71.19	57.5	69.5	69.12	62.88	70.44	70.12
General Knowledge
AA-Omniscience (accuracy)	10.03	10.08	10.03	10.17	9.93	9.7	10.2	10.4	10.2	9.85	9.7	10.05
AA-Omniscience (non-hallucination)	76.42	79.74	79.38	77.44	81.96	82.24	75.13	79.95	80.31	76.17	80.29	82.25

Performance. Table 5 reports measured inference speedups for Dory1.0-4B on NVIDIA DGX Spark for MXFP8 and NVFP4 datatypes relative to the BF16 baseline. Quantization accelerates inference by up to 2.44× over the BF16 baseline.

Input sequence length (ISL, tokens)	Output sequence length (OSL, tokens)	Batch size	MXFP8 (W8A16) speedup vs. BF16	NVFP4 (W4A16) speedup vs. BF16
2560	300	1	1.69×	2.17×
128	1024	1	1.84×	2.44×

Inference Optimization

We explore two ways to improve inference efficiency in Dory1.0-4B. Last-loop KV caching makes attention KV memory independent of loop count, while self-speculative decoding uses shallow drafting and deeper verification to accelerate generation. We evaluate these optimizations separately, using per-loop KV caches for self-speculative decoding.

Last-Loop KV Caching

With standard per-loop caching, each recurrent attention layer maintains a separate KV cache for every loop: a loop-$i$ query attends to earlier tokens’ loop-$i$ keys and values. The recurrent block’s KV memory therefore grows linearly with the loop count $k$, while the input and output blocks each retain one cache. Last-loop KV caching (Ladhak et al. [16]) instead maintains a single cache per recurrent attention layer, overwriting its entries at each loop. After each forward step, the cache retains the final-loop keys and values. Queries attend to final-loop entries for previously cached tokens and current-loop entries for tokens processed in the same step. This keeps attention KV memory at its one-loop footprint for every $k$, without retraining, but exposes earlier-loop queries to final-loop keys and values rather than the same-loop entries used during training.

We compare both cache policies on the final post-trained checkpoint at $k = 4$, with $k = 1$ as a reference, where the policies coincide (Table 6). Serving uses prefix caching and chunked prefill. Last-loop caching retains gains from additional loops, with score changes of $-2.12$ to $+0.42$ points relative to per-loop caching at $k = 4$. Mean output lengths change little, from 1.0% fewer tokens on GPQA Diamond to 3.4% more on Terminal-Bench v2.1. Last-loop caching therefore makes attention KV memory independent of loop count while retaining gains from deeper execution, with little change in generation length.

Task	Metric	Per-loop $k = 1$	Per-loop $k = 4$	Last-loop $k = 4$	$\Delta$ Score	Output Tokens
$\tau^{3}$-Banking	task success rate, $n = 8$	11.60	14.69	14.30	−0.39	1.00×
Terminal-Bench v2.1	pass@1, $n = 8$	12.50	22.19	20.22	−1.97	1.03×
GPQA Diamond	accuracy, no tools, $n = 8$	69.19	72.16	72.10	−0.06	0.99×
SciCode	subtask accuracy, $n = 8$	27.30	35.11	35.37	+0.26	1.01×
AA-Omniscience (accuracy)	judge accuracy, $n = 10$	10.03	10.03	10.45	+0.42	1.03×
AA-Omniscience (non-halluc.)	$1 -$ hallucination rate, $n = 10$	76.42	79.38	79.25	−0.13	1.03×
AA-LCR	judge accuracy, $n = 16$	63.06	72.94	70.81	−2.12	1.02×

Self-Speculative Decoding

Dory1.0-4B uses its variable execution depth for self-speculative decoding, with the same checkpoint drafting at $k = 1$ and verifying at $k = 4$. The drafter proposes $\gamma= 4$ tokens autoregressively, which the verifier evaluates in a batched forward pass. Rejection sampling uses both models’ probabilities to accept proposals and correct rejections, preserving the verifier’s sampling distribution in exact arithmetic without a separate draft model.

With per-loop caching, drafting and verification share the KV caches of the input block and first recurrent loop. We also reuse the corresponding draft activations during verification, avoiding redundant computation for token positions already processed by the drafter. Drafting and verification execute sequentially, so the acceleration comes from accepting multiple tokens per verification step and reusing computation.

We first measure steady-state decode throughput across batch sizes and prompt lengths (Figure 6). With KV and activation reuse, self-speculation achieves $1.37$–$1.71 \times$ the throughput of autoregressive decoding at $k = 4$ across all measured configurations. Gains generally decrease at larger batch sizes but persist through batch size 256 for shorter prompts and batch size 64 for 16K-token prompts.

Self-speculative decoding reduces mean prefill-plus-decode request duration relative to autoregressive decoding at $k = 4$, using the same checkpoint, benchmark sampling settings, and serving configuration (Table 7). Across six benchmarks, self-speculative decoding achieves $1.11$–$1.53 \times$ speedups in mean prefill-plus-decode request duration, with gains depending on the workload and serving conditions.

Task	Prefill + Decode Speedup
$\tau^{3}$-Banking	1.11×
Terminal-Bench v2.1	1.17×
GPQA Diamond	1.34×
SciCode	1.35×
AA-Omniscience	1.53×
AA-LCR	1.44×

Conclusion

We present Dory1.0-4B, a 4-billion-parameter looped language model with elastic depth, in which a single checkpoint serves different inference compute budgets by varying the number of times its shared recurrent block is applied. Dory1.0-4B uses a normalized Transformer backbone with interleaved sliding-window and full-attention layers. It was pretrained on 6 trillion tokens, the first 4.5 trillion in NVFP4, extended to a 256K-token context, and post-trained with SFT and RLVR using a difficulty-aware loop curriculum. Additional loops improve accuracy, with the largest gains on coding, agentic, and long-context benchmarks; with two or more loops, Dory1.0-4B outperforms larger models such as Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks. Looping also makes Dory1.0-4B more token-efficient: four loops generate 1.3–2.5× fewer tokens than one loop on agentic benchmarks, and the second loop gives the largest accuracy gain per unit of added compute. Last-loop KV caching keeps the KV-cache footprint equal to that of a non-looped model at any loop count, and MXFP8, NVFP4, and GGUF quantized checkpoints further reduce deployment cost. We open-source the BF16 and quantized checkpoints of Dory1.0-4B.