WorksComputer Science
Late Entry in Looped Language Models
Abstract
Looped language models deepen reasoning at a compact parameter count by iterating a shared layer stack T times per token, turning recurrence depth directly into sequential decoding latency. Conventional early-exit decoding halts after initial passes without cache completion, truncating cached key-value states of past tokens to shallow depth. We study the opposite strategy, late entry, where new tokens skip initial passes and enter directly into the final K < T passes over a deep context cache. Once lightweight fine-tuning resolves hidden-state norm mismatch at the entry boundary, deep cached context substitutes for skipped token-local passes, allowing late entry to outperform matched early exit across model families and pass budgets. To turn this asymmetry into wall-clock speedups without stalling to build deep caches, candidate blocks verify through only the final K passes on the critical path while previously committed tokens complete passes 1 -> T alongside subsequent steps at zero extra serial calls. Projecting these finished T-pass wake states inside the existing output GEMM (Wake verification, governed by logit-margin tolerance tau in [0, inf]) unifies greedy-exact T-pass decoding (tau = 0, matching K = T block decoding within bf16 reduction order), margin-gated verification (tau > 0), and non-rollback lossy late entry (tau = inf) inside a single static CUDA graph. On Ouro-2.6B (T = 4), K = 2 Wake reaches 5.2x greedy-exact speedup at tau = 0 (+26% over K = 4 block decoding on GSM8K) and 6.8x at tau = inf (+16.9 points over trained early exit at matched speed). Scaling test-time recurrence across T in {8, 16, 32, 64} on Huginn-3.5B cuts K = T/4 rollbacks 21.5x with depth, driving greedy-exact (tau = 0) speedups to 6.1x (T = 32) and 8.4x (T = 64, +153% over K = 64 block) and tau = inf speedups to 10.6 to 12.5x.
Introduction
Looped language models scale test-time compute without increasing model size by passing each token repeatedly through a shared Transformer stack for recurrent passes (Dehghani et al. [8]; Geiping et al. [12]; Zhu et al. [30]). During autoregressive decoding, however, running all passes per generated token turns recurrence depth directly into sequential latency. When each new token is restricted to serial passes, which passes should it execute? As shown in Figure 1, conventional early exit without cache completion (Schuster et al. [23]) runs the first passes and halts (Figure 1b), leaving past key-value (KV) caches truncated at shallow depth so all future tokens attend to immature context states. We explore the opposite choice, late entry, which skips initial passes and feeds each new token’s embedding directly into the final passes over a deep context KV cache (Figure 1c).
Figure 1. Figure 1. Early exit vs. late entry in looped language models. (a) Full depth (■■■■): runs all passes. (b) Early exit (■□□□): halts after pass 1, truncating context KV caches to shallow depth. (c) Late entry (□□□■): skips passes 1 to 3 on the new token (clips) and enters pass 4 over a full-depth context cache completed in parallel (§ 3). (d) Unified Wake and late-entry frontier (GSM8K, , single H100): reusing completed -pass wake states lifts throughput beyond the block baseline with greedy-exact fidelity at (5.2×, +26% on Ouro-2.6B; 6.1 to 8.4×, +82% to +153% on Huginn-3.5B across ) and reaches 6.8× and 10.6× (12.5× on MBPP) at .
At first glance, skipping initial passes seems destined to fail: on off-the-shelf Ouro-1.4B and Ouro-2.6B () (Zhu et al. [30]), feeding raw embeddings into a late pass breaks predictions immediately because late passes expect normalized hidden states from earlier passes. Once lightweight fine-tuning aligns late passes to accept raw embeddings, however, a clear asymmetry emerges between token depth and context depth: one final pass over a four-pass context matches full-depth predictions more closely than two initial passes over a two-pass context (+10+ accuracy points in sequential decoding). Under identical training data, hyperparameters, and steps, spending serial passes at the end of the recurrence consistently outperforms spending them at the beginning across both Ouro scales (; § 2) and 32-pass middle-recurrent Huginn-3.5B (; § 4.4), showing that deep cached context retains much of the benefit of recurrent depth.
Turning this finding into wall-clock speedups requires building deep context states without stalling generation. In late-entry block decoding paired with Wake verification (§ 3), parameter-free candidate blocks (Liu et al. [20]; Jin et al. [16]) execute only the final passes on the critical path to emit a tentatively accepted frontier and provisional KV states, while earlier tentative tokens complete true passes 1 through as piggybacked batch rows at zero extra serial calls (using spare per-call batch capacity in memory-bound decoding rather than reducing FLOPs). Because that wake finishes exact causal -pass states steps later, projecting them inside the same output GEMM (Wake verification, § 3.5) promotes matching tokens into a permanently verified -pass prefix and rolls back rare mismatches, unifying greedy-exact (), margin-gated (), and lossy () decoding in one CUDA graph.
Key Takeaway
In looped language models, deep cached context substitutes for token-local early passes. Whereas existing looped speculative decoders (Ha and Kim [14]; Cho et al. [6]; Li et al. [18]; Liu et al. [22]) accelerate drafting via shallow initial passes while keeping full -pass verification on every accepted token, Late Entry + Wake Verification compresses critical-path verification itself to the final passes over a pipelined full-depth cache and reuses completed 1 wake states in the same output GEMM to preserve -pass greedy equivalence () or trade controlled margin tolerance () for higher throughput.
Our study is organized around three concrete contributions:
Pass-allocation asymmetry and error localization. Across Ouro-1.4B, Ouro-2.6B (), and Huginn-3.5B (), adapted late entry consistently outperforms matched early exit; fp32 interventions trace this gain to pass-coherent context pairs and show that discrepancies concentrate inside the 6% uncertain tail (§ 2).
Unified Late Entry + Wake Verification pipeline. We pair -pass critical-path block verification with cross-step cache completion and same-GEMM Wake verification, proving exact -pass greedy equivalence and finite-step termination at while spanning margin-gated () and lossy () regimes in one engine (Proposition 1; § 3 and § 3.5).
Scaling across benchmarks, recurrence depths, and hardware load. Across 21,403 examples on Ouro-2.6B () and (36 260 effective layers) on Huginn-3.5B, Wake verification delivers 5.2× to 8.4× greedy-exact () speedups and up to 12.5× lossy () speedups as rollbacks drop 21.5× with depth, while an Roofline law calibrated strictly on predicts unseen multi-request throughput across at 8.3% MAPE (§ 4 and § 4.7).
Early Exit vs. Late Entry: Where Should Passes Be Spent?
Before building an inference system, we isolate a central pass-allocation question under idealized context conditions: when each new token is limited to serial passes, is it better to spend them on the initial passes or on the final passes? We study this question on Ouro-1.4B (24 Transformer layers) and Ouro-2.6B (48 Transformer layers), two pretrained looped language models that iterate a shared layer stack for passes with pass-specific KV caches and untied input-output embeddings ([30]). For clarity, § 2 and § 3 develop our analysis and pipeline on Ouro before § 4.4 scales both to 32-pass middle-recurrent Huginn-3.5B ().
In free autoregressive generation, a single altered prediction changes every subsequent token, confounding single-step accuracy with trajectory drift. To measure per-token prediction quality cleanly, we first evaluate teacher-forced agreement (Appendix § 7). We generate reference completions greedily at full depth (), replay the exact same token prefixes under a reduced-pass configuration, and record the fraction of positions whose greedy next-token prediction matches full depth.
Making Late Entry Feasible
What happens if we apply late entry directly to an off-the-shelf looped language model without any adaptation? We test the most extreme case on released Ouro-1.4B and Ouro-2.6B: preceding context tokens retain their full four-pass KV caches, while the current token skips passes 1 to 3 and feeds its raw token embedding directly into pass 4 (). As shown in the first row of Table 1, prediction agreement with full depth drops to .552 on Ouro-1.4B and .751 on Ouro-2.6B, well below simply truncating the entire model to pass 1 (.938 and .949).
| Input state supplied to pass 4 ( over four-pass context) | Released Ouro-1.4B | Released Ouro-2.6B |
| Raw token embedding (unadapted late entry) | .552 | .751 |
| Closed-form linear projection of the token embedding | .816 | .919 |
| True pass-3 hidden state (diagnostic oracle ceiling) | .991 | .992 |
| Reference: 1-pass early exit (pass 1 only for all tokens) | .938 | .949 |
Table 1. Table 1. Diagnosing raw late-entry failure on off-the-shelf purely looped checkpoints. Each row reports GSM8K teacher-forced next-token agreement with full-depth () decoding across 78,693 token positions on Ouro-1.4B and 70,101 positions on Ouro-2.6B. In the top three rows, the current token executes only pass 4 () over a four-pass context cache; the bottom row runs pass 1 only for both token and context. A closed-form linear map recovers 60% to 70% of the gap to the pass-3 oracle without changing a single model weight.
Inspecting internal activations reveals why raw late entry fails: pass 1 expects raw token embeddings (median norm 0.76), whereas pass 4 expects layer-normalized hidden states emitted by pass 3 (norms 37 to 51). To test whether this failure reflects an input representation mismatch rather than missing early-pass computation, we fit a closed-form linear projection from raw token embeddings to pass-4 inputs on held-out states while keeping all model weights frozen. This linear probe alone raises agreement to .816 on Ouro-1.4B and .919 on Ouro-2.6B (Table 1), recovering 60% and 70% of the gap to the diagnostic oracle that supplies the true pass-3 state (.991 and .992).
Still, frozen input adapters on an unadapted model plateau at or below one-pass truncation (; extended adapter diagnostics in Appendix § 9). To align late passes with raw embeddings directly, we fine-tune Ouro on Dolci-Instruct-SFT ([26]) in two one-epoch stages (Appendix § 7): Stage 1 runs standard full-depth instruction SFT, and Stage 2 exposes half of the training segments directly to raw token embeddings at pass () over a full-depth prefix cache (jointly formatted for parallel block decoding in § 3.2; full recipe in Appendix § 7. Ablations in Appendix § 8 confirm that late-entry pass exposure itself resolves the mismatch: Stage 2 lifts plain-embedding late-entry agreement to .962 on Ouro-2.6B, compared with .699 after Stage 1 alone and .738 under identical Stage-2 training without late-entry exposure. With the input mismatch removed, we can now test whether deep context compensates for skipped token passes.
Deep Context Compensates for Skipped Token Passes
Why can a single late pass predict so accurately once the model is adapted? As illustrated in Figure 2a, passes 1 through progressively refine prefix representations and store multi-pass contextual reasoning inside the deep KV cache . Once Stage 2 aligns pass to form valid attention queries directly from a raw token embedding , the new token can read out that distilled four-pass context in a single pass rather than re-deriving depth locally, whereas early exit forces every attention head to read immature prefix states (concurrently, Li et al. [18] show in a training-free drafting setting that pairing shallow initial-pass queries with full-depth prefix values improves draft predictions). To test this mechanism quantitatively on fine-tuned Ouro-2.6B, we independently vary two quantities under teacher forcing: the cached context depth , where all preceding tokens run passes to populate the pass-aligned KV cache, and the current-token pass budget , where the current token enters at pass and executes passes . Under this definition, runs passes 1 and 2 on both token and context (standard two-pass early exit), whereas enters directly at pass 2 over a two-pass context.
Figure 2. Why a single adapted late pass suffices: deep context KV substitutes for token-local passes on fine-tuned Ouro-2.6B. (a) Mechanism: under shallow early exit (, , ▪▪▫▫), token executes two passes but attends only to unrefined two-pass prefix caches; under adapted late entry (, , ▪▪▪▪), multi-pass contextual reasoning is already stored in the four-pass prefix cache, allowing a single aligned pass 4 to query deep context states directly from . (b) Teacher-forced decoupling: varying current-token passes (executing passes ) and context depth on GSM8K ( is 1.000 by definition) reveals that one pass over deep context (.962) beats two passes over shallow context (.954). (c) Free decoding: in single-token greedy generation at , completing four-pass context caches lifts task accuracy by +10.4 points on GSM8K and +12.0 points on IFEval.
Figure 2b isolates each factor in turn. First, holding the current token’s budget fixed at passes, deepening the context cache from to raises agreement from .954 to .984. Second, holding a full-depth context fixed at , reducing the current token from (.992) to (.984) and even (.962) retains high agreement. Together, these two effects produce a clear cross-diagonal reversal: running just pass over a four-pass context (.962) outperforms running passes over a two-pass context (.954). Fine-tuned Ouro-1.4B exhibits the same reversal, reaching .948 at versus .922 at with non-overlapping bootstrap confidence intervals.
Small per-step differences compound substantially over free-running generation (Figure 2c). To isolate the end-to-end impact of context depth without block drafting, we run diagnostic one-token-at-a-time greedy decoding at passes per token. Truncating every token and its cache to the first two passes () scores .749 on GSM8K and .588 on لفEval. Running the final two passes per token while synchronously completing each emitted token’s four-pass cache before the next step () lifts accuracy to .853 (+10.4 points) on GSM8K and .708 (+12.0 points) on IFEval.
Why late passes work and where errors remain. Inspecting internal attention and hidden-state geometry in fp32 across 30,736 GSM8K and 53,184 IFEval positions (Figure 3; full attention-entropy, sink-mass, and distance metrics in Appendix § 9) reveals two structural properties. First, Stage-2 exposure aligns hidden states directly onto the pass-4 manifold (linear CKA .961 vs. .273 to .365 unadapted), and cross-pass KV interventions confirm that late queries require coherent same-pass pairs rather than deep values alone: pairing pass-1 keys with pass-4 values collapses predictions to <> at 100% of positions (.000 agreement), while pairing pass-4 keys with pass-1 values drops agreement to .296 (Figure 3a; tokens more than 64 positions back supply 53% of IFEval’s total context-depth gain). Second, stratifying positions by full-depth top-1 confidence (Figure 3b) shows that on the 68% of GSM8K tokens where full depth is confident (), adapted late entry already agrees at .998 () and 1.000 (), confining the remaining 1.6% discrepancy at to the 6% uncertain tail (, .827 agreement). This concentration motivates both halves of § 3: completing exact context caches in a cross-step pipeline so generation never stalls (§ 3.3), and reusing those finished wake states to catch rare uncertain-tail slips without slowing down the other 98%+ of tokens (§ 3.5).
Figure 3. Pass-coherent addressing and confidence localization on Ouro-2.6B (fp32 probes across 30,736 GSM8K positions). (a) Cross-pass interventions: swapping keys and values across passes collapses agreement to .000 (Pass-1 + Pass-4 ) or .296 (Pass-4 + Pass-1 ), whereas Stage-2 alignment reaches linear CKA .961 with true pass-4 states and .962 () to .984 () agreement over coherent deep caches. (b) Confidence-binned agreement: adapted late entry reaches 1.000 () and .998 () on the 68% of confident tokens (), isolating residual discrepancies to the 6% uncertain tail () targeted by Wake verification.
Late Entry vs. Early Exit Across Scales and Matched Fine-Tuning
We first compare early-exit and late-entry pass placements on the exact same Stage-2 checkpoints across all reduced budgets , both model sizes, and both GSM8K and IFEval. Table 2 compares truncating to the initial passes (*First , depth- context) against entering for the final passes (*Last , four-pass context). Across all 12 cells, spending passes at the end beats spending them at the beginning on identical weights, with margins of +6.4 to +14.1 points at . Note that each agreement score is measured against that checkpoint’s own full-depth output: although unadapted Ouro-2.6B showed .949 one-pass self-agreement in Table 1 versus .898 here, the unadapted base model achieves only .729 full-depth GSM8K accuracy (.442 on IFEval; Appendix § 8), lower than even two-pass truncated decoding on our adapted model (.747 and .597).
| Passes / token | Ouro-2.6B (48 layers, ) | Ouro-1.4B (24 layers, ) | ||||||
| GSM8K | IFEval | GSM8K | IFEval | |||||
| First | Last | First | Last | First | Last | First | Last | |
| (🟧□□ vs. □□🟦) | .898 | .962 | .757 | .887 | .842 | .948 | .743 | .884 |
| (🟧🟧□ vs. □🟦🟦) | .954 | .984 | .866 | .943 | .922 | .976 | .839 | .941 |
| (🟧🟧🟧 vs. 🟦🟦🟦) | .982 | .992 | .949 | .969 | .967 | .990 | .938 | .968 |
Table 2. Same-checkpoint comparison of First versus Last passes across scales and tasks. Entries report teacher-forced next-token agreement with full-depth () decoding on identical Stage-2 checkpoints (70,101 GSM8K positions on Ouro-2.6B; 28,939 GSM8K and 61,723 IFEval positions on Ouro-1.4B). “First ” (🟧□□) runs passes over a depth- context; “Last ” (□□🟦) runs passes over a four-pass () context.
Because Table 2 evaluates checkpoints exposed to late-entry blocks during Stage 2, we next test whether giving early exit its own dedicated Stage-2 training closes the gap. Starting from the exact same Stage-1 checkpoint with identical data order, optimization steps, and random seed, we train a matched early-exit model whose reduced-budget blocks execute the first passes instead of the last (Appendix § 7). Dedicated early-exit training raises truncation agreement on Ouro-2.6B from .898 to .925 on GSM8K and from .757 to .798 on IFEval (while unexposed late entry stays at .755 and .619), yet it still trails trained late entry by wide margins (.962 vs. .925 on GSM8K and .887 vs. .798 on IFEval), confirming across both identical weights and matched fine-tuning that cached context depth outweighs token-local early passes.
Late-Entry Decoding
While § 2 assumed idealized full-depth context states, a practical decoder must construct those states for newly generated tokens without stalling generation. Late-entry decoding achieves this by maintaining deep context with a short pipeline lag across decoding steps. For a concrete mental model at and (previewed in Figure 5), each decoding step performs three coordinated actions:
Figure 5. Multi-step pipeline of late-entry decoding (, ). Each step executes serial forward calls. Tentatively accepted token group A ( tokens) runs late-entry passes 3 and 4 in Step 1 (writing provisional KV states P3*, P4*), and then recomputes true passes as parallel batch rows inside the calls of Step 2 (passes 1, 2) and Step 3 (passes 3, 4), adding zero extra serial forward calls.
Tentative late-pass verification: new candidate tokens skip passes 1 and 2 and execute only passes 3 and 4 ( serial calls) to emit a tentatively accepted token group.
Provisional KV exposure: newly accepted tokens temporarily populate slots 3 and 4 with provisional KV states computed from their embeddings so the next step can proceed immediately without stalling.
Pipelined cache completion and Wake verification: over the next two decoding steps, those tentative tokens ride along inside ongoing forward calls to compute true passes 1 → 2 and 3 → 4 at zero extra serial calls, overwriting provisional KV entries and verifying finished -pass states inside the same output GEMM to advance a permanently verified -pass prefix.
We formalize looped pass mechanics in § 3.1, describe within-step block verification in § 3.2, detail cross-step cache completion in § 3.3, and unify greedy-exact and lossy decoding via Wake verification in § 3.5 (hardware Roofline and concurrency scaling follow in § 4.7).
Looped Forward Pass and Late-Entry Formulation
Given a prefix , let denote the input embedding of token , and let denote the shared stack of Transformer layers iterated across passes . At pass , maps the preceding normalized hidden state (initialized at ) over the pass-aligned KV cache of earlier positions :
where denotes layer normalization. When a token executes a contiguous range of passes , the model’s output readout maps the executed states through the unembedding projection to next-token probabilities without requiring hidden states from skipped passes (via a pass-invariant stick-breaking gate normalized over on Ouro, Appendix § 7, Equation (4); or directly through a non-recurrent Coda on middle-recurrent Huginn, § 4.4).
Conventional early exit runs (🟧🟧⬜) and leaves empty for . Late entry instead runs the final passes (⬜⬜🟦) of Equation (1), feeding directly into pass while attending to context caches for .
Notice why pairing early exit with background cache completion does not yield a distinct competitive baseline. Under Equation (1)’s pass-aligned routing, passes read only and never attend to , so completing passes in the background leaves Ouro’s -pass early-exit predictions bit-for-bit unchanged (on middle-recurrent Huginn, where only the 2-layer Coda reads , supplying an oracle 32-pass cache lifts 8-pass early exit only from .263 to .288 vs. .396 for late entry; Appendix H, Table 15). Conversely, rewiring those passes to query deep caches instead of collapses algebraically into late entry itself because all recurrent passes share identical weights (including Ouro’s pass-invariant exit gate), and forfeiting true states on the critical path means background completion must still compute all passes.
Table 15. Single-token sequential GSM8K evaluation on Huginn-3.5B (, batched greedy decoding; gaps computed prior to rounding). “Shallow ” runs Core passes on both token and context; “Oracle-cache early” runs initial passes over a 32-pass context; “Late entry” runs final passes over a 32-pass context (full depth scores .396 at 132 serial layers/token, vs. 4 + serial layers/token at budget ).
| Core budget | Serial layers / tok | Shallow | Oracle-cache early | Late entry | Token agree. w/ | Late vs. () |
| 20 (6.6×) | .039 | .043 | .315 | .957 | +27.7 () | |
| 36 (3.7×) | .263 | .288 | .396 | .989 | +13.3 () | |
| 52 (2.5×) | .350 | .368 | .393 | .995 | +4.3 () | |
| 68 (1.9×) | .378 | .383 | .400 | .997 | +2.2 (.020) |
Table 15.
Late-Pass Block Verification
To emit multiple tokens per step and amortize cache completion cleanly, we use an existing parameter-free mask-placeholder block decoder ([20, 16]) as our execution vehicle. We retain its single-call parallel verification and lookahead layout unchanged, modifying only which recurrent passes execute on the critical path (passes instead of ) and how committed KV caches are completed afterward.
Figure 4 illustrates one step for , , and block size . Let denote the last committed anchor token (“3”) and let denote candidate tokens proposed for the upcoming positions (“red”, “apples”). To draft the next step’s proposals inside the same forward calls, we append a branch of mask-initialized placeholders behind and behind each candidate , forming parallel rows in total. A structured causal mask restricts candidate to attend only to and so verification never leaks future tokens, while the -th placeholder behind attends to that prefix plus its preceding sibling placeholders to look positions ahead.
Figure 4. Inside a single late-entry block step (, , block size ). The last committed anchor token (“3”) and candidate proposals (“red”, “apples”) each append a branch of mask placeholders, forming parallel rows that enter directly at pass 3. Prefix verification accepts “red” (✓), replaces the rejected proposal with bonus token “balls” (×), and reuses the placeholder predictions behind “red” (“and”, “2”) as draft proposals to be verified from anchor “balls” in the next step.
All rows enter directly at pass and execute only the final passes ( serial forward calls). After pass , we evaluate the greedy late-entry prediction at each position. Let be the length of the longest matching prefix where . We tentatively accept those proposals plus one bonus token (“balls” in Figure 4), advancing the tentatively accepted frontier by tokens, and read the next step’s draft proposals directly from the placeholder branch behind accepted position with zero separate drafting calls. Because that placeholder branch was evaluated alongside rather than the newly realized bonus token , its outputs serve strictly as approximate draft candidates; the next step sets as the new anchor and verifies those proposals from scratch.
Pipelined Cache Completion and Cost Profile
When tentatively accepted tokens advance the frontier after running only passes , their KV cache slots for passes are still empty, and slots hold provisional keys and values computed from raw embeddings (P3’s, P4’s in Figure 5). Pausing generation to recompute passes serially would add extra serial calls per step and erase the latency savings of late entry.
Wake Verification: Unifying Exact and Lossy Late Entry
Instead, we complete full-depth KV caches via cross-step pipelining (Figure 5). Here, asynchronous refers to delayed cache completion across decoding steps inside the same GPU forward calls, rather than a separate background CUDA stream. Because a looped language model shares identical layer weights across all passes, in-flight tentative token groups and the active -row candidate block can execute different pass indices side by side as parallel batch rows inside the same forward calls, each attending to its own pass-aligned cache. Each tentative group advances through true passes at passes per step, finishing within steps (2 steps at , , scaling to 4 steps at , in § 4.4; when , zero catch-up rows are scheduled).
Crucially, this schedule preserves exact causal KV dependencies. Because tentative groups enter the wake queue in chronological order and advance passes per step, whenever a group recomputes pass , every earlier group has already written its exact pass- KV states during a prior step. In Step 3 of Figure 5, for example, Group B computes passes 1 and 2 while attending to Group A’s true pass-1 and pass-2 caches completed in Step 2. Meanwhile, active candidate blocks briefly read provisional KV entries (P3*, P4*) for the one or two most recent groups until true pass- states overwrite them.
Why does appending these catch-up rows add minimal wall-clock latency? It is important to distinguish serial calls from arithmetic work: zero extra serial calls does not mean zero extra FLOPs, because tentatively accepted tokens execute verification passes plus wake passes. In single-sequence (batch-1) decoding, however, GPU latency is memory-bandwidth bound rather than compute bound: once layer weights are streamed from memory for a forward call, extra token rows incur little marginal latency (on Ouro-2.6B on an NVIDIA H100 GPU, forwarding 32 parallel tokens takes only 9% longer than forwarding 1 token). At , , and , the active block has rows while at most two groups of tokens catch up simultaneously, adding at most 18 rows (18% of a 99-row batch; all reported speeds include this work). Higher request concurrency gradually saturates compute under a fixed large unless is reduced alongside load (analyzed via our H100 Roofline law in § 4.7 and Appendix § 16). Finally, cache-completion ablations in Appendix § 13 verify both halves of this design: reading provisional KV states for one or two steps costs zero measurable accuracy compared with synchronous before-next-step repair ( to .77), whereas omitting cache completion altogether drops GSM8K accuracy at by 4.1 points (.839 .798; § 4.4 and Appendix H examine how multi-stage lag scales at ).
Wake Verification: Unifying Exact and Lossy Late Entry
Look closely at what pipelined cache completion in § 3.3 already computes. At any step, generation partitions into a permanently verified -pass prefix (, holding exact KV states) followed by at most in-flight groups along the tentatively accepted frontier. Whenever the oldest tentative group () finishes pass at the back of the wake, every earlier position already holds its exact KV cache and positions have just executed exact causal passes . Hence the finished wake hidden states (or Coda states on Huginn) are mathematically identical to sequential -pass forward states. Whereas basic lossy late entry discards those states after writing KV caches, Wake verification projects them through inside the exact same single output GEMM as the active block rows (90 to 99 rows vs. 81 at , adding zero Transformer layer calls, zero extra kernel launches, and zero extra CUDA graphs).
Figure 6 walks through a concrete rollback at , . Suppose prefix "2 + 3" () is permanently verified. In Step 1, critical-path passes 3 and 4 tentatively accept Group A = ["=", "6"] at positions , where querying from $u = "=" without passes 1 and 2 slipped and guessed . In Step 2, the frontier advances at full speed with [".", " The"] while Group A piggybacks on the same calls to compute exact passes 1 and 2. In Step 3, Group A finishes passes 3 and 4 (writing exact $1 \to 4$ KV states on "=") and projects inside the shared lm_head GEMM, revealing exact with logit gap . On of positions where , Group A simply joins the permanently verified prefix with zero rollback; when a mismatch exceeds tolerance , Wake truncates the tentative tail beyond and commits exact token "5" at .
Figure 6. Worked 3-step walkthrough of Wake verification and rollback (, ). Step 1: over permanently verified prefix "2 + 3" (), passes 3 and 4 tentatively accept Group A = ["=", "6"] at , slipping on position \hat{y}_u = "6"$ instead of "5"). Step 2: the frontier tentatively emits [".", " The"] while Group A computes exact passes 1 and 2 inside the same forward calls. Step 3: Group A finishes passes 3 and 4 and projects inside the same single lm_head GEMM; detecting (), truncates the tentative tail beyond , retains exact KV states on "2 + 3 =" (), commits "5" at , and resumes.
Formally, let denote the exact -pass logit vector at wake position , let be the true -pass greedy next token, and let be the token tentatively accepted on the -pass path. Parameterized by logit-margin tolerance , Wake verification triggers a rollback at the earliest index satisfying
When Equation (2) fires, the decoder truncates the tentative frontier beyond , commits exact -pass token at position (retaining all exact KV caches already written by the wake), resets the wake queue to , and resumes tentative -pass decoding from anchor . When the tentative frontier emits <eos> or hits the token limit while wake groups remain in flight, the decoder runs wake-drain steps before halting so premature stop tokens are verified as well.
Proposition 1: Exactness and Termination of Wake Verification. In exact arithmetic, Wake verification at with EOS wake drain produces bit-for-bit identical token sequences to sequential -pass greedy decoding for any and , and terminates in at most verified steps:
Inductive prefix exactness: By induction on completed wake groups, every position equals the sequential -pass greedy prefix and holds exact KV states. Causal masking within ensures is exact; if for all , then is likewise exact. Hence either the entire group matches sequential -pass decoding, or rollback fires at the exact first mismatch and replaces with the exact token .
Strict progress: Every rollback advances the permanently verified -pass prefix from to , ruling out livelock, while wake drain verifies the final groups before termination (verified across 258/258 adversarial fp64 tests spanning divisible and non-divisible and premature-<eos> sequences).
Varying along a single continuous knob unifies three regimes inside one engine: (i) (Exact Wake) guarantees -pass greedy equivalence in exact arithmetic (and matches block decoding within bf16 kernel reduction order at 0.007% to 0.09% token discrepancy, below the 0.25% to 0.33% sequential bf16 floor; Appendix § 11) while running the critical path at serial passes whenever tentative guesses agree (> 98% of positions; § 2.2); (ii) (Margin-Gated Wake) ignores benign near-tie bf16 or phrasing flips () while rolling back confident reasoning deviations (); and (iii) recovers non-rollback lossy late entry. Finally, because Wake verification and late entry modify only which passes verify a block, they factorize cleanly from how candidate tokens are drafted: pairing the identical verifier with a second parameter-free 4/3/2-gram context suffix drafter ([10, 17]) alongside trained mask-placeholder proposals directly tests whether the throughput multiplier from holds across distinct acceptance regimes at zero extra batch rows or model parameters.
Experiments
We organize our evaluation around four progressive questions:
Pure pass-placement gain without rollback (, § 4.1 to § 4.4): Before activating Wake rollbacks, how much speedup and accuracy does non-rollback late entry achieve on its own across six benchmarks (21,403 items) on Ouro-2.6B (), against matched trained early exit and full-depth () speculative decoders, and on 32-pass middle-recurrent Huginn-3.5B ()?
Recovering greedy-exact fidelity (, § 4.5): When Wake verification reuses finished wake states inside the same output GEMM, how fast are greedy-exact () and margin-gated () decoding across models and candidate drafters?
Scaling with recurrence depth (, Section 4.6): Holding Huginn-3.5B weights fixed, how do Wake rollback rates and speedups scale as test-time loop depth grows from 36 to 260 effective layers?
Hardware Roofline and multi-request serving (, § 4.7): Why do wake rows stay nearly free at batch 1, and how does throughput scale as concurrent load saturates GPU compute?
Unless noted in § 4.7, all runs use standard single-sequence inference (batch size 1, greedy decoding, bf16 precision, single NVIDIA H100 80GB GPU) across the full test sets of six benchmarks totaling 21,403 examples: GSM8K (1,319 math word problems; Cobbe et al., [7]), IFEval (541 strict instruction-following prompts; Zhou et al., [29]), MATH500 (500 competition math problems; Hendrycks et al., [15]; Lightman et al., [19]), MBPP (500 Python synthesis tasks; Austin et al., [1]), BBH (6,511 multi-step reasoning problems; Suzgun et al., [25]), and MMLU-Pro (12,032 ten-choice questions; Wang et al., [27]), scored via lm-evaluation-harness (Gao et al., [11]) with gold-verified MATH500 and MBPP extractors (Appendix § 14). Accuracy differences use two-sided paired sign tests. In every single-engine table (Table 3, Table 6, Table 7, and Table 8), speedups and accuracy gaps are paired against full-depth sequential decoding measured inside the same job on the same GPU (minor sequential reference differences such as .873 at 63.4 tokens/s in Table 3 vs. .868 at 62.3 tokens/s in Table 7 reflect 6/1,319 bf16 reduction-order ties across engine harnesses; Appendix § 11); in the cross-engine comparison of Table 5, all methods are normalized by the fastest sequential engine across the three codebases (62.4 to 67.2 tokens/s; Appendix § 14) so no external baseline is penalized by harness overhead.
| Benchmark | Acc. | Full depth | (full) Gap | (full) Speedup | (late entry) Gap | (late entry) Speedup (vs. ) | (late entry) Gap | (late entry) Speedup (vs. ) | |
| GSM8K | 1,319 | .873 | −0.6 | 4.05× | −2.2† | 5.09× (+26%) | −3.4† | 6.73× (+66%) | |
| IFEval | 541 | .732 | +0.7 | 2.16× | +0.2 | 2.82× (+31%) | −3.3† | 3.78× (+75%) | |
| MATH500 | 500 | .532 | −0.6 | 4.06× | +0.4 | 4.99× (+23%) | −4.0† | 6.49× (+60%) | |
| MBPP | 500 | .616 | +1.0 | 3.65× | +0.2 | 4.48× (+23%) | −1.4 | 5.80× (+59%) | |
| BBH | 6,511 | .577 | 0.0 | 3.30× | −1.3† | 4.30× (+30%) | −2.8† | 6.16× (+87%) | |
| MMLU-Pro | 12,032 | .504 | +0.1 | 3.49× | −0.3 | 4.45× (+28%) | −1.6† | 5.91× (+69%) |
Table 3. Full-dataset accuracy and throughput on Ouro-2.6B () across six benchmarks at block size (21,403 total items, single H100 GPU). “Full depth” is sequential accuracy. “Gap” is accuracy minus full-depth sequential accuracy in percentage points († marks paired sign-test ). “Speedup” divides decode tokens/s by same-job full-depth sequential tokens/s, while (+X%) isolates the incremental throughput from late entry over block decoding.
| Benchmark | Full depth (tok/s) | LoopSpec | LoopAhead | Ours | ||
| GSM8K | 67.2 (.873) | 2.84× | 2.94× | 3.82× .867 | 4.80× .851 | 6.35× .839 |
| IFEval | 64.7 (.732) | 2.34× | 2.67× | 2.12× .739 | 2.75× .734 | 3.68× .699 |
| MATH500 | 62.7 (.532) | 2.85× | 2.59× | 4.11× .526 | 5.07× .536 | 6.63× .492 |
| MBPP | 65.1 (.616) | 2.86× | 2.90× | 3.64× .626 | 4.51× .618 | 5.95× .602 |
| BBH | 65.2 (.577) | 2.62× | 3.25× | 3.29× .577 | 4.35× .564 | 6.33× .549 |
| MMLU-Pro | 62.4 (.504) | 2.75× | 2.44× | 3.56× .504 | 4.56× .501 | 6.12× .488 |
Table 5. Full-depth () speculative decoders versus non-rollback () late entry on Ouro-2.6B across all six benchmarks (21,403 total items, single H100 GPU). Speedups divide each method’s decode tokens/s by the fastest measured full-depth sequential decoder on that benchmark. LoopSpec, LoopAhead, and our block baseline run full-depth verification; and () trade bounded accuracy shifts (small numbers report full-set accuracy, with full-depth sequential accuracy beside each denominator) for higher throughput, while Section 4.5 activates Wake to surpass with greedy-exact fidelity.
| Benchmark | Full depth () Acc. | Full depth () tok/s | (full block) Gap | (full block) Speedup | (late entry) Gap | (late entry) Speedup (vs. ) | (late entry) Gap | (late entry) Speedup (vs. ) | |
| GSM8K | 1,319 | .393 | 21.7 | +0.4 | 3.37× | +1.1 | 5.84× (+74%) | −3.9† | 9.13× (+171%) |
| IFEval | 541 | .551 | 21.6 | 0.0 | 2.93× | +1.5 | 5.24× (+79%) | −0.9 | 8.96× (+206%) |
| BBH | 6,511 | .283 | 22.6 | +0.2 | 3.79× | +0.3 | 6.65× (+75%) | −1.0† | 10.86× (+186%) |
| MBPP | 500 | .234 | 22.4 | +0.2 | 4.46× | −0.2 | 7.76× (+74%) | −0.6 | 12.41× (+178%) |
Table 6. Full-dataset accuracy and throughput on Huginn-3.5B () across all four capability-active benchmarks at block size (8,871 total items, single H100 GPU, unified CUDA-graphed paged FlashAttention-3 engine; definitions match Table 3).
| Benchmark | Budget | Seq. Acc. / (tok/s) | Block Acc. | Spd. | Exact Wake (, greedy-exact) Acc. | rb | Spd. (vs. ) | Margin Wake () Acc. () | rb | Spd. (vs. ) | Lossy () Acc. / Spd. |
| GSM8K | .868 (62.3) | .867 | 4.13× | .867 | 1.14 | 4.19× (+2%) | .870 (.25) | 0.29 | 4.55× (+10%) | .853 / 5.18× | |
| GSM8K | .868 (62.3) | .867 | 4.13× | .867 | 2.02 | 5.20× (+26%) | .877 (.15) | 1.26 | 5.52× (+34%) | .842† / 6.82× | |
| MATH500 | .532 (62.7) | .526 | 4.08× | .526 | 2.11 | 5.23× (+28%) | .548 (.30) | 0.96 | 5.83× (+43%) | .492† / 6.61× | |
| IFEval | .732 (63.8) | .739 | 2.15× | .739 | 5.98 | 2.66× (+24%) | .719 (.15) | 4.08 | 2.94× (+36%) | .699† / 3.76× | |
| MBPP | .616 (63.4) | .626 | 3.79× | .626 | 2.00 | 4.61× (+22%) | .618 (.30) | 0.70 | 5.13× (+35%) | .602 / 6.20× | |
| GSM8K | .391 (21.9) | .394 | 3.35× | .396 | 0.48 | 5.21× (+56%) | .404 (.25) | 0.05 | 5.42× (+62%) | .399 / 5.80× | |
| GSM8K | .391 (21.9) | .394 | 3.35× | .392 | 2.22 | 6.09× (+82%) | .397 (.25) | 0.88 | 7.15× (+114%) | .353† / 9.13× | |
| IFEval | .560 (21.5) | .547 | 2.91× | .547 | 0.37 | 4.85× (+66%) | .560 (.25) | 0.02 | 5.10× (+75%) | .562 / 5.21× | |
| IFEval | .560 (21.5) | .547 | 2.91× | .556 | 2.03 | 6.46× (+122%) | .543 (.25) | 0.59 | 7.59× (+161%) | .542 / 8.85× | |
| MBPP | .234 (21.9) | .236 | 4.47× | .236 | 0.25 | 6.64× (+49%) | .234 (.25) | 0.03 | 6.79× (+52%) | .232 / 7.80× | |
| MBPP | .234 (21.9) | .236 | 4.47× | .236 | 1.41 | 7.69× (+72%) | .236 (.25) | 0.56 | 8.93× (+100%) | .228 / 12.50× |
Table 7. Wake verification spectrum () across Ouro-2.6B () and Huginn-3.5B () at block size (full test sets, single H100 GPU). “rb” denotes Wake rollbacks per 100 emitted tokens; (+X%) measures incremental throughput over same-job full-depth block decoding († marks paired sign-test vs. sequential -pass decoding). Ouro-2.6B ( passes; teacher-forced audit diff: 0.007% GSM8K, 0.09% MATH500, 0.025% MBPP). Huginn-3.5B ( Core passes; unified CUDA-graphed paged FlashAttention-3 engine).
| Depth (Eff. Layers) | Seq. -Pass Acc. | Seq. -Pass tok/s | Spd. | Half Depth () Acc. / rb | Half Depth () Spd. | Half Depth () Acc. / Spd. | Quarter Depth () Acc. | Quarter Depth () rb | Quarter Depth () Spd. | Quarter Depth () Acc. / Spd. |
| (36 layers) | .267 | 78.6 | 2.86× | .265 / 5.56 | 2.63× | .167† / 3.78× | .262 | 12.46 | 1.96× | .030† / 4.82× |
| (68 layers) | .382 | 42.1 | 3.32× | .379 / 1.85 | 4.21× | .361 / 5.15× | .375 | 6.25 | 3.47× | .146† / 6.56× |
| (132 layers) | .391 | 21.9 | 3.35× | .396 / 0.48 | 5.21× | .399 / 5.80× | .392 | 2.22 | 6.09× | .353† / 9.13× |
| (260 layers) | .397 | 11.1 | 3.34× | .397 / 0.26 | 5.42× | .394 / 5.97× | .395 | 0.58 | 8.42× | .392 / 10.57× |
Table 8. Intra-model recurrence depth scaling on Huginn-3.5B across on full GSM8K (, , identical checkpoint and engine). “rb” reports rollbacks per 100 tokens († marks paired sign-test vs. -pass sequential decoding at the same ).
Accuracy and Throughput Across Six Benchmarks
Our total speedup combines two distinct mechanisms: lossless parallel block decoding reduces the number of sequential decoding steps while still executing all passes per step (captured by our block baseline), whereas lossy late entry further cuts the number of serial recurrent passes per step ( and ) at a controlled accuracy trade-off. Figure 7 plots the accuracy-throughput frontiers on GSM8K and IFEval across and block sizes (full grid in Appendix § 10). In Figure 7, the points serve as our lossless full-depth block reference (minor accuracy differences from sequential decoding arise from bf16 reduction order; Appendix § 11), and LoopSpec marks an external lossless speculative reference point on GSM8K ahead of our six-benchmark boundary analysis in § 4.3. Table 3 reports both total speedups over sequential decoding and incremental throughput gains over block decoding at .
Figure 7. Accuracy-throughput frontier of late-entry decoding on Ouro-2.6B across the full GSM8K () and IFEval () test sets for pass budgets and block sizes ( appears in Figure 8 and Appendix § 10). LoopSpec marks its fastest released lossless () reference point on GSM8K.
At , every candidate block executes all four passes starting from pass 1 without skipping. Block decoding alone already accelerates generation by 2.16× on IFEval and up to 4.05× and 4.06× on GSM8K and MATH500, with zero statistically significant accuracy shift on any task (0.01% token discrepancy vs. a 0.27% bf16 floor on GSM8K; Appendix § 11).
Reducing the critical path to isolates the incremental contribution of late entry on top of this block decoder. Skipping pass 1 boosts throughput by an additional 23% to 31% across all six benchmarks (reaching 2.82 to 5.09× total speedup, or 322.5 tokens/s vs. 63.4 tokens/s on GSM8K), while accuracy changes by at most 0.4 points on four of the six tasks (IFEval, MATH500, MBPP, and MMLU-Pro) and trades 1.3 points on BBH and 2.2 points on GSM8K.
Skipping half the passes at raises block throughput by 59% to 87% over (3.78 to 6.73× total speedup; 426.7 tokens/s on GSM8K) with bounded accuracy drops of 1.4 to 4.0 points. Dropping further to , however, triggers a steep accuracy decline (.629 at 544.0 tokens/s on GSM8K) because a single pass both drafts and verifies its own proposals without cross-pass refinement. Thus, (near-full-depth quality) and (high throughput) define the practical operating regime of late-entry decoding.
Controlled Comparison Against Early Exit
Does the late-entry advantage from § 2 carry over to end-to-end block decoding through the identical engine? We test this at two levels: first on identical model weights to isolate execution strategy alone, and second across matched fine-tuned checkpoints to evaluate each paradigm with dedicated Stage-2 adaptation.
On identical weights, we take our adapted Ouro-2.6B checkpoint and compare running its first passes (early-exit truncation) against running its last passes with pipelined cache completion (late entry). At , truncating this checkpoint gives the highest early-exit accuracy we measure on any model (.747 at 412 tokens/s on GSM8K and .597 on IFEval). Even on these exact same weights, late entry at outperforms early-exit truncation by +9.2 points on GSM8K (.839 vs. .747) and +10.2 points on IFEval (.699 vs. .597), and leads by +1.7 and +3.3 points at .
Next, Table 4 and Figure 8 compare our late-entry checkpoint against the matched trained early-exit checkpoint across . Notice first at that forcing shallow passes to emit final predictions during Stage 2 lowers the early-exit model’s full-depth ceiling on GSM8K (.822 vs. .867). Even relative to each checkpoint’s own ceiling, dropping from to costs late entry only 2.8 points on GSM8K (4.0 on IFEval) compared with 15.2 points for trained early exit (13.5 on IFEval; additional adaptive-exit controls in Appendix § 14). Across all eight cells, trained late entry dominates the overlapping speed range, leading by +16.9 accuracy points at matched speed around 430 tokens/s on GSM8K (.839 vs. .669 at ) and by +4.3 points around 180 tokens/s on IFEval (.734 vs. .691 at ).
Table 4. End-to-end comparison of trained early exit and late entry on Ouro-2.6B under matched Stage-2 fine-tuning, decoded through the identical block engine on full test sets († marks paired sign-test ; gaps computed prior to rounding). At , early exit runs passes over a depth- cache, whereas late entry runs passes with pipelined full-depth cache completion.
| Passes / token | Early exit, trained: Acc. | Early exit, trained: tok/s | Late entry, trained: Acc. | Late entry, trained: tok/s | Gap (pts) | Sign-test |
| GSM8K (1,319 test problems) | ||||||
| (🟧□□□ vs. □□□🟦) | .330 | 657 | .629 | 544 | +29.9† | |
| (🟧🟧□□ vs. □□🟦🟦) | .669 | 431 | .839 | 427 | +16.9† | |
| (🟧🟧🟧□ vs. □🟦🟦🟦) | .785 | 314 | .851 | 323 | +6.5† | |
| (full depth) | .822 | 246 | .867 | 257 | +4.5† | |
| IFEval (541 test prompts) | ||||||
| (🟧□□□ vs. □□□🟦) | .442 | 561 | .645 | 386 | +20.3† | |
| (🟧🟧□□ vs. □□🟦🟦) | .586 | 286 | .699 | 239 | +11.3† | |
| (🟧🟧🟧□ vs. □🟦🟦🟦) | .691 | 183 | .734 | 178 | +4.3† | .023 |
| (full depth) | .721 | 138 | .739 | 137 | +1.8 | .34 |
Table 4. End-to-end comparison of trained early exit and late entry on Ouro-2.6B under matched Stage-2 fine-tuning, decoded through the identical block engine on full test sets († marks paired sign-test ; gaps computed prior to rounding). At , early exit runs passes over a depth- cache, whereas late entry runs passes with pipelined full-depth cache completion.
Figure 8. Accuracy versus decode throughput for trained late entry and trained early exit on Ouro-2.6B at (plotting Table 4). Vertical calipers mark matched-throughput operating points (+16.9 points at ∼430 tokens/s on GSM8K; +4.3 and +1.8 points on IFEval).
Comparison with Full-Depth () Speculative Decoders
To situate non-rollback late entry (, ) relative to methods that verify every token through all passes, we evaluate two speculative decoders tailored to looped language models across the full test sets of all six benchmarks (21,403 examples in total): LoopSpec (official implementation, fastest released configuration; [6]) and LoopAhead (best of three reproduced training arms; [22]). Because LoopSpec, LoopAhead, and our block baseline verify every drafted token through all passes, they map out the throughput reachable under full-depth critical-path verification. On each task, every method’s decode tokens/s is normalized by the fastest measured full-depth sequential decoder across all three engines (62.4 to 67.2 tokens/s; protocol details in Appendix § 14).
Table 5 separates full-depth verification from critical-path pass reduction. Within full-depth verification, LoopSpec (2.34 to 2.86×), LoopAhead (2.44 to 3.25×), and our block baseline (2.12 to ) map out the speed range reachable when every step executes all four passes (with leading on five tasks and trailing on IFEval). Reducing critical-path verification to under non-rollback late entry () reaches 2.75 to 5.07× at (≤ 0.4-point accuracy change on four tasks; 1.3 and 2.2 points on BBH and GSM8K) and 3.68 to 6.63× at (1.4 to 4.0-point trade-offs). Moreover, full- critical-path verification is not the ceiling of greedy-exact decoding: as § 4.5 demonstrates next, activating Exact Wake () at also surpasses block throughput (reaching 5.20× on GSM8K and 5.23× on MATH500) while preserving greedy-exact -pass predictions.
Scaling to Deep Middle-Recurrence: Huginn-3.5B ()
To test whether late entry generalizes beyond shallow four-pass recurrence and Ouro’s stick-breaking readout, we evaluate Huginn-3.5B (huginn-0125; Geiping et al., [12]), a middle-recurrent architecture comprising a 2-layer non-recurrent Prelude, a 4-layer Core block iterated times via a learned input-injection adapter, and a 2-layer Coda (132 effective Transformer layers per token, with only 8 unique parameter layers). Under late entry at budget , each candidate block executes the 2-layer Prelude once, skips the initial Core passes to enter directly at pass from Huginn’s standard initialized latent state (, drawn from a fixed-seed truncated Gaussian) over a 32-pass context cache, and exits through the 2-layer Coda; previously committed tokens complete true Core passes and Coda across subsequent steps without recomputing Prelude KV states (Appendix H, Equation (5)). Following our Ouro protocol, we run one epoch of full-depth Stage-1 SFT on Dolci-Instruct-SFT followed by matched 7,406-step Stage-2 block fine-tuning () for both late entry and a dedicated early-exit control, and benchmark all sequential, late-entry, synchronous, and early-exit decoders inside a unified CUDA-graphed paged FlashAttention-3 engine so wall-clock comparisons are free of dispatch overhead (Appendix H). We evaluate the four benchmarks in our suite on which the 8-layer huginn-0125 base model exhibits non-trivial capability (GSM8K, IFEval, BBH, and MBPP; 8,871 test items), omitting ten-choice MMLU-Pro and competition-level MATH500 where the base model lacks task capability and scores near chance. Notably, because Prelude already maps raw embeddings into normalized contextual anchors and re-injects at every Core step, Huginn provides a natural architectural validation of § 2.1’s norm-mismatch diagnosis: even on the early-exit-trained checkpoint (which never saw late entry in Stage 2), late entry beats that checkpoint’s own trained early exit on GSM8K (.327 vs. .271, ), before dedicated late-entry training aligns Core to and lifts accuracy to .354.
Figure 9 and Table 6 confirm that all three properties observed at strengthen at . First, under teacher forcing (Figure 9a), just final Core passes (1/8 of the recurrence) reach .954 agreement with full 32-pass predictions, surpassing 8 trained early-exit passes (.944), while single-token late entry matches 32-pass GSM8K accuracy with zero degradation (.396 vs. .396, +13.3 points over 8-pass truncation; Appendix H, Table 15). Second, whereas at parallel block verification supplies the larger share of total speedup, scaling to (21.6 to 22.6 tokens/s sequential) inverts this balance so late entry drives the majority of wall-clock gain (Table 6): halving Core depth to adds +74% to +79% over block decoding (5.24 to 7.76× total) with zero statistically significant accuracy drop on any task ( reaches 6.44 to 9.75× within 0.6 points; Appendix H), and cutting to adds +171% to +206% (8.96 to 12.41× total, up to 278.3 tokens/s on MBPP with no significant drop on IFEval or MBPP).
Figure 9. Generalization to 32-pass middle-recurrent Huginn-3.5B (, matched 7,406-step Stage-2 checkpoints). (a) Teacher-forced probe: across all pass budgets on GSM8K, late entry over a 32-pass context outperforms trained early exit, with 4 late passes (.954) exceeding 8 trained early-exit passes (.944). (b) End-to-end GSM8K accuracy-throughput frontier (, ): inside our unified CUDA-graphed paged FlashAttention-3 engine, late entry outperforms trained early exit over a broad range of practical operating points, reaching 5.84× (126.8 tokens/s, .405) at , 7.27× (157.8 tokens/s, .395) at $K = 12, 9.13× (198.1 tokens/s, .354) at four-stage , and 9.93× (215.5 tokens/s, .369) under a two-stage partial-cache variant (+9.7 points over trained early exit at tokens/s).
Across matched wall-clock speeds, late entry consistently dominates trained early exit (Figure 9b; full grid in Table 16): on GSM8K, late entry (.395 at 157.8 tokens/s) is both faster and more accurate than 16-pass early exit (.372 at 140.3 tokens/s), while leads 8-pass early exit by +8.3 points (), and on IFEval late entry leads by +4.8 points around 225 tokens/s (). The sole non-rollback trade-off at on GSM8K (−3.9 points) stems from four-stage pipeline lag leaving three recent blocks provisional: shrinking the lag via a two-stage partial-cache variant (17 32) recovers half the gap at 9.93 (+9.7 points over 8-pass early exit; Table 17), while activating Exact Wake () in § 4.5 eliminates this lag completely.
Table 16. Trained early exit versus late entry on Huginn-3.5B () under matched Stage-2 fine-tuning inside the unified CUDA-graphed paged FlashAttention-3 engine at block size across full GSM8K () and strict IFEval (), following Table 4 († marks paired sign-test ; gaps computed prior to rounding; “ (2-stg partial)” completes passes 17 32 and is compared against 8-pass early exit at matched 220 tokens/s).
| Core passes / token | Early exit, trained | Late entry, trained | Gap (pts) | Sign-test | ||
| Acc. | tok/s | Acc. | tok/s | |||
| GSM8K (1,319 test problems; full-depth sequential: .393 at 21.7 tok/s) | ||||||
| (4-stg full) | .271 | 223 | .354 | 198 | +8.3† | |
| (2-stg partial) | .271 | 223 | .369 | 215 | +9.7† | |
| .342 | 174 | .395 | 158 | +5.3† | ||
| .372 | 140 | .405 | 127 | +3.3† | .015 | |
| (full) | .382 | 73 | .397 | 73 | +1.5 | .28 |
| IFEval (541 test prompts; full-depth sequential: .551 at 21.6 tok/s) | ||||||
| .407 | 364 | .530 | 268 | +12.4† | ||
| .490 | 284 | .560 | 223 | +7.0† | .0018 | |
| (4-stg full) | .512 | 227 | .542 | 193 | +3.0 | .16 |
| (2-stg partial) | .512 | 227 | .553 | 205 | +4.1 | .053 |
| .567 | 158 | .545 | 139 | −2.2 | .30 | |
| .549 | 119 | .566 | 113 | +1.7 | .48 | |
| (full) | .543 | 62 | .551 | 63 | +0.7 | .79 |
Table 16.
| Cache-completion policy ( of 32) | Stages (lag) | GSM8K () tok/s (speedup) | GSM8K () Acc. | IFEval () tok/s (speedup) | IFEval () Acc. | |
| Synchronous (blocking , CUDA-graphed) | ( blks) | () | n/a | n/a | ||
| 2-stage partial (passes ) | ( blk) | () | () | |||
| 3-stage partial (passes ) | ( blks) | () | () | |||
| 4-stage full (passes , default) | ( blks) | () | () | |||
| Never (keep provisional 8-pass cache) | n/a () | () | () |
Table 17. Cache-completion lag and depth ablation on Huginn-3.5B at , , across full GSM8K (; sequential is at 21.7 tokens/s) and strict IFEval (; sequential is at 21.6 tokens/s; gaps prior to rounding). Unlike 4-stage full completion (), the 2- and 3-stage policies are partial-cache approximations that omit early passes to shrink the provisional lag window, narrowing the GSM8K gap against synchronous repair from 3.3 to 1.9 points while increasing throughput.
Unifying Lossless and Lossy Decoding via Wake Verification
Sections 4.1 to 4.4 established non-rollback late entry () as a high-speed lossy regime, leaving small accuracy trade-offs at aggressive compression (−2.7 to −4.0 points at on Ouro math tasks, and −3.9 points under four-stage lag at on Huginn GSM8K). We now evaluate Wake verification (§ 3.5), which projects the finished cache-completion wake states inside the same output GEMM to sweep at zero extra layer calls. Figure 10 and Table 7 report full-dataset accuracy, decode throughput, rollback rates per 100 emitted tokens (rb), and teacher-forced fidelity audits across Ouro-2.6B () and Huginn-3.5B ().
Figure 10. Unified Wake verification spectrum () and drafter factorization on GSM8K (, ). (a) Ouro-2.6B (): Exact Wake at lifts throughput +26% beyond the block baseline (4.13 5.20) with identical .867 accuracy and 0.007% committed-token discrepancy, while margin gating () reaches 5.52 to 5.77 (+34% to +40%) at .877 to .864 accuracy before connecting smoothly to (6.82). (b) Huginn-3.5B (): Exact Wake eliminates four-stage lag completely, reaching 5.21 (+56%) at , 6.09 (+82%) at , and 7.15 (+114%) at with zero accuracy drop.
Table 7 highlights three findings. First, at , Wake verification accelerates decoding beyond the block baseline while preserving greedy-exact -pass fidelity: on Ouro-2.6B, Exact Wake matches block accuracy to four decimal places across tasks (with committed-token audit discrepancies of 0.007% to 0.09%, below the 0.25% to 0.33% sequential bf16 floor) while adding +22% to +28% incremental throughput over (5.20 on GSM8K, outpacing even non-rollback lossy at 5.18, and 5.23 on MATH500). On Huginn-3.5B (), because late-entry predictions already agree on of positions, rollbacks stay sparse (0.25 to 2.22 per 100 tokens), lifting greedy-exact speedups to **4.85 to 6.64 (+49% to +66% over block) at ( reaches 5.87, +75%) and **6.09 to 7.69 (+72% to +122% over block) at , completely eliminating four-stage GSM8K lag (.353 .392). Second, margin gating () filters benign near-tie flips (), cutting rollbacks 2× to 16× and sustaining full accuracy at 5.52 to 5.83× on Ouro GSM8K and MATH500 (5.77× at and 6.02× at on GSM8K, ) and at 5.10 to 6.79× () and 7.15 to 8.93× () on Huginn. Third, testing our second parameter-free 4/3/2-gram context-suffix drafter alongside trained mask placeholders confirms the drafter-verifier factorization analyzed in § 4.7 (Figure 10, violet curves): although unconditional short-ngram override slightly lowers raw block acceptance on math and code relative to trained 48-layer placeholder lookahead (3.93 vs. 5.02 tokens/step on Ouro GSM8K; 4.49 vs. 5.34 on Huginn MBPP; while matching or slightly outpacing it on Ouro IFEval at 2.68× vs. 2.66×), the multiplicative speedup gain from Wake verification holds or strengthens across both drafters (1.37× at and 1.48× at vs. 1.26× and 1.40× on Ouro GSM8K; 1.89× vs. 1.72× on Huginn MBPP; 2.25× vs. 2.22× on Huginn IFEval).
Intra-Model Recurrence Depth Scaling ()
Comparing Ouro-2.6B () and Huginn-3.5B () leaves recurrence depth partially confounded with model family and readout architecture. Because Huginn’s recurrent Core shares weights across iterations and generalizes to arbitrary test-time loop counts ([12]), we isolate directly by evaluating the exact same Huginn-3.5B checkpoint across four recurrence depths (36, 68, 132, and 260 effective Transformer layers per token) on full GSM8K () inside the identical CUDA-graphed engine.
Figure 11 and Table 8 reveal a clean depth-scaling law. As test-time recurrence doubles from , sequential GSM8K accuracy rises monotonically (.267 .382 .391 .397) while sequential throughput halves at each step (78.6 11.1 tokens/s). Across all four depths, full-depth block speedup saturates flatly between 2.86× and 3.35× because block acceptance is bounded by single-step predictability rather than . By contrast, late entry scales nearly linearly with : consistent with progressive late-pass stabilization as recurrence deepens ([12]), rollbacks per 100 emitted tokens fall 21.5× across at both (5.56 1.85 0.48 0.26) and
Figure 11. Intra-model recurrence depth scaling on Huginn-3.5B across (36 260 effective Transformer layers/token, identical checkpoint and engine, full GSM8K ). (a) Decode speedup vs. : while full-depth block speedup plateaus near 3.3×, greedy-exact Wake scales from 1.96× () to 8.42× (, +153% over block) and late entry reaches 10.57×. (b) Wake rollback collapse: rollbacks per 100 emitted tokens fall 21.5× across at both (12.5 0.58) and (5.6 0.26).
(12.46 6.25 2.22 0.58; Figure 11b). At shallow , Core passes is too shallow for unverified readout and drops to .030; yet Wake catches every deviation and restores .262 accuracy (+23.2 points), throttling speed to 1.96× when rollback cost exceeds the threshold derived next in § 4.7. Once , greedy-exact Wake surpasses the ceiling and climbs to 5.42× () and 8.42× (, +153% over block, with at 9.17×) at , where even non-rollback () retains .392 accuracy () at 10.57× speedup.
When Does Late Entry Remain Fast? Roofline and Multi-Request Scaling
Finally, to explain from first principles why piggybacked wake rows stay nearly free at batch 1 and how throughput scales as concurrent serving load saturates GPU compute (full continuous-batching sweeps in Appendix I), we model single-pass GPU latency for active query rows across concurrent requests via an Roofline law:
where is single-pass HBM weight-streaming latency, is KV-cache read overhead, is Tensor-Core compute time, and is net committed tokens per step after Wake rollbacks. Calibrating ( ms, ms, ms/row, ) exclusively on full-depth modes ( points, zero latency fits) locates the hardware ridge point at rows/pass (Figure 12a). Equation (3) ties together three empirical behaviors observed across § 4.1 to 4.6: (1) at , , so wake completion and Wake verification execute inside the weight-streaming floor (), making greedy-exact Wake strictly faster than block decoding whenever rollback overhead stays below (explaining both the +22% to +153% gains at and the graceful throttle at , ); (2) candidate drafting ( in the numerator) decouples from pass-budget verification ( in the denominator), explaining why Wake multipliers transfer across trained mask-placeholder and parameter-free n-gram drafters (§ 4.5); and (3) across 24 unseen multi-request points (; Figure 12b), Equation (3) predicts throughput zero-shot at 8.3% MAPE and identifies the optimal block-size crossover on 11/12 (91.7%) operating points, guiding dynamic profile selection up to (2.31× decode, 1.67× end-to-end; Appendix § 16).
Figure 12. Physical H100 Roofline and zero-shot multi-request throughput law on Ouro-2.6B (). (a) Single-pass Roofline: at batch size , appending wake catch-up rows ( rows) sits deep inside the HBM weight-streaming plateau well below the hardware ridge point rows/pass. (b) Zero-shot concurrency prediction: calibrating Equation (3) strictly on full-depth modes predicts unseen continuous-batching throughput across with 8.3% MAPE and selects the optimal block size on 11/12 (91.7%) operating points.
Related Work
Looped and recursive language models iterate a weight-tied layer stack to scale latent reasoning depth at a compact parameter footprint ((Dehghani et al., 2019; Bae et al., 2025a,b; Geiping et al., 2025; Zhu et al., 2025)). In standard non-recurrent Transformers, SkipDecode ([9]) skips bottom layers in favor of upper layers under a monotonically decreasing per-step layer budget so KV caches never require recomputation, while in shared-weight recursive models Relaxed Recursive Transformers ([2]) introduce cross-request depth-wise batching and Mixture-of-Recursions ([3]) learns token-level routing across initial passes. Our analysis addresses a distinct regime: because a looped model shares weights across all passes, every token can enter uniformly at the final passes without decaying position budgets, while within-sequence pipelined cache completion reconstructs exact context caches in the background wake.
To accelerate autoregressive decoding, speculative and block-decoding methods verify multiple drafted tokens in parallel using auxiliary drafters, n-gram suffix lookups, or single-pass mask placeholders [17, 5, 4, 10, 13, 20, 16]. For looped language models specifically, WaveFront [14], LoopSpec [6], DAS [18], and LoopAhead [22] accelerate drafting via shallow initial passes: DAS and LoopAhead verify candidates synchronously through a full -pass target call, whereas WaveFront and LoopSpec pipeline single-pass shallow guesses diagonally into deeper passes. Crucially, because those methods only speed up proposal generation while still routing every accepted token through shallow-to-deep passes on the verification path, full -pass verification bounds their speedup as recurrence deepens. Late entry paired with Wake verification is complementary and structurally distinct along three axes: (1) our optimistic frontier executes the final passes over a deep context cache rather than initial passes over shallow caches, cutting mismatch and rollback rates by an order of magnitude (0.25 to 2.2 rollbacks per 100 tokens at ); (2) in-pass block verification filters candidate branches at depth so only the committed tokens enter the wake, saving passes on all rejected branches; and (3) a single static CUDA graph sweeps from greedy-exact () to margin-gated () and lossy () decoding while factorizing orthogonally from the choice of candidate drafter.
Finally, adaptive early-exit methods halt computation after initial layers or passes when confidence is high [23], while loop-contrastive methods contrast early and late passes to sharpen reasoning accuracy [28, 21]. We show that halting early without cache completion starves a looped language model of coherent pass-aligned context depth, whereas optimistic late entry paired with Wake verification preserves full-depth context and exactness while advancing at serial passes per step.
Limitations and Conclusion
Our study has three main limitations. First, as formalized by our Roofline law in § 4.7, single-request speedups exploit HBM weight-streaming headroom on an H100 GPU ( rows/pass). Under multi-request continuous batching on Ouro-2.6B (Figure 12b; Appendix § 16), speedup compression across stems primarily from block expansion rather than late entry itself: full-depth block decoding drops from 3.90× to 0.77×, whereas late entry’s incremental gain over same- block decoding holds steady (1.65× 1.45× at and 1.84× 1.48× at for ) because skipping passes on rejected draft branches cuts total arithmetic work per step (198 vs. 324 row-passes at ). Guided by Equation (3) (8.3% zero-shot MAPE, 11/12 optimal matches), shrinking with load retains 2.31× decode (1.67× end-to-end including online prefills) at for (1.71× and 1.47× for ). Second, while Wake verification unifies greedy-exact (), margin-gated (), and lossy () decoding inside one engine, Proposition 1 targets greedy decoding (extending Wake rollback to temperature sampling via speculative rejection sampling [17, 5] is left to future work), and speedups depend on optimistic agreement: at shallow recurrence (), frequent rollbacks (12.5 per 100 tokens) preserve full accuracy (+23.2 points over ) at the cost of falling below block speed (1.96× vs. 2.86×), whereas for (0.26 to 2.2 rollbacks per 100 tokens) scales up to 8.42×. Third, unlocking late entry on checkpoints not pretrained with pass-skipping requires a one-time Stage-2 SFT pass (7,395 steps, B tokens on one 8×H100 node, comparable to training lightweight speculative heads such as Medusa or LoopAhead while adding zero inference parameters; Appendix § 8, motivating native late-entry exposure during pretraining and per-token adaptive pass scheduling in future work.
In conclusion, saving passes from the two ends of a looped language model is fundamentally asymmetric: cached context depth substitutes for token-local early passes across Ouro-1.4B, Ouro-2.6B (), and middle-recurrent Huginn-3.5B (). Pairing late-pass block verification with pipelined cache completion and zero-extra-pass Wake verification turns this asymmetry into a unified family that accelerates both greedy-exact -pass decoding (up to 5.2× at and 8.4× at ) and controlled lossy decoding (up to 6.8× and 12.5×). Engine code, training scripts, and checkpoints are available at https://github.com/Xalp/LateComer and https://huggingface.co/collections/Xalphinions/late-entry (project page: https://xalp.net/late-entry/).
Fine-Tuning Recipe and Diagnostic Probe Protocol
Two-stage fine-tuning. We initialize Ouro-2.6B (and Ouro-1.4B for scale replication) from its released pretrained checkpoint [30] and fine-tune on Dolci-Instruct-SFT [26] in two one-epoch stages. Stage 1 performs standard full-depth () instruction SFT at sequence length 1,024. Stage 2 runs one epoch of MARS-style mask-placeholder block fine-tuning [16] using the block structure of § 3.2: blocks of 16 token positions with mask-placeholder branches, learning rate with 3% linear warmup, global batch size 256 in bf16, and sequence length 1,024 (7,395 optimization steps, B tokens per stage, adding zero extra parameters at inference). During Stage 2, half of the blocks run all four passes starting from pass 1, while the other half enter directly at pass () from raw token embeddings. Our matched early-exit control branches from the exact same Stage-1 checkpoint with identical data order, seed, and hyperparameters, reversing only the reduced-budget subset so tokens execute the first passes instead of the last .
Ouro stick-breaking readout and teacher-forced probe protocol. When Ouro executes a contiguous pass range , its learned stick-breaking exit gate emits a scalar gate probability at each executed pass , assigning weight to passes and all remaining mass to pass (so without requiring skipped passes):
To measure single-step agreement without trajectory drift, we first decode full test prompts greedily at full depth (, batch size 1) to obtain fixed reference completions (70,101 GSM8K token positions on Ouro-2.6B and 78,693 on Ouro-1.4B, with the identical procedure on IFEval). We then replay each reference sequence under teacher forcing using Equation (4) and record the fraction of positions whose greedy prediction matches the full-depth arg max.
Training-Stage Ablation
Table 9 disentangles the effects of instruction tuning, block drafting, and late-entry exposure on Ouro-2.6B. Neither Stage-1 SFT alone (.699 probe agreement) nor Stage-2 block training without late-entry exposure (.738) enables late entry. Explicit late-entry exposure during Stage 2 lifts probe agreement to .962 and prevents the severe decoding collapse seen in the unexposed block model (.636 vs. .839 on GSM8K; .529 vs. .699 on IFEval). It also preserves full-depth GSM8K accuracy (.873 vs. .880 after Stage 1, whereas the unadapted base model scores .729 and generates 243 mean tokens compared with 114 after SFT).
| Training configuration | Full-depth acc. (GSM8K / IFEval) | Late-entry probe | Late-entry decode () |
| None (released base model) | .729 / .442 | .751 | n/a (no drafters) |
| Stage 1 only (full-depth SFT) | .880 / .728 | .699 | n/a (no drafters) |
| Stage 1 + Stage 2, late entry off | .820 / .708 | .738 | .636 / .529 |
| Stage 1 + Stage 2 (our full recipe) | .873 / .732 | .962 | .839 / .699 |
Table 9. Training-stage ablation on Ouro-2.6B across full GSM8K () and strict IFEval (). The probe column reports teacher-forced agreement at , ; the rightmost column reports end-to-end block decoding accuracy at , .
Extended Entry-State and Attention-Geometry Diagnostics
Table 10 expands the off-the-shelf diagnostics from Table 1. Naively rescaling normalized embeddings (0.76 norm) to match pass-4 state norms (37 to 51) worsens Ouro-2.6B agreement to .218. Feeding the preceding token’s deep hidden state alone causes severe next-token repetition (, although combining it with a linear embedding projection reaches .949). Using unembedding vectors likewise repeats the current token (.01 to .04), while soft probability-weighted embeddings match hard embeddings on over 99% of confident positions. Even when we train two-layer MLP entry adapters while keeping Ouro weights frozen, agreement trails simple -pass truncation by 1.1, 3.4, and 2.9 points at on Ouro-2.6B (and fails to transfer to IFEval on Ouro-1.4B). On IFEval, closed-form linear maps recover 46% to 51% of the oracle gap (68% to 72% when combined with the predecessor state).
| Input state supplied to pass 4 ( over four-pass context) | Released Ouro-1.4B | Released Ouro-2.6B |
| Raw token embedding (unadapted late entry) | .552 | .751 |
| Normalized and magnitude-rescaled embedding | .245 | .218 |
| Closed-form linear projection of the embedding | .816 | .919 |
| Preceding token’s deep hidden state alone | .030 | .039 |
| True pass-3 hidden state (diagnostic oracle ceiling) | .991 | .992 |
| Reference: 1-pass early exit (pass 1 only) | .938 | .949 |
| Reference: 2-pass early exit (passes 1 and 2 only) | .981 | .985 |
Table 10. Extended pass-4 entry-state diagnostics on off-the-shelf Ouro-1.4B and Ouro-2.6B (GSM8K teacher-forced agreement with full depth; top five rows run pass 4 only over a four-pass context cache).
Attention-entropy, sink-mass, and distance-resolved KV interventions. Complementing Figure 3 in § 2.2, our fp32 attention-geometry probes across 30,736 GSM8K and 53,184 IFEval positions show that on unadapted checkpoints (released base, Stage 1 only, and Stage 2 without late-entry exposure), feeding directly into pass 4 (, ) produces diffuse attention (+0.79 to +1.00 nats higher head-mean entropy than a true pass-4 query and −0.10 to −0.16 less attention-sink mass on position 0), landing off-manifold with linear CKA .273 to .365 against true pass-4 hidden states. Stage-2 late-entry exposure matches true pass-4 attention entropy within +0.07 nats on GSM8K (+0.13 nats on IFEval), restores sink mass within −0.022, and reaches linear CKA .961 (mean cosine similarity .927 to .930). Distance-resolved prefix-depth interventions further show that while adjacent tokens carry the largest per-token marginal effect (+1.6 agreement points at distance 1), deep caches more than 64 tokens back still account for 14% of the total context-depth gain on GSM8K and 53% (+6.9 points) on longer IFEval generations.
Complete (, ) Frontier Grid and Verification Threshold Sweep
Table 11 reports the complete grid of pass budgets and block sizes on GSM8K and IFEval. Using improves throughput by 12% to 22% over on GSM8K while achieving similar throughput and slightly higher accuracy on IFEval. To test whether we could obtain similar speedups without skipping passes, Table 12 keeps full four-pass verification (, ) and relaxes acceptance so any candidate whose full-depth probability exceeds a threshold is accepted. Every threshold is strictly Pareto-dominated on both benchmarks by strict greedy (arg max) verification under late entry at and .
| Setting | GSM8K () tok/s | GSM8K () Speedup | GSM8K () Acc. | IFEval () tok/s | IFEval () Speedup | IFEval () Acc. | |
| Full-depth sequential | n/a | 63.4 | 1.00× | .873 | 63.1 | 1.00× | .732 |
| (■■■■) | 4 | 210.5 | 3.32× | .870 | 137.4 | 2.18× | .719 |
| 8 | 256.6 | 4.05× | .867 | 136.7 | 2.16× | .739 | |
| (□■■■) | 4 | 269.7 | 4.26× | .862 | 175.0 | 2.77× | .728 |
| 8 | 322.5 | 5.09× | .851 | 178.1 | 2.82× | .734 | |
| (□□■■) | 4 | 380.4 | 6.00× | .846 | 247.2 | 3.91× | .697 |
| 8 | 426.7 | 6.73× | .839 | 238.6 | 3.78× | .699 | |
| (□□□■) | 4 | 539.1 | 8.51× | .637 | 385.3 | 6.10× | .625 |
| 8 | 544.0 | 8.59× | .629 | 385.5 | 6.11× | .645 |
Table 11. Complete (, ) accuracy-throughput grid on full GSM8K and IFEval test sets (batch size 1, greedy decoding, single H100 GPU).
| Verification rule | GSM8K Speedup | GSM8K Acc. | IFEval Speedup | IFEval Acc. |
| Strict arg max, | 4.06× | .867 | 2.17× | .739 |
| Relaxed , | 4.14× | .867 | 2.19× | .726 |
| Relaxed , | 4.30× | .861 | 2.26× | .730 |
| Relaxed , | 4.40× | .841 | 2.36× | .719 |
| Relaxed , | 4.49× | .834 | 2.56× | .734 |
| Relaxed , | 4.53× | .823 | 2.69× | .728 |
| Strict arg max, | 5.09× | .851 | 2.82× | .734 |
| Strict arg max, | 6.73× | .839 | 3.78× | .699 |
Table 12. Relaxed probability-threshold verification at full depth (, ) versus strict greedy (arg max) late entry. Accepting proposals whose full-depth probability exceeds yields smaller speedups and lower accuracy than reducing under strict arg max verification.
Committed-Token Fidelity Audit
To verify exactness at and quantify approximation at , we replay all committed sequences through teacher-forced full-depth forward passes (Table 13). At , the token-level discrepancy rate (0.01% to 0.41%, with median top-2 logit margins of .03 to .07) lies at or below the sequential bf16 kernel-order floor (0.25% to 0.64%), and all 3,912/3,912 dense structural checks match identically. Our early-exit block decoder passes the same audit (0.03% discrepancy vs. a 0.23% floor at on GSM8K; 24/24 dense checks). At and without Wake verification (), token discrepancy rises moderately to 1.0% to 5.9%; enabling Exact Wake verification (, § 3.5) restores the exact fidelity floor (0.007% on GSM8K, 0.09% on MATH500, and 0.025% on MBPP).
| Benchmark | Full-depth floor diff | Full-depth floor margin | diff | margin | diff | margin | diff | margin |
| GSM8K | 0.27% | .04 (.10) | 0.01% | .06 (.11) | 1.12% | .11 (.48) | 2.08% | .22 (.87) |
| IFEval | 0.64% | .03 (.09) | 0.41% | .03 (.09) | 3.17% | .13 (.50) | 5.86% | .23 (.77) |
| MATH500 | 0.25% | .05 (.11) | 0.09% | .05 (.12) | 1.16% | .13 (.53) | 2.16% | .23 (1.00) |
| MBPP | 0.33% | .04 (.10) | 0.02% | .07 (.30) | 1.04% | .10 (.39) | 1.87% | .18 (.71) |
| BBH | 0.27% | .04 (.11) | 0.06% | .04 (.12) | 1.33% | .15 (.77) | 2.43% | .33 (1.41) |
| MMLU-Pro | 0.32% | .04 (.11) | 0.09% | .05 (.12) | 1.41% | .13 (.52) | 2.69% | .25 (1.07) |
Table 13. Committed-token fidelity audit at block size . “diff” is the percentage of committed positions whose token differs from a teacher-forced full-depth arg max replay; “margin” reports the median (and 90th percentile) top-2 logit margin at differing positions.
Cache-Completion Ablations and Error Localization
Cache-Completion Ablations and Error Localization
Table 14 compares three cache-completion policies on GSM8K and IFEval. Our lagged pipelined completion matches or slightly exceeds synchronous before-next-step repair ( to ), confirming that reading provisional KV entries for one or two steps introduces zero measurable degradation. Conversely, omitting cache completion altogether (keeping provisional embedding-derived KV states permanently) drops GSM8K accuracy by 4.1 points at (.839 .798).
| Cache-completion policy | Decoding mode | GSM8K | GSM8K | IFEval | IFEval |
| Synchronous (repair before next step) | Sequential | .852 | .864 | .684 | .717 |
| Pipelined (lagged across steps, ours) | Sequential | .859 | .872 | .695 | .723 |
| Pipelined (lagged across steps, ours) | Block () | .839 | .851 | .699 | .734 |
| Never (keep provisional cache) | Block () | .798 | .848 | .688 | .721 |
Table 14. Cache-completion policy ablation on full GSM8K () and IFEval (). Full-depth sequential references are .873 and .732.
Where do the remaining token deviations at occur within a block? Position-resolved audits across 400 GSM8K and 200 IFEval items (Figure 13) show that accepted proposal positions 1 8 average only 1.4% deviation on GSM8K at , while remaining discrepancies localize primarily to low-confidence full-rejection steps where all proposals are rejected (). Exploiting this localization, selectively recomputing only low-confidence (< .5) steps (19% of IFEval steps) at full depth raises IFEval accuracy from .699 to .723 ( vs. .732 full depth) at 196 tokens/s, interpolating smoothly between (239 tokens/s) and (178 tokens/s).
Figure 13. Position-resolved deviation audit at and across 400 GSM8K and 200 IFEval examples. Left subpanels group bonus tokens by accepted count ( means all proposals were rejected); right subpanels show deviation rates across accepted in-block positions .
Reproducibility, Adaptive Controls, and Full-Set Baseline Protocol
Sequential early-exit reproducibility and adaptive exit control. Independent sequential greedy evaluations of the trained early-exit checkpoint run five days apart reproduce to four decimal places across all eight cells: GSM8K .3290, .6732, .7938, .8165 and IFEval .4510, .6026, .6858, .7116 for .
In addition to fixed-depth early exit in Table 4, we also evaluate the trained early-exit model under a deliberately favored adaptive exit control: each token chooses its exit pass via the model’s learned exit gate with thresholds calibrated directly on the test sets, while skipped passes are still computed so subsequent tokens read an oracle full-depth KV cache. Even under these favorable conditions, adaptive exit improves accuracy by less than 0.6 points over fixed-depth early exit at matched mean passes per token on both GSM8K and IFEval.
Gold-answer scorer validation. Running default lm-evaluation-harness extractors on dataset ground-truth answers reveals two bugs: default MATH500 string matching rejects valid equivalent expressions, and default MBPP code extraction strips line 1 of completions (causing 499/500 ground-truth programs to fail). Symbolic expression equivalence on MATH500 and preserved first-line extraction on MBPP restore 100% accuracy on dataset reference solutions.
Full-test-set baseline protocol. Table 5 evaluates the full test set of every benchmark (21,403 prompts in total) with verified token-level prompt identity across engines: every method decodes identical token-ID inputs at batch size 1 with greedy decoding in bf16 on a single NVIDIA H100 80GB GPU per job, and the two largest benchmarks run as contiguous shards pooled exactly by total decode tokens over total decode seconds. On each task, throughputs are normalized by the fastest measured full-depth sequential decoder among ours, LoopSpec’s, and the LoopAhead reproduction’s (62.4 to 67.2 tokens/s; LoopSpec’s engine is fastest on four tasks, ours on MATH500 and MMLU-Pro). LoopSpec uses the fastest of four released configurations selected on a full GSM8K sweep (190.0 vs. 184.8, 173.9, and 119.5 tokens/s). Because official LoopAhead code is unavailable, we report our validated reproduction taking the fastest of three training arms per benchmark (1× and 4× published training budget plus a no-drafter arm), evaluated on the released checkpoint its method requires.
H Huginn-3.5B Architecture Adaptation and Extended Sweeps
Prelude-Core-Coda late-entry formulation and training recipe. Unlike Ouro, which loops its entire Transformer stack and combines pass outputs via a stick-breaking exit gate, Huginn-3.5B (huginn-0125; [12] wraps a recurrent 4-layer Core block between a 2-layer non-recurrent Prelude and a 2-layer non-recurrent Coda . For token , Prelude maps the scaled token embedding to a contextualized input anchor . Full-depth inference initializes a latent state from Huginn’s standard truncated Gaussian state initializer (seeded deterministically at test time) and iterates () via a learned linear adapter :
Under late-entry block decoding with budget , the active block executes Prelude once (writing exact Prelude KV states for all committed tokens, so Prelude is never recomputed during catch-up), initializes , runs only the final Core steps , and reads out through Coda (4 + 4 serial layer calls vs. 132 at full depth). Across the next steps (default full-depth , or an optional partial-cache approximation completing only the final Core steps), committed groups advance at steps per call alongside the active block, fusing Coda refresh into the final catch-up call at zero extra serial forward calls. To ensure hardware-matched wall-clock comparisons across sequential decoding, late entry, synchronous repair, and early exit, all Huginn decoders run inside a unified CUDA-graphed paged FlashAttention-3 engine that packs the active block and all in-flight catch-up stages into a single fixed-shape paged attention call per layer with fused RoPE and RMSNorm kernels. For two-stage adaptation on Dolci-Instruct-SFT, Stage 1 runs one epoch of full-depth () SFT and Stage 2 runs one epoch (7,406 optimization steps, B tokens per stage or of huginn-0125’s 800B-token pretraining budget, adding zero extra inference parameters) of MARS block fine-tuning at global batch size 256 in bf16, sequence length 1,024, learning rate , and truncated backpropagation through the final 8 recurrent steps matching Geiping et al. [12]: 50% of blocks execute all 32 Core passes, while 50% execute passes (final passes for late entry; initial passes for the matched early-exit control).
Single-token curves, matched-throughput frontier, and cache-completion lag ablation. Table 15 reports full-set () single-token sequential GSM8K curves on our adapted checkpoint: running final Core passes over a completed 32-pass cache matches full 32-pass accuracy identically (.396 vs. .396, .989 token agreement), beating 8-pass early exit over a 32-pass oracle cache (.288) by +10.8 points and 8-pass shallow truncation (.263) by +13.3 points (+27.7 points at ). Table 16 compares our matched late-entry and early-exit Stage-2 checkpoints (7,406 steps each) at across GSM8K (, spanning 25% to 100% Core depth in exact correspondence with on Ouro) and IFEval (across all four capability-active benchmarks, intermediate late entry reaches 6.44× at .545 on IFEval, 7.27× at .395 on GSM8K, 8.20× at .281 on BBH, and 9.75× at .234 on MBPP). Because paged FlashAttention-3 eliminates multi-segment dispatch overhead, pipelined catch-up adds only 7% to 12% latency at equal and outperforms trained early exit across all matched-throughput GSM8K operating points in Table 16: late entry (.395 at 157.8 tokens/s) is both faster and +2.4 points higher than 16-pass early exit (.372 at 140.3 tokens/s), late entry (.354 at 198.1 tokens/s) is faster and higher than 12-pass early exit (.342 at 174.4 tokens/s), and our two-stage partial-cache variant (.369 at 215.5 tokens/s) beats 8-pass early exit (.271 at 222.9 tokens/s) by +9.7 points (). On IFEval, the two checkpoints do not differ significantly at (), while above 200 tokens/s late entry retains full-depth accuracy down to 12.5% Core depth and leads trained early exit by +4.8 points at matched tokens/s ( at .560, 223.0 tokens/s vs. at .512, 226.6 tokens/s, ), +7.0 points at (), and +12.4 points at (). Even on the early-exit-trained checkpoint itself (which never saw late entry in Stage 2), Prelude’s normalized anchor allows late entry to beat that checkpoint’s own trained early exit on GSM8K (.327 vs. .271 at 193.9 tokens/s, ). Finally, Table 17 characterizes when provisional KV states degrade at , , by comparing full-depth completion (, 4 stages, 3-block lag) against partial-cache approximations (, 3 and 2 stages, which leave early passes uncomputed in exchange for 2- and 1-block lag). On GSM8K, where each chain-of-thought arithmetic step reads digits emitted in the immediately preceding tokens, 4-stage full completion trails CUDA-graphed synchronous repair by 3.3 points (.354 vs. .387, ), and accuracy varies monotonically with the local lag window (.387 .369 .360 .354 .331 across 0, 1, 2, 3, block lag). The 2-stage partial-cache approximation (17 32, one-block lag 3.5 tokens) narrows this gap to 1.9 points (.369 vs. .387, ; +3.7 points over omitting catch-up at ) while running 3.66× faster than synchronous repair (215.5 vs. 58.9 tokens/s). By contrast, on IFEval, where structural and formatting constraints depend weakly on local provisional refinements, omitting or lagging cache completion leaves accuracy virtually unchanged (.553 at no catch-up and two-stage partial vs. .551 sequential full depth), even though early exit still collapses at small because the active token itself requires late Core passes.
Scaling Across Request Concurrency
How does late-entry block decoding behave when serving concurrent requests on a single GPU? To characterize the transition from memory-bandwidth-bound to compute-bound execution, we extend our paged FlashAttention-3 [24] / CUDA-graph engine on Ouro-2.6B () to multi-request continuous batching across active concurrencies on GSM8K (single NVIDIA H100 80GB GPU, bf16, max_new = 1,024, dynamic paged KV slot allocation with immediate slot refill upon request completion, and zero token discrepancies between CUDA-graph replay and eager execution across all eleven modes). At every concurrency , we normalize against full-depth () continuous-batching autoregressive decoding (AR, , ) at the same , reporting both pure decode throughput and end-to-end serving throughput including online four-pass prefills for newly arriving prompts (Figure 14 and Table 18).
Figure 14. Steady-state concurrency scaling on Ouro-2.6B (, GSM8K, single H100 GPU) across concurrent requests. (a) Speedup over continuous-batching AR at the same : fixed large blocks () decay steeply as rows saturate compute, whereas selecting smaller discrete block profiles with load (, shaded profile envelope) sustains at () and (). (b) System decode throughput vs. per-request speed ( at bottom-right to at top-left along each curve).
| Mode | Regime (rows/req/call) | ||||||
| Pure decode throughput (total decode tokens/s and speedup vs. same- AR) | |||||||
| AR (, ) | Lossless ref. (1) | 64.0 (1.00×) | 121.8 (1.00×) | 233.2 (1.00×) | 455.4 (1.00×) | 848.6 (1.00×) | 1,498.2 (1.00×) |
| , | Lossless ref. (81) | 249.6 (3.90×) | 446.0 (3.66×) | 732.9 (3.14×) | 921.5 (2.02×) | 1,080.1 (1.27×) | 1,152.4 (0.77×) |
| , | Lossless ref. (25) | 207.8 (3.25×) | 385.5 (3.17×) | 729.1 (3.13×) | 1,182.7 (2.60×) | 1,698.2 (2.00×) | 2,011.2 (1.34×) |
| , | Lossy late (99/90/90) | 327.0 (5.11×) | 560.1 (4.60×) | 848.0 (3.64×) | 1,048.3 (2.30×) | 1,212.6 (1.43×) | 1,285.8 (0.86×) |
| , | Lossy late (35/30/30) | 270.2 (4.22×) | 514.1 (4.22×) | 913.3 (3.92×) | 1,420.6 (3.12×) | 1,890.0 (2.23×) | 2,294.5 (1.53×) |
| , | Lossy late (15/12/12) | 196.2 (3.06×) | 370.6 (3.04×) | 681.8 (2.92×) | 1,221.7 (2.68×) | 1,875.7 (2.21×) | 2,560.8 (1.71×) |
| , | Lossy 1-tok (3/2/2) | 79.6 (1.24×) | 153.0 (1.26×) | 293.6 (1.26×) | 545.7 (1.20×) | 952.3 (1.12×) | 1,548.7 (1.03×) |
| , | Lossy late (99) | 410.8 (6.42×) | 706.8 (5.80×) | 1,105.7 (4.74×) | 1,400.9 (3.08×) | 1,575.7 (1.86×) | 1,674.5 (1.12×) |
| , | Lossy late (35) | 382.2 (5.97×) | 732.0 (6.01×) | 1,299.4 (5.57×) | 1,971.0 (4.33×) | 2,573.0 (3.03×) | 2,976.8 (1.99×) |
| , | Lossy late (15) | 282.8 (4.42×) | 537.1 (4.41×) | 976.6 (4.19×) | 1,722.3 (3.78×) | 2,705.5 (3.19×) | 3,466.0 (2.31×) |
| , | Lossy 1-tok (3) | 117.5 (1.83×) | 227.3 (1.87×) | 427.4 (1.83×) | 798.5 (1.75×) | 1,371.6 (1.62×) | 2,222.9 (1.48×) |
| Isolated late-entry multiplier over same- full-depth () block decoding | |||||||
| / at | 198 vs. 324 row-passes/step | 1.65× (+65%) | 1.58× (+58%) | 1.51× (+51%) | 1.52× (+52%) | 1.46× (+46%) | 1.45× (+45%) |
| / at | 70 vs. 100 row-passes/step | 1.84× (+84%) | 1.90× (+90%) | 1.78× (+78%) | 1.67× (+67%) | 1.52× (+52%) | 1.48× (+48%) |
| / at | 95 vs. 100 row-passes/step | 1.30× (+30%) | 1.33× (+33%) | 1.25× (+25%) | 1.20× (+20%) | 1.11× (+11%) | 1.14× (+14%) |
| End-to-end serving throughput including online 4-pass slot-refill prefills (total e2e tokens/s) | |||||||
| AR (, ) | Lossless ref. | 61.5 (1.00×) | 113.2 (1.00×) | 200.3 (1.00×) | 348.6 (1.00×) | 539.6 (1.00×) | 803.5 (1.00×) |
| , | Lossless ref. | 183.2 (2.98×) | 312.0 (2.76×) | 501.4 (2.50×) | 648.5 (1.86×) | 880.7 (1.63×) | 1,100.0 (1.37×) |
| , | Lossy late | 229.0 (3.72×) | 388.9 (3.44×) | 573.2 (2.86×) | 711.2 (2.04×) | 946.0 (1.75×) | 1,185.1 (1.47×) |
| , | Lossy late | 305.8 (4.97×) | 501.8 (4.43×) | 699.0 (3.49×) | 844.1 (2.42×) | 1,080.0 (2.00×) | 1,339.0 (1.67×) |
Table 18. Multi-request continuous-batching throughput and mechanism factorization on Ouro-2.6B (, GSM8K, single H100 GPU) across concurrencies . Top block: total decode tokens/s (and speedup vs. same- AR); middle block: isolated multiplicative gain of late entry over matched- full-depth () block decoding; bottom block: end-to-end serving throughput including online four-pass slot-refill prefills.
Figure 14 and Table 18 highlight three scaling behaviors. First, speedup compression at fixed comes from block expansion rather than late entry: as concurrency grows from to , lossless , block decoding collapses 5.1× (3.90× → 0.77×), whereas late entry’s multiplicative gain over same- full-depth block decoding ( over , middle block of Table 18) stays nearly invariant at 1.65× → 1.45× at and 1.84× → 1.48× at (1.30× → 1.14× for ). Physically, , runs all four passes across all candidate rows (324 row-passes/step), whereas , runs only two passes on the 81 block rows and restricts catch-up passes to the committed tokens (198 row-passes/step, 39% fewer FLOPs per step), preserving late entry’s advantage over full-depth block decoding even when Tensor Cores saturate. Second, selecting smaller discrete block profiles as concurrency rises shifts the crossover substantially (shaded profile envelope in Figure 14a; as in standard multi-bucket serving engines, pre-capturing one CUDA graph per discrete bucket shares the same model weights and paged KV pool, allowing per-step profile selection from active batch size without graph recompilation): is fastest at , (35 rows/request) takes over across to 8 (6.01× to 4.33× at ; 4.22× to 3.12× at ), and (15 rows/request) leads at to 32, reaching 3,466 tokens/s (2.31× decode; 108.3 tokens/s per request, exceeding sequential AR at 64.0 tokens/s) at and 2,561 tokens/s (1.71×) at (vs. 1.34× for lossless , ), with 1.67× and 1.47× end-to-end throughput at including online four-pass prefills. Third, single-token pipelined late entry () uses zero draft branches (3 rows/request/call at , executing total passes per token versus 4 for plain AR) and sustains a flat 1.83× → 1.48× decode speedup across to 32.
References
- [1]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.arxiv.org/abs/2108.07732
- [2]Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, and Tal Schuster. Relaxed recursive Transformers: Effective parameter sharing with layer-wise LoRA. In International Conference on Learning Representations, 2025a.arxiv.org/abs/2410.20672
- [3]Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. In Advances in Neural Information Processing Systems, 2025b.
- [4]Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, 2024.arxiv.org/abs/2401.10774
- [5]Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.arxiv.org/abs/2302.01318
- [6]SangLyul Cho, Langqing Cui, Sehoon Kim, Dongsu Han, and Insu Han. LoopSpec: Pipelined self-speculative decoding for looped Transformers. arXiv preprint arXiv:2609.17184, 2026.arxiv.org/abs/2609.17184
- [7]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Łukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
- [8]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal Transformers. In International Conference on Learning Representations, 2019.arxiv.org/abs/1807.03819
- [9]Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. SkipDecode: Autoregressive skip decoding with batching and caching for efficient LLM inference. arXiv preprint arXiv:2307.02628, 2023.arxiv.org/abs/2307.02628
- [10]Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. In International Conference on Machine Learning, 2024.arxiv.org/abs/2402.02057
- [11]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonnell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. Zenodo, https://doi.org/10.5281/zenodo.10256836, 2023.
- [12]Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2502.05171
- [13]Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In International Conference on Machine Learning, 2024.arxiv.org/abs/2404.19737
- [14]Hyeongju Ha and Jae-Joon Kim. WaveFront Decoding: Parallelized self-speculative decoding for looped language models. arXiv preprint arXiv:2609.23033, 2026.arxiv.org/abs/2609.23033
- [15]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2021.
- [16]Ziqi Jin, Lei Wang, Ziwei Luo, and Aixin Sun. MARS: Enabling autoregressive models multi-token generation. In Advances in Neural Information Processing Systems, 2026.arxiv.org/abs/2604.07023
- [17]Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from Transformers via speculative decoding. In International Conference on Machine Learning, 2023.arxiv.org/abs/2211.17192
- [18]Guanghao Li, Zihan Su, Hao Yu, Jinyang Jiang, Tao Ren, Zehao Li, Feng Lu, Ming Tang, and Chun Yuan. Shallow queries, mature values: Depth-asynchronous self-speculation for looped Transformers. arXiv preprint arXiv:2609.34538, 2026.arxiv.org/abs/2609.34538
- [19]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024.arxiv.org/abs/2305.20050
- [20]Jingyu Liu, Xin Dong, Zhifan Ye, Rishabh Mehta, Yonggan Fu, Vartika Singh, Jan Kautz, Ce Zhang, and Pavlo Molchanov. TiDAR: Think in diffusion, talk in autoregression. In Proceedings of Machine Learning and Systems, 2026a.arxiv.org/abs/2511.08923
- [21]Weihao Liu, Huangjie Zheng, Tianrong Chen, Rohit Dilip, Richard He Bai, Yizhu Jiao, Yuyang Wang, and Ruixiang Zhang. Decoding looped Transformers better for (almost) free. arXiv preprint arXiv:2610.02185, 2026b.arxiv.org/abs/2610.02185
- [22]Zixuan Liu, Nan Xue, Rui Bao, Yuhang Xu, Xiaofeng Cao, Jiangchao Yao, and Zhiyong Chen. LoopAhead: Speculating ahead for faster decoding in looped language models. arXiv preprint arXiv:2610.21594, 2026c.arxiv.org/abs/2610.21594
- [23]Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinnih Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2207.07061
- [24]Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2407.08608
- [25]Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, 2023.arxiv.org/abs/2210.09261
- [26]Team Olmo. Olmo 3. arXiv preprint arXiv:2512.13961, 2025.
- [27]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Ahranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024.arxiv.org/abs/2406.01574
- [28]Byeongho Yu, Junhyuk So, and Eunhyeok Park. LoopCD: Loop-wise contrastive decoding for improving reasoning in looped language models. arXiv preprint arXiv:2609.24196, 2026.arxiv.org/abs/2609.24196
- [29]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.arxiv.org/abs/2311.07911
- [30]Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaij ing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Zhao, Tianle Cai, Ge Zhang, Wenhao Huang, Yoshua Bengio, and Jason Eshraghian. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025.arxiv.org/abs/2510.25741













