betaFeedback

WorksComputer Science

ASCENT: Scale-Complementary External State for Frozen Long-Context Language Models

Abstract

External state should become more valuable as a reader becomes more capable, yet long-context evaluations measure memory utility at a single model scale. This paper formulates scale complementarity: the absolute gain from external state increases when reader capacity and a compact state budget are jointly scaled. ASCENT realizes this principle with a target-blind causal writer, nested typed state, and a bounded query-conditioned readout; every language model remains frozen. Under log loss, the value of a nested refinement equals its target-relevant conditional information; a full factorial makes the reader-by-state interaction identifiable. Official 16K BABILong experiments show Qwen2.5 accuracy gains of 10.50, 78.88, and 82.63 percentage points at 0.5B, 1.5B, and 3B parameters, with both adjacent directional contrasts significant after family-wise correction. At 7B, Foundation accuracy rises to 33.25% while four-slot ASCENT answers every question, which caps the gain at 66.75 points; the 3B-to-7B accuracy increment is -15.88 points [-20.55, -11.20]. The accuracy-side claim is therefore established from 0.5B to 3B; the 7B conclusion rests on the non-saturated negative-log-likelihood factorial. A three-model by three-state factorial yields co-scaled negative log-likelihood gains of 0.388, 2.099, and 7.406 nats at 1.5B, 3B, and 7B; all interaction and gain contrasts retain positive simultaneous lower bounds. Five public 7B-8B readers obtain 66.75-85.63 point gains, and ASCENT exceeds task-trained retrieval checkpoints at the two larger shared scales. Controls, baselines, systems accounting, matched-hardware reproduction, and a cross-architecture sensitivity analysis accompany the experiments. The results establish scale-complementary external state under the evaluated benchmarks, metrics, and state schema.

Introduction

Long-context language-model systems increasingly separate stored evidence from the reader that reasons over it. Sparse attention, retrieval, prompt compression, and recurrent memory each reduce the amount of evidence processed directly, but they are typically judged at one reader size. That protocol leaves a central scaling question unresolved: as the reader becomes more capable, does an external state become more useful in absolute terms? We call a positive answer scale complementarity. It is stronger than showing that memory helps a large model, and it differs from a model-only scaling law: both the reader and the information budget are allowed to grow.

ASCENT is a frozen-foundation interface designed to make this property measurable. A causal target-blind writer converts the pre-query stream into nested typed state; a bounded readout exposes only query-relevant entries to a frozen reader. The state budget co-scales with reader capacity while its byte ratio to model storage falls. This separation creates two independently controllable axes, reader scale and state refinement, and makes the model-by-state interaction identifiable.

The paper unifies theory, design, and evidence at the same estimand. Under Bayes-optimal log loss, the incremental value of a nested state refinement is exactly its conditional mutual information with the target. ASCENT constructs refinements intended to make this quantity positive. A prespecified 3×33 \times3 factorial identifies whether finite readers realize increasingly more of that information, while official free-generation benchmarks establish the corresponding accuracy effect. Model, data, configuration, and analysis hashes are fixed before evaluation; the panel is the unit of inference; and multiplicity is controlled at the family level.

The evidence contains official RULER [1] and BABILong [2] tasks, two ordered model families, five public 7B–8B readers, released task-trained Q-RAG checkpoints [3], raw and neural retrieval controls, representation ablations, systems accounting, cross-node reproduction on matched hardware, and a cross-architecture sensitivity analysis. This design turns scale complementarity from a visual trend into a tested property of a co-scaled language-model system.

The main contributions are:

  • A scale-complementarity theory connects nested state refinement to conditional information and yields a finite-reader interaction estimand.

  • ASCENT supplies a target-blind, bounded, nested external state whose absolute budget grows while its state-to-model storage ratio contracts.

  • Official 16K results establish two positive adjacent accuracy-gain increments over 0.5B, 1.5B, and 3B Qwen2.5 readers.

  • A nine-panel 3×33 \times3 factorial identifies positive model-by-state interactions through 7B with simultaneous inference over the complete contrast family.

  • Five public 7B–8B readers, a direct trained-retrieval comparison, cross-task controls, systems measurements, and cross-node reproduction establish architectural breadth and reproducibility; every reported number is recomputable from the released per-row records.

Related Work

Architectures for Long Sequences

The Transformer established content-addressable global interaction [4]; Transformer-XL and compressive recurrence extended its temporal reach [5, 6]. Sparse and approximate attention reduce the quadratic path through local windows, global tokens, hashing, random features, low-rank projection, Nyström approximation, routing, and structured masks [7, 8, 9, 10, 11, 12, 13, 14]. State-space, convolutional, recurrent, and retention models provide alternative long-sequence substrates [15, 16, 17, 18, 19]`. These works scale the sequence processor itself. ASCENT instead holds each processor frozen and identifies how the value of an external information state changes with reader capacity.

System-level work attacks a complementary bottleneck through IO-aware exact attention, blockwise distribution, paged allocation, sparse prefill, and decoding acceleration [20, 21, 22, 23, 24, 25, 26]. These methods change the cost of executing a context. ASCENT changes the statistical experiment presented to the language-model reader and measures the resulting value rather than treating throughput as the primary endpoint.

External Memory and Retrieval

Learned external memory spans memory networks, differentiable computers, episodic memory, and product-key layers [27, 28, 29, 30, 31]. Long-term memory managers and test-time state extend this line to language models [32, 33, 34, 35, 36, 37]. Retrieval-augmented generation and language modeling use dense, sparse, late-interaction, black-box, self-reflective, corrective, and hierarchical retrieval [38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52]. Recent work in Array combines sparse and dense retrieval with prompt and data processing for large-model reasoning [53]; Q-RAG further trains a multi-step value-based embedder [3]. ASCENT targets a different dependent variable: the adjacent change in external-state gain under co-scaling, with no reader training.

Compression, Evaluation, and Scaling

Prompt filtering and compression improve context efficiency [54, 55, 56]; positional interpolation, rotary embeddings, YaRN, and LongRoPE extend usable windows [57, 58, 59, 60]. Long-context evaluations show that nominal length does not ensure robust evidence use [61, 62, 63, 64, 65, 66, 67]. RULER and BABILong add controlled retrieval, aggregation, and compositional reasoning at long lengths [1, 2]. Neural scaling laws relate loss to model, data, and compute [68, 69, 70, 71]. ASCENT combines these strands by treating state budget as a second scaling coordinate and using comparison of statistical experiments [72] to state when a refinement has positive value.

Training-free extrapolation and cache management preserve or select internal attention state through linear biases, streaming anchors, heavy hitters, persistent importance, query-aware sparsity, and hierarchical cache budgets [73, 74, 75, 76, 77, 78, 79, 80]. Activation beacons, InfLLM, and LongLoRA instead compress, retrieve, or adapt context representations [81, 82, 83]. These approaches motivate strong efficiency controls, but none makes the adjacent external-state gain across reader scale its principal estimand.

Problem Formulation

Let MmM_m be a frozen foundation model of size mm, XX a long input, QQ a query, and YY the target. A causal writer WsW_s produces state Zs=Ws(X)Z_s = W_s(X) under budget ss. A reader RR exposes a bounded projection of ZsZ_s to MmM_m. For loss ℓ\ell, define

L(m,s)=E[ℓ(Y,Mm(Q,R(Zs,Q)))].(1)L(m,s) = \mathbb{E}\left[\ell\left(Y,M_m\left(Q,R(Z_s,Q)\right)\right)\right]. \tag*{(1)}

The external-state gain relative to the foundation-only system is

G(m,s)=L(m,0)−L(m,s).(2)\mathcal{G}(m,s) = L(m,0) - L(m,s). \tag*{(2)}

For ordered model sizes m1<m2<m3m_1 < m_2 < m_3 and a co-scaling rule s(m)s(m), the primary empirical property is

G(m1,s(m1))<G(m2,s(m2))<G(m3,s(m3)).(3)\mathcal{G}\left(m_1,s(m_1)\right) < \mathcal{G}\left(m_2,s(m_2)\right) < \mathcal{G}\left(m_3,s(m_3)\right). \tag*{(3)}

We require s(m)=o(m2)s(m) = o(m^2) and report both absolute state bytes and the ratio of state bytes to model bytes. Equation (3) is a property of the co-scaled system on a task distribution, not a universal model law.

Proposition 1 (No distribution-free strict scaling). No fixed writer, reader, and co-scaling rule can guarantee strict positive adjacent gain increments for every distribution over (X,Q,Y)(X,Q,Y).

Proof. Choose a distribution for which YY is conditionally independent of XX given QQ, or for which the foundation-only predictor is already Bayes optimal. Then every external state has zero Bayes gain. Alternatively, choose a state that contains the exact answer at the smallest scale; bounded accuracy then prevents strictly increasing absolute gain. Thus strict distribution-free scaling is impossible.

For log loss, define the Bayes risk of state ZsZ_s as

L∗(s)=inf⁡pE[−log⁡p(Y∣Q,Zs)]=H(Y∣Q,Zs).(4)L^*(s) = \inf_p \mathbb{E}\left[-\log p(Y \mid Q,Z_s)\right] = H(Y \mid Q,Z_s). \tag*{(4)}

Proposition 2 (Information value of a refinement). If Zs′Z_{s'} refines ZsZ_s, the Bayes-risk improvement due to the refinement is

L∗(s)−L∗(s′)=I(Y;Zs′∣Q,Zs).(5)L^*(s) - L^*(s') = I(Y; Z_{s'} \mid Q, Z_s). \tag*{(5)}

It is strict if and only if the refinement contains target-relevant conditional information.

Proof. Because ZsZ_s is a function of Zs′Z_{s'}, the entropy chain rule gives H(Y∣Q,Zs)−H(Y∣Q,Zs′)=I(Y;Zs′∣Q,Zs)H(Y \mid Q, Z_s) - H(Y \mid Q, Z_{s'}) = I(Y; Z_{s'} \mid Q, Z_s). □\square

This identity supplies a constructive rule: each state expansion should add nonredundant compositional evidence, while model scale determines how much of that evidence a finite reader converts into reduced predictive loss.

Proposition 3 (Strict complementarity on fresh episodic information). Suppose the episode target is created after foundation training and is conditionally independent of every foundation given QQ. Let Z1=π1(Z2)Z_1 = \pi_1(Z_2) and Z2=π2(Z3)Z_2 = \pi_2(Z_3) be nested state signals. If both the no-state and state-conditioned predictors output their Bayes-optimal conditionals and

I(Y;Zi+1∣Q,Zi)>0,i∈{1,2},(6)I(Y; Z_{i+1} \mid Q, Z_i) > 0, \qquad i \in\{1,2\}, \tag*{(6)}

then the Bayes log-loss gains satisfy G(m1,s1)<G(m2,s2)<G(m3,s3)G(m_1, s_1) < G(m_2, s_2) < G(m_3, s_3).

Proof. Fresh-information independence makes the no-state Bayes loss H(Y∣Q)H(Y \mid Q) at every scale. Proper posterior decoding gives state loss H(Y∣Q,Zi)H(Y \mid Q, Z_i), so the gain is I(Y;Zi∣Q)I(Y; Z_i \mid Q). By the chain rule and nesting,

G(mi+1,si+1)−G(mi,si)=I(Y;Zi+1∣Q,Zi)>0.(7)G(m_{i+1}, s_{i+1}) - G(m_i, s_i) = I(Y; Z_{i+1} \mid Q, Z_i) > 0. \tag*{(7)}

□\square

The indices mim_i in Proposition 3 carry no force: with oracle decoders the gain depends only on the nested state. The reader-dependent content of scale complementarity is isolated in Proposition 4, whose interaction term is what the factorial estimates.

The preceding statement concerns an oracle decoder. To connect it to frozen finite readers, define the nonnegative excess risk

ϵ(m,s)=L(m,s)−H(Y∣Q,Zs).(8)\epsilon(m, s) = L(m, s) - H(Y \mid Q, Z_s). \tag*{(8)}

The observed gain admits the exact decomposition

G(m,s)=I(Y;Zs∣Q)+ϵ(m,0)−ϵ(m,s).(9)\mathcal{G}(m,s)=I(Y;Z_s\mid Q)+\epsilon(m,0)-\epsilon(m,s). \tag*{(9)}

It separates information present in the state from the reader’s ability to convert that information into predictive performance.

Proposition 4 (Finite-reader interaction). For two frozen readers ma<mbm_a<m_b evaluated on the same nested refinement Zs⪯Zs′Z_s\preceq Z_{s'}, let

Dm(s,s′)=G(m,s′)−G(m,s).(10)D_m(s,s')=\mathcal{G}(m,s')-\mathcal{G}(m,s). \tag*{(10)}

Then the model-by-state interaction is

Dmb(s,s′)−Dma(s,s′)=[ϵ(mb,s)−ϵ(mb,s′)]−[ϵ(ma,s)−ϵ(ma,s′)].(11)D_{m_b}(s,s')-D_{m_a}(s,s')=[\epsilon(m_b,s)-\epsilon(m_b,s')]-[\epsilon(m_a,s)-\epsilon(m_a,s')]. \tag*{(11)}

Consequently, a positive interaction identifies that the larger reader realizes more of the same state refinement as excess-risk reduction.

Proof. Apply Eq. (9) at ss and s′s' for each reader. The shared conditional-information increment I(Y;Zs′∣Q,Zs)I(Y;Z_{s'}\mid Q,Z_s) cancels between readers, leaving Eq. (11).

This identity motivates the full factorial rather than a diagonal-only scaling curve. Diagonal contrasts establish the co-scaled system property; the crossed interactions determine whether model capacity changes the value extracted from an identical state expansion.

Proposition 5 (Vanishing relative footprint). If state bytes obey B(d)=O(dα)B(d)=O(d^\alpha) for 0<α≤10<\alpha\le1, while a dense LL-layer foundation has N=Ω(Ld2)N=\Omega(Ld^2) parameters with L=Ω(1)L=\Omega(1), then B(d)/N→0B(d)/N\to0 as width d→∞d\to\infty.

Proof. Direct substitution yields B(d)/N=O(1/(Ld2−α))B(d)/N=O(1/(Ld^{2-\alpha})); since 2−α≥12-\alpha\ge1 and LL is bounded below, the ratio converges to zero.

Thus the construction permits increasing absolute information value while the relative state footprint vanishes. The finite-reader interaction experiment tests the key empirical departure from the oracle premise: whether larger frozen language models realize more of each identical nested refinement.

ASCENT

Figure 1 summarizes the interface and the empirical co-scaling target. The writer and foundation model remain frozen throughout; the deterministic state budget alone follows the co-scaling rule. The design therefore attributes the gain curve to reader capacity and state refinement, with training held constant across endpoints.

Figure 1. Theory, mechanism, and identification meet at the external-state gain. (A) A target-blind causal writer converts long evidence into nested, bounded state for query-conditioned use by a frozen reader. (B) Proposition 2 equates the value of a strict refinement with its target-relevant conditional information. (C) The complete 3×33 \times3 factorial identifies the finite-reader consequence in Proposition 4: co-scaled gain rises from .39 to 2.10 to 7.41 nats and both model-by-state interactions are positive.

Execution Contract

ASCENT is the deterministic composition

Zs=Ws(X),Es,Q=Rs(Q,Zs),Y^m,s=DMm(P(Q,Es,Q)),(12)Z_s = W_s(X), \qquad E_{s,Q} = R_s(Q,Z_s), \qquad\widehat{Y}_{m,s} = D_{M_m}(P(Q,E_{s,Q})), \tag*{(12)}

where WsW_s is the causal state writer, RsR_s is a bounded readout, PP is a fixed serializer, and DMmD_{M_m} is greedy decoding by the frozen reader. The foundation arm is exactly s=0s = 0, for which E0,Q=∅E_{0,Q} = \varnothing. For s<s′s < s', a deterministic projection πs′→s\pi_{s' \to s} satisfies Zs=πs′→s(Zs′)Z_s = \pi_{s' \to s}(Z_{s'}), so every larger budget is a strict refinement rather than a different representation.

Each endpoint executes the same five operations: (i) write state from XX before QQ and YY are revealed; (ii) validate registered types, ordering, hashes, alignment, and budget constraints; (iii) select at most ss query-relevant entries; (iv) serialize those entries with the fixed canonical template; and (v) decode with the unchanged model and scorer. Validation is fail-closed: a missing, malformed, or mismatched state aborts the invocation. Equation (12) isolates the two experimental coordinates: mm changes only the frozen decoder, and ss changes only a nested external-state refinement. It is the executable counterpart of Propositions 2 and 4.

Target-Blind State Construction

The writer processes the document before observing the final target. It has three parts, which Table 1 lists for every evaluated document family. A recognizer scans the input that precedes the question and emits typed records; each record carries its fields, its character position, and the verbatim input span from which it was read. A state update folds the records, in document order, into bounded state built from three primitives: a last-write map, an append-only log, and a counter. A question map identifies the question form and names the state keys from which the readout may select. The record types, their composition into state, and the question forms constitute a family’s schema; across the evaluated families the schemas record typed events, assignment edges, occurrence-indexed key–value evidence, and count summaries. Everything outside the schema is common to all families: the write boundary at the question, position-and-span provenance, registered slot bounds, nested budgets fixed before evaluation, and fail-closed validation.

FamilyDocumentRecognizer →\to recordState (bound)Question →\to keysBudget s(m)s(m)
BABILong QA1–QA3Book text with interleaved event sentences4 patterns over closed person, place, and object sets →\to move, pick-up, drop, transfer (person; place, object, recipient)Last-write maps person→\toplace and object→\toholder, joined to object→\toplace; per-object place log (256)3 forms →\to person; object; object and place1/2/3 fact slots; 4 at 7B–8B
BABILong QA7–QA8As aboveSame recordsPer-person inventory log; count of held objects2 forms →\to person1/2/8 fact slots
RULER multi-query retrievalEssay text with key–value needles1 pattern →\to (key, value)Last-write key→\tovalue map (256)Keys named in the question40/69/91 replay tokens (Qwen2.5); 37/55/97 (SmolLM2)
RULER common-word aggregationNumbered list of words1 pattern →\to list itemExact counter (512 or 1,024; overflow aborts)Frequency rank3/7/10 words (Qwen2.5); 3/5/10 (SmolLM2)

Table 1. Writer structure for each evaluated document family. The recognizer, the state composition, and the question map form the family’s schema; the write boundary, provenance, slot bounds, nested budgets, and fail-closed validation are shared. Bounds and budgets are the registered configuration values, listed in order of reader size.

Two of the four families are not bAbI-style. RULER multi-query retrieval embeds key–value needles in essay text, and RULER common-word aggregation presents a numbered list of words; for each, the recognizer is a single regular expression and the state a single primitive, a key–value map or an exact counter. Adapting ASCENT to a new document family therefore amounts to supplying its schema. The rest of the pipeline sees a recognizer only through its output, so any extractor, rule-based or learned, that emits position-anchored verbatim records before the question reuses the same state primitives, budget, and validation unchanged.

The implementation contains no instance answer. Parser checks precede model decoding, and state entries preserve document order under deterministic tie-breaking. Each released BABILong result row records the fields the writer read, that the reference answer was inaccessible until the prompt had been rendered, and that every retained fact is a verbatim span of the input at its recorded position (§ 8).

Co-Scaled Budget

The budget s(m)s(m) is fixed before evaluation. It may grow in the number of events, slots, or replay tokens, but remains small relative to model storage. Fixed-state arms separate larger-reader effects from richer-state effects; the full factorial estimates their interaction.

Readout and Representation

The reader supplies only query-relevant state. For event-state QA, a canonical deictic-free form separates semantic entities, relations, and temporal order. Matched representation experiments identify canonical serialization as the effective interface: at 3B it raises official 16K accuracy by 15.38 points over raw event wording while leaving every Foundation prediction unchanged. The canonical interface was selected on an 8K panel and fixed before official 16K evaluation.

Implementation Details

For the principal BABILong experiment, the writer extracts occurrence-indexed entities, relations, and temporal events from the context before the question is presented to the model. The budgets are one, two, and three query-relevant fact slots for the 0.5B, 1.5B, and 3B Qwen2.5 readers, respectively. The readout uses a fixed canonical serialization and at most eight new decoding tokens. Models are loaded from pinned revisions, are never fine-tuned, and decode greedily. Foundation and ASCENT predictions share the same tokenizer, answer normalizer, and scoring implementation.

Experimental Protocol

Models and Benchmarks

We use frozen Qwen2.5-Instruct [84] readers at 0.5B, 1.5B, 3B, and 7B parameters and SmolLM2 [85] readers at 135M, 360M, and 1.7B parameters. A separately frozen breadth extension contains Mistral-7B-Instruct-v0.3 [86], Falcon3-7B-Instruct [87], Granite-3.3-8B-Instruct [88], and Qwen3-8B in nonthinking mode [89]. Official RULER tasks cover multi-query retrieval and common-word aggregation. Official BABILong tasks cover retrieval, composition, and aggregation at 4K, 8K, and 16K contexts. All decoding is greedy and uses the same normalization path within a comparison.

Experimental Design and Inference

Configurations, panel hashes, model revisions, decision rules, and analysis code are fixed before scores are inspected. The main Qwen2.5 16K study contains ten panels, each with 40 QA2 and 40 QA3 examples. The two directional adjacent gain contrasts form one family and are tested with panel-clustered tt statistics and Holm family-wise error control at 0.05. All panels and tasks are retained.

The panel, rather than the individual question, is the unit of inference. For each endpoint, we first average QA2 and QA3 accuracy within each of ten independently selected panels and then compute the paired Ascent–Foundation gain. Student-tt intervals use the ten panel estimates (9 degrees of freedom). The two one-sided adjacent gain-increment tests were specified before decoding and corrected jointly by Holm’s step-down procedure. Endpoint intervals and two-sided adjacent intervals characterize effect size; the prespecified directional family supplies the primary decision rule.

A separate mechanism study crosses Qwen2.5 1.5B, 3B, and 7B with nested state sizes K∈{2,3,5}K \in\{2,3,5\}. Interface selection uses one independent panel; the factorial analysis uses nine disjoint panels, each containing 256 new examples from an independent categorical observation channel. Two latent values in {0,…,7}\{0,\ldots,7\} must be composed by ordinary addition. To avoid parse and arbitrary-token-frequency confounds, we score the exact joint likelihood of the 15 equal-length two-token continuations 00,...,14 after one fixed assistant prefix. The fifteen continuations tokenize to equal length at every row and endpoint. The gain is the reduction in candidate-normalized target NLL versus Foundation. In addition to three co-scaled gains and two adjacent increments, the interactions compare the value of K2→K3K_2 \to K_3 at 3B versus 1.5B and the value of K3→K5K_3 \to K_5 at 7B versus 3B. All eight two-sided t8t_8 intervals must have positive lower bounds; a Bonferroni family supplies simultaneous 95% intervals over all eight contrasts.

The semantic validation set is selected score-blind from the public BABILong training-generation split. We exclude any row whose task, normalized question, and target-blind parsed-event sequence matches any row in the full official 8K test pool. The resulting 800 rows have zero such semantic fingerprint overlap. It supplies a distribution-matched replication alongside the official held-out evaluation.

Baselines and Controls

The baseline suite includes the foundation-only reader; fixed versus co-scaled state; exact and byte-matched FIFO memory; lexical retrieval; BGE-M3 dense plus BM25 reciprocal-rank fusion and reranking; same-FLOP static replay; irrelevant and corrupted state; certified-only, latent-only, and fused paths; and released task-trained Q-RAG QA2/QA3 checkpoints. Q-RAG is evaluated on the same 800 examples and with the same frozen Qwen readers, preserving its task-specific training advantage.

Theory–Experiment Correspondence

Table 2 links each principal claim to a distinct observable quantity. Exact-posterior transitions establish Proposition 2. Diagonal contrasts test the co-scaled system property, while crossed interactions identify Proposition 4 at matched state refinements. Official generation establishes the task-level consequence. Systems measurements instantiate Proposition 5.

Theoretical claimEmpirical quantityQuantitative resultConclusion
Refinement value (Prop. 2)Exact-posterior loss under each nested KK transition13,824 of 13,824 endpoint–transition row checks: each larger state extends the smaller one and adds an observation (true by construction); panel-mean exact-posterior NLL decreases on every transition in every panel.Design check1
Co-scaled complementarity (Eq. (3))Diagonal gain and two adjacent increments.388→2.099→7.406.388 \to2.099 \to7.406 nats; increments 1.7111.711 [1.540,1.8821.540,1.882] and 5.3075.307 [5.053,5.5615.053,5.561].Established
Finite-reader interaction (Prop. 4)Two crossed interaction contrasts.781.781 [.539,1.023.539,1.023] and .968.968 [.671,1.264.671,1.264]; simultaneous 95% CIs (Bonferroni).Identified
Task-level consequenceOfficial 16K generation accuracy gain10.50→78.88→82.6310.50 \to78.88 \to82.63 points; both directional contrasts remain significant after Holm correction.Established
Relative footprint (Prop. 5)External-state bytes and state/model ratio589.0→624.6→640.6589.0 \to624.6 \to640.6 bytes while the ratio falls 2.189×10−6→8.631×10−7→1.872×10−72.189 \times10^{-6} \to8.631 \times10^{-7} \to1.872 \times10^{-7}.Established

Table 2. Theory–experiment correspondence. Each proposition is paired with its identifying estimand and empirical confirmation. Mechanism intervals are Bonferroni-simultaneous over the eight-estimand family; official accuracy uses the prespecified Holm-corrected directional family.

Nested state differences estimate the information supplied by refinement; adjacent diagonal differences test Eq. (3); crossed second differences isolate the reader-dependent extraction term in Eq. (11); and normalized state bytes test the footprint statement. Each theoretical operator has its own observable.

Results

Qwen2.5 Scaling on Official BABILong

Figure 2 shows every panel-level observation. Both evaluation families produce the same sharply increasing curve. The direct-peer margin becomes positive at 1.5B and increases further at 3B.

Figure 2. Scale-complementary gains across three ordered Qwen2.5 readers. Gray points are the ten task-panel clusters; colored symbols are means and vertical bars are 95% Student-tt intervals. (A) Official 16K BABILong. (B) Score-blind validation with zero semantic-fingerprint overlap with the pinned official pool. (C) Direct comparison with released task-trained Q-RAG on identical rows. The official and zero-overlap curves reproduce the same strict adjacent ordering; the peer margin crosses zero at 1.5B and widens at 3B.

Table 3 reports the two prespecified Qwen2.5 curves. The official 16K curve increases from 10.50 to 78.88 to 82.63 percentage points. The adjacent increments are 68.38 points (pHolm=3.43×10−11p_{\mathrm{Holm}} = 3.43 \times10^{-11}) and 3.75 points (pHolm=0.0255p_{\mathrm{Holm}} = 0.0255). Both adjacent directional contrasts remain significant under family-wise correction; the second is significant but marginal (Appendix B).

SplitReaderFoundationASCENTGain [95% CI]
Official 16K0.5B.0563.1613.1050 [.0680,.1420]
Official 16K1.5B.1888.9775.7888 [.7616,.8159]
Official 16K3B.1613.9875.8263 [.8097,.8428]
Semantic holdout0.5B.1263.1613.0350 [.0065,.0635]
Semantic holdout1.5B.2025.9750.7725 [.7419,.8031]
Semantic holdout3B.1763.9838.8075 [.7668,.8482]

Table 3. Qwen2.5 task-balanced QA2+QA3 accuracy. Intervals are panel-clustered 95% intervals for ASCENT–Foundation gain.

The semantic-holdout curve increases from 3.50 to 77.25 to 80.75 points. Its adjacent increments are 73.75 points (pHolm=1.65×10−13p_{\mathrm{Holm}} = 1.65 \times10^{-13}) and 3.50 points (pHolm=0.02897p_{\mathrm{Holm}} = 0.02897), reproducing both positive adjacent increments.

At 3B on the official 16K panels, the canonical representation reaches .9875 accuracy versus .8338 for the frozen raw representation, a 15.38-point difference with 95% CI [13.69,17.06][13.69,17.06]. Foundation predictions are identical row-by-row, isolating the state representation.

Direct Full-Factorial Scale Interaction

Figure 3 reports the complete factorial mechanism study. The co-scaled NLL gains increase from .388 to 2.099 to 7.406 nats (Figure 3). Their adjacent increments are 1.711 [1.540,1.882][1.540,1.882] and 5.307 [5.053,5.561][5.053,5.561]. Both model-by-state interactions are positive: .781 [.539,1.023][.539,1.023] and .968 [.671,1.264][.671,1.264]. These Bonferroni-simultaneous intervals retain positive lower bounds over the full eight-estimand family. Exact nesting, target isolation, Foundation probability identity across KK, and all 13,824 endpoint–transition row checks hold across the factorial. The row checks hold by construction of the nested writer

Figure 3. Direct identification of scale complementarity. (A) The light trajectories expose all nine independent panels; colored summaries give the mean and 95% interval in every reader–state cell, and the navy starred line is the co-scaled diagonal. (B) Bonferroni-simultaneous 95% intervals for the two diagonal increments, two model-by-state interactions, and largest-reader refinement. Their positive lower bounds jointly establish the diagonal property and the finite-reader mechanism in Proposition 4.

(Table 2, note); the per-row exact-posterior tallies are in Appendix § 16. Reruns of factorial panels on other instances are reported in § 8.

Public 7B–8B Breadth

Every planned public 7B–8B reader has complete ten-panel official 16K BABILong QA2/QA3 results (Figure 4). All five gain intervals exclude zero: Qwen2.5-7B gains 66.75 points, Qwen3-8B 72.00, Mistral-7B 80.75, Falcon3-7B 80.88, and Granite-3.3-8B 85.63. The gain persists across five architectures in the 7B–8B range.

Figure 4. Official 16K BABILong breadth across five public 7B–8B readers. Gray points are the ten task-panel gains; symbols are means and bars are panel-clustered 95% intervals. All five architectures realize large positive gains, extending the task-level consequence beyond the Qwen2.5 scaling family.

Cross-Family and Cross-Task Evidence

The cross-task evidence matrix extends the primary result across model families, task structures, and context lengths. For SmolLM2 on official RULER common-word aggregation, gains increase from .2628 to .4378 to .9561, with both adjacent intervals strictly positive. Qwen2.5 at 16K also has increasing means, .2417 to .6144 to .7100, with a positive log-scale slope. On official RULER multi-query retrieval, both Qwen2.5 (.0438 to .1208 to .3188) and SmolLM2 (.1042 to .2167 to .5229) have positive intervals for both adjacent increments. Official BABILong 4K and 8K studies cover QA1–QA3, while QA7–QA8 provide aggregation evidence.

Direct Current-Peer Comparison

Against released task-trained Q-RAG checkpoints on identical QA2/QA3 panels, frozen ASCENT leads by 22.75 points [.1885,.2666] at 1.5B and by 26.25 points [.2307,.2943] at 3B. The increasing positive margin at the two larger shared scales demonstrates that scale-complementary state can surpass task-trained retrieval without updating the foundation reader.

Controls and Systems Evidence

ASCENT outperforms byte-matched FIFO, lexical retrieval, and BGE-M3 hybrid retrieval on matched SmolLM2 panels. Irrelevant and corrupted-memory controls isolate the benefit of structured evidence from prompt length. Across 18 clean processes, median isolated decode-time ratios are .4130, .4247, and .2791 for the three ordered readers. Fixed write-plus-read state occupies 589.0, 624.6, and 640.6 bytes, while the state-to-model byte ratio decreases with scale. Supported-operator profiles show 83.7×\times, 79.4×\times, and 78.2×\times fewer FLOPs than the foundation generate path.

Limitations

Scope of the claim. Scale complementarity is demonstrated here for Qwen2.5 readers on official BABILong and RULER tasks, under accuracy and under candidate-normalized negative log-likelihood, with a structured-fact state schema. It is not shown to be a general property of frozen long-context readers with external memory. Within this scope three results are unfavorable and are reported as such: the accuracy gain decreases from 3B to 7B; the upper adjacent interval for Qwen2.5 common-word aggregation includes zero; and the SmolLM2 same-refinement interaction intervals include zero.

Schema scope. The writer schema used here (Table 1) records typed events, assignment edges, occurrence-indexed key–value evidence, and count summaries. These are the structures present in the evaluated structured-fact-retrieval benchmarks. The present evidence therefore establishes scale complementarity for external state constructed against such structures, and does not establish it as a general property of frozen readers with arbitrary external memory. Appendix D reports the identical pipeline with three schema-free writers (a sentence-recency window, a lexical retrieval chain, and BGE-M3 hybrid retrieval with reranking) in place of the structured-fact writer. None satisfies the registered co-scaling rule: at the same one-to-three-slot budget the generic state is worse than no state at every scale (gains of −5 to −18 points), and the structured-fact writer’s advantage over each generic writer is 15 points at 0.5B and 89–99 points at 1.5B and 3B. The pattern reported here is therefore a property of external state constructed against this schema. Extending the writer to non-templated natural text is the principal open direction.

Unfavorable and marginal results. The claim is specific to scale and estimand. Official BABILong accuracy establishes increasing gains from 0.5B to 3B; the second increment (1.5B→3B, 3.75 points) is significant but marginal after Holm correction (pHolm=0.0255p_{\mathrm{Holm}} = 0.0255), and Appendix § 16 shows that the ten-panel design detects an increment of this size with power of about .52, against a minimum detectable increment of about 5.3–5.8 points at 80% power. At 7B, Foundation accuracy rises to .3325 and four-slot ASCENT reaches its ceiling; the gain is .6675, yielding a 3B-to-7B increment of −.1588 [−.2055, −.1120]. The 7B conclusion therefore rests on the non-saturated NLL factorial, where both model-by-state interactions have positive simultaneous lower bounds. Likewise, Qwen2.5 common-word aggregation establishes a positive scaling slope, while its upper adjacent interval is .0956 [−.0043, .1954]. The task-trained Q-RAG comparison favors ASCENT at 1.5B and 3B, but not at 0.5B. These cases match Proposition 1: complementarity requires both informative refinement and reader headroom. The SmolLM2 factorials reproduce increasing co-scaled gains, while their same-refinement interaction intervals overlap zero; the Qwen NLL factorial therefore supplies the direct reader-by-state interaction result.

The architecture study covers autoregressive Transformers rather than recurrent or linear readers. The distribution-matched semantic set originates from a training-generation split and is consequently secondary to the official held-out results. The direct interaction study uses a controlled compositional task and teacher-forced likelihood, so natural-text generation remains an important extension. Finally, the systems result establishes bounded state, decreasing relative storage, isolated decode gains, and 78.2–83.7×\times supported-operator FLOP reductions; its profiler-enclosed end-to-end wall-time ratios of .9008, .8651, and .7262 are reported as measured.

Reproducibility and Ethics

Public record. The code, frozen configurations, per-row result records, analysis files, and audit reports are released at https://github.com/XingjianZhang-dev/ascent and archived at Zenodo (version DOI 10.5281/zenodo.22865847). One CPU command (make reproduce) re-verifies the SHA-256 manifests, recomputes every analysis file from the per-row records, rerenders every table in this paper byte for byte, and prints every reported number; VERIFICATION.md maps each value to its records. The freeze commit precedes the first score for all 25 frozen configurations with retained scores (PREREGISTRATION_TIMELINE.md).

Reproduction on other hardware. Every formal run executed on NVIDIA RTX PRO 6000 Blackwell instances. Reruns on matched hardware reproduced three audit cells identically, at the level of prediction and, where floats are stored, IEEE-754 binary64 identity: the factorial development panel (256 records, node 1 versus node 2), official 16K panel 1 for Qwen2.5-7B (80 rows, node 1 versus node 2), and all ten official 16K panels for Qwen3-8B (800 rows, third instance). The three instances have distinct GPU serial numbers, and the first two have distinct host driver versions. The Qwen3-8B rerun is a post-hoc code audit and enters no reported number; it ran from a one-commit repository snapshot whose working tree also held uncommitted manuscript-packaging edits, none of which the runner imports, and the diff is retained in the repository. Every other formal cell was run once.

A rerun on a separately provisioned instance with a different GPU architecture (NVIDIA A100, Ampere) and identical software pins reproduced the reported conclusions; stored values were not bit-identical. Across all sixty official-16K and semantic-holdout cells (4,800 rows, both arms), 124 of 9,600 scored outputs changed (103 Foundation, 21 Ascent); twenty cells were unchanged, and single 80-row panels moved by up to 5.00 accuracy points (four rows). The ten-panel means moved by at most 1.00 point (official 16K: 10.50/78.88/82.63 → 9.75/78.88/83.25 points; semantic holdout: 3.50/77.25/80.75 → 4.50/77.00/80.63), inside every reported interval half-width (1.66–3.70 points), and every registered directional contrast still passes (second official increment 4.38 points, pHolm=0.0028p_{\mathrm{Holm}} = 0.0028; semantic holdout 3.63 points, pHolm=0.022p_{\mathrm{Holm}} = 0.022). On factorial panels 2 and 3, one to two of 256 argmax candidates per arm flipped and the panel NLL gains moved by approximately 0.01 nats against a reported half-width of 0.16 nats. Stored floating-point values differ in their low-order bits, as expected across kernel implementations. Same-instance controls were bit-identical, so the differences follow from the architecture change. The numbers reported in this paper are the Blackwell values.

Determinism. Decoding is greedy in bfloat16 with pinned weights, revisions, panels, seeds, and batch sizes; torch.use_deterministic_algorithms and a TF32 policy were not set (reproduction/DETERMINISM_POLICY.md). The flag requests algorithms that return the same output for the same input when run on the same software and hardware [90]. The pipeline already has that property without it: the three matched-hardware audit cells, two complete executions of the pre-registered A100 audit cells, and six A100 cells run with and without the target-blindness instrumentation are each identical at the prediction level and, where floats are stored, at binary64. Strict same-hardware determinism was thus obtained with the default kernels, at the throughput of the runs as executed; the flag was not needed, and a flag-enabled run was not timed. The flag does not extend across GPU architectures. cuBLAS guarantees bit-wise identical results only on GPUs with the same architecture and the same number of streaming multiprocessors [91], and PyTorch does not guarantee reproducibility across platforms [92]; the Blackwell and A100 instances execute architecture-specific matrix-multiplication and attention kernels whose floating-point reductions need not round identically. The 124 changed outputs arise from this difference, and enabling the flag would not have removed them. Agreement across architectures is accordingly assessed where the conclusions are stated, in panel means and registered contrasts, and it holds there.

Target-blindness. The BABILong writer never receives the target; it is called with the input and the question only. The writer, readout, and prompt builder run behind a blinded row view that raises on any access to the answer field until the prompt has been rendered, and every result row records the fields read, the number of blocked accesses (zero), and a provenance check that each retained fact is the verbatim input span at its recorded position. All 4,800 rows of the official 16K and semantic-holdout studies were run with this instrumentation; on the same instance the instrumented and uninstrumented runners produced identical outputs.

Interruptions and licenses. The RULER common-word confirmation had two pre-score interruptions, a missing legacy configuration field and a node without the SmolLM2-1.7B weights; neither produced a score, and both are logged in the release. The released per-row records omit benchmark row bodies and model weights, which remain under their original licenses; panel identifiers, hashes, pinned revisions, and construction scripts are released instead.

Conclusion

On the evaluated tasks, external state becomes more valuable as a frozen language-model reader scales. Conditional information theory specifies the value of nested refinement, and the factorial interaction shows that larger readers extract increasingly more of that value. Official RULER and BABILong evaluations reproduce the increasing-gain pattern across model families, tasks, and context lengths; five public 7B–8B readers retain large absolute gains. Direct-peer comparisons, systems measurements, representation controls, and cross-node reproduction complete the theory-to-evidence chain. Scale complementarity is therefore identifiable, achievable, and reproducible in the evaluated frozen long-context language-model systems, within the scope stated in Section 7.

CRediT authorship contribution statement

Xingjian Zhang: Conceptualization, Methodology, Software, Formal analysis, Investigation, Validation, Data curation, Visualization, Writing – original draft, Writing – review and editing, Project administration.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

The author declares that he has no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

Code and analysis artifacts are publicly available at https://github.com/XingjianZhang-dev/ascent and archived at https://doi.org/10.5281/zenodo.22865847. Panel identifiers, hashes, and construction scripts are released for BABILong and RULER; the underlying benchmark corpora are not redistributed and remain under their original licenses (Apache-2.0 and BSD components; see THIRD_PARTY_NOTICES.md in the repository). Third-party model weights are not redistributed and remain subject to their original licenses.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

During the preparation and revision of this work, the author used Chat-GPT and Codex and, during the revision, Claude Code as assistive tools for implementing author-specified code modifications, conducting routine checks, and improving the clarity and wording of the manuscript and response to reviewers. No AI tool produced any reported measurement. All research questions, methodological decisions, experimental design, validation, interpretation of results, and final written content were determined, reviewed, and approved by the author, who takes full responsibility for the work. A per-file record is provided as AI_ASSISTANCE.md in the repository.

[90] PyTorch Contributors, torch.use_deterministic_algorithms, PyTorch 2.8 documentation (2025). URL https://docs.pytorch.org/docs/2.8/generated/torch.use_deterministic_algorithms.html

[91] NVIDIA Corporation, cuBLAS library user guide, release 12.8: Results reproducibility (section 2.1.4), CUDA Toolkit 12.8 documentation (2025). URL https://docs.nvidia.com/cuda/archive/12.8.0/cublas/index.html

[92] PyTorch Contributors, Reproducibility, PyTorch 2.8 documentation, Notes (2025). URL https://docs.pytorch.org/docs/2.8/notes/randomness.html

Complete Factorial Cell Table

Table A.1 lists the mean candidate-normalized NLL gain and its 95% Student-t8t_8 interval in every reader×state cell of the numeric factorial (nine independent 256-row panels per cell; Figure 3 plots the same cells). Starred cells form the co-scaled diagonal. Reading along a row isolates the state-budget component at fixed reader; reading down a column isolates the reader component at fixed state; the diagonal increments and the two crossed interactions in Table A.2 are second differences of these entries.

ReaderK=2K=2K=3K=3K=5K=5
1.5B.388 [.340,.437]⋆.295 [.247,.342].304 [.248,.359]
3B1.412 [1.288,1.535]2.099 [2.004,2.194]⋆2.587 [2.536,2.638]
7B3.709 [3.460,3.958]5.950 [5.637,6.264]7.406 [7.246,7.566]⋆

Table A.1. Mean NLL gain (nats) with 95% t8t_8 intervals in all nine reader×state cells. ⋆ marks the co-scaled diagonal.

ReaderState KKGain (nats)Simultaneous 95% CI
1.5B2.388[.311,.465]
3B32.099[1.948,2.251]
7B57.406[7.151,7.661]
Model×\timesstate: 1.5B→\rightarrow3B−-.781[.539,1.023]
Model×\timesstate: 3B→\rightarrow7B−-.968[.671,1.264]

Table A.2. Registered factorial estimands with Bonferroni-simultaneous 95% intervals over the eight-estimand family (t1−.05/16,8=3.677t_{1-.05/16,8}=3.677).

Statistical Formulation, Power, and Consistency

Estimands. Let gm,k,pg_{m,k,p} denote the paired Foundation-minus-ASCENT loss difference for reader mm, state size kk, and panel pp. The co-scaled curve is gˉ1.5,2,gˉ3,3,gˉ7,5\bar{g}_{1.5,2},\bar{g}_{3,3},\bar{g}_{7,5}; the diagonal increments are its adjacent differences; the interactions are δ1=(gˉ3,3−gˉ3,2)−(gˉ1.5,3−gˉ1.5,2)\delta_1=(\bar{g}_{3,3}-\bar{g}_{3,2})-(\bar{g}_{1.5,3}-\bar{g}_{1.5,2}) and δ2=(gˉ7,5−gˉ7,3)−(gˉ3,5−gˉ3,3)\delta_2=(\bar{g}_{7,5}-\bar{g}_{7,3})-(\bar{g}_{3,5}-\bar{g}_{3,3}); the eighth estimand is gˉ7,5−gˉ7,3\bar{g}_{7,5}-\bar{g}_{7,3}. All quantities are computed within panel and averaged over nine panels; ordinary intervals use t8t_8 and simultaneous intervals use t1−.05/16,8t_{1-.05/16,8}. The official Qwen2.5 BABILong study registers two one-sided adjacent accuracy-gain contrasts over ten panels with Holm step-down control at .05; the smallest Holm threshold is therefore .025 one-sided. No row, panel, endpoint, or outlier is removed.

Retrospective power. Table B.1 recomputes, from the panel-level evidence files released with the code, the between-panel SD, SE, tt statistic and raw one-sided pp of every registered contrast, the minimum detectable effect (MDE) at the registered per-test level and 80% power (exact non-central tt), and the achieved power for the observed effect. Two facts follow. The official-16K design is well powered for the first increment (t9=37.4t_9=37.4) and underpowered for the second: with the observed between-panel SD, the MDE at 80% power is 5.25 points on the second contrast’s SE (5.75 on the first’s), so the observed 3.75-point 1.5B→\rightarrow3B increment was detected at power ≈.52\approx.52 and is described as significant but marginal (pHolm=.0255p_{\mathrm{Holm}}=.0255). The very small pp-values elsewhere follow from large effects with small between-panel variance (Table B.2: a 68.38-point increment with SE 1.83 points gives t9=37.4t_9=37.4). Retrospective power uses the observed SE as if it were the true SD; it characterizes the sensitivity of the design, not the probability that a result is true.

Table B.1: Registered contrasts: effect, SE, tt, raw one-sided pp, minimum detectable effect at the registered per-test level and 80% power, and achieved power. Official-16K and semantic-holdout rows use t9t_9 and the Holm step level .025; factorial rows use t8t_8 and the Bonferroni level .05/16 one-sided.

StudyContrastEffectSEttprawp_{\mathrm{raw}}MDE80_{80}Power
Official 16K0.5B →\rightarrow 1.5B68.38 pts1.8337.441.72×10−111.72 \times10^{-11}5.75 pts1.00
Official 16K1.5B →\rightarrow 3B3.75 pts1.672.250.02555.25 pts0.52
Semantic holdout0.5B →\rightarrow 1.5B73.75 pts1.0967.888.26×10−148.26 \times10^{-14}3.42 pts1.00
Semantic holdout1.5B →\rightarrow 3B3.50 pts1.612.170.02905.08 pts0.49
FactorialDiagonal gain: 1.5B/K20.388 nats0.02118.523.73×10−83.73 \times10^{-8}0.098 nats1.00
FactorialDiagonal gain: 3B/K32.099 nats0.04150.911.23×10−111.23 \times10^{-11}0.194 nats1.00
FactorialDiagonal gain: 7B/K57.406 nats0.069106.713.32×10−143.32 \times10^{-14}0.326 nats1.00
FactorialDiagonal: 1.5B to 3B1.711 nats0.04736.741.65×10−101.65 \times10^{-10}0.219 nats1.00
FactorialDiagonal: 3B to 7B5.307 nats0.06976.904.56×10−134.56 \times10^{-13}0.324 nats1.00
Factorial7B refinement: K3 to K51.456 nats0.10713.654.00×10−74.00 \times10^{-7}0.501 nats1.00
FactorialInteraction: 1.5B to 3B0.781 nats0.06611.871.17×10−61.17 \times10^{-6}0.309 nats1.00
FactorialInteraction: 3B to 7B0.968 nats0.08112.001.07×10−61.07 \times10^{-6}0.379 nats1.00

Table B.1.

Table B.2: Statistical-consistency check: the SE and tt implied by each reported value, recomputed from the panel-level data.

Reported quantityReported valueImplied SEImplied ttdf
Official 16K 0.5B →\rightarrow 1.5B increment68.38 pts, pHolm=3.43×10−11p_{\mathrm{Holm}}=3.43 \times10^{-11}1.83 pts37.449
Official 16K 1.5B →\rightarrow 3B increment3.75 pts, pHolm=0.0255p_{\mathrm{Holm}}=0.02551.67 pts2.259
Official 16K 0.5B endpoint gain0.1050 [0.0680, 0.1420]0.01646.419
Official 16K 1.5B endpoint gain0.7888 [0.7616, 0.8159]0.012065.759
Official 16K 3B endpoint gain0.8263 [0.8097, 0.8428]0.0073112.819
Factorial Diagonal: 1.5B to 3B1.711 [1.540, 1.882] (Bonf.)0.04736.748
Factorial Interaction: 1.5B to 3B0.781 [0.539, 1.023] (Bonf.)0.06611.878
Factorial Diagonal: 3B to 7B5.307 [5.053, 5.561] (Bonf.)0.06976.908
Factorial Interaction: 3B to 7B0.968 [0.671, 1.264] (Bonf.)0.08112.008

Table B.2.

Row-level transition audit. The factorial’s 13,824 row checks (three readers ×\times two transitions ×\times nine panels ×\times 256 rows) verify that the state retained at the larger KK extends the smaller state as a prefix and adds an observation; both hold by construction of the nested writer and are reported as implementation checks. The released file reports/row_audits/FACTORIAL_TRANSITION_ROW_AUDIT_13824.csv adds, per row, the exact posterior NLL and the reader’s candidate NLL before and after each transition: under the noisy channel the exact posterior improves on 10,791 of 13,824 transitions and the reader’s NLL on 7,435, so the reported gains are panel averages rather than row-wise guarantees.

BABILong 4K/8K Results and SmolLM2 Factorials

Section 6 describes the SmolLM2 official BABILong 4K and 8K studies and the two SmolLM2 factorials qualitatively. Table C.1 gives the panel-clustered gains and adjacent increments (three panels per context, t2t_2 intervals), and Table C.2 gives all nine cells of the free-generation (QA1–QA3, 1/2/3 slots) and certified-readout (QA7–QA8, 1/2/8 slots) SmolLM2 factorials with their diagonal increments and same-refinement interactions. The certified readout converts each state refinement into the same deterministic accuracy increment at every reader scale, which is why its interactions are exactly zero (see Table C.3).

ContextPanelsGain 135MGain 360MGain 1.7BIncr. 135M→360MIncr. 360M→1.7B
4K3.2611 [.1569,.3653].5056 [.4535,.5577].6750 [.5848,.7652].2444 [.1066,.3823].1694 [.0653,.2736]
8K3.2806 [.2079,.3533].5111 [.4514,.5709].6889 [.5511,.8267].2306 [.1989,.2622].1778 [.0399,.3156]

Table C.1. SmolLM2 official BABILong QA1–QA3 accuracy gains and adjacent increments at 4K and 8K (panel-clustered 95% t2t_2 intervals).

StudyReaderSlot 1Slot 2Slot 3/8
Free generation QA1–QA3 (8K, 3 panels)135M.2417 [.1589,.3245]⋆.4472 [.3063,.5881].4972 [.3832,.6112]
360M.2528 [.2289,.2767].5000 [.4586,.5414]⋆.4972 [.4494,.5450]
1.7B.1583 [−.0568,.3735].6389 [.4653,.8125].6694 [.5197,.8192]⋆
Certified QA7–QA8 (8K, 5 panels)135M.4750 [.4244,.5256]⋆.7250 [.6744,.7756].9750 [.9244,1.0256]
360M.4438 [.3932,.4943].6938 [.6432,.7443]⋆.9438 [.8932,.9943]
1.7B.4063 [.3514,.4611].6563 [.6014,.7111].9063 [.8514,.9611]⋆

Table C.2. SmolLM2 3×33 \times3 factorials: accuracy gain in every cell (⋆: co-scaled diagonal).

StudyDiag. incr. 1Diag. incr. 2Interaction 1Interaction 2
Free generation.2583 [.1488,.3679].1694 [.0554,.2835].0417 [−.0618,.1452].0333 [−.0081,.0747]
Certified.2188 [.1462,.2913].2125 [.1549,.2701].0000 [.0000,.0000].0000 [.0000,.0000]

Table C.3. SmolLM2 factorials: diagonal increments and same-refinement interactions (panel-clustered 95% intervals).

Schema-Blind Writer Ablation

To separate “structured external state helps frozen readers, and more so as they scale” from “this parser solves this benchmark”, the identical pipeline—the same ten official 16K panels, the same per-reader slot budget s(m)=1/2/3s(m) = 1/2/3, the same bullet-list serialization and prompt, the same frozen Qwen2.5 readers, greedy decoding, and scorer—was rerun with the structured-fact writer replaced by three schema-free state constructions: (i) a sentence window, the last s(m)s(m) generic sentences of the text in document order, which reads neither the question nor any task vocabulary; (ii) a lexical chain, a deterministic BM25-style retrieval over generic sentences; and (iii) BGE-M3 hybrid retrieval with the official cross-encoder reranker over three target-blind hops. The interpretation rule was fixed before any score: a generic writer “survives” if its three endpoint gains increase strictly with reader scale and both adjacent one-sided increments pass Holm at .05, the registered rule of the canonical study. Both arms were rerun on the same instance (an NVIDIA A100, cf. § 8); the Foundation arm is identical across conditions and the canonical-versus-generic contrast is paired over panels on the same hardware, so no architecture difference enters it.

With the sentence window writer the ten-panel gains are −5.25, −17.50, −16.00 points at 0.5B, 1.5B, and 3B (95% intervals 0.5B: [−7.39,−3.11][-7.39, -3.11]; 1.5B: [−19.73,−15.27][-19.73, -15.27]; 3B: [−17.88,−14.12][-17.88, -14.12]); the adjacent increments are −12.25 and 1.50 points with pHolm=1.000p_{\mathrm{Holm}} = 1.000 and 0.265, so the registered co-scaling rule collapses. The generic state is worse than no state at every scale: Ascent accuracy is 1.00, 1.13, 0.13 against Foundation accuracy of 6.25, 18.63, 16.13. The structured-fact writer’s advantage in Ascent accuracy over this writer, paired over panels on the same instance, is 0.5B: 15.00 [12.54,17.46][12.54, 17.46]; 1.5B: 96.38 [95.08,97.67][95.08, 97.67]; 3B: 99.25 [98.62,99.88][98.62, 99.88] points. Foundation outputs differ from the same-instance canonical run in 0 of 2,400 rows.

With the lexical chain writer the ten-panel gains are −5.00, −16.38, −11.63 points at 0.5B, 1.5B, and 3B (95% intervals 0.5B: [−7.46,−2.54][-7.46, -2.54]; 1.5B: [−18.92,−13.83][-18.92, -13.83]; 3B: [−14.01,−9.24][-14.01, -9.24]); the adjacent increments are −11.38 and 4.75 points with pHolm=1.000p_{\mathrm{Holm}} = 1.000 and 0.004, so the registered co-scaling rule collapses. The generic state is worse than no state at every scale: Ascent accuracy is 1.25, 2.25, 4.50 against Foundation accuracy of 6.25, 18.63, 16.13. The structured-fact writer’s advantage in Ascent accuracy over this writer, paired over panels on the same instance, is 0.5B: 14.75 [12.49,17.01][12.49, 17.01]; 1.5B: 95.25 [93.19,97.31][93.19, 97.31]; 3B: 94.88 [93.27,96.48][93.27, 96.48] points. Foundation outputs differ from the same-instance canonical run in 0 of 2,400 rows.

With the BGE-M3 hybrid retrieval writer the ten-panel gains are −5.50, −11.00, −5.38 points at 0.5B, 1.5B, and 3B (95% intervals 0.5B: [−7.97,−3.03][-7.97, -3.03]; 1.5B: [−15.02,−6.98][-15.02, -6.98]; 3B: [−8.27,−2.48][-8.27, -2.48]); the adjacent increments are −5.50 and 5.63 points with pHolm=0.987p_{\mathrm{Holm}} = 0.987 and 0.023, so the registered co-scaling rule collapses. The generic state is worse than no state at every scale: Ascent accuracy is 0.75, 7.63, 10.75 against Foundation accuracy of 6.25, 18.63, 16.13. The structured-fact writer’s advantage in Ascent accuracy over this writer, paired over panels on the same instance, is 0.5B: 15.25 [12.66, 17.84]; 1.5B: 89.88 [87.00, 92.75]; 3B: 88.63 [86.92, 90.33] points. Foundation outputs differ from the same-instance canonical run in 0 of 2,400 rows.

No schema-free writer satisfies the registered co-scaling rule (Table D.1). At the same one-to-three-slot budget, generic state constructions rarely retain the supporting facts, so the reader answers worse from them than from the full 16K context, the gains are negative at every scale, and they do not increase with reader scale in the registered sense. The co-scaling pattern reported in the main text is therefore a property of external state constructed against the evaluated structured-fact schema; whether a schema-free writer with a much larger budget, or a learned extractor, would recover it is not tested here. Section 7 scopes the claim accordingly.

WriterGain 0.5BGain 1.5BGain 3BpHolmp_{\mathrm{Holm}} (1)pHolmp_{\mathrm{Holm}} (2)Co-scaling
Sentence window−5.25−17.50−16.001.0000.265collapses
Lexical BM25 chain−5.00−16.38−11.631.0000.004collapses
BGE-M3 hybrid + reranker−5.50−11.00−5.380.9870.023collapses

Table D.1. Schema-blind writer ablation on official 16K BABILong: ASCENT gain (points) per reader for each generic writer, Holm-corrected one-sided pp for the two adjacent increments, and whether the registered co-scaling rule survives.

Pinned Model Revisions

Every checkpoint is loaded from the Hugging Face revision in Table E.1; the frozen configurations additionally record each weight shard’s byte size and SHA-256, and every runner re-hashes the shards before loading and aborts on mismatch. The Qwen2.5 readers are the Instruct variants; Qwen3-8B runs in non-thinking mode.

CheckpointHugging Face revisionShardsBytes
EleutherAI/pythia-1.4b9cc5c8c8148a––
EleutherAI/pythia-1bc0fea4b9176112,090,751,965
EleutherAI/pythia-2.8bdbe7ae300a54––
EleutherAI/pythia-410mbba6a464f54b––
HuggingFaceTB/SmolLM2-1.7B-Instruct31b70e2e869a13,422,777,952
HuggingFaceTB/SmolLM2-135M-Instruct12fd25f773661269,060,552
HuggingFaceTB/SmolLM2-360M-Instructa10cc1512eab1723,674,912
Qwen/Qwen2.5-0.5B-Instruct7ae557604adf1988,097,824
Qwen/Qwen2.5-1.5B-Instruct989aa7980e4c13,087,467,144
Qwen/Qwen2.5-3B-Instructaa8e7253799326,171,926,992
Qwen/Qwen2.5-7B-Instructa09a35458c70415,231,271,888
Qwen/Qwen3-8Bb968826d9c46516,381,516,776
ibm-granite/granite-3.3-8b-instruct51dd4bc2ade4416,341,771,384
mistralai/Mistral-7B-Instruct-v0.3c170c708c41d314,496,080,928
tiiuae/Falcon3-1B-Instruct28ba2251970a13,338,836,632
tiiuae/Falcon3-3B-Instruct411bb94318f926,455,333,640
tiiuae/Falcon3-7B-Instruct1e57a0ecd176414,911,130,600

Table E.1. Pinned checkpoint revisions (first twelve characters of the commit hash), shard counts, and total weight bytes as recorded in configs/. “–”: the configuration pins the revision without a shard manifest; the corresponding result records carry the loaded shard hashes.

References

  1. [1]C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, B. Ginsburg, Ruler: What’s the real context size of your long-context language models?, in: Conference on Language Modeling, 2024.arxiv.org/abs/2404.06654
  2. [2]Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, M. Burtsev, Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, in: Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, Vol. 37, 2024, pp. 106519–106554. doi:10.52202/079017-3381. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/c0d62e70dbc659cc9bd44cbcf1cb652f-Abstract-Datasets_and_Benchmarks_Track.htmlDOI
  3. [3]A. Sorokin, N. Buzun, A. Anokhin, E. Vedernikov, P. Anokhin, M. Burtsev, E. Burnaev, Q-rag: Long context multi-step retrieval via value-based embedder training, in: International Conference on Learning Representations, 2026, oral presentation. URL https://openreview.net/forum?id=MS9nWFY7LGarxiv.org/abs/2511.07328
  4. [4]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, arXiv preprint arXiv:1706.03762 (2017). URL https://arxiv.org/abs/1706.03762
  5. [5]Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, R. Salakhutdinov, Transformer-xl: Attentive language models beyond a fixed-length context, arXiv preprint arXiv:1901.02860 (2019). URL https://arxiv.org/abs/1901.02860
  6. [6]J. W. Rae, A. Potapenko, S. M. Jayakumar, T. P. Lillicrap, Compressive transformers for long-range sequence modelling, arXiv preprint arXiv:1911.05507 (2019). URL https://arxiv.org/abs/1911.05507
  7. [15]A. Gu, T. Dao, Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752 (2023). URL https://arxiv.org/abs/2312.00752
  8. [16]T. Dao, A. Gu, Transformers are smms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv:2405.21060 (2024). URL https://arxiv.org/abs/2405.21060
  9. [17]M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, C. Re, Hyena hierarchy: Towards larger convolutional language models, arXiv preprint arXiv:2302.10866 (2023). URL https://arxiv.org/abs/2302.10866
  10. [18]B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV, X. He, H. Hou, J. Lin, P. Kazienko, J. Kocon, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, G. Song, X. Tang, B. Wang, J. S. Wind, S. Wozniak, R. Zhang, Z. Zhang, Q. Zhao, P. Zhou, Q. Zhou, J. Zhu, R.-J. Zhu, Rwkv: Reinventing rnns for the transformer era, arXiv preprint arXiv:2305.13048 (2023). URL https://arxiv.org/abs/2305.13048
  11. [19]Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, F. Wei, Retentive network: A successor to transformer for large language models, arXiv preprint arXiv:2307.08621 (2023). URL https://arxiv.org/abs/2307.08621
  12. [20]T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Re, Flashattention: Fast and memory-efficient exact attention with io-awareness, arXiv preprint arXiv:2205.14135 (2022). URL https://arxiv.org/abs/2205.14135
  13. [21]T. Dao, Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.08691 (2023). URL https://arxiv.org/abs/2307.08691
  14. [22]H. Liu, M. Zaharia, P. Abbeel, Ring attention with blockwise transformers for near-infinite context, arXiv preprint arXiv:2310.01889 (2023). URL https://arxiv.org/abs/2310.01889
  15. [23]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, Efficient memory management for large language model serving with pagedattention, arXiv preprint arXiv:2309.06180 (2023). URL https://arxiv.org/abs/2309.06180
  16. [24]H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. H. Abdi, D. Li, C.-Y. Lin, Y. Yang, L. Qiu, Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention, arXiv preprint arXiv:2407.02490 (2024). URL https://arxiv.org/abs/2407.02490
  17. [25]Y. Leviathan, M. Kalman, Y. Matias, Fast inference from transformers via speculative decoding, arXiv preprint arXiv:2211.17192 (2022). URL https://arxiv.org/abs/2211.17192
  18. [26]C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper, Accelerating large language model decoding with speculative sampling, arXiv preprint arXiv:2302.01318 (2023). URL https://arxiv.org/abs/2302.01318
  19. [27]J. Weston, S. Chopra, A. Bordes, Memory networks, arXiv preprint arXiv:1410.3916 (2014). URL https://arxiv.org/abs/1410.3916
  20. [28]S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus, End-to-end memory networks, arXiv preprint arXiv:1503.08895 (2015). URL https://arxiv.org/abs/1503.08895
  21. [29]A. Graves, G. Wayne, I. Danihelka, Neural turing machines, arXiv preprint arXiv:1410.5401 (2014). URL https://arxiv.org/abs/1410.5401
  22. [30]A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, T. Lillicrap, One-shot learning with memory-augmented neural networks, arXiv preprint arXiv:1605.06065 (2016). URL https://arxiv.org/abs/1605.06065
  23. [31]G. Lample, A. Sablayrolles, M. Ranzato, L. Denoyer, H. Jegou, Large memory layers with product keys, arXiv preprint arXiv:1907.05242 (2019). URL https://arxiv.org/abs/1907.05242
  24. [32]W. Wang, L. Dong, H. Cheng, X. Liu, X. Yan, J. Gao, F. Wei, Augmenting language models with long-term memory, arXiv preprint arXiv:2306.07174 (2023). URL https://arxiv.org/abs/2306.07174
  25. [33]C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, J. E. Gonzalez, Memgpt: Towards llms as operating systems, arXiv preprint arXiv:2310.08560 (2023). URL https://arxiv.org/abs/2310.08560
  26. [34]W. Zhong, L. Guo, Q. Gao, H. Ye, Y. Wang, Memorybank: Enhancing large language models with long-term memory, arXiv preprint arXiv:2305.10250 (2023). URL https://arxiv.org/abs/2305.10250
  27. [35]Y. Wu, M. N. Rabe, D. Hutchins, C. Szegedy, Memorizing transformers, in: International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TrjbxzRcnf-arxiv.org/abs/2203.08913
  28. [36]Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, C. Guestrin, Learning to (learn at test time): Rnns with expressive hidden states, arXiv preprint arXiv:2407.04620 (2024).arxiv.org/abs/2407.04620
  29. [37]A. Behrouz, P. Zhong, V. Mirrokni, Titans: Learning to memorize at test time, arXiv preprint arXiv:2501.00663 (2025).arxiv.org/abs/2501.00663
  30. [38]P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W.-t. Yih, T. Rocktaschel, S. Riedel, D. Kiela, Retrieval-augmented generation for knowledge-intensive nlp tasks, arXiv preprint arXiv:2005.11401 (2020). URL https://arxiv.org/abs/2005.11401
  31. [39]K. Guu, K. Lee, Z. Tung, P. Pasupat, M.-W. Chang, Realm: Retrieval-augmented language model pre-training, arXiv preprint arXiv:2002.08909 (2020). URL https://arxiv.org/abs/2002.08909
  32. [40]V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, W.-t. Yih, Dense passage retrieval for open-domain question answering, arXiv preprint arXiv:2004.04906 (2020). URL https://arxiv.org/abs/2004.04906
  33. [41]U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, M. Lewis, Generalization through memorization: Nearest neighbor language models, arXiv preprint arXiv:1911.00172 (2019). URL https://arxiv.org/abs/1911.00172
  34. [42]S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. M. van den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al., Improving language models by retrieving from trillions of tokens, in: Proceedings of the 39th International Conference on Machine Learning, Vol. 162 of Proceedings of Machine Learning Research, PMLR, 2022, pp. 2206–2240. URL https://proceedings.mlr.press/v162/borgeaud22a.htmlarxiv.org/abs/2112.04426
  35. [43]G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, E. Grave, Atlas: Few-shot learning with retrieval augmented language models, arXiv preprint arXiv:2208.03299 (2022). URL https://arxiv.org/abs/2208.03299
  36. [44]W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, W.-t. Yih, Replug: Retrieval-augmented black-box language models, arXiv preprint arXiv:2301.12652 (2023). URL https://arxiv.org/abs/2301.12652
  37. [45]A. Asai, Z. Wu, Y. Wang, A. Sil, H. Hajishirzi, Self-rag: Learning to retrieve, generate, and critique through self-reflection, arXiv preprint arXiv:2310.11511 (2023). URL https://arxiv.org/abs/2310.11511
  38. [46]S.-Q. Yan, J.-C. Gu, Y. Zhu, Z.-H. Ling, Corrective retrieval augmented generation, arXiv preprint arXiv:2401.15884 (2024). URL https://arxiv.org/abs/2401.15884
  39. [47]P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, C. D. Manning, Raptor: Recursive abstractive processing for tree-organized retrieval, arXiv preprint arXiv:2401.18059 (2024). URL https://arxiv.org/abs/2401.18059
  40. [48]L. Gao, X. Ma, J. Lin, J. Callan, Precise zero-shot dense retrieval without relevance labels, arXiv preprint arXiv:2212.10496 (2022). URL https://arxiv.org/abs/2212.10496
  41. [49]G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, E. Grave, Unsupervised dense information retrieval with contrastive learning, arXiv preprint arXiv:2112.09118 (2021). URL https://arxiv.org/abs/2112.09118
  42. [50]O. Khattab, M. Zaharia, Colbert: Efficient and effective passage search via contextualized late interaction over bert, arXiv preprint arXiv:2004.12832 (2020). URL https://arxiv.org/abs/2004.12832
  43. [51]G. Izacard, E. Grave, Leveraging passage retrieval with generative models for open domain question answering, arXiv preprint arXiv:2007.01282 (2020). URL https://arxiv.org/abs/2007.01282
  44. [52]K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, M. Zaharia, Colbertv2: Effective and efficient retrieval via lightweight late interaction, arXiv preprint arXiv:2112.01488 (2021). URL https://arxiv.org/abs/2112.01488
  45. [53]Y. Wang, Y. Wan, X. Lei, Q. Chen, H. Hu, A retrieval augmented generation based optimization approach for medical knowledge understanding and reasoning in large language models, Array 28 (2025) 100504. doi:10.1016/j.array.2025.100504. URL https://doi.org/10.1016/j.array.2025.100504
  46. [54]H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, L. Qiu, Llmlingua: Compressing prompts for accelerated inference of large language models, arXiv preprint arXiv:2310.05736 (2023). URL https://arxiv.org/abs/2310.05736
  47. [55]H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, L. Qiu, Longllm-lingua: Accelerating and enhancing llms in long context scenarios via prompt compression, arXiv preprint arXiv:2310.06839 (2023). URL https://arxiv.org/abs/2310.06839
  48. [56]Y. Li, Unlocking context constraints of llms: Enhancing context efficiency of llms with self-information-based content filtering, arXiv preprint arXiv:2304.12102 (2023). URL https://arxiv.org/abs/2304.12102
  49. [57]S. Chen, S. Wong, L. Chen, Y. Tian, Extending context window of large language models via positional interpolation, arXiv preprint arXiv:2306.15595 (2023). URL https://arxiv.org/abs/2306.15595
  50. [58]J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, Y. Liu, Roformer: Enhanced transformer with rotary position embedding, arXiv preprint arXiv:2104.09864 (2021). URL https://arxiv.org/abs/2104.09864
  51. [59]B. Peng, J. Quesnelle, H. Fan, E. Shippole, Yarn: Efficient context window extension of large language models, arXiv preprint arXiv:2309.00071 (2023). URL https://arxiv.org/abs/2309.00071
  52. [60]Y. Ding, L. L. Zhang, C. Zhang, Y. Xu, N. Shang, J. Xu, F. Yang, M. Yang, Longrope: Extending llm context window beyond 2 million tokens, arXiv preprint arXiv:2402.13753 (2024). URL https://arxiv.org/abs/2402.13753
  53. [61]N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, Lost in the middle: How language models use long contexts, arXiv preprint arXiv:2307.03172 (2023). URL https://arxiv.org/abs/2307.03172
  54. [62]Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, J. Li, Longbench: A bilingual, multitask benchmark for long context understanding, arXiv preprint arXiv:2308.14508 (2023). URL https://arxiv.org/abs/2308.14508
  55. [63]C. An, S. Gong, M. Zhong, X. Zhao, M. Li, J. Zhang, L. Kong, X. Qiu, L-eval: Instituting standardized evaluation for long context language models, arXiv preprint arXiv:2307.11088 (2023). URL https://arxiv.org/abs/2307.11088
  56. [64]X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, M. Sun, Qobench: Extending long context evaluation beyond 100k tokens, arXiv preprint arXiv:2402.13718 (2024). URL https://arxiv.org/abs/2402.13718
  57. [65]M. Li, S. Zhang, T. Zhang, H. Duan, Y. Liu, K. Chen, Needlebench: Evaluating llm retrieval and reasoning across varying information densities, arXiv preprint arXiv:2407.11963 (2024). URL https://arxiv.org/abs/2407.11963
  58. [66]Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, D. Metzler, Long range arena: A benchmark for efficient transformers, arXiv preprint arXiv:2011.04006 (2020). URL https://arxiv.org/abs/2011.04006
  59. [67]G. Qin, Y. Feng, B. Van Durme, The nlp task effectiveness of long-range transformers, arXiv preprint arXiv:2202.07856 (2022). URL https://arxiv.org/abs/2202.07856
  60. [68]J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361 (2020).arxiv.org/abs/2001.08361
  61. [69]J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Ruther-ford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, in: Advances in Neural Information Processing Systems, Vol. 35, 2022, pp. 30016–30030. doi:10.52202/068431-2176. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract.htmlDOI
  62. [70]D. Hernandez, J. Kaplan, T. Henighan, S. McCandlish, Scaling laws for transfer, arXiv preprint arXiv:2102.01293 (2021). URL https://arxiv.org/abs/2102.01293
  63. [71]N. Muennighoff, A. M. Rush, B. Barak, T. L. Scao, A. Piktus, N. Tazi, S. Pyysalo, T. Wolf, C. Raffel, Scaling data-constrained language models, arXiv preprint arXiv:2305.16264 (2023). URL https://arxiv.org/abs/2305.16264
  64. [72]D. Blackwell, Equivalent comparisons of experiments, The Annals of Mathematical Statistics 24 (2) (1953) 265–272.DOI
  65. [73]O. Press, N. A. Smith, M. Lewis, Train short, test long: Attention with linear biases enables input length extrapolation, arXiv preprint arXiv:2108.12409 (2021). URL https://arxiv.org/abs/2108.12409
  66. [74]C. Han, Q. Wang, H. Peng, W. Xiong, Y. Chen, H. Ji, S. Wang, Lm-infinite: Zero-shot extreme length generalization for large language models, arXiv preprint arXiv:2308.16137 (2023). URL https://arxiv.org/abs/2308.16137
  67. [75]G. Xiao, Y. Tian, B. Chen, S. Han, M. Lewis, Efficient streaming language models with attention sinks, arXiv preprint arXiv:2309.17453 (2023). URL https://arxiv.org/abs/2309.17453
  68. [76]Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Re, C. Barrett, Z. Wang, B. Chen, H_2O: Heavy-hitter oracle for efficient generative inference of large language models, arXiv preprint arXiv:2306.14048 (2023). URL https://arxiv.org/abs/2306.14048
  69. [77]Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, A. Shrivastava, Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time, arXiv preprint arXiv:2305.17118 (2023). URL https://arxiv.org/abs/2305.17118
  70. [78]J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasicki, S. Han, Quest: Query-aware sparsity for efficient long-context llm inference, arXiv preprint arXiv:2406.10774 (2024). URL https://arxiv.org/abs/2406.10774
  71. [79]Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, D. Chen, Snapkv: Llm knows what you are looking for before generation, arXiv preprint arXiv:2404.14469 (2024). URL https://arxiv.org/abs/2404.14469
  72. [80]Z. Cai, Y. Zhang, B. Gao, Y. Liu, Y. Li, T. Liu, K. Lu, W. Xiong, Y. Dong, J. Hu, W. Xiao, Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, arXiv preprint arXiv:2406.02069 (2024). URL https://arxiv.org/abs/2406.02069
  73. [81]P. Zhang, Z. Liu, S. Xiao, N. Shao, Q. Ye, Z. Dou, Long context compression with activation beacon, arXiv preprint arXiv:2401.03462 (2024). URL https://arxiv.org/abs/2401.03462
  74. [82]C. Xiao, P. Zhang, X. Han, G. Xiao, Y. Lin, Z. Zhang, Z. Liu, M. Sun, Infllm: Training-free long-context extrapolation for llms with an efficient context memory, arXiv preprint arXiv:2402.04617 (2024). URL https://arxiv.org/abs/2402.04617
  75. [83]Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, J. Jia, Longlora: Efficient fine-tuning of long-context large language models, arXiv preprint arXiv:2309.12307 (2023). URL https://arxiv.org/abs/2309.12307
  76. [84]Qwen Team, Qwen2.5 technical report, arXiv preprint arXiv:2412.15115 (2024).
  77. [85]L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, A. Marafoti, et al., Smollm2: When smol goes big—data-centric training of a small language model, arXiv preprint arXiv:2502.02737 (2025).
  78. [86]A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023).arxiv.org/abs/2310.06825
  79. [87]Technology Innovation Institute, Falcon3-7b-instruct model card, Hugging Face, pinned revision 1e57a0ecd17 6c7c139f289c60a74e57f887c3dfb (2024). URL https://huggingface.co/tiiuae/Falcon3-7B-Instruct
  80. [88]IBM Granite Team, Granite-3.3-8b-instruct model card, Hugging Face, pinned revision 51dd4bc2ade4059a6bd87649d68aa11e4fb2529b (2025). URL https://huggingface.co/ibm-granite/granite-3.3-8binstruct
  81. [89]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, et al., Qwen3 technical report, arXiv preprint arXiv:2505.09388 (2025).arxiv.org/abs/2505.09388
  82. [90]PyTorch Contributors, torch.use_deterministic_algorithms, PyTorch 2.8 documentation (2025). URL https://docs.pytorch.org/docs/2.8/generated/torch.use_deterministic_algorithms.html
  83. [91]NVIDIA Corporation, cuBLAS library user guide, release 12.8: Results reproducibility (section 2.1.1.4), CUDA Toolkit 12.8 documentation (2025). URL https://docs.nvidia.com/cuda/archive/12.8.0/cublas/index.html
  84. [92]PyTorch Contributors, Reproducibility, PyTorch 2.8 documentation, Notes (2025). URL https://docs.pytorch.org/docs/2.8/notes/randomness.html

Paper details

vlnc:W_mv2xta9u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Chat-GPT, Codex, Claude Code

View statement
Looking for related papers…
Contents