WorksComputer Science
ASCENT: Scale-Complementary External State for Frozen Long-Context Language Models
Abstract
External state should become more valuable as a reader becomes more capable, yet long-context evaluations measure memory utility at a single model scale. This paper formulates scale complementarity: the absolute gain from external state increases when reader capacity and a compact state budget are jointly scaled. ASCENT realizes this principle with a target-blind causal writer, nested typed state, and a bounded query-conditioned readout; every language model remains frozen. Under log loss, the value of a nested refinement equals its target-relevant conditional information; a full factorial makes the reader-by-state interaction identifiable. Official 16K BABILong experiments show Qwen2.5 accuracy gains of 10.50, 78.88, and 82.63 percentage points at 0.5B, 1.5B, and 3B parameters, with both adjacent directional contrasts significant after family-wise correction. At 7B, Foundation accuracy rises to 33.25% while four-slot ASCENT answers every question, which caps the gain at 66.75 points; the 3B-to-7B accuracy increment is -15.88 points [-20.55, -11.20]. The accuracy-side claim is therefore established from 0.5B to 3B; the 7B conclusion rests on the non-saturated negative-log-likelihood factorial. A three-model by three-state factorial yields co-scaled negative log-likelihood gains of 0.388, 2.099, and 7.406 nats at 1.5B, 3B, and 7B; all interaction and gain contrasts retain positive simultaneous lower bounds. Five public 7B-8B readers obtain 66.75-85.63 point gains, and ASCENT exceeds task-trained retrieval checkpoints at the two larger shared scales. Controls, baselines, systems accounting, matched-hardware reproduction, and a cross-architecture sensitivity analysis accompany the experiments. The results establish scale-complementary external state under the evaluated benchmarks, metrics, and state schema.
Introduction
Long-context language-model systems increasingly separate stored evidence from the reader that reasons over it. Sparse attention, retrieval, prompt compression, and recurrent memory each reduce the amount of evidence processed directly, but they are typically judged at one reader size. That protocol leaves a central scaling question unresolved: as the reader becomes more capable, does an external state become more useful in absolute terms? We call a positive answer scale complementarity. It is stronger than showing that memory helps a large model, and it differs from a model-only scaling law: both the reader and the information budget are allowed to grow.
ASCENT is a frozen-foundation interface designed to make this property measurable. A causal target-blind writer converts the pre-query stream into nested typed state; a bounded readout exposes only query-relevant entries to a frozen reader. The state budget co-scales with reader capacity while its byte ratio to model storage falls. This separation creates two independently controllable axes, reader scale and state refinement, and makes the model-by-state interaction identifiable.
The paper unifies theory, design, and evidence at the same estimand. Under Bayes-optimal log loss, the incremental value of a nested state refinement is exactly its conditional mutual information with the target. ASCENT constructs refinements intended to make this quantity positive. A prespecified factorial identifies whether finite readers realize increasingly more of that information, while official free-generation benchmarks establish the corresponding accuracy effect. Model, data, configuration, and analysis hashes are fixed before evaluation; the panel is the unit of inference; and multiplicity is controlled at the family level.
The evidence contains official RULER [1] and BABILong [2] tasks, two ordered model families, five public 7B–8B readers, released task-trained Q-RAG checkpoints [3], raw and neural retrieval controls, representation ablations, systems accounting, cross-node reproduction on matched hardware, and a cross-architecture sensitivity analysis. This design turns scale complementarity from a visual trend into a tested property of a co-scaled language-model system.
The main contributions are:
A scale-complementarity theory connects nested state refinement to conditional information and yields a finite-reader interaction estimand.
ASCENT supplies a target-blind, bounded, nested external state whose absolute budget grows while its state-to-model storage ratio contracts.
Official 16K results establish two positive adjacent accuracy-gain increments over 0.5B, 1.5B, and 3B Qwen2.5 readers.
A nine-panel factorial identifies positive model-by-state interactions through 7B with simultaneous inference over the complete contrast family.
Five public 7B–8B readers, a direct trained-retrieval comparison, cross-task controls, systems measurements, and cross-node reproduction establish architectural breadth and reproducibility; every reported number is recomputable from the released per-row records.
Related Work
Architectures for Long Sequences
The Transformer established content-addressable global interaction [4]; Transformer-XL and compressive recurrence extended its temporal reach [5, 6]. Sparse and approximate attention reduce the quadratic path through local windows, global tokens, hashing, random features, low-rank projection, Nyström approximation, routing, and structured masks [7, 8, 9, 10, 11, 12, 13, 14]. State-space, convolutional, recurrent, and retention models provide alternative long-sequence substrates [15, 16, 17, 18, 19]`. These works scale the sequence processor itself. ASCENT instead holds each processor frozen and identifies how the value of an external information state changes with reader capacity.
System-level work attacks a complementary bottleneck through IO-aware exact attention, blockwise distribution, paged allocation, sparse prefill, and decoding acceleration [20, 21, 22, 23, 24, 25, 26]. These methods change the cost of executing a context. ASCENT changes the statistical experiment presented to the language-model reader and measures the resulting value rather than treating throughput as the primary endpoint.
External Memory and Retrieval
Learned external memory spans memory networks, differentiable computers, episodic memory, and product-key layers [27, 28, 29, 30, 31]. Long-term memory managers and test-time state extend this line to language models [32, 33, 34, 35, 36, 37]. Retrieval-augmented generation and language modeling use dense, sparse, late-interaction, black-box, self-reflective, corrective, and hierarchical retrieval [38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52]. Recent work in Array combines sparse and dense retrieval with prompt and data processing for large-model reasoning [53]; Q-RAG further trains a multi-step value-based embedder [3]. ASCENT targets a different dependent variable: the adjacent change in external-state gain under co-scaling, with no reader training.
Compression, Evaluation, and Scaling
Prompt filtering and compression improve context efficiency [54, 55, 56]; positional interpolation, rotary embeddings, YaRN, and LongRoPE extend usable windows [57, 58, 59, 60]. Long-context evaluations show that nominal length does not ensure robust evidence use [61, 62, 63, 64, 65, 66, 67]. RULER and BABILong add controlled retrieval, aggregation, and compositional reasoning at long lengths [1, 2]. Neural scaling laws relate loss to model, data, and compute [68, 69, 70, 71]. ASCENT combines these strands by treating state budget as a second scaling coordinate and using comparison of statistical experiments [72] to state when a refinement has positive value.
Training-free extrapolation and cache management preserve or select internal attention state through linear biases, streaming anchors, heavy hitters, persistent importance, query-aware sparsity, and hierarchical cache budgets [73, 74, 75, 76, 77, 78, 79, 80]. Activation beacons, InfLLM, and LongLoRA instead compress, retrieve, or adapt context representations [81, 82, 83]. These approaches motivate strong efficiency controls, but none makes the adjacent external-state gain across reader scale its principal estimand.
Problem Formulation
Let be a frozen foundation model of size , a long input, a query, and the target. A causal writer produces state under budget . A reader exposes a bounded projection of to . For loss , define
The external-state gain relative to the foundation-only system is
For ordered model sizes and a co-scaling rule , the primary empirical property is
We require and report both absolute state bytes and the ratio of state bytes to model bytes. Equation (3) is a property of the co-scaled system on a task distribution, not a universal model law.
Proposition 1 (No distribution-free strict scaling). No fixed writer, reader, and co-scaling rule can guarantee strict positive adjacent gain increments for every distribution over .
Proof. Choose a distribution for which is conditionally independent of given , or for which the foundation-only predictor is already Bayes optimal. Then every external state has zero Bayes gain. Alternatively, choose a state that contains the exact answer at the smallest scale; bounded accuracy then prevents strictly increasing absolute gain. Thus strict distribution-free scaling is impossible.
For log loss, define the Bayes risk of state as
Proposition 2 (Information value of a refinement). If refines , the Bayes-risk improvement due to the refinement is
It is strict if and only if the refinement contains target-relevant conditional information.
Proof. Because is a function of , the entropy chain rule gives .
This identity supplies a constructive rule: each state expansion should add nonredundant compositional evidence, while model scale determines how much of that evidence a finite reader converts into reduced predictive loss.
Proposition 3 (Strict complementarity on fresh episodic information). Suppose the episode target is created after foundation training and is conditionally independent of every foundation given . Let and be nested state signals. If both the no-state and state-conditioned predictors output their Bayes-optimal conditionals and
then the Bayes log-loss gains satisfy .
Proof. Fresh-information independence makes the no-state Bayes loss at every scale. Proper posterior decoding gives state loss , so the gain is . By the chain rule and nesting,
The indices in Proposition 3 carry no force: with oracle decoders the gain depends only on the nested state. The reader-dependent content of scale complementarity is isolated in Proposition 4, whose interaction term is what the factorial estimates.
The preceding statement concerns an oracle decoder. To connect it to frozen finite readers, define the nonnegative excess risk
The observed gain admits the exact decomposition
It separates information present in the state from the reader’s ability to convert that information into predictive performance.
Proposition 4 (Finite-reader interaction). For two frozen readers evaluated on the same nested refinement , let
Then the model-by-state interaction is
Consequently, a positive interaction identifies that the larger reader realizes more of the same state refinement as excess-risk reduction.
Proof. Apply Eq. (9) at and for each reader. The shared conditional-information increment cancels between readers, leaving Eq. (11).
This identity motivates the full factorial rather than a diagonal-only scaling curve. Diagonal contrasts establish the co-scaled system property; the crossed interactions determine whether model capacity changes the value extracted from an identical state expansion.
Proposition 5 (Vanishing relative footprint). If state bytes obey for , while a dense -layer foundation has parameters with , then as width .
Proof. Direct substitution yields ; since and is bounded below, the ratio converges to zero.
Thus the construction permits increasing absolute information value while the relative state footprint vanishes. The finite-reader interaction experiment tests the key empirical departure from the oracle premise: whether larger frozen language models realize more of each identical nested refinement.
ASCENT
Figure 1 summarizes the interface and the empirical co-scaling target. The writer and foundation model remain frozen throughout; the deterministic state budget alone follows the co-scaling rule. The design therefore attributes the gain curve to reader capacity and state refinement, with training held constant across endpoints.
Figure 1. Theory, mechanism, and identification meet at the external-state gain. (A) A target-blind causal writer converts long evidence into nested, bounded state for query-conditioned use by a frozen reader. (B) Proposition 2 equates the value of a strict refinement with its target-relevant conditional information. (C) The complete factorial identifies the finite-reader consequence in Proposition 4: co-scaled gain rises from .39 to 2.10 to 7.41 nats and both model-by-state interactions are positive.
Execution Contract
ASCENT is the deterministic composition
where is the causal state writer, is a bounded readout, is a fixed serializer, and is greedy decoding by the frozen reader. The foundation arm is exactly , for which . For , a deterministic projection satisfies , so every larger budget is a strict refinement rather than a different representation.
Each endpoint executes the same five operations: (i) write state from before and are revealed; (ii) validate registered types, ordering, hashes, alignment, and budget constraints; (iii) select at most query-relevant entries; (iv) serialize those entries with the fixed canonical template; and (v) decode with the unchanged model and scorer. Validation is fail-closed: a missing, malformed, or mismatched state aborts the invocation. Equation (12) isolates the two experimental coordinates: changes only the frozen decoder, and changes only a nested external-state refinement. It is the executable counterpart of Propositions 2 and 4.
Target-Blind State Construction
The writer processes the document before observing the final target. It has three parts, which Table 1 lists for every evaluated document family. A recognizer scans the input that precedes the question and emits typed records; each record carries its fields, its character position, and the verbatim input span from which it was read. A state update folds the records, in document order, into bounded state built from three primitives: a last-write map, an append-only log, and a counter. A question map identifies the question form and names the state keys from which the readout may select. The record types, their composition into state, and the question forms constitute a family’s schema; across the evaluated families the schemas record typed events, assignment edges, occurrence-indexed key–value evidence, and count summaries. Everything outside the schema is common to all families: the write boundary at the question, position-and-span provenance, registered slot bounds, nested budgets fixed before evaluation, and fail-closed validation.
| Family | Document | Recognizer record | State (bound) | Question keys | Budget |
| BABILong QA1–QA3 | Book text with interleaved event sentences | 4 patterns over closed person, place, and object sets move, pick-up, drop, transfer (person; place, object, recipient) | Last-write maps personplace and objectholder, joined to objectplace; per-object place log (256) | 3 forms person; object; object and place | 1/2/3 fact slots; 4 at 7B–8B |
| BABILong QA7–QA8 | As above | Same records | Per-person inventory log; count of held objects | 2 forms person | 1/2/8 fact slots |
| RULER multi-query retrieval | Essay text with key–value needles | 1 pattern (key, value) | Last-write keyvalue map (256) | Keys named in the question | 40/69/91 replay tokens (Qwen2.5); 37/55/97 (SmolLM2) |
| RULER common-word aggregation | Numbered list of words | 1 pattern list item | Exact counter (512 or 1,024; overflow aborts) | Frequency rank | 3/7/10 words (Qwen2.5); 3/5/10 (SmolLM2) |
Table 1. Writer structure for each evaluated document family. The recognizer, the state composition, and the question map form the family’s schema; the write boundary, provenance, slot bounds, nested budgets, and fail-closed validation are shared. Bounds and budgets are the registered configuration values, listed in order of reader size.
Two of the four families are not bAbI-style. RULER multi-query retrieval embeds key–value needles in essay text, and RULER common-word aggregation presents a numbered list of words; for each, the recognizer is a single regular expression and the state a single primitive, a key–value map or an exact counter. Adapting ASCENT to a new document family therefore amounts to supplying its schema. The rest of the pipeline sees a recognizer only through its output, so any extractor, rule-based or learned, that emits position-anchored verbatim records before the question reuses the same state primitives, budget, and validation unchanged.
The implementation contains no instance answer. Parser checks precede model decoding, and state entries preserve document order under deterministic tie-breaking. Each released BABILong result row records the fields the writer read, that the reference answer was inaccessible until the prompt had been rendered, and that every retained fact is a verbatim span of the input at its recorded position (§ 8).
Co-Scaled Budget
The budget is fixed before evaluation. It may grow in the number of events, slots, or replay tokens, but remains small relative to model storage. Fixed-state arms separate larger-reader effects from richer-state effects; the full factorial estimates their interaction.
Readout and Representation
The reader supplies only query-relevant state. For event-state QA, a canonical deictic-free form separates semantic entities, relations, and temporal order. Matched representation experiments identify canonical serialization as the effective interface: at 3B it raises official 16K accuracy by 15.38 points over raw event wording while leaving every Foundation prediction unchanged. The canonical interface was selected on an 8K panel and fixed before official 16K evaluation.
Implementation Details
For the principal BABILong experiment, the writer extracts occurrence-indexed entities, relations, and temporal events from the context before the question is presented to the model. The budgets are one, two, and three query-relevant fact slots for the 0.5B, 1.5B, and 3B Qwen2.5 readers, respectively. The readout uses a fixed canonical serialization and at most eight new decoding tokens. Models are loaded from pinned revisions, are never fine-tuned, and decode greedily. Foundation and ASCENT predictions share the same tokenizer, answer normalizer, and scoring implementation.
Experimental Protocol
Models and Benchmarks
We use frozen Qwen2.5-Instruct [84] readers at 0.5B, 1.5B, 3B, and 7B parameters and SmolLM2 [85] readers at 135M, 360M, and 1.7B parameters. A separately frozen breadth extension contains Mistral-7B-Instruct-v0.3 [86], Falcon3-7B-Instruct [87], Granite-3.3-8B-Instruct [88], and Qwen3-8B in nonthinking mode [89]. Official RULER tasks cover multi-query retrieval and common-word aggregation. Official BABILong tasks cover retrieval, composition, and aggregation at 4K, 8K, and 16K contexts. All decoding is greedy and uses the same normalization path within a comparison.
Experimental Design and Inference
Configurations, panel hashes, model revisions, decision rules, and analysis code are fixed before scores are inspected. The main Qwen2.5 16K study contains ten panels, each with 40 QA2 and 40 QA3 examples. The two directional adjacent gain contrasts form one family and are tested with panel-clustered statistics and Holm family-wise error control at 0.05. All panels and tasks are retained.
The panel, rather than the individual question, is the unit of inference. For each endpoint, we first average QA2 and QA3 accuracy within each of ten independently selected panels and then compute the paired Ascent–Foundation gain. Student- intervals use the ten panel estimates (9 degrees of freedom). The two one-sided adjacent gain-increment tests were specified before decoding and corrected jointly by Holm’s step-down procedure. Endpoint intervals and two-sided adjacent intervals characterize effect size; the prespecified directional family supplies the primary decision rule.
A separate mechanism study crosses Qwen2.5 1.5B, 3B, and 7B with nested state sizes . Interface selection uses one independent panel; the factorial analysis uses nine disjoint panels, each containing 256 new examples from an independent categorical observation channel. Two latent values in must be composed by ordinary addition. To avoid parse and arbitrary-token-frequency confounds, we score the exact joint likelihood of the 15 equal-length two-token continuations 00,...,14 after one fixed assistant prefix. The fifteen continuations tokenize to equal length at every row and endpoint. The gain is the reduction in candidate-normalized target NLL versus Foundation. In addition to three co-scaled gains and two adjacent increments, the interactions compare the value of at 3B versus 1.5B and the value of at 7B versus 3B. All eight two-sided intervals must have positive lower bounds; a Bonferroni family supplies simultaneous 95% intervals over all eight contrasts.
The semantic validation set is selected score-blind from the public BABILong training-generation split. We exclude any row whose task, normalized question, and target-blind parsed-event sequence matches any row in the full official 8K test pool. The resulting 800 rows have zero such semantic fingerprint overlap. It supplies a distribution-matched replication alongside the official held-out evaluation.
Baselines and Controls
The baseline suite includes the foundation-only reader; fixed versus co-scaled state; exact and byte-matched FIFO memory; lexical retrieval; BGE-M3 dense plus BM25 reciprocal-rank fusion and reranking; same-FLOP static replay; irrelevant and corrupted state; certified-only, latent-only, and fused paths; and released task-trained Q-RAG QA2/QA3 checkpoints. Q-RAG is evaluated on the same 800 examples and with the same frozen Qwen readers, preserving its task-specific training advantage.
Theory–Experiment Correspondence
Table 2 links each principal claim to a distinct observable quantity. Exact-posterior transitions establish Proposition 2. Diagonal contrasts test the co-scaled system property, while crossed interactions identify Proposition 4 at matched state refinements. Official generation establishes the task-level consequence. Systems measurements instantiate Proposition 5.
| Theoretical claim | Empirical quantity | Quantitative result | Conclusion |
| Refinement value (Prop. 2) | Exact-posterior loss under each nested transition | 13,824 of 13,824 endpoint–transition row checks: each larger state extends the smaller one and adds an observation (true by construction); panel-mean exact-posterior NLL decreases on every transition in every panel. | Design check1 |
| Co-scaled complementarity (Eq. (3)) | Diagonal gain and two adjacent increments | nats; increments [] and []. | Established |
| Finite-reader interaction (Prop. 4) | Two crossed interaction contrasts | [] and []; simultaneous 95% CIs (Bonferroni). | Identified |
| Task-level consequence | Official 16K generation accuracy gain | points; both directional contrasts remain significant after Holm correction. | Established |
| Relative footprint (Prop. 5) | External-state bytes and state/model ratio | bytes while the ratio falls . | Established |
Table 2. Theory–experiment correspondence. Each proposition is paired with its identifying estimand and empirical confirmation. Mechanism intervals are Bonferroni-simultaneous over the eight-estimand family; official accuracy uses the prespecified Holm-corrected directional family.
Nested state differences estimate the information supplied by refinement; adjacent diagonal differences test Eq. (3); crossed second differences isolate the reader-dependent extraction term in Eq. (11); and normalized state bytes test the footprint statement. Each theoretical operator has its own observable.
Results
Qwen2.5 Scaling on Official BABILong
Figure 2 shows every panel-level observation. Both evaluation families produce the same sharply increasing curve. The direct-peer margin becomes positive at 1.5B and increases further at 3B.
Figure 2. Scale-complementary gains across three ordered Qwen2.5 readers. Gray points are the ten task-panel clusters; colored symbols are means and vertical bars are 95% Student- intervals. (A) Official 16K BABILong. (B) Score-blind validation with zero semantic-fingerprint overlap with the pinned official pool. (C) Direct comparison with released task-trained Q-RAG on identical rows. The official and zero-overlap curves reproduce the same strict adjacent ordering; the peer margin crosses zero at 1.5B and widens at 3B.
Table 3 reports the two prespecified Qwen2.5 curves. The official 16K curve increases from 10.50 to 78.88 to 82.63 percentage points. The adjacent increments are 68.38 points () and 3.75 points (). Both adjacent directional contrasts remain significant under family-wise correction; the second is significant but marginal (Appendix B).
| Split | Reader | Foundation | ASCENT | Gain [95% CI] |
| Official 16K | 0.5B | .0563 | .1613 | .1050 [.0680,.1420] |
| Official 16K | 1.5B | .1888 | .9775 | .7888 [.7616,.8159] |
| Official 16K | 3B | .1613 | .9875 | .8263 [.8097,.8428] |
| Semantic holdout | 0.5B | .1263 | .1613 | .0350 [.0065,.0635] |
| Semantic holdout | 1.5B | .2025 | .9750 | .7725 [.7419,.8031] |
| Semantic holdout | 3B | .1763 | .9838 | .8075 [.7668,.8482] |
Table 3. Qwen2.5 task-balanced QA2+QA3 accuracy. Intervals are panel-clustered 95% intervals for ASCENT–Foundation gain.
The semantic-holdout curve increases from 3.50 to 77.25 to 80.75 points. Its adjacent increments are 73.75 points () and 3.50 points (), reproducing both positive adjacent increments.
At 3B on the official 16K panels, the canonical representation reaches .9875 accuracy versus .8338 for the frozen raw representation, a 15.38-point difference with 95% CI . Foundation predictions are identical row-by-row, isolating the state representation.
Direct Full-Factorial Scale Interaction
Figure 3 reports the complete factorial mechanism study. The co-scaled NLL gains increase from .388 to 2.099 to 7.406 nats (Figure 3). Their adjacent increments are 1.711 and 5.307 . Both model-by-state interactions are positive: .781 and .968 . These Bonferroni-simultaneous intervals retain positive lower bounds over the full eight-estimand family. Exact nesting, target isolation, Foundation probability identity across , and all 13,824 endpoint–transition row checks hold across the factorial. The row checks hold by construction of the nested writer
Figure 3. Direct identification of scale complementarity. (A) The light trajectories expose all nine independent panels; colored summaries give the mean and 95% interval in every reader–state cell, and the navy starred line is the co-scaled diagonal. (B) Bonferroni-simultaneous 95% intervals for the two diagonal increments, two model-by-state interactions, and largest-reader refinement. Their positive lower bounds jointly establish the diagonal property and the finite-reader mechanism in Proposition 4.
(Table 2, note); the per-row exact-posterior tallies are in Appendix § 16. Reruns of factorial panels on other instances are reported in § 8.
Public 7B–8B Breadth
Every planned public 7B–8B reader has complete ten-panel official 16K BABILong QA2/QA3 results (Figure 4). All five gain intervals exclude zero: Qwen2.5-7B gains 66.75 points, Qwen3-8B 72.00, Mistral-7B 80.75, Falcon3-7B 80.88, and Granite-3.3-8B 85.63. The gain persists across five architectures in the 7B–8B range.
Figure 4. Official 16K BABILong breadth across five public 7B–8B readers. Gray points are the ten task-panel gains; symbols are means and bars are panel-clustered 95% intervals. All five architectures realize large positive gains, extending the task-level consequence beyond the Qwen2.5 scaling family.
Cross-Family and Cross-Task Evidence
The cross-task evidence matrix extends the primary result across model families, task structures, and context lengths. For SmolLM2 on official RULER common-word aggregation, gains increase from .2628 to .4378 to .9561, with both adjacent intervals strictly positive. Qwen2.5 at 16K also has increasing means, .2417 to .6144 to .7100, with a positive log-scale slope. On official RULER multi-query retrieval, both Qwen2.5 (.0438 to .1208 to .3188) and SmolLM2 (.1042 to .2167 to .5229) have positive intervals for both adjacent increments. Official BABILong 4K and 8K studies cover QA1–QA3, while QA7–QA8 provide aggregation evidence.
Direct Current-Peer Comparison
Against released task-trained Q-RAG checkpoints on identical QA2/QA3 panels, frozen ASCENT leads by 22.75 points [.1885,.2666] at 1.5B and by 26.25 points [.2307,.2943] at 3B. The increasing positive margin at the two larger shared scales demonstrates that scale-complementary state can surpass task-trained retrieval without updating the foundation reader.
Controls and Systems Evidence
ASCENT outperforms byte-matched FIFO, lexical retrieval, and BGE-M3 hybrid retrieval on matched SmolLM2 panels. Irrelevant and corrupted-memory controls isolate the benefit of structured evidence from prompt length. Across 18 clean processes, median isolated decode-time ratios are .4130, .4247, and .2791 for the three ordered readers. Fixed write-plus-read state occupies 589.0, 624.6, and 640.6 bytes, while the state-to-model byte ratio decreases with scale. Supported-operator profiles show 83.7, 79.4, and 78.2 fewer FLOPs than the foundation generate path.
Limitations
Scope of the claim. Scale complementarity is demonstrated here for Qwen2.5 readers on official BABILong and RULER tasks, under accuracy and under candidate-normalized negative log-likelihood, with a structured-fact state schema. It is not shown to be a general property of frozen long-context readers with external memory. Within this scope three results are unfavorable and are reported as such: the accuracy gain decreases from 3B to 7B; the upper adjacent interval for Qwen2.5 common-word aggregation includes zero; and the SmolLM2 same-refinement interaction intervals include zero.
Schema scope. The writer schema used here (Table 1) records typed events, assignment edges, occurrence-indexed key–value evidence, and count summaries. These are the structures present in the evaluated structured-fact-retrieval benchmarks. The present evidence therefore establishes scale complementarity for external state constructed against such structures, and does not establish it as a general property of frozen readers with arbitrary external memory. Appendix D reports the identical pipeline with three schema-free writers (a sentence-recency window, a lexical retrieval chain, and BGE-M3 hybrid retrieval with reranking) in place of the structured-fact writer. None satisfies the registered co-scaling rule: at the same one-to-three-slot budget the generic state is worse than no state at every scale (gains of −5 to −18 points), and the structured-fact writer’s advantage over each generic writer is 15 points at 0.5B and 89–99 points at 1.5B and 3B. The pattern reported here is therefore a property of external state constructed against this schema. Extending the writer to non-templated natural text is the principal open direction.
Unfavorable and marginal results. The claim is specific to scale and estimand. Official BABILong accuracy establishes increasing gains from 0.5B to 3B; the second increment (1.5B→3B, 3.75 points) is significant but marginal after Holm correction (), and Appendix § 16 shows that the ten-panel design detects an increment of this size with power of about .52, against a minimum detectable increment of about 5.3–5.8 points at 80% power. At 7B, Foundation accuracy rises to .3325 and four-slot ASCENT reaches its ceiling; the gain is .6675, yielding a 3B-to-7B increment of −.1588 [−.2055, −.1120]. The 7B conclusion therefore rests on the non-saturated NLL factorial, where both model-by-state interactions have positive simultaneous lower bounds. Likewise, Qwen2.5 common-word aggregation establishes a positive scaling slope, while its upper adjacent interval is .0956 [−.0043, .1954]. The task-trained Q-RAG comparison favors ASCENT at 1.5B and 3B, but not at 0.5B. These cases match Proposition 1: complementarity requires both informative refinement and reader headroom. The SmolLM2 factorials reproduce increasing co-scaled gains, while their same-refinement interaction intervals overlap zero; the Qwen NLL factorial therefore supplies the direct reader-by-state interaction result.
The architecture study covers autoregressive Transformers rather than recurrent or linear readers. The distribution-matched semantic set originates from a training-generation split and is consequently secondary to the official held-out results. The direct interaction study uses a controlled compositional task and teacher-forced likelihood, so natural-text generation remains an important extension. Finally, the systems result establishes bounded state, decreasing relative storage, isolated decode gains, and 78.2–83.7 supported-operator FLOP reductions; its profiler-enclosed end-to-end wall-time ratios of .9008, .8651, and .7262 are reported as measured.
Reproducibility and Ethics
Public record. The code, frozen configurations, per-row result records, analysis files, and audit reports are released at https://github.com/XingjianZhang-dev/ascent and archived at Zenodo (version DOI 10.5281/zenodo.22865847). One CPU command (make reproduce) re-verifies the SHA-256 manifests, recomputes every analysis file from the per-row records, rerenders every table in this paper byte for byte, and prints every reported number; VERIFICATION.md maps each value to its records. The freeze commit precedes the first score for all 25 frozen configurations with retained scores (PREREGISTRATION_TIMELINE.md).
Reproduction on other hardware. Every formal run executed on NVIDIA RTX PRO 6000 Blackwell instances. Reruns on matched hardware reproduced three audit cells identically, at the level of prediction and, where floats are stored, IEEE-754 binary64 identity: the factorial development panel (256 records, node 1 versus node 2), official 16K panel 1 for Qwen2.5-7B (80 rows, node 1 versus node 2), and all ten official 16K panels for Qwen3-8B (800 rows, third instance). The three instances have distinct GPU serial numbers, and the first two have distinct host driver versions. The Qwen3-8B rerun is a post-hoc code audit and enters no reported number; it ran from a one-commit repository snapshot whose working tree also held uncommitted manuscript-packaging edits, none of which the runner imports, and the diff is retained in the repository. Every other formal cell was run once.
A rerun on a separately provisioned instance with a different GPU architecture (NVIDIA A100, Ampere) and identical software pins reproduced the reported conclusions; stored values were not bit-identical. Across all sixty official-16K and semantic-holdout cells (4,800 rows, both arms), 124 of 9,600 scored outputs changed (103 Foundation, 21 Ascent); twenty cells were unchanged, and single 80-row panels moved by up to 5.00 accuracy points (four rows). The ten-panel means moved by at most 1.00 point (official 16K: 10.50/78.88/82.63 → 9.75/78.88/83.25 points; semantic holdout: 3.50/77.25/80.75 → 4.50/77.00/80.63), inside every reported interval half-width (1.66–3.70 points), and every registered directional contrast still passes (second official increment 4.38 points, ; semantic holdout 3.63 points, ). On factorial panels 2 and 3, one to two of 256 argmax candidates per arm flipped and the panel NLL gains moved by approximately 0.01 nats against a reported half-width of 0.16 nats. Stored floating-point values differ in their low-order bits, as expected across kernel implementations. Same-instance controls were bit-identical, so the differences follow from the architecture change. The numbers reported in this paper are the Blackwell values.
Determinism. Decoding is greedy in bfloat16 with pinned weights, revisions, panels, seeds, and batch sizes; torch.use_deterministic_algorithms and a TF32 policy were not set (reproduction/DETERMINISM_POLICY.md). The flag requests algorithms that return the same output for the same input when run on the same software and hardware [90]. The pipeline already has that property without it: the three matched-hardware audit cells, two complete executions of the pre-registered A100 audit cells, and six A100 cells run with and without the target-blindness instrumentation are each identical at the prediction level and, where floats are stored, at binary64. Strict same-hardware determinism was thus obtained with the default kernels, at the throughput of the runs as executed; the flag was not needed, and a flag-enabled run was not timed. The flag does not extend across GPU architectures. cuBLAS guarantees bit-wise identical results only on GPUs with the same architecture and the same number of streaming multiprocessors [91], and PyTorch does not guarantee reproducibility across platforms [92]; the Blackwell and A100 instances execute architecture-specific matrix-multiplication and attention kernels whose floating-point reductions need not round identically. The 124 changed outputs arise from this difference, and enabling the flag would not have removed them. Agreement across architectures is accordingly assessed where the conclusions are stated, in panel means and registered contrasts, and it holds there.
Target-blindness. The BABILong writer never receives the target; it is called with the input and the question only. The writer, readout, and prompt builder run behind a blinded row view that raises on any access to the answer field until the prompt has been rendered, and every result row records the fields read, the number of blocked accesses (zero), and a provenance check that each retained fact is the verbatim input span at its recorded position. All 4,800 rows of the official 16K and semantic-holdout studies were run with this instrumentation; on the same instance the instrumented and uninstrumented runners produced identical outputs.
Interruptions and licenses. The RULER common-word confirmation had two pre-score interruptions, a missing legacy configuration field and a node without the SmolLM2-1.7B weights; neither produced a score, and both are logged in the release. The released per-row records omit benchmark row bodies and model weights, which remain under their original licenses; panel identifiers, hashes, pinned revisions, and construction scripts are released instead.
Conclusion
On the evaluated tasks, external state becomes more valuable as a frozen language-model reader scales. Conditional information theory specifies the value of nested refinement, and the factorial interaction shows that larger readers extract increasingly more of that value. Official RULER and BABILong evaluations reproduce the increasing-gain pattern across model families, tasks, and context lengths; five public 7B–8B readers retain large absolute gains. Direct-peer comparisons, systems measurements, representation controls, and cross-node reproduction complete the theory-to-evidence chain. Scale complementarity is therefore identifiable, achievable, and reproducible in the evaluated frozen long-context language-model systems, within the scope stated in Section 7.
CRediT authorship contribution statement
Xingjian Zhang: Conceptualization, Methodology, Software, Formal analysis, Investigation, Validation, Data curation, Visualization, Writing – original draft, Writing – review and editing, Project administration.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Declaration of competing interest
The author declares that he has no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Data availability
Code and analysis artifacts are publicly available at https://github.com/XingjianZhang-dev/ascent and archived at https://doi.org/10.5281/zenodo.22865847. Panel identifiers, hashes, and construction scripts are released for BABILong and RULER; the underlying benchmark corpora are not redistributed and remain under their original licenses (Apache-2.0 and BSD components; see THIRD_PARTY_NOTICES.md in the repository). Third-party model weights are not redistributed and remain subject to their original licenses.
Declaration of generative AI and AI-assisted technologies in the manuscript preparation process
During the preparation and revision of this work, the author used Chat-GPT and Codex and, during the revision, Claude Code as assistive tools for implementing author-specified code modifications, conducting routine checks, and improving the clarity and wording of the manuscript and response to reviewers. No AI tool produced any reported measurement. All research questions, methodological decisions, experimental design, validation, interpretation of results, and final written content were determined, reviewed, and approved by the author, who takes full responsibility for the work. A per-file record is provided as AI_ASSISTANCE.md in the repository.
[90] PyTorch Contributors, torch.use_deterministic_algorithms, PyTorch 2.8 documentation (2025). URL https://docs.pytorch.org/docs/2.8/generated/torch.use_deterministic_algorithms.html
[91] NVIDIA Corporation, cuBLAS library user guide, release 12.8: Results reproducibility (section 2.1.4), CUDA Toolkit 12.8 documentation (2025). URL https://docs.nvidia.com/cuda/archive/12.8.0/cublas/index.html
[92] PyTorch Contributors, Reproducibility, PyTorch 2.8 documentation, Notes (2025). URL https://docs.pytorch.org/docs/2.8/notes/randomness.html
Complete Factorial Cell Table
Table A.1 lists the mean candidate-normalized NLL gain and its 95% Student- interval in every reader×state cell of the numeric factorial (nine independent 256-row panels per cell; Figure 3 plots the same cells). Starred cells form the co-scaled diagonal. Reading along a row isolates the state-budget component at fixed reader; reading down a column isolates the reader component at fixed state; the diagonal increments and the two crossed interactions in Table A.2 are second differences of these entries.
| Reader | |||
| 1.5B | .388 [.340,.437]⋆ | .295 [.247,.342] | .304 [.248,.359] |
| 3B | 1.412 [1.288,1.535] | 2.099 [2.004,2.194]⋆ | 2.587 [2.536,2.638] |
| 7B | 3.709 [3.460,3.958] | 5.950 [5.637,6.264] | 7.406 [7.246,7.566]⋆ |
Table A.1. Mean NLL gain (nats) with 95% intervals in all nine reader×state cells. ⋆ marks the co-scaled diagonal.
| Reader | State | Gain (nats) | Simultaneous 95% CI |
| 1.5B | 2 | .388 | [.311,.465] |
| 3B | 3 | 2.099 | [1.948,2.251] |
| 7B | 5 | 7.406 | [7.151,7.661] |
| Modelstate: 1.5B3B | .781 | [.539,1.023] | |
| Modelstate: 3B7B | .968 | [.671,1.264] |
Table A.2. Registered factorial estimands with Bonferroni-simultaneous 95% intervals over the eight-estimand family ().
Statistical Formulation, Power, and Consistency
Estimands. Let denote the paired Foundation-minus-ASCENT loss difference for reader , state size , and panel . The co-scaled curve is ; the diagonal increments are its adjacent differences; the interactions are and ; the eighth estimand is . All quantities are computed within panel and averaged over nine panels; ordinary intervals use and simultaneous intervals use . The official Qwen2.5 BABILong study registers two one-sided adjacent accuracy-gain contrasts over ten panels with Holm step-down control at .05; the smallest Holm threshold is therefore .025 one-sided. No row, panel, endpoint, or outlier is removed.
Retrospective power. Table B.1 recomputes, from the panel-level evidence files released with the code, the between-panel SD, SE, statistic and raw one-sided of every registered contrast, the minimum detectable effect (MDE) at the registered per-test level and 80% power (exact non-central ), and the achieved power for the observed effect. Two facts follow. The official-16K design is well powered for the first increment () and underpowered for the second: with the observed between-panel SD, the MDE at 80% power is 5.25 points on the second contrast’s SE (5.75 on the first’s), so the observed 3.75-point 1.5B3B increment was detected at power and is described as significant but marginal (). The very small -values elsewhere follow from large effects with small between-panel variance (Table B.2: a 68.38-point increment with SE 1.83 points gives ). Retrospective power uses the observed SE as if it were the true SD; it characterizes the sensitivity of the design, not the probability that a result is true.
Table B.1: Registered contrasts: effect, SE, , raw one-sided , minimum detectable effect at the registered per-test level and 80% power, and achieved power. Official-16K and semantic-holdout rows use and the Holm step level .025; factorial rows use and the Bonferroni level .05/16 one-sided.
| Study | Contrast | Effect | SE | MDE | Power | ||
| Official 16K | 0.5B 1.5B | 68.38 pts | 1.83 | 37.44 | 5.75 pts | 1.00 | |
| Official 16K | 1.5B 3B | 3.75 pts | 1.67 | 2.25 | 0.0255 | 5.25 pts | 0.52 |
| Semantic holdout | 0.5B 1.5B | 73.75 pts | 1.09 | 67.88 | 3.42 pts | 1.00 | |
| Semantic holdout | 1.5B 3B | 3.50 pts | 1.61 | 2.17 | 0.0290 | 5.08 pts | 0.49 |
| Factorial | Diagonal gain: 1.5B/K2 | 0.388 nats | 0.021 | 18.52 | 0.098 nats | 1.00 | |
| Factorial | Diagonal gain: 3B/K3 | 2.099 nats | 0.041 | 50.91 | 0.194 nats | 1.00 | |
| Factorial | Diagonal gain: 7B/K5 | 7.406 nats | 0.069 | 106.71 | 0.326 nats | 1.00 | |
| Factorial | Diagonal: 1.5B to 3B | 1.711 nats | 0.047 | 36.74 | 0.219 nats | 1.00 | |
| Factorial | Diagonal: 3B to 7B | 5.307 nats | 0.069 | 76.90 | 0.324 nats | 1.00 | |
| Factorial | 7B refinement: K3 to K5 | 1.456 nats | 0.107 | 13.65 | 0.501 nats | 1.00 | |
| Factorial | Interaction: 1.5B to 3B | 0.781 nats | 0.066 | 11.87 | 0.309 nats | 1.00 | |
| Factorial | Interaction: 3B to 7B | 0.968 nats | 0.081 | 12.00 | 0.379 nats | 1.00 |
Table B.1.
Table B.2: Statistical-consistency check: the SE and implied by each reported value, recomputed from the panel-level data.
| Reported quantity | Reported value | Implied SE | Implied | df |
| Official 16K 0.5B 1.5B increment | 68.38 pts, | 1.83 pts | 37.44 | 9 |
| Official 16K 1.5B 3B increment | 3.75 pts, | 1.67 pts | 2.25 | 9 |
| Official 16K 0.5B endpoint gain | 0.1050 [0.0680, 0.1420] | 0.0164 | 6.41 | 9 |
| Official 16K 1.5B endpoint gain | 0.7888 [0.7616, 0.8159] | 0.0120 | 65.75 | 9 |
| Official 16K 3B endpoint gain | 0.8263 [0.8097, 0.8428] | 0.0073 | 112.81 | 9 |
| Factorial Diagonal: 1.5B to 3B | 1.711 [1.540, 1.882] (Bonf.) | 0.047 | 36.74 | 8 |
| Factorial Interaction: 1.5B to 3B | 0.781 [0.539, 1.023] (Bonf.) | 0.066 | 11.87 | 8 |
| Factorial Diagonal: 3B to 7B | 5.307 [5.053, 5.561] (Bonf.) | 0.069 | 76.90 | 8 |
| Factorial Interaction: 3B to 7B | 0.968 [0.671, 1.264] (Bonf.) | 0.081 | 12.00 | 8 |
Table B.2.
Row-level transition audit. The factorial’s 13,824 row checks (three readers two transitions nine panels 256 rows) verify that the state retained at the larger extends the smaller state as a prefix and adds an observation; both hold by construction of the nested writer and are reported as implementation checks. The released file reports/row_audits/FACTORIAL_TRANSITION_ROW_AUDIT_13824.csv adds, per row, the exact posterior NLL and the reader’s candidate NLL before and after each transition: under the noisy channel the exact posterior improves on 10,791 of 13,824 transitions and the reader’s NLL on 7,435, so the reported gains are panel averages rather than row-wise guarantees.
BABILong 4K/8K Results and SmolLM2 Factorials
Section 6 describes the SmolLM2 official BABILong 4K and 8K studies and the two SmolLM2 factorials qualitatively. Table C.1 gives the panel-clustered gains and adjacent increments (three panels per context, intervals), and Table C.2 gives all nine cells of the free-generation (QA1–QA3, 1/2/3 slots) and certified-readout (QA7–QA8, 1/2/8 slots) SmolLM2 factorials with their diagonal increments and same-refinement interactions. The certified readout converts each state refinement into the same deterministic accuracy increment at every reader scale, which is why its interactions are exactly zero (see Table C.3).
| Context | Panels | Gain 135M | Gain 360M | Gain 1.7B | Incr. 135M→360M | Incr. 360M→1.7B |
| 4K | 3 | .2611 [.1569,.3653] | .5056 [.4535,.5577] | .6750 [.5848,.7652] | .2444 [.1066,.3823] | .1694 [.0653,.2736] |
| 8K | 3 | .2806 [.2079,.3533] | .5111 [.4514,.5709] | .6889 [.5511,.8267] | .2306 [.1989,.2622] | .1778 [.0399,.3156] |
Table C.1. SmolLM2 official BABILong QA1–QA3 accuracy gains and adjacent increments at 4K and 8K (panel-clustered 95% intervals).
| Study | Reader | Slot 1 | Slot 2 | Slot 3/8 |
| Free generation QA1–QA3 (8K, 3 panels) | 135M | .2417 [.1589,.3245]⋆ | .4472 [.3063,.5881] | .4972 [.3832,.6112] |
| 360M | .2528 [.2289,.2767] | .5000 [.4586,.5414]⋆ | .4972 [.4494,.5450] | |
| 1.7B | .1583 [−.0568,.3735] | .6389 [.4653,.8125] | .6694 [.5197,.8192]⋆ | |
| Certified QA7–QA8 (8K, 5 panels) | 135M | .4750 [.4244,.5256]⋆ | .7250 [.6744,.7756] | .9750 [.9244,1.0256] |
| 360M | .4438 [.3932,.4943] | .6938 [.6432,.7443]⋆ | .9438 [.8932,.9943] | |
| 1.7B | .4063 [.3514,.4611] | .6563 [.6014,.7111] | .9063 [.8514,.9611]⋆ |
Table C.2. SmolLM2 factorials: accuracy gain in every cell (⋆: co-scaled diagonal).
| Study | Diag. incr. 1 | Diag. incr. 2 | Interaction 1 | Interaction 2 |
| Free generation | .2583 [.1488,.3679] | .1694 [.0554,.2835] | .0417 [−.0618,.1452] | .0333 [−.0081,.0747] |
| Certified | .2188 [.1462,.2913] | .2125 [.1549,.2701] | .0000 [.0000,.0000] | .0000 [.0000,.0000] |
Table C.3. SmolLM2 factorials: diagonal increments and same-refinement interactions (panel-clustered 95% intervals).
Schema-Blind Writer Ablation
To separate “structured external state helps frozen readers, and more so as they scale” from “this parser solves this benchmark”, the identical pipeline—the same ten official 16K panels, the same per-reader slot budget , the same bullet-list serialization and prompt, the same frozen Qwen2.5 readers, greedy decoding, and scorer—was rerun with the structured-fact writer replaced by three schema-free state constructions: (i) a sentence window, the last generic sentences of the text in document order, which reads neither the question nor any task vocabulary; (ii) a lexical chain, a deterministic BM25-style retrieval over generic sentences; and (iii) BGE-M3 hybrid retrieval with the official cross-encoder reranker over three target-blind hops. The interpretation rule was fixed before any score: a generic writer “survives” if its three endpoint gains increase strictly with reader scale and both adjacent one-sided increments pass Holm at .05, the registered rule of the canonical study. Both arms were rerun on the same instance (an NVIDIA A100, cf. § 8); the Foundation arm is identical across conditions and the canonical-versus-generic contrast is paired over panels on the same hardware, so no architecture difference enters it.
With the sentence window writer the ten-panel gains are −5.25, −17.50, −16.00 points at 0.5B, 1.5B, and 3B (95% intervals 0.5B: ; 1.5B: ; 3B: ); the adjacent increments are −12.25 and 1.50 points with and 0.265, so the registered co-scaling rule collapses. The generic state is worse than no state at every scale: Ascent accuracy is 1.00, 1.13, 0.13 against Foundation accuracy of 6.25, 18.63, 16.13. The structured-fact writer’s advantage in Ascent accuracy over this writer, paired over panels on the same instance, is 0.5B: 15.00 ; 1.5B: 96.38 ; 3B: 99.25 points. Foundation outputs differ from the same-instance canonical run in 0 of 2,400 rows.
With the lexical chain writer the ten-panel gains are −5.00, −16.38, −11.63 points at 0.5B, 1.5B, and 3B (95% intervals 0.5B: ; 1.5B: ; 3B: ); the adjacent increments are −11.38 and 4.75 points with and 0.004, so the registered co-scaling rule collapses. The generic state is worse than no state at every scale: Ascent accuracy is 1.25, 2.25, 4.50 against Foundation accuracy of 6.25, 18.63, 16.13. The structured-fact writer’s advantage in Ascent accuracy over this writer, paired over panels on the same instance, is 0.5B: 14.75 ; 1.5B: 95.25 ; 3B: 94.88 points. Foundation outputs differ from the same-instance canonical run in 0 of 2,400 rows.
With the BGE-M3 hybrid retrieval writer the ten-panel gains are −5.50, −11.00, −5.38 points at 0.5B, 1.5B, and 3B (95% intervals 0.5B: ; 1.5B: ; 3B: ); the adjacent increments are −5.50 and 5.63 points with and 0.023, so the registered co-scaling rule collapses. The generic state is worse than no state at every scale: Ascent accuracy is 0.75, 7.63, 10.75 against Foundation accuracy of 6.25, 18.63, 16.13. The structured-fact writer’s advantage in Ascent accuracy over this writer, paired over panels on the same instance, is 0.5B: 15.25 [12.66, 17.84]; 1.5B: 89.88 [87.00, 92.75]; 3B: 88.63 [86.92, 90.33] points. Foundation outputs differ from the same-instance canonical run in 0 of 2,400 rows.
No schema-free writer satisfies the registered co-scaling rule (Table D.1). At the same one-to-three-slot budget, generic state constructions rarely retain the supporting facts, so the reader answers worse from them than from the full 16K context, the gains are negative at every scale, and they do not increase with reader scale in the registered sense. The co-scaling pattern reported in the main text is therefore a property of external state constructed against the evaluated structured-fact schema; whether a schema-free writer with a much larger budget, or a learned extractor, would recover it is not tested here. Section 7 scopes the claim accordingly.
| Writer | Gain 0.5B | Gain 1.5B | Gain 3B | (1) | (2) | Co-scaling |
| Sentence window | −5.25 | −17.50 | −16.00 | 1.000 | 0.265 | collapses |
| Lexical BM25 chain | −5.00 | −16.38 | −11.63 | 1.000 | 0.004 | collapses |
| BGE-M3 hybrid + reranker | −5.50 | −11.00 | −5.38 | 0.987 | 0.023 | collapses |
Table D.1. Schema-blind writer ablation on official 16K BABILong: ASCENT gain (points) per reader for each generic writer, Holm-corrected one-sided for the two adjacent increments, and whether the registered co-scaling rule survives.
Pinned Model Revisions
Every checkpoint is loaded from the Hugging Face revision in Table E.1; the frozen configurations additionally record each weight shard’s byte size and SHA-256, and every runner re-hashes the shards before loading and aborts on mismatch. The Qwen2.5 readers are the Instruct variants; Qwen3-8B runs in non-thinking mode.
| Checkpoint | Hugging Face revision | Shards | Bytes |
| EleutherAI/pythia-1.4b | 9cc5c8c8148a | – | – |
| EleutherAI/pythia-1b | c0fea4b91761 | 1 | 2,090,751,965 |
| EleutherAI/pythia-2.8b | dbe7ae300a54 | – | – |
| EleutherAI/pythia-410m | bba6a464f54b | – | – |
| HuggingFaceTB/SmolLM2-1.7B-Instruct | 31b70e2e869a | 1 | 3,422,777,952 |
| HuggingFaceTB/SmolLM2-135M-Instruct | 12fd25f77366 | 1 | 269,060,552 |
| HuggingFaceTB/SmolLM2-360M-Instruct | a10cc1512eab | 1 | 723,674,912 |
| Qwen/Qwen2.5-0.5B-Instruct | 7ae557604adf | 1 | 988,097,824 |
| Qwen/Qwen2.5-1.5B-Instruct | 989aa7980e4c | 1 | 3,087,467,144 |
| Qwen/Qwen2.5-3B-Instruct | aa8e72537993 | 2 | 6,171,926,992 |
| Qwen/Qwen2.5-7B-Instruct | a09a35458c70 | 4 | 15,231,271,888 |
| Qwen/Qwen3-8B | b968826d9c46 | 5 | 16,381,516,776 |
| ibm-granite/granite-3.3-8b-instruct | 51dd4bc2ade4 | 4 | 16,341,771,384 |
| mistralai/Mistral-7B-Instruct-v0.3 | c170c708c41d | 3 | 14,496,080,928 |
| tiiuae/Falcon3-1B-Instruct | 28ba2251970a | 1 | 3,338,836,632 |
| tiiuae/Falcon3-3B-Instruct | 411bb94318f9 | 2 | 6,455,333,640 |
| tiiuae/Falcon3-7B-Instruct | 1e57a0ecd176 | 4 | 14,911,130,600 |
Table E.1. Pinned checkpoint revisions (first twelve characters of the commit hash), shard counts, and total weight bytes as recorded in configs/. “–”: the configuration pins the revision without a shard manifest; the corresponding result records carry the loaded shard hashes.
References
- [1]C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, B. Ginsburg, Ruler: What’s the real context size of your long-context language models?, in: Conference on Language Modeling, 2024.arxiv.org/abs/2404.06654
- [2]Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, M. Burtsev, Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, in: Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, Vol. 37, 2024, pp. 106519–106554. doi:10.52202/079017-3381. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/c0d62e70dbc659cc9bd44cbcf1cb652f-Abstract-Datasets_and_Benchmarks_Track.htmlDOI
- [3]A. Sorokin, N. Buzun, A. Anokhin, E. Vedernikov, P. Anokhin, M. Burtsev, E. Burnaev, Q-rag: Long context multi-step retrieval via value-based embedder training, in: International Conference on Learning Representations, 2026, oral presentation. URL https://openreview.net/forum?id=MS9nWFY7LGarxiv.org/abs/2511.07328
- [4]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, arXiv preprint arXiv:1706.03762 (2017). URL https://arxiv.org/abs/1706.03762
- [5]Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, R. Salakhutdinov, Transformer-xl: Attentive language models beyond a fixed-length context, arXiv preprint arXiv:1901.02860 (2019). URL https://arxiv.org/abs/1901.02860
- [6]J. W. Rae, A. Potapenko, S. M. Jayakumar, T. P. Lillicrap, Compressive transformers for long-range sequence modelling, arXiv preprint arXiv:1911.05507 (2019). URL https://arxiv.org/abs/1911.05507
- [15]A. Gu, T. Dao, Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752 (2023). URL https://arxiv.org/abs/2312.00752
- [16]T. Dao, A. Gu, Transformers are smms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv:2405.21060 (2024). URL https://arxiv.org/abs/2405.21060
- [17]M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, C. Re, Hyena hierarchy: Towards larger convolutional language models, arXiv preprint arXiv:2302.10866 (2023). URL https://arxiv.org/abs/2302.10866
- [18]B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV, X. He, H. Hou, J. Lin, P. Kazienko, J. Kocon, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, G. Song, X. Tang, B. Wang, J. S. Wind, S. Wozniak, R. Zhang, Z. Zhang, Q. Zhao, P. Zhou, Q. Zhou, J. Zhu, R.-J. Zhu, Rwkv: Reinventing rnns for the transformer era, arXiv preprint arXiv:2305.13048 (2023). URL https://arxiv.org/abs/2305.13048
- [19]Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, F. Wei, Retentive network: A successor to transformer for large language models, arXiv preprint arXiv:2307.08621 (2023). URL https://arxiv.org/abs/2307.08621
- [20]T. Dao, D. Y. Fu, S. Ermon, A. Rudra, C. Re, Flashattention: Fast and memory-efficient exact attention with io-awareness, arXiv preprint arXiv:2205.14135 (2022). URL https://arxiv.org/abs/2205.14135
- [21]T. Dao, Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.08691 (2023). URL https://arxiv.org/abs/2307.08691
- [22]H. Liu, M. Zaharia, P. Abbeel, Ring attention with blockwise transformers for near-infinite context, arXiv preprint arXiv:2310.01889 (2023). URL https://arxiv.org/abs/2310.01889
- [23]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, Efficient memory management for large language model serving with pagedattention, arXiv preprint arXiv:2309.06180 (2023). URL https://arxiv.org/abs/2309.06180
- [24]H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. H. Abdi, D. Li, C.-Y. Lin, Y. Yang, L. Qiu, Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention, arXiv preprint arXiv:2407.02490 (2024). URL https://arxiv.org/abs/2407.02490
- [25]Y. Leviathan, M. Kalman, Y. Matias, Fast inference from transformers via speculative decoding, arXiv preprint arXiv:2211.17192 (2022). URL https://arxiv.org/abs/2211.17192
- [26]C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper, Accelerating large language model decoding with speculative sampling, arXiv preprint arXiv:2302.01318 (2023). URL https://arxiv.org/abs/2302.01318
- [27]J. Weston, S. Chopra, A. Bordes, Memory networks, arXiv preprint arXiv:1410.3916 (2014). URL https://arxiv.org/abs/1410.3916
- [28]S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus, End-to-end memory networks, arXiv preprint arXiv:1503.08895 (2015). URL https://arxiv.org/abs/1503.08895
- [29]A. Graves, G. Wayne, I. Danihelka, Neural turing machines, arXiv preprint arXiv:1410.5401 (2014). URL https://arxiv.org/abs/1410.5401
- [30]A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, T. Lillicrap, One-shot learning with memory-augmented neural networks, arXiv preprint arXiv:1605.06065 (2016). URL https://arxiv.org/abs/1605.06065
- [31]G. Lample, A. Sablayrolles, M. Ranzato, L. Denoyer, H. Jegou, Large memory layers with product keys, arXiv preprint arXiv:1907.05242 (2019). URL https://arxiv.org/abs/1907.05242
- [32]W. Wang, L. Dong, H. Cheng, X. Liu, X. Yan, J. Gao, F. Wei, Augmenting language models with long-term memory, arXiv preprint arXiv:2306.07174 (2023). URL https://arxiv.org/abs/2306.07174
- [33]C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, J. E. Gonzalez, Memgpt: Towards llms as operating systems, arXiv preprint arXiv:2310.08560 (2023). URL https://arxiv.org/abs/2310.08560
- [34]W. Zhong, L. Guo, Q. Gao, H. Ye, Y. Wang, Memorybank: Enhancing large language models with long-term memory, arXiv preprint arXiv:2305.10250 (2023). URL https://arxiv.org/abs/2305.10250
- [35]Y. Wu, M. N. Rabe, D. Hutchins, C. Szegedy, Memorizing transformers, in: International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TrjbxzRcnf-arxiv.org/abs/2203.08913
- [36]Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, C. Guestrin, Learning to (learn at test time): Rnns with expressive hidden states, arXiv preprint arXiv:2407.04620 (2024).arxiv.org/abs/2407.04620
- [37]A. Behrouz, P. Zhong, V. Mirrokni, Titans: Learning to memorize at test time, arXiv preprint arXiv:2501.00663 (2025).arxiv.org/abs/2501.00663
- [38]P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W.-t. Yih, T. Rocktaschel, S. Riedel, D. Kiela, Retrieval-augmented generation for knowledge-intensive nlp tasks, arXiv preprint arXiv:2005.11401 (2020). URL https://arxiv.org/abs/2005.11401
- [39]K. Guu, K. Lee, Z. Tung, P. Pasupat, M.-W. Chang, Realm: Retrieval-augmented language model pre-training, arXiv preprint arXiv:2002.08909 (2020). URL https://arxiv.org/abs/2002.08909
- [40]V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, W.-t. Yih, Dense passage retrieval for open-domain question answering, arXiv preprint arXiv:2004.04906 (2020). URL https://arxiv.org/abs/2004.04906
- [41]U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, M. Lewis, Generalization through memorization: Nearest neighbor language models, arXiv preprint arXiv:1911.00172 (2019). URL https://arxiv.org/abs/1911.00172
- [42]S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. M. van den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al., Improving language models by retrieving from trillions of tokens, in: Proceedings of the 39th International Conference on Machine Learning, Vol. 162 of Proceedings of Machine Learning Research, PMLR, 2022, pp. 2206–2240. URL https://proceedings.mlr.press/v162/borgeaud22a.htmlarxiv.org/abs/2112.04426
- [43]G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, E. Grave, Atlas: Few-shot learning with retrieval augmented language models, arXiv preprint arXiv:2208.03299 (2022). URL https://arxiv.org/abs/2208.03299
- [44]W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, W.-t. Yih, Replug: Retrieval-augmented black-box language models, arXiv preprint arXiv:2301.12652 (2023). URL https://arxiv.org/abs/2301.12652
- [45]A. Asai, Z. Wu, Y. Wang, A. Sil, H. Hajishirzi, Self-rag: Learning to retrieve, generate, and critique through self-reflection, arXiv preprint arXiv:2310.11511 (2023). URL https://arxiv.org/abs/2310.11511
- [46]S.-Q. Yan, J.-C. Gu, Y. Zhu, Z.-H. Ling, Corrective retrieval augmented generation, arXiv preprint arXiv:2401.15884 (2024). URL https://arxiv.org/abs/2401.15884
- [47]P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, C. D. Manning, Raptor: Recursive abstractive processing for tree-organized retrieval, arXiv preprint arXiv:2401.18059 (2024). URL https://arxiv.org/abs/2401.18059
- [48]L. Gao, X. Ma, J. Lin, J. Callan, Precise zero-shot dense retrieval without relevance labels, arXiv preprint arXiv:2212.10496 (2022). URL https://arxiv.org/abs/2212.10496
- [49]G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, E. Grave, Unsupervised dense information retrieval with contrastive learning, arXiv preprint arXiv:2112.09118 (2021). URL https://arxiv.org/abs/2112.09118
- [50]O. Khattab, M. Zaharia, Colbert: Efficient and effective passage search via contextualized late interaction over bert, arXiv preprint arXiv:2004.12832 (2020). URL https://arxiv.org/abs/2004.12832
- [51]G. Izacard, E. Grave, Leveraging passage retrieval with generative models for open domain question answering, arXiv preprint arXiv:2007.01282 (2020). URL https://arxiv.org/abs/2007.01282
- [52]K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, M. Zaharia, Colbertv2: Effective and efficient retrieval via lightweight late interaction, arXiv preprint arXiv:2112.01488 (2021). URL https://arxiv.org/abs/2112.01488
- [53]Y. Wang, Y. Wan, X. Lei, Q. Chen, H. Hu, A retrieval augmented generation based optimization approach for medical knowledge understanding and reasoning in large language models, Array 28 (2025) 100504. doi:10.1016/j.array.2025.100504. URL https://doi.org/10.1016/j.array.2025.100504
- [54]H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, L. Qiu, Llmlingua: Compressing prompts for accelerated inference of large language models, arXiv preprint arXiv:2310.05736 (2023). URL https://arxiv.org/abs/2310.05736
- [55]H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, L. Qiu, Longllm-lingua: Accelerating and enhancing llms in long context scenarios via prompt compression, arXiv preprint arXiv:2310.06839 (2023). URL https://arxiv.org/abs/2310.06839
- [56]Y. Li, Unlocking context constraints of llms: Enhancing context efficiency of llms with self-information-based content filtering, arXiv preprint arXiv:2304.12102 (2023). URL https://arxiv.org/abs/2304.12102
- [57]S. Chen, S. Wong, L. Chen, Y. Tian, Extending context window of large language models via positional interpolation, arXiv preprint arXiv:2306.15595 (2023). URL https://arxiv.org/abs/2306.15595
- [58]J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, Y. Liu, Roformer: Enhanced transformer with rotary position embedding, arXiv preprint arXiv:2104.09864 (2021). URL https://arxiv.org/abs/2104.09864
- [59]B. Peng, J. Quesnelle, H. Fan, E. Shippole, Yarn: Efficient context window extension of large language models, arXiv preprint arXiv:2309.00071 (2023). URL https://arxiv.org/abs/2309.00071
- [60]Y. Ding, L. L. Zhang, C. Zhang, Y. Xu, N. Shang, J. Xu, F. Yang, M. Yang, Longrope: Extending llm context window beyond 2 million tokens, arXiv preprint arXiv:2402.13753 (2024). URL https://arxiv.org/abs/2402.13753
- [61]N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, Lost in the middle: How language models use long contexts, arXiv preprint arXiv:2307.03172 (2023). URL https://arxiv.org/abs/2307.03172
- [62]Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, J. Li, Longbench: A bilingual, multitask benchmark for long context understanding, arXiv preprint arXiv:2308.14508 (2023). URL https://arxiv.org/abs/2308.14508
- [63]C. An, S. Gong, M. Zhong, X. Zhao, M. Li, J. Zhang, L. Kong, X. Qiu, L-eval: Instituting standardized evaluation for long context language models, arXiv preprint arXiv:2307.11088 (2023). URL https://arxiv.org/abs/2307.11088
- [64]X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, M. Sun, Qobench: Extending long context evaluation beyond 100k tokens, arXiv preprint arXiv:2402.13718 (2024). URL https://arxiv.org/abs/2402.13718
- [65]M. Li, S. Zhang, T. Zhang, H. Duan, Y. Liu, K. Chen, Needlebench: Evaluating llm retrieval and reasoning across varying information densities, arXiv preprint arXiv:2407.11963 (2024). URL https://arxiv.org/abs/2407.11963
- [66]Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, D. Metzler, Long range arena: A benchmark for efficient transformers, arXiv preprint arXiv:2011.04006 (2020). URL https://arxiv.org/abs/2011.04006
- [67]G. Qin, Y. Feng, B. Van Durme, The nlp task effectiveness of long-range transformers, arXiv preprint arXiv:2202.07856 (2022). URL https://arxiv.org/abs/2202.07856
- [68]J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361 (2020).arxiv.org/abs/2001.08361
- [69]J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Ruther-ford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, in: Advances in Neural Information Processing Systems, Vol. 35, 2022, pp. 30016–30030. doi:10.52202/068431-2176. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract.htmlDOI
- [70]D. Hernandez, J. Kaplan, T. Henighan, S. McCandlish, Scaling laws for transfer, arXiv preprint arXiv:2102.01293 (2021). URL https://arxiv.org/abs/2102.01293
- [71]N. Muennighoff, A. M. Rush, B. Barak, T. L. Scao, A. Piktus, N. Tazi, S. Pyysalo, T. Wolf, C. Raffel, Scaling data-constrained language models, arXiv preprint arXiv:2305.16264 (2023). URL https://arxiv.org/abs/2305.16264
- [72]D. Blackwell, Equivalent comparisons of experiments, The Annals of Mathematical Statistics 24 (2) (1953) 265–272.DOI
- [73]O. Press, N. A. Smith, M. Lewis, Train short, test long: Attention with linear biases enables input length extrapolation, arXiv preprint arXiv:2108.12409 (2021). URL https://arxiv.org/abs/2108.12409
- [74]C. Han, Q. Wang, H. Peng, W. Xiong, Y. Chen, H. Ji, S. Wang, Lm-infinite: Zero-shot extreme length generalization for large language models, arXiv preprint arXiv:2308.16137 (2023). URL https://arxiv.org/abs/2308.16137
- [75]G. Xiao, Y. Tian, B. Chen, S. Han, M. Lewis, Efficient streaming language models with attention sinks, arXiv preprint arXiv:2309.17453 (2023). URL https://arxiv.org/abs/2309.17453
- [76]Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Re, C. Barrett, Z. Wang, B. Chen, H_2O: Heavy-hitter oracle for efficient generative inference of large language models, arXiv preprint arXiv:2306.14048 (2023). URL https://arxiv.org/abs/2306.14048
- [77]Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, A. Shrivastava, Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time, arXiv preprint arXiv:2305.17118 (2023). URL https://arxiv.org/abs/2305.17118
- [78]J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasicki, S. Han, Quest: Query-aware sparsity for efficient long-context llm inference, arXiv preprint arXiv:2406.10774 (2024). URL https://arxiv.org/abs/2406.10774
- [79]Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, D. Chen, Snapkv: Llm knows what you are looking for before generation, arXiv preprint arXiv:2404.14469 (2024). URL https://arxiv.org/abs/2404.14469
- [80]Z. Cai, Y. Zhang, B. Gao, Y. Liu, Y. Li, T. Liu, K. Lu, W. Xiong, Y. Dong, J. Hu, W. Xiao, Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, arXiv preprint arXiv:2406.02069 (2024). URL https://arxiv.org/abs/2406.02069
- [81]P. Zhang, Z. Liu, S. Xiao, N. Shao, Q. Ye, Z. Dou, Long context compression with activation beacon, arXiv preprint arXiv:2401.03462 (2024). URL https://arxiv.org/abs/2401.03462
- [82]C. Xiao, P. Zhang, X. Han, G. Xiao, Y. Lin, Z. Zhang, Z. Liu, M. Sun, Infllm: Training-free long-context extrapolation for llms with an efficient context memory, arXiv preprint arXiv:2402.04617 (2024). URL https://arxiv.org/abs/2402.04617
- [83]Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, J. Jia, Longlora: Efficient fine-tuning of long-context large language models, arXiv preprint arXiv:2309.12307 (2023). URL https://arxiv.org/abs/2309.12307
- [84]Qwen Team, Qwen2.5 technical report, arXiv preprint arXiv:2412.15115 (2024).
- [85]L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, A. Marafoti, et al., Smollm2: When smol goes big—data-centric training of a small language model, arXiv preprint arXiv:2502.02737 (2025).
- [86]A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023).arxiv.org/abs/2310.06825
- [87]Technology Innovation Institute, Falcon3-7b-instruct model card, Hugging Face, pinned revision 1e57a0ecd17 6c7c139f289c60a74e57f887c3dfb (2024). URL https://huggingface.co/tiiuae/Falcon3-7B-Instruct
- [88]IBM Granite Team, Granite-3.3-8b-instruct model card, Hugging Face, pinned revision 51dd4bc2ade4059a6bd87649d68aa11e4fb2529b (2025). URL https://huggingface.co/ibm-granite/granite-3.3-8binstruct
- [89]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, et al., Qwen3 technical report, arXiv preprint arXiv:2505.09388 (2025).arxiv.org/abs/2505.09388
- [90]PyTorch Contributors, torch.use_deterministic_algorithms, PyTorch 2.8 documentation (2025). URL https://docs.pytorch.org/docs/2.8/generated/torch.use_deterministic_algorithms.html
- [91]NVIDIA Corporation, cuBLAS library user guide, release 12.8: Results reproducibility (section 2.1.1.4), CUDA Toolkit 12.8 documentation (2025). URL https://docs.nvidia.com/cuda/archive/12.8.0/cublas/index.html
- [92]PyTorch Contributors, Reproducibility, PyTorch 2.8 documentation, Notes (2025). URL https://docs.pytorch.org/docs/2.8/notes/randomness.html



