WorksComputer Science
How Much Information Can We Fit inside a Transformer?
Abstract
How efficiently can transformers store factual knowledge? The answer to this question has implications for both model design and understanding the limits of transformer-based systems. Prior studies observe a linear relationship between the number of facts memorized and the number of model parameters required, substantially worse than what theoretical analyses predict is achievable. We re-examine empirical scaling: through rigorous and systematic experiments, including extensive hyperparameter search via Bayesian optimization, we determine the smallest model that can perfectly memorize a given set of facts, and measure how this minimum size scales with the number of facts. On synthetic datasets of random, independent (subject, relation, object) triples—e.g., (Paris, capital_of, France) —drawn at random with no exploitable structure, we find that memorization capacity, the size of the largest dataset that a model memorizes completely, grows exponentially with the transformer dimension. Equivalently, the minimum transformer dimension required to memorize facts grows logarithmically in . Across 11 dataset sizes spanning to , a lack-of-fit F-test does not reject the logarithmic fit (with an offset), but it does reject all other tested alternatives including power-law. Along the depth axis, however, at fixed width, the maximum memorizable dataset size grows sublinearly with the number of layers over the measured range. We further extend our exploration to structured data containing symmetric relations or two-hop composition. In this setting, we find that transformers exploit these regularities to further improve capacity. All code to reproduce our results is available at https://github.com/Magronox/transformer_information_capacity.
Introduction
Large language models (LLMs) exhibit a remarkable ability to memorize facts and associations, sometimes to the point of verbatim recall [21, 22, 23]. As transformer models scale to trillions of parameters, critical questions arise: What is the maximum number of associations a model of given size can memorize? How does this capacity scale with model size? Answering these questions enables practitioners to select appropriate architectures/hyperparameters for knowledge-intensive tasks and predict when transformers will successfully memorize versus fail to learn training data due to insufficient capacity [1]. Our central research questions are: (1) how does the minimum transformer model dimension required to perfectly memorize facts scale with ? and (2) at fixed model dimension, how does the maximum number of memorizable facts scale with the number of layers ?
Prior empirical studies of transformer memorization measure how many bits of factual knowledge are learned per parameter. [20] train GPT-2, LLaMA, and Mistral models on synthetic biographies composed of (name, attribute, value) tuples and establish, through an information-theoretic bit-complexity analysis of their trained model, that sufficiently trained transformers store bits of information per parameter (i.e., linear scaling). [29] train GPT-style models on synthetic random sequences and report bits per parameter. [5], training Qwen-style LLMs directly on relational triples, report approximately linear scaling between fact capacity and number of parameters, estimating that memorizing Wikidata would require billion parameters; however, their measurements use 50–200 training epochs while their own analysis indicates that capacity saturates only at around 1000 epochs and hyperparameters are only partially tuned at different model sizes.
Theoretical work has established bounds on transformer memorization capacity. The result most directly relevant to our work is [7], who study the next-token prediction setting (a single label per input sequence, equivalent to ours) and prove that parameters are sufficient for a transformer to memorize facts, and that parameters are necessary for memorizing every -labeling of facts; these upper and lower bounds are the same up to log factors and are much better than the linear scaling observed in empirical studies. Other upper-bound constructions for related settings give larger parameter counts [9, 10, 11, 8] while Hopfield-network and Johnson–Lindenstrauss theoretical results predict transformer dimension needs to scale logarithmically with for memorization in idealized settings; we discuss both in § 2. Our results show that the standard fixed-depth softmax transformer can memorize facts with the transformer dimension scaling as .
We believe that the gap with prior empirical work has two main sources. Prior studies that report a linear relationship between parameter count and the number of memorized facts [5, 29] use limited hyperparameter search and limited training compute, both of which prevent the model from reaching its maximum information capacity at larger dataset sizes (§ 2). [20] demonstrate that the number of times each fact is seen during training (i.e., exposure) directly governs measured capacity, confirming that insufficient training duration causes underestimation of the model’s information capacity.
In this work we show that standard transformers empirically achieve substantially better scaling than the linear relationship between number of parameters and number of facts reported in prior empirical work [20, 5]. We train transformers on synthetic relational triples [26] with uniformly random structure, since we aim to test the model’s memorization capability while controlling for generalization. We find that the number of facts that a transformer can memorize1 grows exponentially with transformer dimension. Equivalently, the minimum transformer dimension required to memorize facts grows logarithmically in . In this setting, the logarithmic form is the only form that is not rejected by a lack-of-fit F-test (§ 4.1). At fixed width (i.e., transformer dimension) and a fixed training budget, the maximum memorizable dataset size grows with depth (i.e., number of layers), but sublinearly; a lack-of-fit F-test rejects linear, (), affine, (), and superlinear forms (), while a logarithm and a power-law with exponent 0.65 are not rejected, and our data do not distinguish between them (§ 4.2).
In addition, we extend our analysis beyond unstructured uniformly-distributed data. For data with logical structure, such as data containing symmetric relations ( is symmetric if then for any , ) and two-hop compositional relations (if and then for any , , ), we find that transformers exploit these regularities to achieve improved scaling. To probe whether distribution skew alone affects capacity—a relevant question because real-world fact distributions are typically Zipf-like rather than uniform—we additionally test power-law-distributed data, finding that they show similar scaling behavior to uniformly-distributed data.
Our key contributions are:
We demonstrate that memorization capacity grows exponentially with the transformer dimension: at fixed depth , the minimum transformer dimension needed to memorize facts grows logarithmically in , and a lack-of-fit F-test rejects the power-law and other alternatives (§ 4.1, Table tab:width-fits).
At fixed width and a fixed training budget, capacity grows sublinearly with the number of layers: linear and superlinear growth are rejected, and the sublinear forms we tested are not distinguished (§ 4.2, Table tab:depth-forms).
We show that the gap with prior empirical work stems from training methodology: both insufficient training duration and limited hyperparameter optimization lead to suboptimal scaling trends (§ 4.3).
We characterize how relational structure in the data and the shape of the data distribution affect memorization. Structured data (symmetric relations and two-hop compositions, defined in § 3.6) leads to improved scaling by exploiting regularities, and power-law-distributed data shows a similar scaling behavior to uniformly distributed data (§ 4.1, Figure 1).
These findings suggest that LLMs may have substantially greater knowledge capacity than previous empirical estimates indicate, with implications for understanding how transformers store knowledge [16] and for predicting capacity requirements at scale.
Related Work
Scaling Laws.
Empirical scaling laws characterize how language model performance varies with computational resources. [2] established power-law relationships between loss and model parameters, dataset size, and training compute; [3] revised their compute-allocation recommendations, showing parameters and training tokens should scale equally. [4] extended scaling laws to multimodal domains. Extensive work on scaling laws has since followed, and we refer the reader to a survey [1]. These scaling laws characterize average performance (cross-entropy loss) rather than memorization capacity.
Empirical Memorization Studies.
Earlier work by [24] showed that larger transformer models memorize training data faster and can memorize a larger fraction of the training data before overfitting, i.e., before their validation loss begins to increase. Three recent studies measure transformer memorization capacity, with complementary methodologies and findings. [5] train Qwen-style LLMs on relational triples and report approximately linear scaling between parameters and fact capacity, estimating that memorizing Wikidata would require billion parameters; their scaling measurements use 50–200 training epochs, while their own analysis indicates that the fraction of facts recalled correctly continues to improve up to 1000 epochs, suggesting that longer training might yield better scaling behavior. [20] introduce an information-theoretic framework, training GPT-2, LLaMA, and Mistral models on synthetic biography datasets composed of (name, attribute, value) tuples—closely analogous to the relational triples studied by [5] and in our work. They establish bits/param for sufficiently trained transformers. They further show that the number of times each fact is seen during training critically affects measured capacity: with 1000 exposures per fact, transformers achieve 2 bits/param, but with only 100 exposures, they achieve 1 bit/param. [29] estimate capacity for GPT-style models at bits per parameter. Crucially, the bits/param measurements of [20] and [29] are measurements for sufficiently trained transformers for a specific dataset size, not assertions about how capacity scales with dataset size more broadly. Our work is complementary: rather than reporting a bits-per-parameter estimate for a single dataset size, we fit a precise functional form , the minimum transformer dimension that achieves perfect memorization, across a wide range of values of , finding that grows logarithmically in across this range.
Theoretical Bounds on Memorization Capacity.
Theoretical analyses of transformer memorization build on contextual mapping constructions [12]. These results are typically worst-case bounds over arbitrary labelings, and assumptions vary across settings (e.g., the prediction task, label space, and separation conditions on inputs). Most directly relevant to our setting, [7] prove that parameters are sufficient to memorize sequences with a single label each in the next-token prediction setting, and that this bound is tight up to log factors. Earlier upper-bound constructions for related but distinct settings give different parameter-count scalings: (1) parameters for sequence-to-sequence prediction with one-layer transformers with transformer dimension and input sequence length [9], (2) parameters for sequence-to-sequence prediction with layers [8], (3) parameters for next-token distribution prediction over distinct contexts with a vocabulary of size [10], and (4) parameters for a multi-head attention layer with heads to memorize linearly independent inputs [11].
[28] study a synthetic factual-recall task with fixed random embeddings and prove that a single-layer transformer reaches perfect accuracy when either its self-attention or its MLP parameters scale linearly with the number of facts (up to log factors). Our embeddings are jointly trained, and the number of non-embedding parameters grow as in , rather than the linear .
[28] further prove that a single self-attention layer followed by an MLP stores facts at a rate proportional to its parameter count, and [30] find on synthetic sequence tasks that one attention layer suffices for memorization while reasoning and generalization need two or three layers. Neither result predicts what gradient training can achieve when layers are added at a fixed width and a fixed training budget. We measure this in our depth experiment.
Two theoretical results predict that storing items in a -dimensional space is achievable when . The Johnson–Lindenstrauss lemma [13] establishes that near-orthogonal unit vectors fit in -dimensional Euclidean space. Equivalently, suffices to fit such vectors. Modern Hopfield-network analyses [14, 15] show that softmax attention, viewed as an associative memory over stored patterns, has retrieval capacity exponential in the pattern dimension. Both results concern idealized settings—vector geometry and a single softmax retrieval step over fixed patterns, respectively—rather than trained transformers with FFN, residuals, and learned embeddings.
Our results on memorization capacity scaling with respect to transformer dimension are consistent with concurrent work by [33], who fit a logarithmic curve to the smallest embedding dimension that reaches 95% training accuracy on random bijections. However, they do not report a statistical test and do not compare the logarithmic fit with other forms such as a power-law. Our work, on the other hand, systematically studies the scaling of memorization capacity across 11 dataset sizes and compares logarithmic, power-law and linear forms with a lack-of-fit F-test (§ 4.1).
Background and Methods
Throughout, we use to denote the number of facts in the dataset, for the transformer dimension (which equals the embedding dimension in our setting, by standard convention), for the number of layers, and for the transformer (non-embedding) parameter count. The total number of parameters is , where is the size of the embedding table; in our experimental setup scales with .
Our goal is to investigate how transformer memorization capacity scales with model dimension (width) and number of layers (depth) under realistic training constraints.
We measure capacity along two axes: (1) given a dataset, we find the smallest dimension that achieves perfect memorization (100% accuracy); (2) given a model, we find the largest dataset size it can perfectly memorize. We optimize training hyperparameters in each setting to maximize measured capacity. For detailed mathematical formulation, see Appendix app:optimization.
We make a simplifying assumption that all facts have the form , where is the subject entity, is the relation, and is the object entity. For example, the fact “Paris is the capital of France” can be expressed as . This representation is widely adopted in empirical and theoretical studies of memorization and factual recall in language models [26, 23, 18, 17, 19, 20, 5, 28]. In this controlled setting, on unstructured data, the model can only achieve high accuracy by memorizing facts. The abstraction corresponds to these representations. Further, it is easy to generate an arbitrary number of triples with a high degree of control over their distribution and structure. Studying the ability of transformers to memorize triples facilitates our measurement of the fundamental capacity of the associative memory mechanisms that transformers employ for factual knowledge, enabling us to probe the model’s memorization capacity independently of its information extraction ability.
Task Formulation
We measure memorization capacity using knowledge graph triples where the subject and relation together uniquely specify the object . Each fact query is tokenized as a sequence of two input tokens and the model must learn to map each input to its unique corresponding output token . This task requires perfect memorization of all associations in the dataset.
Definition (memorization and capacity).
A trained model memorizes a fact if, given the two-token input , its highest-probability output token is (i.e., greedy decoding). A model memorizes a dataset of facts if it memorizes every fact—that is, it reaches 100% training accuracy. Capacity is defined as the largest dataset size that a given model can memorize completely. The transformer width and depth scaling experiments measure this same quantity in two different ways. Let denote the model obtained by training a transformer of dimension and depth on a dataset of facts with hyperparameters (Table tab:hparam-search), until it reaches 100% training accuracy or exhausts the training budget. We consider two methods for measuring the capacity of the transformer: (1) We fix the dataset size and search for the smallest transformer model dimension such that memorizes the dataset for some setting of hyperparameters (Eq. (1)); this finds the smallest model whose capacity is at least . (2) We fix the model and search for the largest such that memorizes the dataset (Eq. (2)); this measures the model’s capacity directly. For brevity, we focus on the first method for the width scaling experiments, where we fix , but we demonstrate that both methods yield the same results in Appendix sec:exp-maxn-width. For the depth scaling experiments, since is fixed, the first method is not feasible and so we apply the second method. All reported values use the hyperparameter configuration selected by the search described in Appendix app:hyperopt, with 60 completed trials per learning-rate schedule.
Model Architecture
We use the GPT-2 [6] architecture, with the following differences: a single attention head (), the Swish activation function in place of GELU, no dropout, and no positional embeddings; since every input has the fixed form , position is implicit in the input structure. In preliminary experiments, models trained with and without learned positional embeddings required the same minimum dimension (Appendix sec:Appendix-results). The vocabulary has tokens, one for each entity and relation, in a single embedding table of size , shared between input and output.
Following standard convention [2], the embedding dimension is , so parameters are split into embedding () and non-embedding (, attention and feed-forward weights); the latter is for our single-head architecture, which equals at . We report and the embedding parameters separately throughout, since the embedding table is determined by vocabulary and is determined by the architecture alone.
Hyperparameter Optimization
We optimize training hyperparameters using Bayesian optimization (BO) with tree-structured Parzen estimator (TPE) sampling via the Optuna library [27, 25]. We stratify the search by learning-rate schedule (linear or cosine): a separate Bayesian optimization is run for each schedule and the best across schedules is reported. The search space is given in Table tab:hparam-search (Appendix app:hyperopt); each candidate is evaluated on three seeds, with 60 completed trials per schedule for every study.
Training Protocol
We train in bfloat16 mixed precision using the Adam optimizer [31] to minimize cross-entropy loss with the weight decay parameter obtained via Bayesian optimization. Training is full-batch: every fact appears in every iteration. We cap training at 40,000 iterations (excluding warmup)—up to 40,000 exposures per fact—based on our convergence analysis (Appendix sec:Appendix-results), with early stopping only after 100% accuracy holds for 50 consecutive iterations, i.e., after memorization is complete. That is 40 the 1,000-exposure saturation regime of [20].
Capacity Search Procedures
A training run fails if it does not reach 100% training accuracy and maintain it for 50 consecutive iterations within the 40,000-iteration budget, and succeeds otherwise. For the experiments where we aim to find the smallest model dimension that leads to successful memorization, we perform a search to find —the smallest dimension at which training succeeds—for each hyperparameter configuration. To this end, we perform a two-phase search: In the first phase, we repeatedly perform geometric halving, starting from , until the first failure (or doubling until the first success if the starting estimate fails). In the second phase, we perform binary search to pinpoint . For the experiments where we search for the largest dataset that leads to successful memorization, we similarly perform the same two-phase search procedure to find —the largest dataset size at which training succeeds. The full procedure is in Appendix app:hyperopt.
Dataset Construction
For all experiments, we use synthetic datasets comprising triples. We generate both unstructured and structured datasets to investigate whether transformers can exploit structure to achieve higher memorization capacity. To create a dataset, we specify two parameters: the number of entities and the number of relations .
We generate four datasets: Uniform, Symmetric, 2Hop, and PowerLaw.
To remain consistent with our deterministic full-batch methodology, each batch contains every fact. For Uniform, Symmetric, and 2Hop, each fact appears once per batch. For PowerLaw (Zipf with exponent ), each fact appears times per batch, where is its Zipf probability and the multiplier is chosen so that the rarest fact appears exactly once per batch. Frequent facts therefore appear more times per batch than rare ones; full details for generating PowerLaw are in Appendix app:powerlaw. We conduct experiments with entities and relations across all dataset families (Symmetric up to and 2Hop up to ). For PowerLaw, we only consider since grows with and the batch becomes prohibitively large otherwise.
Uniform Dataset (unstructured): For Uniform, we create a dataset such that, for any subject and for any relation , there is a triple in the dataset where the object is uniformly sampled from the set . For any subject-relation pair , we have a unique object . Therefore, the dataset size is .
Symmetric Dataset (structured): For Symmetric, for each relation we shuffle the entities uniformly at random and pair them in order, the first with the second, the third with the fourth, and so on (when is odd, the last entity is paired with itself). Each entity is in exactly one pair, and for each pair we add both and to the dataset (a single fact if ). Every relation is therefore symmetric: if a triple is in the dataset, then its counterpart is also in the dataset (e.g., if is a sibling of , then is a sibling of ).
2Hop Dataset (structured): For 2Hop, we use relations and dedicate the last relation to the two-hop compositions. For relations and , objects are sampled uniformly as in Uniform; then, for each subject , we add the triple , where and are the entities such that and are in the dataset. This is analogous to real-world relations that have the same multi-hop compositional property (e.g., if is the mother of and is the parent of , then is the grandmother of ). This format preserves the uniqueness property, i.e., for any pair , is unique. Step-by-step generation procedures for all dataset families are given in Appendix sec:Appendix-algorithms (Algorithms alg:gen-dataset and alg:powerlaw-batch).
Power-law Dataset: To probe whether non-uniform fact distributions affect memorization capacity, we sample triples from a Zipf distribution: each triple is assigned a rank and sampled with probability .
Results
We characterize how transformer memorization capacity scales with model dimension (width) and number of layers (depth).
Width scaling: Capacity grows exponentially with transformer width
Figure 1. Minimum transformer dimension required for complete memorization vs. dataset size (log scale) at , mean over three seeds. Uniform (blue circles), with the logarithmic-with-offset fit of Table tab:width-fits (light blue line). PowerLaw (amber triangles; Zipf ), tested up to . Symmetric (pink circles), up to , and 2Hop (green circles), up to .
Figure 1 plots the minimum transformer dimension required for complete memorization across all four dataset families at . Two findings are visible.
Exponential capacity scaling on Uniform. As a general observation, across the measured range the number of non-embedding parameters per memorized fact strictly decreases as the dataset grows; the logarithmic fit below is our best quantitative description of this trend within this range. Table tab:width-fits compares candidate fits on the 11 Uniform measurements, spanning to (all fits computed in log space; we compare forms with the lack-of-fit F-test for repeated measurements, [32], using the three seeds at each size, Appendix § 6.0.0.3). The logarithmic-with-offset form achieves , and is not rejected by a lack-of-fit F-test (); the power-law is rejected with or without an offset. The largest dataset sizes are what separate the two forms. Considering only dataset sizes up to , neither power-law is rejected ( without an offset, with one); with the sizes up to , both are rejected ( and ).
| Functional form | Best fit | Lack-of-fit -value | |
| Logarithmic + offset | |||
| Logarithmic | |||
| Power-law + offset | |||
| Power-law | |||
| Linear |
Table 1. Width-scaling fits on Uniform (, 11 measured points, to ). All fits are computed in log space, with each fit (except the pure logarithmic fit) constrained so that the fitted is positive for every ; the last column is the lack-of-fit F-test -value ( rejects the functional form).
Structured datasets require smaller dimension. For Symmetric (where the relation is its own inverse) and 2Hop (where we have two-hop compositional relations), grows more slowly than for Uniform. Under our full hyperparameter optimization protocol, this gap is large and consistent from onward (at the three smallest dataset sizes, all families require similar dimensions). When the hyperparameter search is restricted to finding only the learning rate and batch size, as in prior empirical work, the three families require the same dimension, as we discuss in the ablation studies (§ 4.3).
Depth scaling: Capacity grows sublinearly with the number of layers
Figure 2. Maximum memorizable dataset size vs. number of layers at fixed width (Uniform dataset). Grey dots: the three seeds; blue: their mean. Lines: the logarithmic (solid) and power-law (dashed) fits of Table tab:depth-forms, the two forms that the lack-of-fit test does not reject.
Figure 2 and Table tab:depth-fits (Appendix sec:Appendix-results) report the maximum memorizable dataset size as a function of layer count at fixed width on Uniform data. At fixed width, the maximum memorizable dataset size grows with depth up to layers, but sublinearly. Over the measured range a logarithmic form fits best, but the data do not distinguish between logarithmic and power-law scaling (Table tab:depth-forms, Appendix sec:Appendix-results).
Two features of the experiment bear on this result. First, the training budget is the same at every depth: 40,000 iterations after warmup, with a hyperparameter search of 60 trials per learning-rate schedule. The reported capacity at each depth comes from three training runs, one per seed, each at the largest dataset size at which the run does not fail (§ 3.5). The median iteration at which they converge is 51% of the budget at and between 77% and 91% at . This suggests that, at larger values of , a larger budget can increase the measured capacity, which we leave to future work. It’s also interesting to note that training is also less stable at larger depth. The fraction of training runs that reach 100% accuracy but do not hold it for 50 consecutive steps rises from 2.9% at to 10.3% at . Second, memorization is a task that one layer can already do at a rate proportional to its parameter count: [28] prove that a single layer of self-attention followed by an MLP reaches 100% accuracy on factual recall if the parameter count of either component is linear in the number of facts, up to log factors. [30] find that one attention layer suffices for memorization while reasoning and generalization need at least two or three.
Ablation Studies
Impact of suboptimal hyperparameters.
To test whether the training protocol alone can account for the difference in scaling with prior work, we ran the width experiment on the same three dataset families and the same architecture while restricting the hyperparameter search to the learning rate and the batch size, which is the extent of tuning in prior work [5, 29]. Under this restricted search, a power-law fit gives an exponent between 0.73 and 0.77 for each of the three families (Figure 3, Appendix sec:Appendix-results), so grows as to : the number of non-embedding parameters grow superlinearly in , which is even less favorable scaling than the linear relationship reported by prior work. Symmetric and 2Hop require the same dimension as Uniform. Expanding optimization to include weight decay, minimum learning rate, learning rate schedule, and decay duration transforms both findings: Capacity scales exponentially with respect to the transformer dimension for Uniform (Figure 1), and the model’s ability to leverage structure emerges—structured datasets require fewer parameters than Uniform at every dataset size in .
Batching and maximum training batches.
Beyond hyperparameters, two factors proved important. We found that random batch sampling caused loss spikes from adverse batches, requiring many batches to recover and inflating our estimate of the minimum transformer dimension ; deterministic full-batch updates eliminated this issue (Appendix app:sampling). Since each iteration uses the full dataset, each iteration is one exposure per fact, and we use a higher maximum number of exposures per fact (up to 40,000) than [5]. This is necessary because convergence slows dramatically as the model dimension approaches (Appendix E, Figure 4). Full-batch training also keeps the number of exposures per fact the same for every dataset size, because every iteration uses every fact. With a fixed batch size and a fixed number of steps [29], each fact is seen times during training on average, which decreases as the dataset grows.
Generality across architectures: OLMo-3
To test whether our findings are specific to the GPT-2 architecture, we repeated the width scaling experiment on the OLMo-3 architecture at 8 dataset sizes, , , , , , , , and , under the same experimental protocol; the OLMo-3 training details are given in Appendix app:olmo3-details. Over this range, OLMo-3 shows the same scaling behavior as GPT-2 (Table tab:olmo-width, Appendix sec:Appendix-results). The minimum transformer model dimension grows sublinearly with the dataset size, a lack-of-fit F-test rejects a linear fit (), and neither a logarithmic nor a power-law fit is rejected, as with GPT-2 at these sizes. OLMo-3 memorizes each dataset at a smaller model dimension than GPT-2.
Analysis
Our experiments reveal that memorization capacity grows exponentially with transformer dimension, with structured relations reaching a given capacity at a smaller dimension than unstructured data. For the Uniform dataset, the best-fit form for is logarithmic-with-offset, (); a lack-of-fit F-test rejects the power-law alternatives (Table tab:width-fits). Under this fit, non-embedding parameters grow as , which is sublinear in ; equivalently, capacity grows superlinearly in . Structured datasets (Symmetric, 2Hop) achieve further improvements in efficiency. Over the measured range, the logarithmic fit for implies , which grows more slowly than , the theoretical worst-case bound proved by [7], although within our range itself is larger than (4,617 against 223 at ).
Embedding versus non-embedding parameters.
Thus far, our investigation has focused on the effect of non-embedding parameters on memorization capacity. But how much do the embedding parameters contribute to the capacity? In the width experiments on Uniform (, relations, to ), the non-embedding parameters and the embedding table scale differently. The number of non-embedding parameters, , grows quadratically in . Following standard convention (§ 3), the embedding table stores one -dimensional vector per token, resulting in rows, a number that grows linearly in . As grows logarithmically as a function of , the table grows exponentially versus , and its share of all parameters increases from 25% at to 99% at (Table tab:bits-split, Appendix sec:Appendix-results). Counting every parameter, a dataset with bits yields 0.72 to 1.80 bits per parameter at the five smallest dataset sizes () and 2.04 to 2.19 at the six largest dataset sizes . Therefore, bits per parameter increases when is small and shows signs of saturation when is large. A lack-of-fit F-test rejects the two non-saturating forms, a logarithm () and a power-law (), but not the saturating forms and ( and ).
Prior work reports bits per parameter for GPT-2, LLaMA and Mistral models [20] and for GPT-style models [29]. In comparison, we measure 2.04 to 2.19 bits per parameter at six dataset sizes . Both prior studies use a fixed vocabulary, and [20] utilize several GPT-2 tokens to encode each entity. A fixed vocabulary is a practical choice, but composing entities from several tokens could confound the measurement of memorization capacity in terms of the number of facts. However, if each entity corresponds to a vocabulary item, a fixed vocabulary would limit the maximum number of expressible facts and would induce an artificial ceiling on the measured information capacity. Therefore, in our above experiments, we allowed the vocabulary to vary as a function of dataset size, where each entity and relation in the dataset is encoded as a unique token. But this raises the question as to how varies as a function of vocabulary size. To measure this, we conduct an additional experiment where we hold fixed and vary the dataset from 9,216 entities with 1 relation to 96 entities with 96 relations, thereby varying the vocabulary size by a factor of 48 (Appendix sec:exp-sweep). Every dataset remains memorizable, and rises from 6 () to 46.33 () before falling to 45.0 (). We note that this experiment does not test whether still grows logarithmically in under an embedding table of fixed size (and sufficient capacity).
Conclusion and Future Work
We systematically measured transformer memorization capacity along width and depth axes and find that, under fixed budget, capacity scales exponentially with transformer dimension, and sublinearly with transformer depth. The minimum transformer dimension required to memorize facts grows logarithmically in , in contrast to the linear scaling reported by prior empirical studies. Our results further shed light on aspects of scaling law for memorization capacity that were missing in prior work.
Limitations.
Our experiments span . Whether the logarithmic law for extends beyond the measured range is an empirical question. Our measurements use synthetic triples whose information content is exactly known; testing capacity on natural-language datasets such as [20], where the model must both store the facts and learn to parse and express them in natural language, is an interesting next step.
Future work.
Our scaling experiments vary one axis at a time. Measuring capacity over the joint space of width and depth would help to further illuminate scaling behavior and identify potential trade-offs between the axes; we leave this to future work.
AI use statement
In this work, we used generative AI tools (large language models) for writing and editing the text of the paper, and for writing research code. We have reviewed all AI-assisted work; every number in the paper was recomputed from the experiment databases by scripts that the authors inspected, every cited paper was read and its bibliographic entry checked by an author, and AI-written code was audited and tested by the authors before use. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.
Ethics statement
All datasets in this work are synthetic (subject, relation, object) triples generated by the procedures in Appendix sec:Appendix-algorithms. The work involves no human subjects, personal data, or scraped text, and we are not aware of potential conflicts of interest.
Reproducibility statement
The dataset-generation procedures are given in Algorithms alg:gen-dataset and alg:powerlaw-batch; the model and the training protocol in § 3; the hyperparameter search space, the search procedures, and the search objectives in Appendix app:hyperopt; and the statistical test used to compare scaling forms in Appendix § 6.0.0.3. The code to reproduce the experiments and the results is open source and publicly available at https://github.com/Magronox/transformer_information_capacity.
Acknowledgments
We thank the anonymous reviewers for their considerate and helpful feedback. This work was supported in part by the Rosen Center for Advanced Computing (RCAC) resources, services, and staff expertise at Purdue University.
We measure transformer memorization capacity in two complementary ways: in width experiments, we find the minimum transformer dimension for a fixed dataset; in depth experiments, we find the maximum dataset size for a fixed transformer. Both depend critically on training hyperparameters: suboptimal learning rates, weight decay, or schedules can prevent memorization even when the model has sufficient capacity.
Let denote the hyperparameter vector listed in Table tab:hparam-search. Let denote training a transformer of dimension and depth on a dataset of size with hyperparameters until either reaching 100% accuracy or exhausting the training budget.
Width experiments (, dataset size fixed).
We seek for each in our experimental sweep.
Depth experiments (dimension and number of layers fixed).
We seek for each .
Procedural details (per-trial search, multi-seed evaluation, BO objective formula) are in Appendix app:hyperopt.
| Parameter | Search space | Notes |
| Learning rate | (log) | depth: |
| Use weight decay | gate; if False, | |
| Weight decay | (log) | only sampled when gate is True |
| Min-LR ratio | (log) | |
| Schedule-shape: | ||
| Decay end step | step (after warmup) where decay ends | |
| LR decay schedule | stratified (separate study per type) | |
| Warmup steps | fixed; depth: |
Table 2. Hyperparameter search space. Decay schedule is treated as a stratification variable: a separate optimization study is run for each schedule type and the best is reported. Warmup is fixed (not searched), as initial optimization results showed this would not hurt the best value achieved. Discussed in § 3.
Note that log in the table means the optimization was done in the log space.
Per-trial dimension search (width experiments, Algorithm alg:min-dim). For each candidate configuration and seed , we locate as follows:
Geometric halving downward. Start at and halve (), training at each and recording the smallest passing dimension.
Upward fallback. If itself fails, double upward () until a passing dimension is found. If none passes by , the seed is marked as failed.
Binary search. Refine between the last passing and first failing dimension to pinpoint .
Per-trial dataset-size search (depth experiments, Algorithm alg:max-size). For each candidate configuration and seed , we locate as follows:
Geometric doubling/halving. Starting from an initial estimate of , double or halve the dataset size until the first success–failure boundary is found.
Binary search. Refine between the last passing and first failing dataset size to pinpoint .
Multi-seed objective. Each candidate configuration is evaluated jointly on three seeds (10, 20, 30) run in parallel across available GPUs, and is scored by
where is the smallest transformer model dimension at which the training run with seed memorizes the dataset, if seed fails to memorize within the training budget, and is the number of failed seeds. The objective primarily minimizes the transformer model dimension; the two remaining terms serve only as tiebreakers between hyperparameter configurations that reach the same minimum dimension, preferring configurations with more reliable training (fewer failed seeds and less spread across seeds). The coefficients 0.1 and 0.05 are chosen small enough that the tiebreakers cannot outweigh the minimum dimension: because the dimension is an integer, changing the winning configuration would require the tiebreaker terms to add up to at least 1, which in turn would require the seeds to disagree by about 20 dimensions in total. No completed configuration came close to this.
Multi-seed objective (depth experiments). In the depth experiments, each seed runs its own dataset-size search, and each hyperparameter configuration is scored by
where is the largest dataset size that the training run with seed memorizes, if seed memorizes no dataset size, is the number of failed seeds, and facts is the coarse step of the size search. The objective primarily maximizes the dataset size; the two remaining terms prefer configurations with more reliable training (fewer failed seeds and less spread across seeds). Unlike in the width experiments, these terms are measured in facts and can change which configuration scores highest. The objective only guides the search: for each depth we report the configuration with the largest mean over the three seeds (Table tab:depth-fits), which does not use the objective.
Lack-of-fit F-test.
To test whether a functional form with parameters describes a scaling curve measured at points (dataset sizes, or numbers of layers), we use the lack-of-fit F-test for repeated measurements [32]. At each point we have three values , one per seed, where is the logarithm of the measured quantity (, , or bits per parameter). Let be their mean. We fit to the means by least squares, and compute the pure-error and lack-of-fit sums of squares
with and degrees of freedom. If is the correct form, follows an distribution; we report and reject the form when . The test asks whether the fitted curve misses the means by more than the spread across seeds explains, so a large means the form is consistent with the data, not that it is proven.
To evaluate whether logarithmic width scaling is specific to GPT-2 or generalizes across modern transformer architectures, we repeat the width scaling experiment using the OLMo-3 architecture.
To maintain parameter fairness against our GPT-2 baseline (Section 3.2) at width , the OLMo-3 block is aligned via three adjustments:
Disabling Positional Embeddings for position-free processing across all integer widths ;
Using standard full attention matching GPT-2; and
Setting the SwiGLU (Swish-Gated Linear Unit) feed-forward hidden dimension to so that its 3-matrix FFN parameter count matches GPT-2’s standard 2-matrix FFN up to rounding.
Models are trained using full-batch gradient descent, bfloat16 mixed precision, and the Adam optimizer. For each dataset size , hyperparameters are optimized over the search space of Table tab:hparam-search with 60 Optuna TPE trials per learning-rate schedule. Convergence requires maintaining 100% training accuracy for 50 consecutive evaluation steps within a budget of 40,000 batches.
As detailed in Table 3, the minimum required dimension for OLMo-3 grows from at to at ; as for GPT-2, a lack-of-fit F-test rejects a linear fit () while a logarithm () and a power law with exponent 0.22 () both pass.
| Entities () | Facts () | Information (Bits) | Seeds (10 / 20 / 30) | Mean | Best |
| 15 | 45 | 176 | 2 / 2 / 2 | 2.00 | 2 |
| 30 | 90 | 442 | 3 / 4 / 2 | 3.00 | 2 |
| 60 | 180 | 1,063 | 3 / 3 / 3 | 3.00 | 3 |
| 120 | 360 | 2,486 | 4 / 4 / 4 | 4.00 | 4 |
| 260 | 780 | 6,257 | 4 / 4 / 5 | 4.33 | 4 |
| 520 | 1,560 | 14,075 | 5 / 5 / 5 | 5.00 | 5 |
| 1040 | 3,120 | 31,270 | 6 / 6 / 6 | 6.00 | 6 |
| 2080 | 6,240 | 68,780 | 6 / 6 / 6 | 6.00 | 6 |
Table 3. OLMo-3 factual capacity measurements across dataset sizes . is the minimum width achieving 100% memorization accuracy across three random seeds (seeds 10, 20, 30).
Throughout our experiments, we found that memorization—and, consequently, the estimated minimum transformer dimension —was highly sensitive to random sampling of training examples. This sensitivity persisted even when the sample size used per update exceeded the dataset size by factors (random sampling with replacement), indicating that the issue was not whether the sample was small or large, but rather the randomness of the draw itself. In particular, we observed that a single adverse sample (a “bad batch” in the sense of an unlucky random draw) could induce a large, misaligned update that slowed loss descent, as evidenced by spikes or plateaus in the training loss traces (Figure 5). When we removed this source of noise by using deterministic full-dataset updates (i.e., the exact empirical gradient computed over a fixed example order), the minimum feasible dimension dropped significantly, and the region of settings that achieved this minimum became stable across seeds.
Based on this discovery, for all subsequent experiments, we fix the batch size: for Uniform, Symmetric, and 2Hop datasets, we set (the full dataset size), processing all triples in each batch. For PowerLaw distributions, we use Power-Law sampling as mentioned in Appendix app:powerlaw.
To probe whether non-uniform fact distributions affect memorization capacity, we sample triples according to a Zipf distribution. For a dataset with entities and relations ( total triples), each triple is assigned a rank and sampled with probability:
and we use in our experiments.
The replication rule is: each fact appears times per batch, where the multiplier is set so that the rarest fact (smallest ) appears exactly once per batch. The batch is constructed once and reused for every training iteration (Algorithm alg:powerlaw-batch). The resulting per-batch size grows roughly as , which is why we capped tested at (§ 3.6).
Positional embeddings.
In preliminary experiments we trained models with and without learned positional embeddings. Since every input has the fixed form , position is implicit in the input structure, and positional embeddings did not change the minimum transformer model dimension required for memorization; we therefore omit them in all experiments.
Figure 3. Minimum model dimension required for complete memorization vs. dataset size with power-law fit (offset unconstrained) across dataset types (Uniform, Symmetric, 2Hop), when we only optimized learning rate and batch size. The minimum transformer dimension required for complete memorization is way higher than under the full search (Figure 1), highlighting the impact of hyperparameter optimization.
| Body % | ||||
| 45 | 3.33 | 75% | 1.0 | 0.72 |
| 90 | 4.00 | 66% | 1.8 | 1.15 |
| 180 | 5.00 | 54% | 2.8 | 1.54 |
| 360 | 6.67 | 44% | 3.9 | 1.71 |
| 780 | 9.00 | 32% | 5.7 | 1.80 |
| 1,560 | 10.33 | 21% | 9.8 | 2.06 |
| 3,120 | 12.67 | 14% | 14.8 | 2.04 |
| 6,240 | 14.33 | 8% | 25.7 | 2.11 |
| 12,480 | 16.00 | 5% | 45.3 | 2.15 |
| 24,960 | 17.33 | 3% | 84.1 | 2.19 |
| 49,920 | 19.00 | 1% | 151.6 | 2.18 |
Table 4. Non-embedding-vs-embedding parameter breakdown for the smallest single-layer model that memorizes a Uniform dataset of size ( averaged over 3 seeds). Non-embedding: ; embedding: ; bits . Including embedding parameters, capacity rises from 0.72 to bits per parameter, reaching from onward. Discussed in § 5.
| Functional form | Best fit | Lack-of-fit | |
| Linear without intercept | |||
| Linear with intercept | |||
| Power-law, exponent | |||
| Exponential | |||
| Logarithmic | |||
| Power-law |
Table 5. Depth-scaling fits on Uniform (fixed width , 7 depths, 3 seeds). All fits are computed in log space; the last column is the lack-of-fit F-test -value ( rejects the functional form).
| 45 | 90 | 180 | 360 | 780 | 1,560 | 3,120 | 6,240 | |
| OLMo-3 | 2.00 | 3.00 | 3.00 | 4.00 | 4.33 | 5.00 | 6.00 | 6.00 |
| GPT-2 | 3.33 | 4.00 | 5.00 | 6.67 | 9.00 | 10.33 | 12.67 | 14.33 |
Table 6. OLMo-3 width scaling experiment: minimum transformer model dimension needed to memorize all facts, with relations (averaged over 3 seeds; GPT-2 shown for comparison).
| layers | 1 | 2 | 4 | 6 | 8 | 12 | 16 |
| max (mean) | 730 | 1,580 | 2,560 | 3,010 | 3,290 | 4,160 | 4,850 |
| smallest seed | 720 | 990 | 2,310 | 2,940 | 3,180 | 3,990 | 4,530 |
| largest seed | 750 | 2,010 | 2,760 | 3,060 | 3,420 | 4,320 | 5,010 |
Table 7. Depth scaling experiment at transformer model dimension 8: maximum dataset size memorized completely, for each number of layers (Uniform, ). Each entry is the mean over seeds 10, 20, 30 for the hyperparameter configuration with the largest mean; the range is the smallest and largest seed.
§ 4.1 measures width scaling by finding the smallest dimension that memorizes each dataset size . The same scaling can also be measured by finding the largest dataset size that each dimension memorizes. To confirm that the two methods result in similar scaling behavior, we ran the maximum-dataset-size search (Algorithm alg:max-size) at for transformer dimensions to , with seeds 10, 20, 30 and 60 completed trials per width. We only ran this with linear learning-rate schedule, and the fine step for search is 3 facts instead of 30, because the memorization capacities at small are below 100 facts. Table tab:maxn-width compares the measured with the at which the logarithmic-with-offset fit of Table tab:width-fits predicts ; they differ by to . Table tab:maxn-width-fits fits the same forms as Table tab:width-fits, with the same test and constraints. As with the minimum-dimension search (Table tab:width-fits), the logarithmic-with-offset form is the only form that is not rejected (); both power laws, the plain logarithm and the linear form are rejected.
| width | 3 | 4 | 5 | 7 | 8 | 9 | 10 | 13 | 14 | 16 |
| (measured) | 34 | 89 | 214 | 452 | 704 | 918 | 1,584 | 4,433 | 7,379 | 10,129 |
| (Table tab:width-fits fit) | 35 | 85 | 158 | 413 | 628 | 939 | 1,384 | 4,270 | 6,173 | 12,841 |
| difference |
Table 8. Largest memorizable dataset size at for each width (mean over seeds 10, 20, 30), and the predicted by the width scaling experiment’s logarithmic-with-offset fit of Table tab:width-fits.
| Functional form | Best fit | Lack-of-fit | |
| Logarithmic + offset | |||
| Logarithmic | |||
| Power-law + offset | |||
| Power-law | |||
| Linear |
Table 9. Fits to the measurements of Table tab:maxn-width, written as as in Table tab:width-fits. Fits are computed on with the three seeds; the offsets are constrained so that the fitted is positive for every ; the last column is the lack-of-fit F-test -value ( rejects the functional form).
To test whether the growing vocabulary drives our results, we hold the number of facts fixed at and vary the composition of the dataset—from 9,216 entities with 1 relation to 96 entities with 96 relations—shrinking the vocabulary (and with it the embedding table) 48-fold. Every dataset remains fully memorizable, and the minimum transformer model dimension rises from 6.00 with relation to 46.33 with relations, then falls to 45.00 with . A joint fit with the width scaling experiment’s dataset sizes gives (), where is the number of entities and the number of relations; a lack-of-fit F-test rejects this form ().
| entities | 9216 | 4608 | 3072 | 2304 | 1152 | 576 | 288 | 144 | 96 |
| relations | 1 | 2 | 3 | 4 | 8 | 16 | 32 | 64 | 96 |
| min. dimension to memorize all facts | 6.00 | 10.67 | 15.33 | 19.00 | 29.33 | 38.00 | 44.00 | 46.33 | 45.00 |
Table 10. Minimum transformer model dimension needed to memorize facts as we vary the composition of the dataset, from 9,216 entities with 1 relation to 96 entities with 96 relations (averaged over 3 seeds).
Figure 4 illustrates the relationship between the number of batches required for training accuracy vs the transformer dimension across different dataset sizes. We observe a clear "capacity compute tradeoff": a smaller dimension requires many more training batches to achieve the same level of capacity as larger models.
Furthermore, the plot compares 2 different decay types and shows that both schedules eventually converge to a similar minimum dimension. This figure shows that a high number of batches is essential for uncovering the true scaling regime between transformer dimension and dataset size.
Figure 4. Convergence Speed vs. Dimension. The total number of batches required to achieve 100% training accuracy vs transformer dimension for different entity sizes . Plots for both the linear and cosine decay types are shown. The y-axis is on a log scale, showing that as the model approaches the theoretical minimum dimension , the cost required to train for perfect memorization increases exponentially. The shaded bands represent the variation of the number of batches required over 3 seeds.
Figure 5. Training loss vs. number of iterations under random batch sampling (Uniform dataset of size 1500, transformer dimension 95). Sudden spikes correspond to “bad batches”—unlucky random draws whose gradient is misaligned with the loss surface, causing transient divergence and slowing convergence. These episodes inflate measured when the iteration cap is finite. Switching to deterministic full-batch updates eliminates them and reduces the minimum dimension required for memorization.
\caption{Dataset generation: GenerateDataset($n$, $m$, $type$)}
\label{alg:gen-dataset}
\begin{algorithmic}[1]
\REQUIRE number of entities $n$, number of relations $m$, $type \in \{\texttt{Uniform}, \texttt{Symmetric}, \texttt{2Hop}\}$. Each entity and relation is a single vocabulary token ($|V| = n + m$). For \texttt{2Hop}, $m = 3$ and the last relation $r_3$ is dedicated to the two-hop compositions.
\STATE $D \gets \emptyset$
\FORALL{relations $r \in \{1, \ldots, m\}$}
\FORALL{subjects $s \in \{1, \ldots, n\}$}
\IF{$type = \texttt{Uniform}$}
\STATE sample $o$ uniformly from $\{1, \ldots, n\}$; add $(s, r, o)$ to $D$
\ELSIF{$type = \texttt{Symmetric}$}
\STATE add $(s, r, \pi_r(s))$ to $D$, where $\pi_r$ pairs consecutive entities of a uniformly random shuffle of the $n$ entities, drawn once for each relation $r$ (the last entity is paired with itself when $n$ is odd); $(\pi_r(s), r, s)$ is therefore in $D$ as well
\ELSIF{$type = \texttt{2Hop}$ \AND $r \in \{r_1, r_2\}$}
\STATE sample $o$ uniformly from $\{1, \ldots, n\}$; add $(s, r, o)$ to $D$
\ELSIF{$type = \texttt{2Hop}$ \AND $r = r_3$}
\STATE add $(s, r_3, c)$ to $D$, where $b$ and $c$ are the entities such that $(s, r_1, b) \in D$ and $(b, r_2, c) \in D$ (these facts exist because $r_3$ is processed last)
\ENDIF
\ENDFOR
\ENDFOR
\STATE \textbf{return} $D$ \COMMENT{every (subject, relation) pair appears exactly once: $N = n \cdot m$ facts, each a 3-token sequence; the training loss is computed on the object token only}
\end{algorithmic}
\caption{Batch construction for \texttt{PowerLaw}: ConstructBatch($D$, $\alpha$)}
\label{alg:powerlaw-batch}
\begin{algorithmic}[1]
\REQUIRE dataset $D$ of $N$ facts, Zipf exponent $\alpha > 0$. The batch is constructed once and reused for every training iteration.
\STATE assign each fact $i \in D$ a Zipf probability $p_i \propto i^{-\alpha}$ (fixed for the dataset; they do not change across batches)
\STATE $k \gets 1/p_N$, where $p_N$ is the probability of the rarest fact \COMMENT{the rarest fact appears exactly once per batch}
\STATE construct the batch so that each fact $i$ appears $\max(1, \lceil k\, p_i \rceil)$ times
\STATE \textbf{return} the batch \COMMENT{for \texttt{Uniform}, \texttt{Symmetric}, and \texttt{2Hop}, each batch contains each fact exactly once}
\end{algorithmic}
\caption{Per-seed dimension search: MinDim($N$, $\theta$, $s$)}
\label{alg:min-dim}
\begin{algorithmic}[1]
\REQUIRE dataset size $N$, hyperparameters $\theta$, seed $s$; start $d_0 = 32$, cap $d_{\max} = 256$. \textsc{Train}$(d)$ returns pass if $\mathrm{Train}(d, 1, N; \theta)$ with seed $s$ reaches stable 100\% accuracy.
\ENSURE $d^*_s(\theta)$, or $\infty$ if no $d \leq d_{\max}$ passes
\STATE $d \gets d_0$; $d_{\text{ok}} \gets \infty$
\WHILE{$d \geq 1$ \textbf{and} \textsc{Train}$(d)$ passes}
\STATE $d_{\text{ok}} \gets d$; $d \gets \lfloor d/2 \rfloor$ \COMMENT{geometric halving}
\ENDWHILE
\IF{$d_{\text{ok}} = \infty$}
\STATE $d \gets 2 d_0$
\WHILE{$d \leq d_{\max}$ \textbf{and} \textsc{Train}$(d)$ fails}
\STATE $d \gets 2d$ \COMMENT{upward fallback}
\ENDWHILE
\IF{$d > d_{\max}$} \STATE \textbf{return} $\infty$ \ENDIF
\STATE $d_{\text{ok}} \gets d$; $d_{\text{fail}} \gets d/2$
\ELSE
\IF{$d = 0$} \STATE \textbf{return} $d_{\text{ok}}$ \COMMENT{every $d \geq 1$ passed} \ENDIF \STATE $d_{\text{fail}} \gets d$ \COMMENT{first failing dimension}
\ENDIF
\STATE $lo \gets d_{\text{fail}}$ ($d_{\text{fail}} + 1$ after the upward fallback); $hi \gets d_{\text{ok}}$ \COMMENT{both ends included}
\WHILE{$lo \leq hi$}
\STATE $m \gets \lfloor (lo + hi)/2 \rfloor$
\IF{\textsc{Train}$(m)$ passes} \STATE $d_{\text{ok}} \gets m$; $hi \gets m - 1$ \ELSE \STATE $lo \gets m + 1$ \ENDIF
\ENDWHILE
\STATE \textbf{return} $d_{\text{ok}}$
\end{algorithmic}
\caption{Per-seed dataset-size search: MaxSize($d$, $L$, $\theta$, $s$)}
\label{alg:max-size}
\begin{algorithmic}[1]
\REQUIRE width $d$, depth $L$, hyperparameters $\theta$, seed $s$; initial estimate $N_0$; coarse step $S = 1{,}002$ facts; fine step $F = 30$ facts. \textsc{Train}$(N)$ returns pass if $\mathrm{Train}(d, L, N; \theta)$ with seed $s$ reaches stable 100\% accuracy. $\mathrm{round}_S(\cdot)$ rounds to the nearest multiple of $S$, never below $S$; $\mathrm{round}_F(\cdot)$ likewise for $F$.
\ENSURE $N^*_s(\theta)$, or $0$ if no size passes
\IF{\textsc{Train}$(N_0)$ passes}
\STATE $N_{\text{ok}} \gets N_0$; $N \gets \mathrm{round}_S(2 N_0)$
\WHILE{\textsc{Train}$(N)$ passes} \STATE $N_{\text{ok}} \gets N$; $N \gets \mathrm{round}_S(2N)$ \ENDWHILE \COMMENT{doubling}
\STATE $N_{\text{fail}} \gets N$
\ELSE
\STATE $N_{\text{fail}} \gets N_0$; $N \gets \mathrm{round}_S(N_0 / 2)$
\WHILE{\textsc{Train}$(N)$ fails}
\STATE $N_{\text{fail}} \gets N$
\IF{$N/2 < F$} \STATE \textbf{return} $0$ \ENDIF
\STATE $N \gets \mathrm{round}_S(N/2)$ if $N/2 \geq S$, else $\mathrm{round}_F(N/2)$ \COMMENT{halving: coarse grid above $S$, fine grid below}
\ENDWHILE
\STATE $N_{\text{ok}} \gets N$
\ENDIF
\STATE $lo \gets$ the smallest multiple of $F$ above $N_{\text{ok}}$; $hi \gets$ the largest multiple of $F$ below $N_{\text{fail}}$ \COMMENT{binary search on the $F$ grid}
\WHILE{$lo \leq hi$}
\STATE $m \gets$ the multiple of $F$ nearest to $(lo + hi)/2$
\IF{\textsc{Train}$(m)$ passes} \STATE $N_{\text{ok}} \gets m$; $lo \gets m + F$ \ELSE \STATE $hi \gets m - F$ \ENDIF
\ENDWHILE
\STATE \textbf{return} $N_{\text{ok}}$
\end{algorithmic}
References
- [1]Li, M., Kudugunta, S., & Zettlemoyer, L. (2025). (Mis)Fitting Scaling Laws: A Survey of Scaling Law Fitting Techniques in Deep Learning. The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=xI71dsS3o4openreview.net/forum?id=xI71dsS3o4
- [2]Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361
- [3]Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., … Sifre, L. (2022). Training Compute-Optimal Large Language Models. Proceedings of the 36th International Conference on Neural Information Processing Systems.
- [4]Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., & McCandlish, S. (2020). Scaling Laws for Autoregressive Generative Modeling. https://arxiv.org/abs/2010.14701
- [5]Lu, X., Li, X., Cheng, Q., Ding, K., Huang, X., & Qiu, X. (2024). Scaling Laws for Fact Memorization of Large Language Models. https://arxiv.org/abs/2406.15720
- [6]Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9.
- [7]Kajitsuka, T., & Sato, I. (2025). On the Optimal Memorization Capacity of Transformers. Proceedings of the 13th International Conference on Learning Representations (ICLR).
- [8]Kim, J., Kim, M., & Mozafari, B. (2023). Provable Memorization Capacity of Transformers. The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=8JCg5xJCTPRopenreview.net/forum?id=8JCg5xJCTPR
- [9]Kajitsuka, T., & Sato, I. (2024). Are Transformers with One-Layer Self-Attention Universal Approximators?. Proceedings of the 12th International Conference on Learning Representations (ICLR).
- [10]Madden, L., Fox, C., & Thrampoulidis, C. (2025). Next-Token Prediction Capacity: General Upper Bounds and a Lower Bound for Transformers. IEEE Transactions on Information Theory, 71(9), 7134–7148. https://doi.org/10.1109/TIT.2025.3584013
- [11]Mahdavi, S., Liao, R., & Thrampoulidis, C. (2024). Memorization Capacity of Multi-Head Attention in Transformers. Proceedings of the 12th International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2306.02010
- [12]Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, S. J., & Kumar, S. (2020). Are Transformers universal approximators of sequence-to-sequence functions?. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. https://openreview.net/forum?id=ByxRM0Ntvropenreview.net/forum?id=ByxRM0Ntvr
- [13]Johnson, W. B., & Lindenstrauss, J. (1984). Extensions of Lipschitz mappings into a Hilbert space. In Conference in Modern Analysis and Probability (Vol. 26, pp. 189–206). American Mathematical Society.
- [14]Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., et al. (2021). Hopfield Networks is All You Need. International Conference on Learning Representations.
- [15]Lucibello, C., & Mézard, M. (2024). The Exponential Capacity of Dense Associative Memories. Physical Review Letters, 132(7), 077301.
- [16]Geva, M., Schuster, R., Berant, J., & Levy, O. (2021). Transformer Feed-Forward Layers Are Key-Value Memories. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 5484–5495. https://doi.org/10.18653/v1/2021.emnlp-main.446
- [17]Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems, 35, 17359–17372.
- [18]Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022). Knowledge Neurons in Pretrained Transformers. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8493–8502. https://doi.org/10.18653/v1/2022.acl-long.581
- [19]Hernandez, E., Sharma, A. S., Haklay, T., Meng, K., Wattenberg, M., Andreas, J., Belinkov, Y., & Bau, D. (2024). Linearity of Relation Decoding in Transformer Language Models. The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=w7LU2s14kEopenreview.net/forum?id=w7LU2s14kE
- [20]Allen-Zhu, Z., & Li, Y. (2025). Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=FxNNiUgtfaopenreview.net/forum?id=FxNNiUgtfa
- [21]Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., Oprea, A., & Raffel, C. (2021). Extracting Training Data from Large Language Models. 30th USENIX Security Symposium.
- [22]Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., & Zhang, C. (2023). Quantifying Memorization Across Neural Language Models. ICLR.
- [23]Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019). Language Models as Knowledge Bases?. EMNLP-IJCNLP.
- [24]Tirumala, K., Markosyan, A., Zettlemoyer, L., & Aghajanyan, A. (2022). Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models. Advances in Neural Information Processing Systems, 35, 38274–38290.
- [25]Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A Next-generation Hyperparameter Optimization Framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2623–2631. https://doi.org/10.1145/3292500.3330701
- [26]Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., & Yakhnenko, O. (2013). Translating Embeddings for Modeling Multi-relational Data. Advances in Neural Information Processing Systems, 26, 2787–2795.
- [27]Bergstra, J., Bardenet, R., Bengio, Y., & Kégl, B. (2011). Algorithms for Hyper-Parameter Optimization. Advances in Neural Information Processing Systems, 24. https://proceedings.neurips.cc/paper_files/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdfproceedings.neurips.cc/paper_files/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf
- [28]Nichani, E., Lee, J. D., & Bietti, A. (2025). Understanding Factual Recall in Transformers via Associative Memories. The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=hwSmPOAmhkopenreview.net/forum?id=hwSmPOAmhk
- [29]Morris, J. X., Sitawarin, C., Kokhlikyan, N., Guo, C., Suh, G. E., Rush, A. M., Chaudhuri, K., & Mahloujifar, S. (2026). How much can language models memorize?. https://openreview.net/forum?id=NhU661EZ9Copenreview.net/forum?id=NhU661EZ9C
- [30]Chen, X., & Zou, D. (2024). What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks. Forty-first International Conference on Machine Learning. https://openreview.net/forum?id=YNbCbcGyXEopenreview.net/forum?id=YNbCbcGyXE
- [31]Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1412.6980
- [32]Draper, N. R., & Smith, H. (1998). Applied Regression Analysis. Wiley. https://doi.org/10.1002/9781118625590
- [33]Ravfogel, S., Yehudai, G., Bruna, J., & Bietti, A. (2026). Geometric Factual Recall in Transformers. https://arxiv.org/abs/2605.12426




