WorksComputer Science
A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt
Abstract
A black hole retains only a few conserved charges of the matter it formed from. We are inspired to make the analogy and we ask how much an autoregressive transformer retains of its prompt after generating tokens. For two prompts followed by the same continuation, the difference of their hidden states decays approximately as a power law , where is the rate at which a fixed prompt's share of uniform attention shrinks. Beyond the first layers, trained models forget faster (–) than untrained ones of the same architecture (–). For finite-depth pre-LN softmax transformers with bounded attention logits and non-degenerate pre-norm activations, we prove that this offset is for prompt length and depth : the prompt's direct influence is diluted as the context grows, and relay through continuation tokens adds only logarithmic factors. Conditional on the generated text, the prompt's influence on the next token thus vanishes. The bound fixes only the asymptotic shape and is uninformative at the lengths we measure in trained networks; an exactly solvable linear relay model shows that it cannot be improved and accounts for the untrained curves. Cancelling the dilution with a logit bias on the prompt stops the decay early in training, and training suppresses relay by about one billion tokens, before attention sinks form. A toolkit that measures for any Hugging Face causal language model is publicly available.
Introduction
Think of a conversation that runs long with your friends. You open with a remark about the weather: say, Wednesday looks nice, or Thursday looks nice. By the time it has moved through work, a film, a friend’s illness, nobody could tell from the last ten minutes which of the two weekdays you said. The opening line is still in the room, in the sense that nothing was deleted. It has simply stopped mattering. Large language models behave the same way. A system prompt sits at the start of the context, all of its tokens visible to every later query, and yet over a few dialogue turns the persona it sets drifts measurably [10]. An agent whose constraints are given once, at the start of a long trajectory, undergoes the same erosion. Nothing is truncated, and still the model somehow stops listening. We find that the offset between the hidden states of two prompts, measured under a shared continuation, decays close to a power law, and that, counterintuitively, training accelerates this decay: in the layers beyond the first, a trained network forgets its prompt faster than a random-weight network of the same architecture. Whatever information about the prompt persists in the generated text, such as its language, is then plausibly carried by the text itself rather than by a memory in the hidden state; § 8 tests this.
Why should an opening remark fade at all, when it never leaves the context? Physics offers a template. A black hole forgets almost everything about the matter that formed it: an exterior observer can only measure a few charges, and these persist because the exterior field itself can carry them. The corresponding questions for a transformer are then clear.
The core inquiry of this paper is simple to state. Give a language model a prompt, let it write tokens, and compare two prompts that differ in a single word, in phrasing, in language, or in genre. Long after the prompt, can anything inside the model still tell them apart? And if not, why not?
We call the residual amount of the prompt in the hidden state its imprint, and ask three questions about it: how large is the imprint after generated tokens, how fast does it decay, and what properties of the prompt can still be read off from it. Whether an imprint of a given size is readable depends on how it compares with the variation that the continuation itself causes, so we also track a decodability ratio: the imprint measured against the spread of hidden states across different continuations. The two quantities are not the same, but they are linked — a decoder can read the prompt off the state only while the imprint stands out against the continuation — and we show in what follows that in practice they decay at the same rate, which lets a single exponent summarise how quickly a prompt is forgotten.
This paper answers these questions in three steps. We first define a quantity that can be measured: the prompt offset, the difference between the hidden states of two runs that share a continuation but differ in their prompt. We then prove that, under two conditions on the architecture (bounded attention logits and pre-norm activations bounded away from zero), this offset is bounded by a quantity that decays as up to logarithmic factors, and show that the bound is attained by a model that can be solved exactly. Finally, we trace where the prompt’s influence goes as it decays: through the attention path we call relay, which training suppresses by about one billion tokens.
How the theorem and the measurements relate. The theorem fixes only the asymptotic shape of the decay. Its constant is so large for trained networks that the bound is uninformative within any window we measure (Remark 1), so the measured exponents of trained models, –, are not in conflict with the asymptotic rate 1: the bound constrains the decay only for , far beyond our windows. These exponents are measurements, not predictions of the theorem; the solvable model of § 4.5 accounts for the untrained networks.
Contributions. The imprint is measured as a prompt offset, and five results characterise how it decays.
The measured decay. We define the teacher-forced prompt offset and show that it decays approximately as , with logarithmic corrections in untrained networks and close to a pure power law in trained Pythia on coherent text. The spread across continuations stays essentially constant ( except in untrained Qwen), so the rate at which the prompt is forgotten is governed by a single exponent (§ 3).
A theoretical bound. For finite-depth pre-LN transformers with bounded attention logits and non-degenerate pre-norm activations (Assumption 1, verified empirically on the sequences we measure), the offset is ; conditional on the generated text, the prompt’s influence on the next token vanishes at the same rate. For trained networks the constant makes the bound uninformative within our measurement windows. Uniform attention attains the bound asymptotically, and the resulting linear relay model is solved exactly for arbitrary layer gains (§ 4).
The relay mechanism. We decompose the attention contribution to the offset into a direct path (queries reading the prompt) and a relay path (queries reading continuation tokens that
already carry prompt information). The decomposition reproduces the measured within 0.06 at every layer and training stage (§ 6).
From mechanism to training. Compensating the dilution with a logit bias on the prompt removes or reverses forgetting; on Pythia-160m, forgetting sets in between steps 128 and 512 (roughly 0.3–1B training tokens), before attention sinks appear; attention weights alone are a misleading proxy (§ 6.2 and § 7).
An application, with its costs. A mild compensation increases instruction adherence at a small perplexity cost but induces instruction echoing; in multi-turn conversations it shows a consistent trend towards higher adherence, but no more than a constant boost with the same mean bias, so the benefit comes from extra attention to the instruction rather than from the shape of the compensation (§ 9).
Related work
Where this paper sits. Three lines of work are close to ours. One studies the practical failure of long contexts: instruction drift, attention sinks, and the limits of what a model can generate. The second studies information flow through attention, both empirically and as over-squashing. The third studies the statistical structure of long text and the power-law decay of attention. Our question “how much of a prompt remains in the hidden state after generated tokens?” sits between these lines, and our answer borrows from each: the failure mode is from the first, the path decomposition from the second, the power law from the third.
Instruction drift. Li et al. [ [10] show that adherence to a system prompt degrades within a few dialogue rounds, attribute it to attention decay, and propose split-softmax. Our causal experiment agrees on the cause, and our quantitative law explains why the drift occurs: the prompt’s share of attention shrinks as under uniform attention, so the decay is a geometric consequence of a growing context rather than an artefact of any particular dialogue pattern.
Attention sinks and over-mixing. Sinks absorb attention without carrying information [20, 7]; Barbero et al. [ [3] argue that they prevent over-mixing. Our relay path is a form of mixing, and our training-dynamics measurement indicates the ordering in Pythia-160m: forgetting sets in between steps 128 and 512 (roughly 0.3–1B tokens), while the attention mass on the first token stays at or below 0.003 through step 1000 (about 2B tokens), so sinks are not the initial cause of forgetting in this model.
Information flow. Norm-based attention analysis [8] and attention flow [1] motivate our path decomposition; rank collapse [6] is a depth-wise counterpart of the token-wise mixing studied here. Barbero et al. [ [2] show that in decoder-only transformers the sensitivity of the last token to earlier tokens is limited by over-squashing; our theorem makes this loss quantitative for a fixed prompt as generation proceeds, and the linear relay model shows that the rate cannot be improved in the uniform-attention regime.
Signal propagation and mean-field theory. The propagation of a perturbation through depth in randomly initialised networks has been analysed with mean-field methods as a question about correlation lengths and critical initialisations [16, 19]; for transformers, Noci et al. [ [15] study signal propagation and rank collapse at initialisation. In our setting the perturbation is confined to a fixed set of positions (the prompt) and the control parameter is the context length, so the relevant decay is along the token axis, and the exponent plays the role of a decay exponent along that axis. The untrained networks, whose weights carry no statistics of any corpus, are the natural counterpart of those mean-field analyses.
Long-range statistics of text. Mutual information in natural text decays as a power law [11, 5]. A power-law forgetting curve could in principle be inherited from the data. Our random-initialisation baseline rules this out for the existence of the power law: an untrained network, whose weights carry no statistics of any corpus, already forgets as a power law, with exponents that are unchanged (to within 0.01) when the continuation sentences are shuffled. In trained networks the exponent does depend on the continuation text (§ 5.5), but the ordering of the axes and the trained–untrained contrast do not.
Prompt influence on generated text. Marklová et al. [ [12] rescore a fixed generation under different prompts and find that the influence of the prompt fades with accumulated context. Where they measure the effect on the output distribution, we measure it on the hidden state, which lets us track its decay layer by layer and separate the direct path from relay.
Power-law decay of attention. Attention scores to earlier tokens are known to decay roughly as a power law over generation steps, which Koley et al. [ [9] exploit for dynamic sparse attention. Our dilution lemma gives a mechanism for such decay; the path decomposition shows, however, that attention weights are an unreliable proxy for information flow, because sinks attract weight without carrying prompt-specific content, and the decay of the trace in the hidden state is what determines whether the prompt is still readable.
Limits of long generation. Meyer et al. [ [13] bound which output sequences a transformer can generate at all, finding a sharp length threshold beyond which most sequences become inaccessible. Their question is what a model can produce; ours is the complementary one of how the influence of a given prompt on what it produces decays.
Attention fading and length-dependent scaling. Nakanishi [ [14] observes that softmax attention flattens as the context grows (“attention fading”) and proposes Scalable-Softmax, which multiplies the logits by . Our dilution lemma is a quantitative form of the same effect. A logit scaling that grows with context length makes the logits unbounded, so Lemma 1(a) and with it Lemma 2 no longer apply (see the scope paragraph in Appendix § 11); our analysis therefore predicts that such scalings, including Scalable-Softmax, retain prompt information longer.
Setup and measurement
We want to know whether a model that has generated tokens after a prompt still carries that prompt in its hidden state, and how fast this memory fades as grows. Answering it requires three choices: what to compare, so that a difference in hidden states can only come from the prompt (§ 3.1); what to measure, so that the result says whether the prompt can be read off the state (§ 3.2); and how to estimate decay rates reliably (§ 3.3).
What to compare
Let a decoder-only language model receive a prompt of tokens and then generate autoregressively, without further input. After generated tokens, information about can be held in two places. The first is the generated text, which any observer can read: call this the exterior. The second is the residual stream at the current position, which also sees through attention: call this the interior. Only the interior is a memory of the network; the exterior is a record anyone could consult. This distinction is the same one that separates a black hole from its exterior field, and it will return when we ask what survives.
The natural experiment, letting two prompts generate freely and comparing their hidden states, cannot tell these apart: two prompts produce two different texts, and a difference in hidden states may be a memory of the prompt or merely a response to different text. We therefore fix the text: both prompts are followed by the same continuation , written independently of either, and we compare the hidden states along it. With the continuation held fixed, any remaining difference between the two runs must come from the prompt. This is also the quantity bounded by Theorem 1, and the one that must vanish for a no-hair statement to hold.
The measurements in this section and in § 4–§ 9 therefore use what we call teacher-forcing: the continuation is written by a human, not by the model. This is a measurement device, not the failure mode itself. The failure mode is the one that occurs in free generation, when the continuation is produced by the model and the prompt’s influence on the next token is expected to travel mainly through the text it has caused (Corollary 1 and § 8 examine what this does and does not imply). § 8 returns to that setting and asks what survives. In the theoretical sections that follow, we will replace the time index with the distance from the end of the prompt, so that .
What to measure
The most direct imprint of the prompt is the difference between the two hidden states. Fix a layer , and let be the residual stream after block (before normalisation) at time along , where is the hidden dimension and denotes concatenation, so that is the prompt followed by the continuation. To reduce token-level fluctuations we average it over a fixed window of consecutive continuation tokens and assign it the time of the window’s centre.
Definition 1 (Prompt offset). For two prompts and a shared continuation , the prompt offset at layer and time is
We write for the -th coordinate of , and for the -th coordinate of .
A small offset is not by itself evidence of forgetting: what matters is whether it stands out against the variation that the continuation itself causes. If the offset is the same for every continuation, the prompt can be read off the state whatever text follows; if it is swamped by the differences between continuations, it cannot. We therefore split the offset into a part common to all continuations, the signal, and compare it with the spread of hidden states across continuations, the noise. Both are measured in units of , the standard deviation of across continuations in the first window, pooled over both prompts, so that no coordinate dominates because of its scale. We write for the number of continuations per pair, and and for the mean and variance across them.
Definition 2 (Signal, noise and decay exponents). The signal and the noise at layer are
and we summarise how fast each decays by a power-law exponent: and .
The exponent we care about is : it measures how quickly the signal of the prompt fades. The exponent measures how quickly the noise fades as well, and we track it because a prompt is only decodable while the signal stands out against the noise: decodability scales as . We find almost everywhere (§ 5), so the rate at which the prompt is forgotten is the single number .
Estimating decay rates
Sampling times. Decay laws are best seen on a logarithmic time axis, so we space measurements by octaves. For Pythia on real-corpus pairs, windows end at ; for Qwen, at . For the hand-written scenes, which use Pythia and whose continuations are at least 1920 tokens long, the same octave spacing runs to , short of Pythia’s 2048-token context. Each window covers the 16 tokens ending at these points. We label it by its centre, not by its end: end labelling biases the fitted exponent upwards when a window spans a whole octave, as it does in the first octaves. For offsets of the form , the bias is up to 0.19 locally and 0.07 globally with end labels, against less than 0.03 and 0.01 with centre labels; on our data the global exponents drop by 0.05–0.10 when going from end to centre labels.
Signal. The signal is the part of the offset common to all continuations, so we estimate it by averaging over continuations and squaring. At late times this average is small, and the naive estimate is then dominated by its sampling bias , which would make the decay look slower than it is. We therefore use the unbiased estimator
which corrects for this bias by averaging over pairs of continuations rather than squaring the mean. We also report the exponent of the mean squared offset . Unlike , which averages the offset before squaring and therefore cancels offsets whose direction varies across continuations, averages the squared offset, so it captures both the common part and the variation in direction. Comparing the two tells us how much of the decay is decorrelation rather than shrinking norm (Remark 2).
We use continuations per pair for real-corpus pairs and per scene for hand-written scenes. All real-corpus exponents of § 5 use , and so do the path decomposition (§ 6), the compensation experiment (§ 7) and the trade-off (§ 9). The hand-written scenes use the plug-in estimate, because the correction requires storing pairwise products of offsets, which the earlier runs did not record; their late-time exponents are correspondingly biased low.
Exponents. The signal is first averaged over prompt pairs (and, where stated, over axes); the global exponent is the least-squares slope of against over all measurement points, with 95% bootstrap intervals over scenes or prompt pairs. A single exponent can hide a decay that is not a pure power law, so we also compute the local exponent , the same slope between consecutive points. A pure power law has constant ; dilution with logarithmic corrections gives an that rises towards 1 (§ 4). is estimated in the same way from .
Prompts and continuations
Our picture predicts that the decay exponent depends on the architecture and on the length of the prompt, but not on how different the two prompts are: a larger difference should raise the amplitude of the offset, not change how fast it decays. We test this prediction by varying the kind and size of the difference, on two sets of prompts.
Hand-written scenes. Twenty scenes, each with a base prompt and four variants, from the smallest to the largest change: one content word changed (detail), a paraphrase (para), a newswire rewrite (genre) and a translation (lang). Continuations are human-written: sixteen passages per scene from the WikiText-103 validation set, at least 1920 tokens long, shared by all five prompts of the scene. The full list is in Appendix § 13.
Real-corpus pairs. Hand-written prompts may share stylistic regularities, so we repeat the measurement on pairs drawn from public corpora, one hundred per axis: a WikiText sentence and the same sentence with one word replaced (detail), MRPC paraphrase pairs (para), WMT14 English–French pairs (lang), and two unrelated WikiText sentences (unrelated), the largest possible difference. Continuations are passages from the WikiText-103 validation set, eight per pair, shared by both prompts of the pair.
Shared first token. Trained models often concentrate attention on the first token, an attention sink [20]. When two prompts begin with different tokens, the sink itself differs, and its contribution need not dilute. Where stated, we prepend a common token (<|endoftext|>) to both prompts to remove this difference.
Models
Two questions guide the choice of models: when during training does the forgetting rate change, and does the picture depend on architectural details? Pythia [4] answers the first, since it provides training checkpoints of a fixed architecture (LayerNorm, parallel residual): we use Pythia-160m at eight checkpoints from initialisation to the end of training (steps 0, 128, 256, 512, 1000, 4000, 16000, 143000), and Pythia-410m and Pythia-1b at initialisation and after training, to separate the effects of training, width and depth. Qwen2.5 [18] addresses the second with a different architecture (RMSNorm, sequential residual, grouped-query attention): we use Qwen2.5-0.5B, trained and at random initialisation (the library’s default scheme: normal weights with standard deviation 0.02, seed 0; for Pythia, step 0 is the released initial checkpoint), and the trained Qwen2.5-1.5B. All computations run in float32 on a single Apple-silicon laptop.
A conditional no-hair theorem
Setting and notation
Architecture. We work with -layer pre-LN decoders: causal attention, residual connections, and rotary or no positional encoding inside the logits (as in Pythia and Qwen). The pre-LN placement matters here: each block normalises its input inside the residual branch, before attention and before the MLP, so the residual stream itself is never normalised. This is the sequential form, followed by the MLP update, as in Qwen, or the parallel form, where both updates are computed from and added, as in Pythia. The scope covers most current decoders, with LayerNorm or RMSNorm (see below); it excludes post-LN blocks, linear attention and recurrent state.
Notation. We use the same setup as § 3, with one change of notation for the proof. Two prompts are followed by the same continuation , and denote the residual streams at the -th continuation token after blocks. This is the quantity of Definition 1, with the time index replaced by the distance from the end of the prompt. Write . Token embeddings of the continuation coincide, so .
The two prompts have lengths and . From here on, denotes the larger of the two, so that any bound in terms of applies to both. We index continuation positions by their distance from the end of the prompt, so the -th continuation query attends to at most keys. With rotary encodings, positions enter only the logits, through relative offsets, so is allowed; with absolute position embeddings we take .
Finally, we write the normalisation as . Here is the all-ones vector, the identity, and the projection onto the subspace orthogonal to it, so that with the mean of . Thus LN first removes the mean and then rescales to length . The gain and bias of the layer are absorbed into the adjacent linear maps and do not appear separately. RMSNorm is the special case , in which the mean is not removed.
The proof needs the normalisation layers to be well conditioned:
Assumption 1 (Non-degenerate pre-norm activations). There is such that at every position and in both runs, whenever is the input of blocks , the input of the final norm, or, in a sequential block, the post-attention residual that the MLP reads.
The assumption is what gives the normalisation Lipschitz: the constant in Lemma 1 is , and it degenerates if can vanish. In a pre-LN residual stream it cannot do so in a harmful way. The stream is never normalised, and each block adds its update to it, so the norm grows with depth rather than collapsing; the only vectors that come close to the degenerate case are the token embeddings, which are excluded from the assumption. On the sequences we measured (up to 1024 continuation tokens), the assumption holds at block inputs with a comfortable margin: (Pythia-160m, initialisation), (160m, trained), (1b, trained) and (Qwen2.5-0.5B), against medians of order – (Appendix § 12). Qwen’s blocks are sequential, so the input of the post-attention norm enters as well; its minimum is in the first block of the trained model and or more in all later blocks. For Qwen2.5-0.5B the constant of Assumption 1 is therefore , set by this first post-attention norm: the assumption holds, with a small margin in that block only. These checks are empirical: they cover the measured sequences and lengths and do not establish the assumption for arbitrary continuations or contexts. All other constants follow from the weights.
Dilution
Theorem 1 rests on one geometric fact. A prompt of length occupies a fixed number of positions, while a query at distance sees keys; as grows, the prompt’s share of the query’s attention can only shrink. This is what we call dilution. We write for the logit of query against key , so that the attention weights are . We first collect the constants that the architecture imposes (Lemma 1), then make dilution precise (Lemma 2), and finally turn it into a recursion across layers (Lemma 3).
Lemma 1 (Consequences of the architecture). Let denote spectral norms of the weight matrices and the head dimension.
(a) Bounded logits, unconditionally: (plus bias terms), since and rotary matrices are orthogonal.
(b) LayerNorm is Lipschitz away from zero: under Assumption 1, .
(c) Hence value-output vectors are bounded by and -Lipschitz with ; logits are -Lipschitz in each argument with ; the MLP update is -Lipschitz with , where is the MLP nonlinearity, its weight matrices and its Lipschitz constant.
Proof. (a) The logit is . By Cauchy–Schwarz, we have: . Since and rotary matrices are orthogonal, this gives .
(b) is an orthogonal projection, hence non-expansive, so . For ,
using in the second step and in the last.
(c) The value-output vector has norm at most , and is -Lipschitz with , by (b). The logit is -Lipschitz in each argument with , again by (b). The MLP update is -Lipschitz with . The constants are summed over heads in Appendix § 11.
Lemma 2 (Dilution). Let denote the attention weights of a head. For every query and every set of keys, , where is the number of keys visible to the query. In particular, a continuation query at distance pays at most to the prompt, in either run.
Proof. By definition, and a prompt of length gives .
Lemma 3 (One-layer recursion). Throughout, are the multi-head constants of Appendix § 11, already summed over heads. For ,
with , and for a sequential block; for a parallel block the same bound holds with , which is no larger.
Proof. We give the main steps; the full proof, with the constants per head and the alignment of positions, is in Appendix § 11. Fix a head and a continuation query , and let be the value-output vectors of the two runs. The prompts may have different lengths, so we separate the prompt from the continuation, whose keys are the same in both runs and sit at the same relative positions. Let and for the continuation keys , and write the head output as , with and ; both have norm at most , and similarly for the primed run. Then
Prompt terms. By Lemma 2, ; hence also , and the last three terms are at most .
Continuation term. The weights are a softmax over the same keys in both runs, with logits in , so by Lemma 4 in Appendix § 11 we have , also along the linear interpolation between the logits of the two runs. Write
The first sum (i) is at most . For the second sum (ii), the derivative of softmax gives (Lemma 4 in Appendix § 11), and , so (ii) is at most .
Since , summing over heads (absorbed in the constants ) gives
Residual and MLP. The residual adds . In a sequential block the MLP reads the post-attention residual, whose difference is at most , and multiplies it by at most , so . In a parallel block the MLP reads and the resulting bound is smaller. Collecting terms gives (2) with the constants stated, for the sequential case; the parallel case follows from the smaller above. □
In words, (2) says that the offset at layer and position is bounded by three contributions: the offset at the same position one layer earlier, the average offset over all earlier positions (the relay term), and the prompt’s direct contribution, which is . Only the last term shrinks with ; the relay term is what carries prompt information through depth.
The conditional no-hair theorem
Now we give the following theorem where the no-hair analogy becomes precise.
Theorem 1 (Conditional no-hair theorem). Under Assumption 1, for every ,
where we recall that is the norm of the prompt offset at the final layer after generated tokens, that bounds the length of the two prompts, and that is the depth. The theorem bounds this offset by a constant that depends on the architecture but not on or , and a decay in that is at worst up to logarithmic factors.
Proof. We prove by induction on that , with .
For , , and (2) gives .
For , put . The function is decreasing on , since and . Hence
With and for the other two terms, (2) yields
Unrolling gives . Since , each term in the sum is at least 1, so the sum is at most and . Since , the asymptotic form is .
Remark 1 (What the constant does and does not control). The constant is large and grows with depth. can grow exponentially with depth, and the bound is stated in terms of , which is large. Lemma 2 only uses the spread of the logits seen by each query, so can be replaced by ; but in trained networks with attention sinks is large, and the gain is limited. Intuitively, relay acts as a memory buffer: the prompt’s information is carried forward by the generated text itself, but it is subject to repeated attenuation and mixing at each layer.
What Theorem 1 fixes is the asymptotic shape, not a numerical prediction. The bound is informative only once , which for trained networks lies beyond any window we measure; the fitted exponents of trained models, which are below 1, are therefore neither in conflict with the theorem nor predicted by it. What the theorem and the solvable model establish together is an asymptotic statement: no decay is slower than up to at most powers of , one per relay round, and for uniform attention this cannot be improved. The shift inside the logarithm only matters for small . For the finite windows of the untrained networks, the relevant description is the exactly solvable linear relay (Proposition 2); for trained networks we have no corresponding quantitative model, and their exponents are measurements (§ 5).
How large can be is observed in practice. For Pythia-160m we measure, over late queries of 400 prompt–continuation pairs, the spread of each query’s logits, with . At initialisation its median is 1.9–2.3 (maximum 4.3), so is of order and the dilution regime is reached within our windows. After training the median is 17–18 in layers 1–3, so ; in the deep layers the spread reaches the float32 limit (, where the smallest attention weights underflow), so there it is only bounded from below. In all these layers the bound is not informative at any accessible .
Remark 2 (From to the measured signal). Since , the signal of § 3 obeys , so Theorem 1 bounds it as well (averaging over a window only moves within the window). The converse fails: offsets whose direction depends on the continuation partly cancel in , so can decay faster than . We therefore also report the exponent of (§ 3.3), which is the window-averaged up to the coordinate scaling.
The exterior carries the hair
Corollary 1 (The exterior carries the hair). Let be the next-token distribution given the history (the prefix of the continuation up to time ), the unembedding, and , where is the operator norm from to and TV the total-variation distance. For every history ,
and for every the joint laws of the next tokens given differ in total variation by at most .
Proof. The final LayerNorm is Lipschitz by Lemma 1(b), and softmax is 1-Lipschitz from logits to total variation; Theorem 1 holds for every continuation, hence for every history. For the joint law, couple the two runs token by token, maximally at each step given a common history: the probability that the coupled sequences first differ at step is at most . Finally is non-increasing in , by the monotonicity argument in the proof of Theorem 1.
The corollary says what happens over a short window. Over a window short compared with , the next token is determined, up to , by the text generated so far, whichever prompt produced it.
It says nothing about long horizons. The bound applies step by step; the sum diverges, so it does not exclude a cumulative influence of the prompt that bypasses the text. Nor does it show that such an influence exists.
Whether the prompt’s influence at late times is in fact carried by the text is therefore an empirical statement, which § 8 tests, not a consequence of Theorem 1.
This is the sense in which the no-hair theorem holds. In general relativity the quantities that survive are those measurable at infinity, carried by the exterior field; here, step by step, the channel through which a prompt influences what comes next is the exterior record. § 8 is consistent with this picture: the language of the prompt remains decodable from free generations, but under a shared continuation its imprint in the hidden state still decays, at a rate comparable to that of a one-word change.
An exactly solvable model
Theorem 1 gives an upper bound, and the bound contains logarithmic powers. Are they an artefact of the proof, or a feature of the architecture? To answer this, we turn to a simplified model that can be solved exactly, which is, the linear relay model. In this model, every head attends uniformly, each block multiplies the offset by a constant gain, and the prompt enters uniformly. The model is solvable in closed form, and its solution has the same shape as the bound, which shows that the logarithmic powers cannot be removed. It also gives a parameter-free prediction for untrained networks, which we compare with the measured curves at the end of this subsection.
The model. Consider the first-order dynamics of a small offset around a fixed background: every head attends uniformly (, the regime of a randomly initialised network, whose logits are close to zero), the value-output path of block multiplies the offset by a gain along a fixed direction , and normalisation and MLPs are replaced by their Jacobians, which we absorb into . The two prompts differ by at every prompt position, and we keep this prompt offset fixed through depth (if it grows, every offset below grows, so lower bounds remain valid). The offset at the -th continuation token after blocks is then with
The first result says below that the bound from Theorem 1 is attained in this regime: with equal gains, the offset decays with exactly logarithmic powers, so the exponent cannot be lowered.
Proposition 1 (Uniform attention attains the bound). In the linear relay model with equal gains , for all ,
Hence as : the power in Theorem 1 cannot be lowered.
Proof. Since , (4) is of the form (2) with and , and the induction in the proof of Theorem 1 applies with replaced by . The lower bound is the term of the exact solution (6) below, with and ; all other terms are non-negative.
Layer-dependent gains. The model can also be solved exactly for arbitrary gains, which turns the depth dependence of untrained networks into a prediction.
Proposition 2 (Exact solution with layer gains). Let . We write and for the elementary and the complete homogeneous symmetric polynomials in and in (, ). In the linear relay model, for all ,
Moreover, with , for every
so the continuum form holds up to corrections of order times terms of lower degree in . For equal gains and the leading power is .
Proof. We first derive (6), then the bound (7).
Deriving (6). Put
so that and . Since , (4) reads
Applying and adding 1 gives , hence
By induction on , , and . Dividing by gives (6).
Deriving (7). Let and . Over multisets of size , and with . The difference is non-negative and involves only multisets with a repeated index , whose monomials appear in , hence
Comparing with gives , so
Combining the two estimates gives (7).
Corollary 2 (Small gains). Let and . Then . If , and (the three error terms of Corollary 3; note that alone does not suffice once is large), then
with a modified Bessel function, and the local exponent is
Depth enters only through the accumulated gain ; Proposition 3 and Corollary 3 in Appendix § 11 bound the error of (8) explicitly. Moreover (a) as , and (b) at fixed , does not increase as grows, hence decreases with depth. Over a finite window the measured exponent of an untrained network is therefore below 1, lower in deeper layers, and creeps up as the window is extended.
Proof. For the bound on , write over multisets of size and as the sum over those without repeated index; a multiset with a repeated index contributes at most times a term of . Inserting and (Proposition 2) into (6) gives (8); the terms of the series peak at , so the truncation at is immaterial. Since , , and the expansion follows from . For (a), . For (b), write with , so with . Now is the mean of under weights proportional to , and its derivative in is the corresponding variance, so is non-decreasing and is non-decreasing in .
Comparison with experiment. The model makes two parameter-free predictions for the first blocks. In layer 1, , so the local exponent is exactly . In layer 2, , so is linear in with slope exactly 1, whatever the gains. Untrained Pythia is consistent with both (§ 5: the layer-1 law with close to the prompt length, fitted power 1.07–1.33 in layer 2); untrained Qwen is consistent with neither, so its first blocks are not in the linear-relay regime. In deeper layers the effective power seen over a finite window depends on the gains, not only on the layer index, so the coincidence in layers 2–4 of Pythia should not be read as a general law.
Small-gain fit. Fitting (9) without the factor and the term to untrained Pythia-1b on real-corpus pairs (, unbiased signal, centre-labelled windows) gives in layers 3–15, with in layers 3, 4, 8, 12, 15 and increments of roughly per layer, i.e. . Repeating the fit in all fifteen layers gives (, largest residual 0.05) and per-layer increments ; the fitted logarithmic power follows (). Over our windows, however, (and in layer 3), outside the range of (9), so these values are effective parameters rather than estimates of the accumulated gain.
Exact-solution fit. The exact solution (6) turns them into estimates. We fit it to the same curves layer by layer, with the exact recursion and one free amplitude per layer (the coordinate normalisation of is layer dependent), so that each layer is fitted on the shape of its curve only. Layer 1 fixes whatever the gains; it gives , against a mean prompt length of 25.7 tokens, with and a largest deviation of the local exponent of 0.026. Since depends on the gains only through the symmetric functions , layer 2 constrains only ; we fit jointly on layers 2–3 and then, from layer 4 on, one new gain per layer with the earlier ones fixed, so that a single parameter predicts the whole curve of each layer. All fifteen curves are reproduced with ; the largest deviation between measured and predicted local exponents grows from 0.05 in layer 2 to 0.08 in layer 15, so the one-gain-per-layer model captures the curves well but not perfectly in the deep layers.
Identifying the first two gains. Layer 2 alone constrains and only through ; together with layer 3 they are determined as an unordered pair, since every is symmetric in , so and cannot be told apart. The fit returns equal values, .
A test of the symmetry. This is not an artefact of the starting point: restarting the joint fit of from six widely different initial values, including strongly asymmetric ones (e.g. and ), reaches the same minimum at the same cost each time, and on synthetic curves generated with an unequal pair the same fit, started on the symmetric diagonal, leaves it and recovers the unequal values, so the equality is a property of the data and not of the optimiser.
The remaining gains. The gains then decrease as (layers 4–15), to 0.02 in layer 15, with $G_{15}=2.18; a joint fit of all gains gives the same values to within 0.09. The first gains are not small, which is why the effective of the small-gain fit differ from these estimates.
Remark 3 (Running exponent). For and , Eq. (9) gives with . The approach to the asymptotic exponent 1 is therefore logarithmically slow, with leading correction whose amplitude grows with depth. In the language of critical phenomena, is a running exponent, and the accumulated gain , which our fits give as growing roughly like with the layer index (an effective parameter, see above), acts as a depth-dependent coupling that sets the crossover scale below which the decay is not yet in its asymptotic regime. Over our windows, , so the untrained networks stay in this crossover regime.
Figure 1. Decay of the prompt offset under a shared continuation (Pythia-160m, real-corpus pairs, unbiased signal normalised to the first window; bands: 95% bootstrap over pairs). Dotted: layer 1; solid: last layer. Training makes the deep-layer offset decay much faster; the dashed line is .
Remark 4 (Physics analogy). Lemma 2 is a Gauss-law statement: a fixed charge (here the prompt, a fixed number of positions) spread over a growing volume (the visible keys), so that its share of any query’s attention falls as . Relay plays the role of back-scattering off the background curvature, which in general relativity produces late-time power-law tails [17]: a perturbation that has left the region of direct view returns through intermediate positions, and in the linear relay model each relay round contributes one factor of . The analogy is structural, not quantitative. We make no claim that the exponent or the amplitude of the tail corresponds to a quantity of the black-hole problem, and the identification of a prompt with a conserved charge is only suggestive, since a transformer has no conservation law that protects any feature of its prompt. Trained networks scatter little; untrained ones scatter a lot. We use the analogy as a guide only, and no result of this paper depends on it.
Forgetting is close to a power law
We now turn to the measurement. The theory of § 4 predicts the shape of the decay but not its rate; the rate is the exponent defined in § 3, and this section reports the corresponding results.
Hand-written scenes
Table 1 gives the exponent for Pythia-160m and 410m at initialisation and after training. Five patterns are visible, and they carry over to the real-corpus pairs of § 5.2.
| L1 | L2 | L3 | L6 | L9 | L11 | |
| Pythia-160m (12 layers) | ||||||
| init (step 0) | 0.82 [.81,.82] | 0.65 [.65,.65] | 0.57 [.57,.58] | 0.47 [.47,.47] | 0.42 [.42,.43] | 0.40 [.40,.41] |
| trained (143k) | 1.05 [1.04,1.07] | 0.92 [.90,.93] | 0.63 [.57,.71] | 0.59 [.55,.62] | 0.64 [.62,.67] | 0.66 [.64,.68] |
| Pythia-410m (24 layers) | ||||||
| L1 | L2 | L6 | L12 | L18 | L23 | |
| init (step 0) | 0.82 [.81,.82] | 0.65 [.65,.66] | 0.47 [.47,.48] | 0.40 [.39,.40] | 0.36 [.36,.37] | 0.35 [.34,.35] |
| trained (143k) | 1.01 [1.00,1.03] | 0.89 [.88,.90] | 0.70 [.69,.72] | 0.57 [.56,.58] | 0.60 [.59,.61] | 0.60 [.59,.61] |
Table 1. Decay exponent of the prompt offset, hand-written scenes (20 scenes, 16 continuations each), , windows labelled by their centre, plug-in signal (not the unbiased estimator of § 3.3: late-time exponents are biased low, so the unbiased values of Table 3 and Table 4 are the reference), 95% bootstrap intervals over scenes. in all cells.
Noise does not decay. for every Pythia model, layer and checkpoint (untrained Qwen is the exception, § 5.4), so all the change in decodability comes from . Since is the spread of hidden states across continuations, this is what stationarity of the continuation text implies; we report it as a check rather than as a finding.
Layer 1 is pure dilution. After the first attention block the continuation tokens are still identical in the two runs, so no relay is possible and only the direct path exists. In untrained
models the local exponent follows the dilution law of a uniform head, between consecutive points, with a single fitted to within 0.03 at every octave; the fitted value tracks the prompt length ( for the hand-written scenes, for the longer real-corpus prompts). Trained models give in layer 1 (1.05 and 1.01 here, 0.94–0.97 on real-corpus pairs).
Relay enters after layer 2 in trained models. Layers 1–2 stay at 0.9–1.05; deeper layers settle at 0.57–0.70 (both sizes) and are roughly flat with depth.
At initialisation, depends on the layer index, not on width. At matched layers the two untrained models agree exactly (L1 0.82/0.82, L2 0.65/0.65, L6 0.47/0.47). In layer 1 this is automatic: pure dilution depends only on the prompt length, which is the same for both models (same tokenizer). The agreement in deeper layers is informative: in the linear relay model (Proposition 2) it means that the gains at a given layer index do not depend on the width. This is not because the initial weights match. The released step-0 checkpoints follow small-init for the input projections (standard deviation : 0.0228, 0.0198, 0.0140 for 160m, 410m, 1b) and wang-init for the output projections (: 0.0060, 0.0026, 0.0028), so the absolute size of each block’s write differs between the three models and depends on their total depth. A scale-free mechanism is needed. A natural candidate is that what matters is the write of a block relative to the residual stream it adds to, which at initialisation is itself made of the earlier writes: the pre-norm activations grow as with depth (Pythia-160m: to within 0.01 from block 1 to block 11), so the relative write of block , and with it the gain, depends on but not on the absolute scale.
Content-agnostic vs. content-dependent. At initialisation the axes have the same to within 0.04; after training they separate, in the same order for both sizes (deep layers on hand-written scenes: paraphrase 0.50–0.57, detail 0.65–0.75).
Local exponents rise towards 1. If , the local exponent is for some reference time . Then is linear in with slope , and we use as a descriptive effective power. For untrained networks rises monotonically with in every layer, as Corollary 2 predicts: in Pythia-160m, from 0.47 to 0.99 in layer 1, and from 0.21 to 0.53 in layer 11. The linear form fits with (Table 2). In layer 2 the fitted power is close to 1 (1.07–1.08), the value the linear relay model gives for any gains (§ 4.5, *Comparison with experiment). In layer 3 it is close to 2 (1.96), and from layer 6 on it grows much more slowly than (–2.2 in both models), as expected when the gains are small. These fits ignore the factor of (9); the exact solution (6) is the appropriate model for the full curves. The small-gain form of Corollary 2 fits the same local exponents at least as well, and we compare the two on the real-corpus pairs, where the unbiased signal is available.
| layer | |||||
| Pythia-160m, hand-written scenes | 2 | 3 | 6 | 9 | 11 |
| fitted | 1.07 | 1.96 | 3.59 | 4.65 | 5.10 |
| relay rounds | 1 | 2 | 5 | 8 | 10 |
| 0.985 | 0.996 | 0.995 | 0.993 | 0.996 | |
| Pythia-410m, hand-written scenes | |||||
| layer | 2 | 6 | 12 | 18 | 23 |
| fitted | 1.08 | 3.54 | 5.26 | 6.21 | 6.76 |
| relay rounds | 1 | 5 | 11 | 17 | 22 |
| 0.988 | 0.996 | 0.995 | 0.997 | 0.996 | |
| Pythia-1b, real-corpus pairs (unbiased signal) | |||||
| layer | 2 | 3 | 4 | 8 | 15 |
| fitted | 1.33 | 2.14 | 2.78 | 4.24 | 5.52 |
| 0.969 | 0.986 | 0.994 | 0.996 | 0.997 | |
| fitted (Cor. 2) | 0.17 | 0.36 | 0.52 | 0.95 | 1.35 |
| 0.966 | 0.990 | 0.999 | 0.998 | 0.996 |
Table 2. Effective logarithmic power and accumulated gain , fitted from the local exponents of the untrained models (windows labelled by their centre). The are effective parameters of the small-gain form; the estimates from the exact solution, which differ because the first gains are not small, are in § 4.5.
Real-corpus prompt pairs
To rule out an artefact of hand-written scenes we repeat the measurement on 100 pairs per axis drawn from real corpora: a WikiText-103 test sentence vs. the same sentence with one content word swapped (detail); MRPC paraphrase pairs (para); WMT14 English sentences vs. their French translations (lang); and two random WikiText sentences (unrelated), the largest possible difference. Continuations are WikiText-103 validation chunks, , 8 continuations per pair, unbiased signal; intervals are over prompt pairs (Table 3).
| layer | init (step 0) | trained (step 143000) | ||||||
| detail | para | lang | unrel. | detail | para | lang | unrel. | |
| L1 | 0.75 | 0.74 | 0.69 | 0.73 | 0.92 | 0.98 | 0.99 | 0.97 |
| L2 | 0.59 | 0.58 | 0.54 | 0.57 | 0.80 | 0.81 | 0.80 | 0.82 |
| L3 | 0.52 | 0.51 | 0.47 | 0.50 | 0.67 | 0.52 | 0.56 | 0.67 |
| L6 | 0.42 | 0.42 | 0.39 | 0.41 | 0.61 | 0.54 | 0.54 | 0.63 |
| L9 | 0.39 | 0.38 | 0.35 | 0.37 | 0.67 | 0.58 | 0.61 | 0.67 |
| L11 | 0.37 | 0.37 | 0.33 | 0.35 | 0.69 | 0.59 | 0.65 | 0.70 |
Table 3. on real-corpus prompt pairs, Pythia-160m, 100 pairs per axis, unbiased signal, windows labelled by their centre. 95% intervals over pairs have half-width at initialisation and after training; .
Every finding carries over. At initialisation falls with depth and the local exponent rises monotonically with in every layer; the fitted powers are in layers 2 and 3 (), then ; the layer-1 local exponent follows the dilution law with , close to the length of these prompts (26–32 tokens on average), against for the shorter hand-written ones: the prompt’s share of the context scales with its length. After training, layers 1–2 stay at 0.8–1 and deeper layers at 0.52–0.70, with paraphrase and language forgotten most slowly.
Two descriptions of the untrained curves. Corollary 2 predicts that is linear in with slope . On the real-corpus pairs this holds with from layer 3 on (Pythia-160m: in layers 3, 6, 9, 11, ; Pythia-1b in Table 2), at least as well as the pure logarithmic form. In Pythia-1b the increments of per layer are about , i.e. , as expected if the gain of a layer shrinks as the residual stream it writes into grows; these are effective parameters rather than estimates of the accumulated gain. Over our windows the two descriptions cannot be told apart.
Figure 2. Local exponents between consecutive octaves. Left: every layer of untrained Pythia-1b; the exponent rises towards 1 in every layer and decreases with depth; dashed: the small-gain form of Corollary 2, fitted as (the offset absorbs the finite- corrections, § 4.5). Right: the trained model, where no systematic trend remains in the deep layers.
Scale: Pythia-1b. Table 4 repeats the real-corpus measurement on Pythia-1b (16 layers, width 2048) with the same 100 pairs per axis. At initialisation the exponents coincide with those of Pythia-160m at matched layers to two decimals (L1–L3: 0.73, 0.57, 0.50 for both), the local exponent rises monotonically in every layer, and the fitted logarithmic powers are , 2.14, 2.78 in layers 2, 3, 4, close to (), with –2.1 beyond. After training, layer 1 is at 0.87, layer 2 at 0.79, and deep layers settle at 0.58–0.61; paraphrase is again forgotten most slowly (0.52 in the last layer) and the unrelated axis again matches the one-word detail axis. Across a 6× increase in parameters we see no qualitative change.
| model | L1 | L2 | L3 | last | ||||
| 160m | init | 0.73 | 0.57 | 0.50 | L6 0.41 | L9 0.37 | L11 0.36 | |
| 160m | trained | 0.97 | 0.81 | 0.61 | L6 0.58 | L9 0.63 | L11 0.66 | |
| 1b | init | 0.73 | 0.57 | 0.50 | L4 0.46 | L8 0.38 | L12 0.35 | L15 0.33 |
| 1b | trained | 0.87 | 0.79 | 0.71 | L4 0.69 | L8 0.61 | L12 0.61 | L15 0.58 |
| Qwen-0.5B | init | 0.63 | 0.45 | 0.37 | L6 0.28 | L12 0.22 | L18 0.20 | L23 0.18 |
| Qwen-0.5B | trained | 0.74 | 0.69 | 0.69 | L6 0.62 | L12 0.55 | L18 0.54 | L23 0.58 |
| Qwen-1.5B | trained | 0.95 | 0.81 | 0.61 | L7 0.63 | L14 0.62 | L21 0.56 | L27 0.55 |
Table 4. on real-corpus prompt pairs (mean over the four axes; per-axis 95% intervals have half-width ), unbiased signal, windows labelled by their centre. Pythia-160m and Pythia-1b: ; Qwen2.5-0.5B and 1.5B: . At initialisation the two Pythia sizes agree at matched layers.
Figure 3. against the layer index (mean over the four axes, real-corpus pairs). Untrained models (orange) depend on the layer index alone: the two Pythia sizes coincide; trained models (green) are flat beyond the first layers.
A second model family: Qwen2.5. Qwen2.5 differs from Pythia in its normalisation (RMSNorm), its residual structure (sequential) and its attention (grouped-query); we measure it on the real-corpus pairs with (Table 4). Trained Qwen2.5-0.5B gives 0.74 in layer 1, 0.69 in layers 2–3 and 0.54–0.62 beyond; Qwen2.5-1.5B gives 0.95 in layer 1, 0.81 in layer 2 and 0.55–0.63 beyond, the same profile as the trained Pythia models. At random initialisation falls from 0.63 (layer 1) to 0.18 (layer 23), and the local exponent rises monotonically with in every layer. In the untrained model a one-character change of the prompt remains linearly decodable after 512 tokens; in the trained model it does not. § 5.4 discusses where Qwen departs from the linear relay picture.
The unrelated axis gives the sharpest test of the geometric picture: two completely different sentences are forgotten at the same rate as a one-word substitution (Pythia-160m, layer 11: 0.35 vs. 0.37 at initialisation, 0.70 vs. 0.69 after training). The size of the difference sets the amplitude of the offset, not its decay exponent.
Signal versus mean squared offset
In untrained Pythia the exponents of and of the mean squared offset coincide to within 0.01 in every layer: the offset points in the same direction whatever the continuation. After training decays more slowly in the deep layers (Pythia-160m: –0.57 against –0.67 in layers 6–11; Qwen2.5-0.5B: 0.52–0.58 against 0.55–0.62), and in untrained Qwen much more slowly (0.10–0.21 against 0.18–0.26 in layers 6–23). Part of the decay of in these models is therefore the decorrelation of the offset’s direction across continuations, as anticipated in Remark 2; the norm of the offset itself decays more slowly.
Where Qwen differs
Two features of the untrained Qwen model do not carry over from Pythia. First, the layer-1 local exponent does not follow the dilution law with a single (residuals up to 0.13), and the fitted powers are in layers 2, 3 (–5 beyond; –4.9). The linear relay model gives exactly the dilution law in layer 1 and a power of exactly 1 in layer 2 for any gains (§ 4.5), so the first blocks of untrained Qwen are not in the linear-relay regime, although the qualitative predictions (local exponent rising towards 1, decreasing with depth) hold. This is not in conflict with Theorem 1, which bounds asymptotically and says nothing about its local slope at finite . Attention in untrained Qwen is as uniform as in untrained Pythia (normalised entropy 0.990–0.993, largest weight 0.006–0.007 in layers 1–3, logit spread median ), so non-uniform attention is not the reason. A difference that is present is the block structure: Pythia uses parallel residual connections, so its first MLP sees only the embedding, which is the same in both runs, and the layer-1 offset is exactly the attention write; Qwen’s blocks are sequential, so the MLP of the first block already processes the attention write nonlinearly, and at initialisation that write dominates the residual (mid-block norm 0.033, against about 0.02 for embeddings drawn with standard deviation 0.02). We take this as the likely reason the linear relay model fails in the first blocks of Qwen, without having tested it. Second, the noise across continuations grows slowly at initialisation ( to in layers 2–23), so the decodability of the prompt decays as rather than .
A late slowdown in trained Qwen. With windows up to 2048, the local exponent of the trained Qwen models decreases at late times. In Qwen2.5-0.5B this happens in every layer, down to 0.23–0.57 in the last octave, including layer 1 (0.24), where only the direct path exists. In Qwen2.5-1.5B layers 1–2 keep decaying fast (last octave 2.2 and 1.3) while layers 3–27 slow down to 0.38–0.53. In layers 14 and 21 of Qwen2.5-1.5B the signal stays below three times its sampling floor at all windows, so those two layers are not interpreted.
Three explanations excluded. We first excluded a statistical explanation: with continuations the plug-in signal is biased upwards by , which would flatten the curve late; with the unbiased estimator the slowdown is unchanged (Qwen2.5-0.5B, last octave: 0.24, 0.42, 0.43 in layers 1–3). We then tested a prompt-specific attention sink: Lemma 2 bounds the attention to the prompt by , which only bites once , and without a BOS token the sink is the first prompt token, which differs between the two prompts of most pairs. Prepending the same token (<|endoftext|>) to both prompts, which makes the sink shared, leaves the slowdown unchanged (last octave 0.24, 0.40, 0.44 in layers 1–3). Finally we tested the structure of the continuation text: WikiText chunks span headings and document boundaries, which affect all prompts alike and are only probed at the largest windows. Replacing the continuations by WikiText sentences in random order, which removes headings, document boundaries and any structure beyond a sentence, leaves the slowdown unchanged as well (last octave 0.23, 0.49, 0.47 in layers 1–3 and 0.18–0.57 in the deep layers, against 0.24, 0.42, 0.43 and 0.23–0.57); the global exponents move by at most 0.06 (slightly lower in the deep layers). Three explanations are thus excluded: estimation bias, a prompt-specific sink, and the long-range structure of the continuation.
A positional oscillation. Since the slowdown appears in layer 1, where only the direct path
| model | layer | detail | para | unrel. | detail | para | unrel. |
| 160m, init | L1 | 0.75 | 0.74 | 0.73 | 0.75 | 0.74 | 0.73 |
| L11 | 0.37 | 0.37 | 0.35 | 0.37 | 0.37 | 0.35 | |
| 160m, trained | L1 | 0.92 | 0.98 | 0.97 | 0.93 | 0.99 | 0.98 |
| L11 | 0.69 | 0.59 | 0.70 | 0.61 | 0.54 | 0.66 | |
| 1b, trained | L1 | 0.90 | 0.86 | 0.88 | 0.90 | 0.86 | 0.88 |
| L15 | 0.60 | 0.52 | 0.62 | 0.48 | 0.41 | 0.53 |
caption: α with contiguous WikiText continuations and with WikiText sentences in random order (real-corpus pairs, unbiased signal, centre-labelled windows).
:::
exists, what remains is the dependence of attention on distance. We measured it directly: in the first block of trained Qwen2.5-0.5B, the attention that continuation queries pay to the prompt follows on average out to , with times the attention mass between 6 and 11 from on, but it oscillates by about around that law from octave to octave (log-slopes , , , , between consecutive windows from ). The local exponents of the layer-1 offset follow these oscillations: the steep octave 512–1024 (attention slope ) is where exceeds 1, and the shallower last octave () is where it drops to 0.24. The late slowdown in layer 1 is therefore an oscillation of the direct path around dilution, of positional origin, rather than a breakdown of dilution; its amplitude in the offset is larger than in the attention mass, so the value vectors contribute as well. In block 2, times the attention mass rises slowly with (12 to 17 between and 2048), a mild departure from at long range. The slowdown is absent from layers 1–2 of Qwen2.5-1.5B, whose attention we have not measured; tying the period of the oscillation to the rotary frequencies is left open. A plausible physical mechanism is phase cancellation in the rotary embeddings: at long distances, the dot product between different rotary frequency components can destructively interfere, causing the query to lose its original directional preference towards the prompt and oscillate around the dilution law. We leave a rigorous proof of this interference to future work, but it aligns with the observation that the oscillation amplitude is larger in the offset than in the raw attention mass, as the value vectors also carry positional phase information.
Continuation structure
For trained networks on contiguous WikiText, shows no systematic trend over 16–1920 in the deep layers of Pythia (a power law over two decades), with a recurring bump at 128–256 in both sizes. Repeating the measurement with shuffled-sentence continuations, which keep the vocabulary and the sentences of WikiText but remove all structure beyond a sentence, separates three effects (Table 5).
Untrained networks do not notice. For untrained Pythia-160m every exponent, every local exponent and every fitted and agrees with the contiguous-text values to within 0.01: geometric dilution and relay at initialisation are independent of what the text says, as the theory requires.
The first layers do not notice either. In trained models layers 1–4, where the direct path dominates, change by at most 0.04. The bump of the layer-1 local exponent at 128–256 in Pythia-160m survives (1.62 against 1.54), so it is positional, not a property of the text.
Figure 4. Path decomposition (Pythia-160m, real-corpus pairs). Left: share of the late offset carried by relay. Right: decay exponents of the direct (circles, solid) and relay (squares, dashed) paths. At initialisation relay carries almost all of the late offset and barely decays.
Deep layers of trained models forget more slowly on unstructured text. The global exponents drop by 0.02–0.11 in layers 6–11 of Pythia-160m and by 0.09–0.18 in layers 8–15 of Pythia-1b (by in Qwen2.5-0.5B). The shape of the local exponent changes in a model-dependent way: it rises from about 0.45 to about 1.0 across the window in Pythia-160m, and falls from about 0.5 to about 0.3 in Pythia-1b.
The pure power law of the trained deep layers is therefore a property of the model on coherent text. A natural reading is that a coherent continuation gives the deep layers relevant recent content to attend to, which displaces the prompt; without that structure the prompt competes for longer. We have not tested this mechanism. The ordering of the axes and the trained–untrained contrast are unaffected. In layers 1–2 of the trained models the local exponents fluctuate above 1 in the last octaves, where the offset has decayed by more than three decades.
Mechanism: direct path and relay
The decomposition
The offset at a given layer is built from the attention writes of the blocks below it. To see where the prompt’s imprint goes as it decays, we split each write into a direct part, in which the query reads the prompt, and a relay part, in which the query reads a continuation token that has already absorbed prompt information.
The write of attention block at query is , where runs over the heads of the block, is the attention weight of head from query to key , its output matrix and the value vector at key . Splitting keys into prompt tokens (direct) and continuation tokens (relay) and differencing the two prompts under the same continuation gives an exact decomposition of the attention contribution to the offset; sinks shared by both prompts (a common first token) cancel.
Table 6 reports the decomposition at three checkpoints. Five patterns stand out.
| step | L1 | L2 | L3 | L6 | L9 | L11 | |
| 0 | measured | 0.71 | 0.56 | 0.49 | 0.40 | 0.36 | 0.35 |
| paths (both) | 0.70 | 0.53 | 0.46 | 0.38 | 0.34 | 0.32 | |
| / | 0.70 / 0.42 | 0.69 / 0.18 | 0.69 / 0.13 | 0.68 / 0.06 | 0.68 / 0.02 | 0.68 / 0.01 | |
| relay share | 0.02 | 0.73 | 0.87 | 0.95 | 0.97 | 0.97 | |
| 512 | measured | 0.70 | 0.74 | 0.79 | 0.77 | 0.66 | 0.67 |
| paths (both) | 0.69 | 0.73 | 0.79 | 0.73 | 0.60 | 0.61 | |
| / | 0.69 / 0.42 | 0.82 / 0.39 | 0.92 / 0.50 | 0.96 / 0.52 | 0.83 / 0.38 | 0.83 / 0.38 | |
| relay share | 0.01 | 0.55 | 0.71 | 0.87 | 0.88 | 0.88 | |
| 143000 | measured | 0.98 | 0.81 | 0.57 | 0.56 | 0.63 | 0.65 |
| paths (both) | 0.98 | 0.83 | 0.62 | 0.61 | 0.63 | 0.63 | |
| / | 0.97 / 0.85 | 0.86 / 0.62 | 0.56 / 0.33 | 0.39 / 0.19 | 0.40 / 0.20 | 0.41 / 0.23 | |
| relay share | 0.03 | 0.41 | 0.50 | 0.56 | 0.53 | 0.53 |
Table 6. Path decomposition on real-corpus prompt pairs (Pythia-160m, 25 pairs per axis, four axes pooled, 8 WikiText continuations per pair, , centre-labelled windows, unbiased signal for every component). Cumulative attention writes of blocks reproduce the measured within 0.06 at every layer and checkpoint. Relay share: relay part of the squared offset in the last window. Bootstrap half-widths of the measured are .
Layer 1 has no relay, as it must: the continuation tokens are still identical after the first block, and the relay share is 0.02–0.03 at every checkpoint.
At initialisation the late offset is 73–97% relay, rising with depth, and the relay component barely decays ( from 0.18 in layer 2 to 0.01 in layer 11), while the direct path decays as a uniform head would (–0.70 in every layer, the layer-1 value). The long tail of untrained networks is relay.
Training makes relay leak and steepens the direct path: at step 512 –0.52 and –0.96 in layers 2–11, while relay still dominates the late offset (0.55–0.88).
After training the components no longer add up simply: in layers 6–11 –0.41 and –0.23, yet their sum decays with –0.63. The two components have different decay rates and a cross term (they are anti-aligned at late times), so the exponent of the sum cannot be read off the exponents of the parts. The same happened on hand-written scenes from step on. Part of it is bookkeeping (softmax weights sum to one, so attention moved off the prompt lands on the continuation); in this regime only the sum is reliable.
Attention weights mislead. Raw (or value-weighted) attention to the prompt predicts only around steps 512–1000; before, it misses relay; after, sinks (token 0 and in-prompt delimiters) dominate the weights while carrying no prompt-specific information.
These patterns describe the mechanism at a fixed checkpoint. We now ask when the mechanism appears during training.
When forgetting appears in training
Table 7 and Figure 5 follow the same 400 prompt pairs through training. Three phases appear.
| step | L1 | L2 | L3 | L6 | L9 | L11 |
| 0 | 0.73 | 0.57 | 0.50 | 0.41 | 0.37 | 0.36 |
| 128 | 0.73 | 0.57 | 0.51 | 0.43 | 0.41 | 0.41 |
| 256 | 0.72 | 0.62 | 0.62 | 0.57 | 0.52 | 0.52 |
| 512 | 0.72 | 0.76 | 0.81 | 0.77 | 0.67 | 0.68 |
| 1000 | 0.72 | 0.75 | 0.79 | 0.67 | 0.63 | 0.66 |
| 4000 | 0.96 | 0.95 | 0.83 | 0.60 | 0.64 | 0.62 |
| 16000 | 1.03 | 0.90 | 0.70 | 0.59 | 0.64 | 0.62 |
| 143000 | 0.97 | 0.81 | 0.61 | 0.58 | 0.63 | 0.66 |
Table 7. across training, Pythia-160m, real-corpus prompt pairs (100 per axis, mean over the four axes, the same pairs at every checkpoint), , centre-labelled windows, unbiased signal. Per-axis 95% intervals have half-width .
Figure 5. Training dynamics of Pythia-160m on the same 400 prompt pairs. Left: per layer. Middle: normalised attention entropy. Right: attention mass on token 0. The shaded band marks steps 128–512, where forgetting sets in and attention beyond layer 1 leaves the uniform regime; the first layer and the sink change only after step 1000. Step 0 is drawn at the left edge.
Onset (steps 128–512, roughly 0.3–1B tokens). Up to step 128 the exponents are those of the untrained network, and so is the attention: normalised entropy in every layer measured. Between steps 128 and 512 every layer from 2 on jumps (layer 11: 0.41 → 0.52 → 0.68); the path decomposition at step 512 shows that this is relay starting to leak (Table 6). Attention leaves the uniform regime at the same time: the normalised entropy falls from 0.98 to 0.85 and 0.65 in layer 3 and from 0.95 to 0.68 and 0.54 in layer 12 between steps 128, 256 and 512, while layer 1 stays at 0.98. Forgetting is strongest at step 512 and relaxes slightly afterwards in the deep layers.
First-layer locality (steps 1000–4000). Layer 1 keeps the uniform-head value 0.72–0.73 until step 1000 and reaches 0.96 by step 4000: the first block becomes recency-biased only after forgetting in deeper layers is established, and its attention entropy falls only then (0.98 at step 1000, 0.82 at step 4000). On the same 400 pairs, the attention mass on token 0 stays
at or below 0.003 in every layer measured (1, 2, 3, 7, 12) up to step 1000 and appears only afterwards (0.031 in layer 7 at step 4000, 0.071 at 16000, 0.089 at the end; 0.05–0.10 in layers 5–11 of the final model).
Settling. By the end of training deep layers sit at 0.58–0.66, and the depth dependence of the untrained network (0.73 → 0.36 from layer 1 to 11) has given way to a profile that is roughly flat beyond layer 3.
Forgetting therefore sets in between steps 128 and 512, earlier than the formation of attention sinks and earlier than the first layer becoming strongly recency-biased. The earliest change observed in training is that relay begins to leak, while layer 1, where only the direct path exists, stays in the untrained regime until step 1000.
Causal test: compensating dilution
The intervention
The decomposition of § 6 shows that the prompt’s imprint decays through dilution, but it is a measurement, not an intervention. To test whether dilution is the cause of the decay, we cancel it.
We add to the logits from continuation queries to prompt tokens , either uniformly () or with per-head fitted so that each head’s attention to a typical prompt token stops decaying. The boost exactly compensates the shrinkage of the prompt’s share of attention under uniform attention.
What the intervention removes
Table 8 shows that compensating dilution removes forgetting at initialisation and at step 512; after training, uniform compensation leaves – and per-head compensation reverses the decay.
| step | L1 | L2 | L3 | L6 | L9 | L11 | |
| 0 | original | 0.71 | 0.56 | 0.49 | 0.40 | 0.36 | 0.35 |
| 0 | uniform | 0.10 | 0.09 | 0.08 | 0.07 | 0.06 | 0.06 |
| 0 | per-head | 0.07 | 0.06 | 0.06 | 0.05 | 0.05 | 0.05 |
| 512 | original | 0.70 | 0.74 | 0.79 | 0.77 | 0.66 | 0.67 |
| 512 | uniform | 0.10 | 0.17 | 0.23 | 0.28 | 0.25 | 0.26 |
| 512 | per-head | 0.07 | 0.08 | 0.09 | 0.11 | 0.12 | 0.13 |
| 143000 | original | 0.98 | 0.81 | 0.57 | 0.56 | 0.63 | 0.65 |
| 143000 | uniform | 0.29 | 0.22 | 0.14 | 0.11 | 0.21 | 0.25 |
| 143000 | per-head | −0.12 | −0.10 | −0.08 | −0.09 | −0.04 | −0.01 |
Table 8. under dilution compensation on real-corpus prompt pairs (same setting as Table 6). Bootstrap half-widths . Per-head calibration: residual median head slope −0.07 (step 0) and −0.08 (step 512), −0.33 (10–90%: [−1.16, −0.10]) at step 143000.
Figure 6. Left: per layer before (solid) and after (dotted) per-head compensation of the dilution, at three checkpoints. Right: retention against cost at the end of training; the last-layer against the perplexity change for a boost on the prompt and for an equally strong placebo boost.
Under uniform compensation the residual matched the residual slope of prompt attention in a preliminary run on three hand-written scenes (steps 0, 256, 512: residual = 0.06–0.08, 0.14–0.15, 0.23–0.28 against residual attention slopes 0.09, 0.16, 0.26); per-head compensation brings to 0.05–0.13 at steps 0 and 512, and overshoots after training (: the prompt’s influence grows), where the attention-based calibration does not converge, a sign that the attention a head pays to the prompt is not the same as the information it carries.
What the test does and does not show
Per-head compensation raises perplexity by 140% or more (§ 9), so the compensated network runs far from its training distribution. The test therefore shows that, in these networks, the forgetting we measure disappears once dilution is compensated: dilution is necessary for it. It does not show that the compensated network computes as the original one does, and it does not show that dilution is sufficient on its own; that it is sufficient is shown by the untrained networks and the linear relay model of § 4.5, where uniform attention alone produces the decay. Within this caveat, no second forgetting mechanism is needed. Crucially, the placebo control (Table 9) provides a direct defense against the objection that the compensation simply destroys the model. While both the prompt boost and the placebo increase perplexity, only the prompt boost substantially flattens the decay of . This asymmetry confirms that the retention is driven specifically by drawing attention back to the prompt, rather than by a global distortion of the next-token distribution.
| \multicolumn{3}{c}{boost on the prompt} | \multicolumn{3}{c}{placebo boost} | |||||
| L6 | L11 | PPL | L6 | L11 | PPL | |
| 0 | 0.56 | 0.65 | — | 0.56 | 0.65 | — |
| 0.25 | 0.45 | 0.58 | +0.7% | 0.53 | 0.64 | +0.2% |
| 0.5 | 0.31 | 0.48 | +2.6% | 0.48 | 0.63 | +1.3% |
| 0.75 | 0.20 | 0.36 | +7.1% | 0.41 | 0.59 | +4.2% |
| 1 | 0.11 | 0.25 | +16.1% | 0.34 | 0.53 | +10.3% |
Table 9. Trade-off of the uniform boost on Pythia-160m (step 143000), real-corpus pairs (25 per axis, four axes pooled; centre-labelled windows, unbiased signal; intervals 0.06). PPL: perplexity change on WikiText continuations of 40 real-corpus prompts. The placebo boosts a same-length span at the start of the continuation.
In a preliminary run on the hand-written scenes, continuations generated by the model from the prompt gave the same as human-written ones (within 0.1; identical early in training), so the rate of forgetting is not inherited from the statistics of the model’s own text; all real-corpus results above use human-written continuations.
Exterior: what survives in the generated text
The question. So far the continuation has been written by a human, so the exterior was the same for both prompts. In free generation the exterior is itself a product of the model, and the prompt can influence later tokens only through the text it has already caused. What properties of the prompt can still be read off from that text?
The probe. The observer sees only text: a window of generated tokens ending at (32 tokens for Qwen, 8 for Pythia) is turned into a hashed unigram histogram (4096 bins, square-root normalised), and a logistic-regression probe (inputs standardised, projected on 16 principal components, penalty ) tells the two prompts of a pair apart under 5-fold stratified cross-validation. The numbers are accuracies averaged over three scenes, with 32 freely generated trajectories per prompt (pure sampling at temperature 1). Chance is 0.5; the 95th percentile of a null obtained by splitting one prompt’s trajectories at random is 0.58–0.63. These runs predate the main pipeline and use three hand-written scenes (Chinese for Qwen, English with a French language variant for Pythia).
What survives. Details, paraphrase and genre become undecodable from the text, while language remains decodable (0.81 for Qwen2.5-0.5B, 0.84 for Pythia-160m at ). Language is not conserved: in Qwen, 4–11 of 32 trajectories per scene switch language, mostly towards Chinese, the model’s dominant mode. The teacher-forced offset for the language axis still decays (–0.65 in layers 3–11 of trained Pythia-160m, against 0.61–0.69 for a one-word detail; Table 3), so language survives through the trajectory itself, not through memory of the prompt.
Why language survives. The two observations are complementary rather than contradictory. Corollary 1 concerns the sensitivity of the next-token distribution to the prompt given the same history; in free generation the history is itself a function of the prompt. Language therefore survives along an exterior channel: the prompt tilts the first tokens towards its language, and every later token is predicted from a history already written in that language, which keeps the trajectory there until it switches by chance. This is precisely the sense of the analogy: what persists is what the exterior field carries.
Application: a dilution knob, and its costs
The analysis above says that the prompt’s imprint fades because its share of attention shrinks as . This suggests a simple intervention: boost the logits from the prompt by a factor that cancels the shrinkage, and see whether forgetting slows. We test this on Pythia-160m with a uniform boost on the prompt, and compare it with a placebo of the same mean strength applied to the start of the continuation.
The trade-off
A mild boost is cheap: lowers by 0.17–0.25 for +2.6% perplexity. The placebo also lowers , because the boosted continuation tokens follow the prompt directly and already carry its imprint through relay. At matched retention the prompt boost is much cheaper than the placebo (: +2.6% at vs. +10.3% at ). At matched strength it is costlier on these pairs (+16.1% vs. +10.3% at ), whereas in a preliminary run on three hand-written scenes, whose continuations were generated from the prompt, the two cost about the same (: +2.7% vs. +2.2% on human-written text, +1.3% vs. +1.7% on the model’s own text; : +14.0% vs. +14.8% and +8.3% vs. +10.8%). The cost of remembering the prompt thus depends on its relevance to the continuation: here the WikiText continuations are unrelated to the prompt, and attending to it more hurts prediction.
Per-head calibrated compensation is too strong to be practical (+140% to +376% perplexity).
Instruction adherence and its side effects
Qwen2.5-0.5B-Instruct, system prompt “answer only in French”, 2048 generated tokens: share of samples fully in French 31% → 94% at . Qwen2.5-1.5B-Instruct: 84% → 94–100% for all boosts; a constant boost with the same mean bias does as well, with less echo.
Multi-turn conversation (Qwen2.5-1.5B-Instruct, 8 turns, English user turns asking for the next chapter, natural reply endings, 32 conversations per condition). Without boost the French share drops from 0.90 to 0.81–0.87 after the first turn and 78% of conversations stay in French throughout. With boosts of strength 0.1 and 0.25: last-three-turn French share 0.90 and 0.96, conversations fully in French 84% and 91%. Constant boosts with the same mean bias: 0.94 and 0.94, 84% and 88%. Echo in late turns stays at the baseline level (the excess is concentrated in the first reply, more so for the constant boost: 12.9 vs. 8.6 leak words per 1000 at the higher strength); repetition does not increase; replies are about 10% shorter. With 32 conversations per condition these differences are trends rather than significant effects, and the shape shows no clear advantage over a matched constant: the benefit comes from extra attention to the system prompt.
Instruction echo. With forced-length single-turn generation, the share of outputs whose second half talks about the instruction rises from 19% to 56–69% (1.5B); much of this text is filler written after the story has ended.
Across all three tests, the pattern is the same: what helps is extra attention to the instruction, not the shape of the correction.
Discussion and conclusion
A transformer’s memory of its prompt, measured as the prompt-induced offset of the hidden state under a shared continuation, decays approximately as a power law whose exponent we can measure, bound and manipulate. The bound follows from geometric dilution of attention and the number of relay rounds through depth; it is attained asymptotically in the uniform-attention regime, whose exact solution with one gain per layer accounts for the slow, depth-dependent forgetting of untrained networks; trained networks suppress relay by about one billion training tokens, before attention sinks appear, and come close to pure dilution. Compensating the dilution removes or reverses forgetting, so dilution is necessary for it in these networks; that it is also sufficient is shown by the untrained networks and the linear relay model, where uniform attention alone produces the decay. Two negative results matter as much: attention weights are not a reliable measurement of the effect, and a log -shaped compensation is no better than a constant boost for keeping an instruction alive; what helps is extra attention to the instruction, not the shape of the correction.
These findings come with limitations. The constant of Theorem 1 can grow exponentially with depth and the bound is informative only for , so the theorem constrains the asymptotic shape of the decay rather than the exponents we fit; the non-degeneracy assumption is verified empirically rather than derived, with a small margin in the first block of Qwen. How the exponents of trained networks relate to the theorem is therefore an open question: for trained models the bound is uninformative within our windows, and the solvable model describes the untrained ones.), and in trained Qwen the late slowdown is an oscillation of layer-1 attention around whose period, and its relation to the rotary frequencies, remain open. The single-turn behavioural test forces a fixed generation length, so its late text partly consists of filler written after the story has ended; the multi-turn test avoids this but has only 32 conversations per condition. Exponents are fitted over at most seven octaves, which does not allow the logarithmic and small-gain descriptions of the untrained curves to be told apart.
Several questions are left open. The most direct is scale: whether the exponents we fit stay in the same range at larger models, and whether the depth dependence of untrained networks survives. A second is the period of the layer-1 oscillation in Qwen, which we have not tied to the rotary frequencies. A third is whether the picture changes under instruction tuning: all our trained models are pretrained only, and the path decomposition across checkpoints suggests that the training process itself is what suppresses relay, so a model trained on dialogue may forget its system prompt at a different rate. Finally, the toolkit we release makes it easy to measure on any Hugging Face model, and we hope the community will extend the measurements beyond the range we could afford.
Conditional on the generated text the prompt is forgotten; what survives, such as the language of the conversation, is carried by the text itself.
Proofs
This appendix gives the steps that § 4 compresses: a perturbation lemma for softmax, the bookkeeping of the constants over heads, the alignment of positions when the two prompts differ in length, a complete proof of Lemma 3, and a quantitative version of Corollary 2 with explicit error terms.
Softmax along a path
Lemma 4. For let and the softmax. Then
If moreover for all , then for all and .
Proof. With and , (with the elementwise product), whose norm is at most . Integrating over gives the first claim. Each coordinate of is a convex combination of and , hence lies in , and .
Constants of a multi-head block
An attention block with heads writes , where is the number of heads, are the attention weights of head , its value matrix, and the block of columns of that reads head . For each head let
with the contribution of query and key biases, if any. The constants of § 4 are to be read as
so that every bound proved for one head with sums over heads into the same bound with ; in particular of Lemma 3 carry no hidden factor . Gains of the normalisation layers are absorbed into the adjacent weight matrices. Lemma 1 then holds head by head with these constants, by the same arguments.
Positions
Lemma 5. With rotary position encodings, the logit between a query at continuation distance and a key at continuation distance depends on the positions only through ; in particular it is the same function of in both runs, whatever the prompt lengths and .
Proof. Rotary encodings give because is orthogonal and , where and are the query and key vectors, their absolute positions and the rotary matrix at position . A continuation token at distance sits at absolute position in one run and in the other, so in both. Positions enter nowhere else in the block.
With absolute position embeddings the continuation tokens would carry different embeddings in the two runs; we then take , which restores at continuation positions.
Scope. The proof covers multi-head attention with full or partial rotary encodings or with no positional term in the logits, grouped-query attention (key and value matrices shared across heads only change the per-head constants), query and key biases, LayerNorm and RMSNorm, and sequential or parallel blocks; absolute position embeddings are covered for . It does not cover additive position biases that grow with distance, such as ALiBi: they make the logits unbounded, so Lemma 1(a) fails, even though they push towards more forgetting rather than less; nor logit scalings that grow with context length, such as Scalable-Softmax [14], which also make the logits unbounded and, unlike ALiBi, can push towards less forgetting; nor sliding-window attention, for which the number of visible keys stops growing and the dilution lemma no longer applies as stated; nor non-softmax attention.
Proof of Lemma 3
Fix layer , a continuation query at distance and a head ; drop the head index. Write and for the second run, so , and at continuation keys. Let , for the continuation keys, and , and similarly with primes. The head output is , and
Prompt terms. The query sees keys in one run and in the other; by Lemma 2, , so and, since , the last three terms have norm at most .
Continuation term. and are softmaxes over the same keys of the logits and , , which lie in ; by Lemma 5 they differ only through the hidden states. Then
By Lemma 4, , so the first sum is at most . For the second, Lemma 4 and (Lemma 1(c)) give
using for the first term and for the second. With and the sum over heads of the previous subsection,
Residual and MLP. In a sequential block, and , with -Lipschitz on non-degenerate inputs (Assumption 1), so . In a parallel block, , so , which is smaller. Inserting the bound on K_1=(1+\mu)(1+2\lambda V)$, and . □
A quantitative form of Corollary 2
Let , so that in (8).
Lemma 6. With , and , and writing for the positive part,
Proof. The lower bound combines and from the proof of Proposition 2. For the upper bound, with (where ). Exponentials of series with non-negative coefficients preserve coefficientwise inequalities, so is at most the coefficient of in , namely . By the AM–GM inequality this is at most , and . □
Proposition 3. In the linear relay model, with , and , for all ,
Proof. Insert into (6) the bounds , which restate from the proof of Corollary 2, and the bounds of Lemma 6 on . All terms are non-negative. □
We checked both inequalities numerically against the exact recursion (4) for 300 random choices of depth (), prompt length () and gains (), at : no violation in 1800 checks.
Corollary 3. Write for the normalised terms of the series, , and for the weight beyond depth . Then
whenever .
Proof. For the lower bound use and for in Proposition 3, drop the factor in favour of the linear bound, and add the missing terms as . For the upper bound use and extend the sum to all . For , the ratio of consecutive terms is for ; sum the geometric series. □
Reading the bound. The weights , are those of the series of ; for they concentrate around (the mean of is , from (9)), so and . The relative error of the small-gain form is then of order
which are the conditions stated in Corollary 2: (many small gains; is not enough, since ), prompts long compared with , and depth large compared with . For , as in the first layers of our measurements, dominates and all corrections are small, but the asymptotic expansion (9) of the local exponent is not valid there; this is why the fitted of § 4.5 are effective parameters. Statements (a) and (b) of Corollary 2 are properties of ; a bound on such as the one above does not by itself bound its logarithmic derivative, so the local exponents of the model are covered by (a) and (b) only to the extent that is well approximated by uniformly over an octave.
Non-degeneracy of pre-norm activations
Table 10 reports at the input of every block and of the final norm, over 80 sequences of length (40 real-corpus prompts 2 WikiText continuations). The minimum is never close to zero on these sequences; this supports Assumption 1 empirically but does not prove it for other continuations or longer contexts. The embeddings (block 0 input) are small but are not used by Theorem 1 (Assumption 1). For the trained Pythia-1b the minimum over blocks 1–15 is 0.49 (median 0.57–2.81) and 0.97 at the final norm, again a wide margin. Qwen2.5-0.5B uses RMSNorm (); its minimum over block inputs 1–23 is 0.105 (median 0.17–2.56) and 1.30 at the final norm. Because its blocks are sequential, the post-attention norms enter Assumption 1 as well; their minimum is 0.012, in the first block, so for this model.
| input of | init (step 0) | trained (step 143000) | ||||
| min | 0.01% | median | min | 0.01% | median | |
| block 0 (embeddings) | 0.021 | 0.021 | 0.023 | 0.018 | 0.018 | 0.028 |
| block 1 | 0.112 | 0.112 | 0.127 | 0.236 | 0.241 | 0.325 |
| block 2 | 0.162 | 0.162 | 0.184 | 0.279 | 0.285 | 0.402 |
| block 3 | 0.203 | 0.208 | 0.229 | 0.320 | 0.331 | 0.477 |
| block 6 | 0.305 | 0.307 | 0.335 | 0.428 | 0.436 | 0.616 |
| block 9 | 0.377 | 0.378 | 0.414 | 0.519 | 0.529 | 0.883 |
| block 11 | 0.416 | 0.418 | 0.460 | 0.605 | 0.607 | 1.114 |
| final norm | 0.433 | 0.437 | 0.479 | 0.811 | 0.938 | 4.319 |
Table 10. for Pythia-160m. The theorem uses blocks and the final norm: (init) and 0.236 (trained).
Prompt sets
The twenty hand-written scenes. Each prompt is followed by the same sixteen WikiText-103 validation passages. Real-corpus pairs are drawn as described in § 3.4; the exact pairs are released with the toolkit.
base It was raining in Paris today, and I stood at the metro entrance, detail It was snowing in Paris today, and I stood at the metro entrance, para Rain was falling over Paris today; I stopped at the door of the metro station, lang Il pleuvait à Paris aujourd’hui, et je me tenais à l’entrée du métro, genre PARIS (Reuters) - Heavy rain hit the French capital on Tuesday, and at one metro entrance
base In my grandmother’s kitchen, rain ran down the window as she detail In my grandmother’s kitchen, snow piled up on the windowsill as she para Rain streaked the glass of my grandmother’s kitchen window while she lang Dans la cuisine de ma grand-mère, la pluie coulait sur la vitre pendant qu’elle genre LONDON (AP) - Days of heavy rain left several homes with leaking kitchen windows, and one elderly resident
base It was raining that day. He waited at the train station for a long time, and finally detail It was snowing that day. He waited at the train station for a long time, and finally para The rain never stopped. He spent hours waiting at the station until, at last, lang Il pleuvait ce jour-là. Il a attendu longtemps à la gare, et finalement genre BERLIN (Reuters) - Rain delayed dozens of trains on Monday, and one passenger who had waited for hours
base The old fisherman pulled his red boat onto the beach and detail The old fisherman pulled his blue boat onto the beach and para Dragging his red boat up onto the sand, the old fisherman lang Le vieux pêcheur tira son bateau rouge sur la plage et genre LISBON (AP) - An elderly fisherman hauled his boat ashore on Sunday as officials warned that
base My sister opened the letter from the university, read the first line, and detail My brother opened the letter from the university, read the first line, and para After tearing open the university’s letter, my sister glanced at the first sentence and lang Ma sœur ouvrit la lettre de l’université, lut la première ligne, et genre BOSTON (Reuters) - Thousands of students received admission letters this week, and one applicant
base The cat jumped onto the piano in the middle of the night, and detail The dog jumped onto the piano in the middle of the night, and para Late at night the cat leapt up onto the piano keys, and lang Le chat sauta sur le piano au milieu de la nuit, et genre VIENNA (AP) - A cat triggered a late-night alarm at a concert hall on Friday after it
base We drove north for six hours until the road ended at a frozen lake, where detail We drove north for nine hours until the road ended at a frozen lake, where para After six hours heading north, the road stopped at the edge of a frozen lake, and there lang Nous avons roulé vers le nord pendant six heures jusqu’à un lac gelé, où genre OSLO (Reuters) - A remote northern road was closed on Thursday at a frozen lake, where
base The baker woke at four in the morning, lit the oven, and detail The baker woke at five in the morning, lit the oven, and para Up at four a.m., the baker got the oven going lang Le boulanger se leva à quatre heures du matin, alluma le four et genre LYON (AFP) - Bakers across the region began work before dawn on Monday as flour prices
base The children found an old map in the attic, and the next morning they detail The children found an old key in the attic, and the next morning they para Up in the attic the kids discovered an ancient map; the following day they lang Les enfants trouvèrent une vieille carte dans le grenier, et le lendemain matin ils genre DUBLIN (AP) - Children discovered a centuries-old map in a family attic, local historians said, and
base The doctor looked at the X-ray for a long moment before she detail The doctor looked at the scan for a long moment before she para For a long while the doctor studied the X-ray, and then she lang La médecin regarda la radiographie un long moment avant de genre GENEVA (Reuters) - Hospital doctors reviewed new imaging results on Wednesday before officials
base The storm knocked out the power, so we lit candles and detail The storm knocked out the phones, so we lit candles and para When the storm cut the electricity, we got out the candles and lang La tempête coupa l’électricité, alors nous avons allumé des bougies et genre MIAMI (AP) - A powerful storm left thousands without electricity on Saturday, and residents
base On the first day of school, the new teacher wrote her name on the board and detail On the last day of school, the new teacher wrote her name on the board and para The new teacher began the first school day by writing her name on the black-board, then lang Le premier jour d’école, la nouvelle maîtresse écrivit son nom au tableau et genre MADRID (Reuters) - Schools reopened on Monday, and at one primary school a new teacher
base The mountain hut was empty except for a wooden table and a note that said detail The mountain hut was empty except for a wooden chair and a note that said para Apart from a wooden table and a short note, the mountain hut was deserted; the note said lang Le refuge de montagne était vide, à part une table en bois et un mot qui disait genre ZURICH (AP) - Rescuers who reached an empty mountain hut on Sunday found only a note that said
base My father fixed the old radio on the kitchen table, and when it finally played, detail My father fixed the old clock on the kitchen table, and when it finally played, para When my father at last got the old kitchen radio working again, lang Mon père répara la vieille radio sur la table de la cuisine, et quand elle joua enfin, genre ROME (Reuters) - Collectors of vintage radios gathered on Saturday, and one restorer said that
base The ship left the harbor at dawn, carrying coffee and three passengers who detail The ship left the harbor at dusk, carrying coffee and three passengers who para At first light the ship sailed out of port with a cargo of coffee and three travellers who lang Le navire quitta le port à l’aube, chargé de café et de trois passagers qui genre SANTOS (Reuters) - A cargo ship carrying coffee left port early on Friday with three passengers who
base The library was closing, but the old man kept reading by the window until detail The museum was closing, but the old man kept reading by the window until para Although the library was about to close, the old man went on reading at the window until lang La bibliothèque fermait, mais le vieil homme continuait à lire près de la fenêtre jusqu’à ce que genre PRAGUE (AP) - The city’s main library shortened its opening hours this week, and one regular reader
base She planted tomatoes in the garden every spring, and this year detail She planted roses in the garden every spring, and this year para Each spring she put tomato plants in the garden; this time, though, lang Elle plantait des tomates dans le jardin chaque printemps, et cette année genre BORDEAUX (AFP) - Home gardeners reported a poor tomato season this year as a late frost
base The orchestra stopped playing when the lights went out, and the conductor detail The orchestra stopped playing when the alarm went off, and the conductor para As soon as the lights failed, the orchestra fell silent and the conductor lang L’orchestre cessa de jouer quand les lumières s’éteignirent, et le chef d’orchestre genre SALZBURG (Reuters) - A power cut halted a sold-out concert on Thursday, and the conductor
base Two hikers got lost in the forest after sunset, and one of them detail Two hikers got lost in the desert after sunset, and one of them para After the sun went down, two walkers lost their way among the trees, and one of them lang Deux randonneurs se perdirent dans la forêt après le coucher du soleil, et l’un d’eux genre DENVER (AP) - Rescue teams searched overnight for two hikers lost in a national forest, and one
base The market was full of oranges and noise, and the woman at the corner stall detail The market was full of lemons and noise, and the woman at the corner stall para Oranges and shouting filled the market, and at the stall on the corner the woman lang Le marché était plein d’oranges et de bruit, et la femme à l’étal du coin genre SEVILLE (Reuters) - Orange prices rose sharply at local markets this week, and one stallholder
Toolkit
The code runs on a single laptop and measures, for any Hugging Face causal language model: the prompt offset under shared continuations and its signal, noise and mean-squared versions, with the unbiased estimator, centre-labelled octave windows, global and local exponents, and bootstrap intervals over prompt pairs (real_run.py; scale_run.py for the hand-written scenes); the non-degeneracy check of Assumption 1 (prenorm_check.py); the exact direct/relay decomposition of the attention writes together with uniform and per-head dilution compensation (real_mech.py); the trade-off between retention and perplexity with a placebo control (tradeoff_real.py); and the single- and multi-turn instruction-adherence tests (behavior_test.py, multiturn_test.py). The toolkit, together with the exact real-corpus prompt pairs, will be released publicly.
References
- [1]Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.arxiv.org/abs/2005.00928
- [2]Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João G. M. Araújo, Alex Vivitvskiy, Razvan Pascanu, and Petar Velicković. Transformers need glasses! information over-squashing in language tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2406.04267
- [3]Federico Barbero et al. Why do LLMs attend to the first token?, 2025.arxiv.org/abs/2504.02732
- [4]Stella Biderman et al. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2304.01373
- [5]Zhuo Chen, Oriol Mayné i Comas, Zhi Jin, Di Luo, and Marin Soljačić. L²M: Mutual information scaling law for long-context language modeling, 2025.arxiv.org/abs/2503.04725
- [6]Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2103.03404
- [7]Xiangming Gu et al. When attention sink emerges in language models: An empirical view. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.10781
- [8]Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.arxiv.org/abs/2004.10102
- [9]Niranjan Koley, Pranav Singhania, and Abhinav Bhatele. Power law guided dynamic sifting for efficient attention, 2025.arxiv.org/abs/2506.05300
- [10]Kenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Measuring and controlling instruction (in)stability in language model dialogs. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2402.10962
- [11]Henry W. Lin and Max Tegmark. Critical behavior in physics and probabilistic formal languages. Entropy, 19(7):299, 2017.DOI
- [12]Alžběta Marklová, Ján Milička, Marta Vokáčová, and Rudolf Rosa. Production and perception in LLMs: A token probability approach, 2026.arxiv.org/abs/2607.11703
- [13]Martin Meyer, Marco Michelessa, Christophe Chaux, and Vincent Y. F. Tan. How many different outputs can a transformer generate? In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2605.22223
- [14]Ken M. Nakanishi. Scalable-softmax is superior for attention, 2025.arxiv.org/abs/2501.19399
- [15]Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2206.03126
- [16]Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems, volume 29, 2016.arxiv.org/abs/1606.05340
- [17]Richard H. Price. Nonspherical perturbations of relativistic gravitational collapse. I. scalar and gravitational perturbations. Physical Review D, 5:2419–2438, 1972.DOI
- [18]Qwen Team. Qwen2.5 technical report, 2025.arxiv.org/abs/2412.15115
- [19]Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2017.arxiv.org/abs/1611.01232
- [20]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2309.17453





