Separating Thinking from Spelling for Efficient Generative Recommendation

Generative recommenders retrieve items by writing their semantic IDs (SIDs) code by code under trie-constrained beam search, an approach now deployed on large web platforms. Standard autoregressive interfaces pay for the SID length $K$ twice: a history of $T$ items enters the backbone as $TK$ code tokens, and every beam-search step runs through the backbone, so returning $B$ candidates costs $O(BK)$ backbone decode tokens per user. The second cost often dominates, and it grows with the backbone, the SID depth and the candidate budget.

We introduce Deco (decoding from concepts), which separates reading the history from writing the identifier. The backbone reads each history item as one fused token and condenses the history once into a compact next-item concept; a light KV-cached decoder conditioned on the concept and the generated prefix runs the entire beam search, while the identifiers, the trie and the autoregressive factorization stay unchanged. The decoder approaches full-backbone accuracy with cross-entropy alone, and distillation improves it further. On seven datasets from e-commerce, movies and music streaming, Deco achieves comparable or better accuracy than both a standard generative recommender and the full-backbone teacher with fused inputs, while delivering 3.2–8.6× their throughput under a 50 ms p95 latency budget in a production-style serving pipeline. Delegating beam search makes larger candidate pools and models practical: Deco generates 1000 candidates faster than its teacher generates 100, and with a 1.5B-parameter LLM backbone, it matches the 69M-parameter teacher’s throughput while achieving higher accuracy.

CCS Concepts • Information systems → Recommender systems.

Keywords recommender systems; generative recommendation; generative retrieval; semantic IDs; efficient inference

ACM Reference Format: Danil Gusak, Anna Volodkevich, Ivan Oseledets, Evgeny Frolov. 2026. Separating Thinking from Spelling for Efficient Generative Recommendation. In . ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

This work is licensed under a Creative Commons Attribution 4.0 International License. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn

Introduction

Recommendation shapes how people find products, films and music on the Web, and every request begins with candidate retrieval: selecting a few hundred to a few thousand items from a large catalog within tens of milliseconds. Generative retrieval casts this stage as sequence generation [37, 44]. Each item receives a semantic ID (SID), a short tuple of discrete codes derived from its content; SIDs let similar items share code prefixes and keep the output vocabulary far smaller than the catalog. A sequence model reads the user’s history and generates SIDs code by code using beam search constrained by a trie of valid SIDs. The SIDs retained at the final step identify the candidate items. Generative recommenders (GR) inherit the architectures and scaling practices of language models [30, 59, 60], already serve industrial traffic [3, 61] and increasingly build on pretrained LLMs with billions of parameters [10, 23].

SID-based retrieval has two sources of inference cost: encoding the history and decoding candidates. In a flat representation (FlatGR, Figure 1a), a history of $T$ items with $K$ codes each occupies $TK$ tokens. Fusing each item’s code embeddings into one token (FusedGR, Figure 1b) reduces the input to $T$ tokens. Both models encode the history once during prefill, but use the full backbone for autoregressive (AR) decoding. Beam search of width $B$ over $K$-level SIDs requires $O(BK)$ backbone decode tokens per user, each attending to the cached history. In our measurements at beam width 100, beam search takes up to 91% of generation time, even with a beam-shared history cache (§ 4.5.2). This cost grows with the SID depth, the candidate budget and the backbone size, so input compression alone does not remove this decoding bottleneck.

Our starting point is to separate modeling the user’s preferences from spelling out candidate SIDs. FlatGR and FusedGR use the same backbone to represent the history and score the next code for every candidate prefix, spending $O(BK)$ full-backbone decode tokens per user. Our experiments show that a compact next-item representation from the backbone lets a much smaller decoder generate SIDs without sacrificing recommendation quality (§ 4.5.1). Preference modeling can thus remain in the backbone, while a light decoder handles every step of candidate generation.

We propose Deco (Decoding from concepts), which makes this division explicit (Figure 1c). The backbone reads the fused history once and emits a compact next-item concept shared by every candidate in the beam. It delegates the entire trie-constrained beam search to a light KV-cached decoder conditioned only on this concept and the generated prefix. The SID codes and trie stay unchanged. The decoder can be trained with cross-entropy alone, and distillation from a full-backbone model improves its accuracy. For repeat-heavy domains such as music streaming, a light pointer over the history retrieves the exact items a user returns to. The light decoder returns final candidates without the backbone verification used in speculative decoding [28], and unlike parallel or single-pass decoders [14, 27], it keeps every step autoregressive and trie-constrained.

Our contributions are as follows:

• We show that SID-based generative retrieval often spends more time spelling identifiers than reading the history: beam-search decoding dominates prefill in both FlatGR and FusedGR. Fusing history items reduces prefill cost, but beam search still requires $O(BK)$ full-backbone decode tokens per user.

• We propose Deco, which combines fused history representations with delegated decoding. The backbone produces a compact next-item concept, and a light decoder runs the entire trie-constrained beam search using only this concept and the generated prefix. Training supervises every next-item transition in a single backbone pass.

• On seven datasets, concept decoding matches or exceeds the accuracy of the corresponding full-backbone baselines for both flat and fused histories. In a production-style pipeline, Deco delivers 2.8–5.1× FusedGR’s throughput at batch size 128 and 3.2–7.3× under a 50 ms p95 latency budget.

• We show that the speedup grows with the candidate budget, SID depth and backbone size. Deco extends to pretrained LLM backbones, reaching a 16× speedup over FusedGR with a 3B model.

Related Work

Generative recommendation with semantic IDs. Generative retrieval represents each document or item by a short sequence of discrete tokens and retrieves it by decoding this sequence [42, 44]. TIGER [37] brought this idea to sequential recommendation with SIDs from residual quantization [25, 58]. Later work improved item tokenization [5, 15, 26, 29, 51, 63], combined generative and dense retrieval [54], adapted pretrained LLMs to SID-based recommendation [2, 10, 23, 60], and scaled generative recommenders to industrial traffic [3, 61]. These works develop item representations and recommendation models; we focus on the computation repeated during candidate decoding. Full-backbone SID recommenders, whether they read the history as a flat sequence of SID code tokens or as one fused token per item [64], generate the codes either in the backbone or, as in encoder–decoder models such as TIGER and OneRec [3], in a decoder of comparable or larger size that cross-attends to the encoded history at every step. OneRec-V2 [61] drops the encoder and lets a large decoder read the context through shared keys and values, cutting the cost of reading; writing, which we target, still runs through the large model. As generative recommenders grow, adopt large decoder-only LLMs, and reach industrial scale, reducing repeated decoding work becomes important for meeting latency budgets.

Efficient decoding for generative recommendation. The cost of beam search is a recognized obstacle to deployment [28, 48]. Serving optimizations include early sorting termination, valid-item filtering and overlap between computation and data transfers [41]. Speculative decoding verifies a small model’s drafts with the backbone [28]; NEZHA uses an autoregressive head over per-level backbone states and filters invalid SIDs afterwards [52]. RPG generates unordered codes in parallel and searches a graph [14], SETRec generates order-agnostic identifiers [27], and LLaDA-Rec uses discrete diffusion [40]. Deco instead separates reading from writing: it runs the entire autoregressive beam search in a light decoder conditioned on a fixed-size, cacheable next-item concept produced by the heavy backbone. Wider beams and deeper SIDs increase decoder computation without further backbone calls. Unlike the parallel and diffusion-based methods above, Deco retains left-to-right autoregressive decoding with trie constraints, ensuring that every retained prefix extends to a catalog item. Among the generative baselines evaluated in Table 3, Deco achieves the highest recommendation accuracy on both datasets and higher throughput than the autoregressive models.

Model	Sports N@10	Sports R@100	Sports users/s, b128	BPC-23 N@10	BPC-23 R@100	BPC-23 users/s, b128
TIGER (KV)	0.0126 ±₀.₀₀₀₈	0.1099 ±₀.₀₀₂₈	523	0.0110 ±₀.₀₀₀₇	0.0804 ±₀.₀₀₂₃	521
LLaDA-Rec	0.0127 ±₀.₀₀₀₃	0.0914 ±₀.₀₀₁₁	46	0.0088 ±₀.₀₀₀₃	0.0627 ±₀.₀₀₀₉	46
RPG$_{\mathrm{graph}}$	0.0161 ±₀.₀₀₀₆	0.1002 ±₀.₀₀₄₀	573	0.0086 ±₀.₀₀₀₁	0.0444 ±₀.₀₀₀₃	461
RPG$_{\mathrm{full}}$	0.0163 ±₀.₀₀₀₁	0.1027 ±₀.₀₀₄₂	21,854	0.0099 ±₀.₀₀₀₁	0.0594 ±₀.₀₀₀₃	5,706
NEZHA	0.0135 ±₀.₀₀₀₈	0.1165 ±₀.₀₀₁₉	2,923	0.0120 ±₀.₀₀₀₆	0.0822 ±₀.₀₀₂₂	2,864
SASRec⁺	0.0184 ±₀.₀₀₀₈	0.1332 ±₀.₀₀₁₁	46,757	0.0104 ±₀.₀₀₀₄	0.0787 ±₀.₀₀₁₇	65,922
Deco	0.0191 ±₀.₀₀₀₇	0.1301 ±₀.₀₀₂₃	6,238	0.0132 ±₀.₀₀₀₃	0.0905 ±₀.₀₀₀₅	6,132

Global–local generation, distillation and copying. Global–local language models use a global model over patches or blocks and a local token decoder [12, 56]; related designs appear in deep-encoder, shallow-decoder translation [19] and concept-level language modeling [4, 24]. Deco applies this pattern to SID retrieval, where returning many candidates typically requires wider beams than in LLM text generation [41]. Sharing one concept across the candidate prefixes keeps this repeated work in the light decoder. To improve retrieval accuracy, we use distillation [11, 20, 43], which can let equal-capacity students surpass their teachers [6] and acts as teacher-informed label smoothing in our level-wise objective [57]. To handle repeat consumption, we add a history pointer, following copy mechanisms [39, 46] and repeat-aware recommendation [38]; its mixture with the generator factorizes over the SID trie.

Method

Semantic-ID Generation and Its Cost

Semantic IDs. A tokenizer maps every item $i$ to a semantic ID $s(i) = (c_1^i, \ldots, c_K^i)$ of $K$ codes from $\{1, \ldots, V\}$ by residual quantization of a content embedding, so the first code places the item in a coarse cluster and later codes refine it [37, 62]. Catalog SIDs form a prefix tree $\mathcal{T}$, and $\mathcal{T}(\pi)$ is the set of valid next codes after a prefix $\pi$. Given a history $H = (i_1, \ldots, i_T)$, a generative recommender factorizes the next-item distribution over code levels,

and retrieves candidates by beam search of width $B$ over $\mathcal{T}$: at every level, each surviving prefix is extended by its valid codes, and the $B$ best extensions are kept.

Flat autoregressive decoding. FlatGR represents the history as a flat sequence of $TK$ code tokens, following TIGER [37] and many subsequent SID models. We use a decoder-only backbone [10, 61] to read this sequence and autoregressively generate the next item’s $K$-code SID. The prefill encodes the $L_{\mathrm{in}}$ history tokens once and caches their keys and values. Beam search then reads the first level from the prefill output, and each of the other $K-1$ levels passes $B$ new tokens through the backbone, each attending to the cached history. To leading order, the compute per user, which bounds throughput under load, is

where $F_{\mathrm{bb}}$ is the cost of the prefill and $D_{\mathrm{bb}}$ is the cost of one backbone decode token, which grows with the history length it attends to. For FlatGR, $L_{\mathrm{in}} = TK$: the model pays for the SID length twice, in what it reads and, multiplied by the beam, in what it writes.

Fused History

A flat history represents each past item with $K$ code tokens. Following prior work on item-level SID representations [14, 64], FusedGR instead uses one token per history item. A small MLP, the fuser, combines the code embeddings into a single item vector:

This shortens the input from $TK$ to $T$ tokens. Dense training is native to a flat model, since every code token is a next-token target; a fused input contains no code tokens, so the codes of the next item must be supplied. On a training sequence $(i_1, \ldots, i_{T+1})$, we append, for every position $t$, the gold prefix of $i_{t+1}$ under a block-causal mask, so that its tokens attend only to $x_1, \ldots, x_t$ and causally to one another. One forward pass over $TK$ tokens, as in a flat model, shares the history computation across targets and evaluates

Fusion, however, changes only what the backbone reads: with $L_{\mathrm{in}} = T$ in (2), the $O(BK)$ decode tokens remain.

Decoding from Concepts

We separate user modeling from SID generation: the backbone produces a next-item concept from the history, and a light decoder generates SIDs conditioned on this concept. With the flat history of FlatGR, Deco-flat isolates the effect of delegating beam search. With fused history, Deco also reduces prefill cost (Figure 1c). The backbone $f_{\theta}$ of width $d$ reads the history once, and a linear map turns its last state $h_T$ (for a flat history, the state at the last code token) into a next-item concept of one or a few vectors ($M=2$ in our experiments; $M$ from 1 to 8 performs similarly):

The concept does not commit to one item: it conditions a distribution over plausible next items, which a small causal transformer $g_{\phi}$ enumerates code by code. The decoder reads the concept, mapped to its width by an MLP adapter $u = \psi(z)$, followed by its own level-specific embeddings $e_j(c_j^i)$ of the codes generated so far. Level-wise heads $W_k$ then score the valid children of the prefix:

The concept is the only interface between the two stages: the decoder never attends to the history and keeps no backbone cache. Beam search therefore runs entirely inside $g_{\phi}$ with its own KV cache (Algorithm 1). Ignoring trie masking and top-$B$ selection, which all models share, the compute per user is

where $F_{\mathrm{dec}}$ and $D_{\mathrm{dec}}$ are the prefill and decode costs of the decoder (by default two layers of width 256, against the backbone’s eight of width 640), which attends to at most $M+K-1$ tokens. A larger backbone or a longer history increases Deco’s one-time encoding cost, while wider beams and deeper SIDs increase computation in the light decoder. As the beam width $B$ grows for a fixed $K$, the compute speedup approaches

a ratio that grows with the backbone size and the history length; a longer history, however, mainly lengthens the prefill, which both pay once.

The concept as a serving state. Since the decoder sees the user only through $z$, its adapted form $u = \psi(z)$ is a complete per-user state for the generator: 1 KiB in bf16 for any backbone, against 0.4–7.8 MiB for the history KV cache of a full-backbone model (Figure 10, Appendix A). This state can be computed in advance

Algorithm 1 Retrieval with Deco for one user
Input: history $(i_1,\ldots,i_T)$, beam width $B$, SID trie $\mathcal{T}$
1: $x_t \leftarrow \operatorname{MLP}([E_1(c_1^t);\ldots;E_K(c_K^t)])$ for $t=1,\ldots,T$ ▷ {xref}`Eq. (3)`
2: $z \leftarrow \operatorname{reshape}(W_z f_\theta(x_{1:T}^T)),\quad u \leftarrow \psi(z)$ ▷ the only backbone pass
3: prefill $g_\phi$ with $u_1,\ldots,u_M$; $\mathcal{B}\leftarrow\{(\varnothing,0)\}$
4: for $k=1,\ldots,K$ do
5: $\mathcal{B}\leftarrow\operatorname{top}_B\{(\pi c,r+\log p_\phi(c\mid\pi,z)):(\pi,r)\in\mathcal{B},c\in\mathcal{T}(\pi)\}$
6: if $k<K$ then
7: feed the last code of each beam to $g_\phi$, reusing the decoder KV cache of its parent
8: end if
9: end for
10: return the SIDs in $\mathcal{B}$ ordered by score $r$, resolved to items

when the history changes, leaving only the light decoder online, although our timings include recomputation for every request.

Training the Light Decoder

Learning from data. The backbone, the fuser and the decoder are trained jointly and densely: a concept $z_t$ is read at every position $t$ (Eq. (5) at $h_t$), and the decoder predicts the codes $c_\star^k$ of $i_{t+1}$ for all $t$ in parallel, as $T$ short sequences of $M+K-1$ tokens. With cross-entropy alone, $\mathcal{L}_{\mathrm{CE}}=-\sum_t\sum_k\log p_\phi^k(c_\star^k)$, where $p_\phi^k=p_\phi(\cdot\mid c_\star^{<k},z_t)$, Deco needs nothing beyond interaction data; we call this teacher-free variant Deco-CE. A full-backbone model can provide a distribution over valid continuations, giving the light decoder richer supervision than hard next-item labels alone.

Learning from a full-backbone model. To further improve retrieval quality, we regularize the decoder with knowledge distillation (KD) [11, 57]. Frozen full-backbone teachers attend to the history at each code level: FusedGR for Deco and FlatGR for Deco-flat. The decoder learns from both the observed code and the teacher’s next-code distribution $q^k$ along the gold prefix $c_\star^{<k}$.

where $\tau$ is a temperature, both distributions are restricted to $\mathcal{T}(c_\star^{<k})$, and the weights $w_{\mathrm{ce}}$ and $w_{\mathrm{kd}}$ are fixed once from the initial loss values (Appendix C).

With cross-entropy $\ell(y,p)=-\sum_c y_c\log p_c$ and $\tau=1$, since $\operatorname{KL}(q\Vert p)=\ell(q,p)-\ell(q,q)$, the objective is, up to a constant,

with $\alpha=w_{\mathrm{kd}}/(w_{\mathrm{ce}}+w_{\mathrm{kd}})$: a single cross-entropy against a target that mixes the teacher’s belief with the observed code. Distillation thus acts as teacher-informed label smoothing [57]: instead of spreading uniformly over all codes, the smoothing mass goes to the trie-valid continuations of the same gold prefix that the teacher finds plausible. The observed code keeps a point mass of $1-\alpha$, so the student is guided by the teacher without being bound to it where the two disagree; the teacher is needed only for training.

Copying from the History

A concept summarizes what comes next, not which past items the user has seen. In domains such as music streaming, however, users often return to items they have consumed before [38, 45]. Such a target is easy to point to but hard to generate: the model must know which past item to repeat, and a fixed-size concept is not meant to carry the identities of a hundred earlier items. Following pointer-generator networks [39, 46], we add a copy pointer over the history as a second distribution over SIDs,

where $a_t$ is the softmax attention weight from the last backbone state $h_T$ to the state of history item $i_t$, and $\lambda=\operatorname{sigmoid}(w_\lambda^T h_T)$ is a learned gate. The pointer is a single attention read over the history, like one cross-attention head, except that its weights are used directly as a distribution over past items rather than to mix their values. Both components live in the same SID space, so the mixture factorizes over the trie exactly. With $p_\phi(\pi)=p_\phi(\pi\mid z)$ the generator probability of a prefix $\pi$ and $A(\pi)$ the attention mass of the history items whose SID starts with $\pi$,

Beam search thus runs unchanged on the mixture, and training maximizes its exact likelihood, with the distillation term of Eq. (9) applied to the full mixture. The pointer is computed once from the backbone states and keeps only the history SIDs, their attention weights and the gate, so the decoder still never attends to the history. We use this pointer with both the full-backbone baselines and Deco on Yambda (§4.2.4).

Experiments

Our experiments address the following research questions:

RQ1 Does delegating the entire beam search to a light decoder preserve recommendation accuracy?

RQ2 Where does generation time go, and how does delegating beam search affect serving latency and throughput?

RQ3 How do the gains scale with the candidate budget, the SID depth, the history length and the backbone size?

RQ4 Is a compact concept a sufficient interface to the history, and where does it fall short?

RQ5 How large does the light decoder need to be, and how should it be trained?

Experimental Setup

Datasets

We use seven public datasets that differ in domain, scale, history length and repeat behavior (Table 1). Toys, Beauty and Sports come from Amazon Reviews 2014 [31], and BPC-23 (Beauty and Personal Care) and Books-23 from the much larger Amazon Reviews 2023 [13]. We use their published 5-core subsets as implicit feedback and describe items by their product metadata. MovieLens is a subset of MovieLens-25M [9], with movies described by title, genres, directors, actors, writers and plot. Yambda [35] is a music-streaming log: we take the last 30 days of Yambda-500M listens, remove exact duplicates, collapse consecutive repeats of a track [47], apply 5/20 user/item thresholds, and describe tracks by the released item embeddings. MovieLens and Yambda have long histories, and Yambda is also repeat-heavy: 29% of its test targets are among the user’s last 100 items.

Dataset	#Users	#Items	#Interact.	Avg. Len.	Median Len	Density (%)
Beauty [31]	22,363	12,101	198,502	8.88	6.0	0.073
Sports [31]	35,598	18,357	296,337	8.32	6.0	0.045
Toys [31]	19,412	11,924	167,597	8.63	6.0	0.072
BPC-23 [13]	729,576	207,649	6,624,441	9.08	7.0	0.004
Books-23 [13]	776,370	495,063	9,488,297	12.22	7.0	0.002
MovieLens [9]	110,456	10,870	16,193,689	146.61	69.0	1.349
Yambda [35]	87,806	79,197	12,818,976	145.99	173.0	0.184

Evaluation Protocol

We use a global temporal split [8, 47], which, unlike leave-one-out, restricts training to interactions that precede the evaluation periods. The latest 10% of interactions by timestamp define the test period and the latest 10% of the rest the validation period (on Yambda, the last day and the day before). Each user’s last interaction in a held-out period is the target. The input history includes earlier interactions both before and within that period [8]. Items the user has interacted with are not filtered from recommendations: repeats are legitimate targets, and in production such filtering typically follows retrieval. We report NDCG@$k$ (N@$k$) and Recall@$k$ (R@$k$) over the full training catalog with beam width 100 for $k \in\{10, 50, 100\}$, and up to $k = 1000$ in the candidate-budget study; main results are means and standard deviations over three seeds [7].

Efficiency is measured on one H100 GPU. Our six models use a production-style pipeline with bf16 weights, compilation, CUDA graphs and a beam-shared history cache (Appendix § 10). Separately, since external baselines have different retrieval implementations, Table 3 times them and Deco in a common eager PyTorch [33] execution setup, each through its native retrieval path. Latency is the GPU time for prefill and beam search per batch; throughput is batch size divided by median latency. We report throughput at the peak of a batch-size sweep, under a 50 ms 95th-percentile (p95) latency budget, and at batch 128, the largest where all full-backbone baselines meet that budget, near their peak (Appendix § 10).

Models and Baselines

Six models share the backbone family, the SIDs and the training budget: FlatGR, a decoder-only TIGER-style model [37]; FlatGR-match, which truncates the flat history to $T/K$ items to read as many tokens as the fused models; FusedGR, which combines each history item’s code embeddings into a single input token; Deco, with delegated decoding and distillation from FusedGR; Deco-flat, its flat-history variant distilled from FlatGR; and Deco-CE, trained without a teacher using cross-entropy alone.

On Sports and BPC-23, a small and a large catalog, we also compare with SID-based generative recommenders tuned under our protocol and trained from scratch: TIGER [37] with a KV cache, LLaDA-Rec [40], RPG [14] with both graph-based candidate search and full-catalog scoring, and NEZHA [52]. We further include SASRec⁺ (SASRec [18] with full cross-entropy), a state-of-the-art sequential recommender [22, 32, 34] that scores the whole catalog instead of generating. Finally, we compare with recent SID-based recommenders on the leave-one-out Beauty benchmark, using the results reported in the literature.

Implementation Details

SIDs have $K = 4$ levels of $V = 2048$ codes, built by RQ-KMeans from multilingual E5-large text embeddings [49, 50] or from the provided 128-dimensional item embeddings on Yambda, with popularity-based collision resolution [62]. The backbone is an 8-layer Qwen2-style decoder [53] of width 640 with 10 heads, and the light decoder has 2 layers of width 256 with 4 heads. The fuser embeds codes and mixes them with a two-layer MLP. We keep input and output code tables untied for flexibility: input representations can include extra item features without changing the output classes; tying gives small gains for most models but does not change the comparison (Appendix § 9). The history is truncated to the last $T = 20$ items on the Amazon datasets and $T = 100$ on MovieLens and Yambda. We train with Adam [21] and a warmup-cosine schedule, tuning the learning rate on validation. We select checkpoints by validation N@10 at beam 10 and test at beam 100. On Yambda, all models use the copy pointer of §3.5. For the LLM backbones, we adapt Qwen2.5 [36] with LoRA [16]; before recommendation training, bidirectional metadata–SID grounding aligns the item representations with the pretrained model [23, 60], and LoRA is merged into the backbone weights before measuring inference latency and throughput.

Results

RQ1: Recommendation Quality

Deco achieves comparable or better accuracy than both FlatGR and FusedGR, with 5.5–6.5× and 2.8–5.1× their throughput at batch 128, respectively. Table 2 separates the effects of history fusion and delegated decoding.

Model	NDCG@10	Recall@50	Recall@100	b128	peak	b128 / peak
FlatGR	0.0211 ± .0018	0.0960 ± .0154	0.1319 ± .0145	6,454	7,083	ref.
FlatGR-match	0.0216 ± .0027	0.0856 ± .0129	0.1178 ± .0163	7,922	8,558	×1.2 / ×1.2
Deco-flat	▲0.0308 ± .0009	▲0.1249 ± .0022	▲0.1671 ± .0036	22,362	29,737	×3.5 / ×4.2
FusedGR	0.0300 ± .0010	0.1147 ± .0029	0.1526 ± .0053	7,876	8,682	×1.2 / ×1.2
Deco-CE	0.0287 ± .0035	0.1081 ± .0027	0.1439 ± .0056	39,938	59,634	×6.2 / ×8.4
Deco	▲0.0352 ± .0009	▲0.1296 ± .0058	▲0.1718 ± .0049	39,853	59,451	×6.2 / ×8.4
FlatGR	0.0138 ± .0024	0.0747 ± .0088	0.1087 ± .0139	6,458	6,993	ref.
FlatGR-match	0.0143 ± .0030	0.0671 ± .0075	0.1050 ± .0070	7,917	8,889	×1.2 / ×1.3
Deco-flat	▲0.0184 ± .0018	▲0.0917 ± .0035	▲0.1307 ± .0069	22,418	29,234	×3.5 / ×4.2
FusedGR	0.0190 ± .0010	0.0841 ± .0051	0.1217 ± .028	7,872	8,531	×1.2 / ×1.2
Deco-CE	0.0173 ± .0006	0.0783 ± .0035	0.1166 ± .0055	39,924	59,682	×6.2 / ×8.5
Deco	▲0.0200 ± .0004	▲0.0912 ± .0052	▲0.1319 ± .0066	39,895	59,629	×6.2 / ×8.5
FlatGR	0.0132 ± .0005	0.0713 ± .0031	0.1053 ± .0036	6,392	7,004	ref.
FlatGR-match	0.0132 ± .0005	0.0676 ± .0022	0.1027 ± .0038	7,814	8,592	×1.2 / ×1.2
Deco-flat	▲0.0160 ± .0009	▲0.0786 ± .0031	▲0.1170 ± .0034	21,953	29,796	×3.4 / ×4.3
FusedGR	0.0160 ± .0003	0.0774 ± .0004	0.1134 ± .0038	7,765	8,547	×1.2 / ×1.2
Deco-CE	0.0164 ± .0004	0.0781 ± .0040	0.1164 ± .0035	38,878	59,632	×6.1 / ×8.5
Deco	▲0.0191 ± .0007	▲0.0900 ± .0013	▲0.1301 ± .0023	37,920	59,473	×5.9 / ×8.5
FlatGR	0.0881 ± .0015	0.2984 ± .0047	0.4301 ± .0077	3,069	3,165	ref.
FlatGR-match	0.0680 ± .0007	0.2602 ± .0011	0.3599 ± .0032	6,154	6,562	×2.0 / ×2.1
Deco-flat	▲0.0865 ± .0017	▲0.3051 ± .0014	▲0.4212 ± .0027	5,981	6,269	×1.9 / ×2.0
FusedGR	0.0887 ± .0016	0.3083 ± .0037	0.4134 ± .0054	6,113	6,604	×2.0 / ×2.1
Deco-CE	0.0808 ± .0013	0.2965 ± .0019	0.3967 ± .0095	20,090	24,622	×6.5 / ×7.8
Deco	▲0.0898 ± .0019	▲0.3157 ± .0010	▲0.4220 ± .0016	20,080	24,745	×6.5 / ×7.8
FlatGR	0.0136 ± .0001	0.0648 ± .0002	0.0907 ± .0008	6,410	6,958	ref.
FlatGR-match	0.0128 ± .0002	0.0624 ± .0012	0.0879 ± .0006	7,854	8,577	×1.2 / ×1.2
Deco-flat	▲0.0144 ± .0001	▲0.0675 ± .0005	▲0.0946 ± .0008	21,938	29,530	×3.4 / ×4.2
FusedGR	0.0129 ± .0001	0.0623 ± .0004	0.0886 ± .0003	7,810	8,497	×1.2 / ×1.2
Deco-CE	0.0127 ± .0001	0.0636 ± .0008	0.0894 ± .0007	38,393	55,533	×6.0 / ×8.3
Deco	▲0.0132 ± .0003	▲0.0635 ± .0002	▲0.0905 ± .0005	38,406	57,576	×6.0 / ×8.3
FlatGR	0.0131 ± .0003	0.0739 ± .0004	0.0955 ± .0004	6,337	6,983	ref.
FlatGR-match	0.0165 ± .0003	0.0671 ± .0002	0.0905 ± .0009	7,718	8,447	×1.2 / ×1.2
Deco-flat	0.0193 ± .0002	▲0.0747 ± .0004	▲0.1002 ± .0008	21,240	28,587	×3.4 / ×4.1
FusedGR	0.0196 ± .0004	0.0736 ± .0008	0.0973 ± .0010	7,682	8,750	×1.2 / ×1.3
Deco-CE	0.0178 ± .0001	0.0670 ± .0004	0.0900 ± .0007	35,885	55,910	×5.7 / ×8.0
Deco	▲0.0197 ± .0005	▲0.0753 ± .0010	▲0.1002 ± .0010	35,926	55,848	×5.7 / ×8.0
FlatGR	0.1746 ± .0008	0.3517 ± .0010	0.4144 ± .0022	2,943	3,023	ref.
FlatGR-match	0.0995 ± .0013	0.2553 ± .0028	0.3116 ± .0036	5,827	6,370	×2.0 / ×2.1
Deco-flat	▲0.1779 ± .0046	▲0.3588 ± .0018	▲0.4229 ± .0015	5,518	5,812	×1.9 / ×1.9
FusedGR	0.1674 ± .0008	0.3478 ± .0004	0.4125 ± .0025	5,688	6,151	×1.9 / ×1.9
Deco-CE	0.1647 ± .0017	0.3466 ± .0010	0.4099 ± .0016	16,073	19,782	×5.5 / ×6.5
Deco	▲0.1712 ± .0006	▲0.3544 ± .0006	▲0.4209 ± .0007	16,069	19,890	×5.5 / ×6.6

Fusion retains comparable quality. At equal item counts, FusedGR is 1.2–2.0× faster than FlatGR at batch 128 and raises NDCG@10 by 21–38% on Amazon-2014, while staying within about 5% on the four larger datasets. At equal token budgets and similar speed, it beats FlatGR-match, whose flat history covers only $T/K$ items, by 30% in NDCG@10 on MovieLens and 68% on Yambda. Fusion thus keeps more history per token, reducing history processing cost while retaining competitive accuracy.

Moving the search preserves accuracy. Deco exceeds FusedGR in NDCG@10 by 5–19% on Amazon-2014 and 0.5–2.3% on the four larger datasets. Compared with the standard generative baseline FlatGR, its NDCG@10 is higher on five datasets and less than 3% lower on BPC-23 and Yambda. Deco-flat also matches or exceeds FlatGR on every dataset. A single backbone read thus suffices to match a teacher that attends to the history at every code level. Deco-CE tests this interface without teacher supervision: it learns from the same observed labels as FusedGR, yet stays within 10% of it in NDCG@10 and 8% in Recall@100, and surpasses it on Sports and in BPC-23 recall. The compact interface thus supports accurate retrieval without teacher supervision. Distillation turns the remaining gap into a consistent gain (§4.2.5). Since Deco and Deco-CE share the same inference path, teacher supervision improves quality without adding backbone calls or decoder work.

Concept decoding helps either input. Neither input dominates: FusedGR leads on Amazon-2014 and MovieLens, FlatGR on BPC-23 and Yambda, and Books-23 is mixed. Deco-flat keeps FlatGR’s flat input and Deco keeps FusedGR’s fused input, so both gains come without changing the history representation. Speedups also compound across stages: on MovieLens, fusion makes generation 2.0× faster and concept decoding another 3.3×, 6.5× overall.

External baselines. In Table 3, Deco leads in NDCG@10 on both datasets and Recall@100 on BPC-23; on Sports, only SASRec⁺ has slightly higher Recall@100. Only two baselines are faster: parallel RPG with full-catalog scoring on Sports (15% lower NDCG@10) and SASRec⁺, which does not generate. On the larger BPC-23 catalog, Deco leads generative baselines in accuracy and speed and beats SASRec⁺ by 27% in NDCG@10 and 15% in Recall@100.

Leave-one-out. While our primary evaluation uses a global temporal split to avoid temporal leakage, we also report leave-one-out results on Beauty (Table 4) for comparison with published SID-based recommenders. Deco reaches 0.0481 NDCG@10 and 0.0839 Recall@10, retaining its advantage under this protocol.

Metric	TIGER [37]	LC-Rec [60]	LETTER [51]	ETRec [29]	DHier [5]	RPG [14]	Deco
N@10	0.0384	0.0397	0.0406	0.0415	0.0431	0.0464	0.0481
R@10	0.0648	0.0675	0.0708	0.0725	0.0752	0.0809	0.0839

RQ2: Serving Efficiency

Fusion shortens prefill by compressing the history, and delegated decoding makes candidate expansion cheaper by assigning it to the light decoder. On BPC-23 at batch 128, fusion cuts prefill from 4.4 ms in FlatGR to 1.5 ms in FusedGR, but beam search still accounts for 91% of generation time. Deco then cuts search from 14.9 to 2.1 ms, a 7× reduction. Decoding therefore limits fusion’s overall speedup, while delegation removes repeated backbone work from candidate expansion.

As the batch grows, (Figure 2), Deco’s p95 latency stays under 8 ms up to 128 users, whereas the full-backbone baselines saturate the GPU at a few hundred users per batch. Under a 50 ms p95 budget, Deco serves 7.1×, 3.9× and 3.2× more users per second than FusedGR on BPC-23, MovieLens and Yambda (6.5–8.6× more than FlatGR), and most of its curve lies at throughputs that no full-backbone model reaches. Thus, the accuracy gains in RQ1 come with greater serving capacity.

Our timings recompute the concept for every request, and on long histories, prefill becomes the main remaining cost. Precomputing the 1KiB concept whenever a history changes (§3.3) leaves only light decoding for online requests. With prefill excluded for both models, Deco decodes 4.8–7.7× faster than its teacher at batch 128 and up to 10.6× at peak.

RQ3: Scaling the Generation Cost Factors.

Candidate budget. Returning the top $k$ items requires a beam of at least $k$, and widening it from 10 to 100 costs Deco 34% of its throughput, against 73% for FusedGR and 61% for FlatGR (Figure 3a). At $k = 1000$, Deco still exceeds the throughput of FusedGR at $k = 100$ and recalls three times as many targets (0.267 vs. 0.088); against FusedGR at $k = 1000$, it is 8.8× faster at equal recall. Wide beams pay off because most targets leave the beam at the first two SID levels (Figure 3b; at $k = 100$, only 38% survive the first and 13% the second), and later levels cannot restore a pruned path. Light decoding makes wider beams affordable, allowing Deco to preserve more early SID branches and return more candidates. At fixed $k$, widening the beam beyond $k$ raises Recall@$k$ a little (Appendix § 8). All models lose targets at similar rates across levels, suggesting that the concept does not make early decisions worse.

SID depth. Longer SIDs describe items more finely. As we vary the depth from $K = 2$ to 8 with nested codebooks of 256 codes, small enough that every added level still splits items (Figure 4), Recall@100 on Sports rises up to about $K = 5$ and then levels off for all models except FlatGR-match, which degrades beyond $K = 4$ as its history shrinks to 20/$K$ items (§4.2.1). With the same compact concept, Deco is the most accurate at every depth, so the concept need not grow with the identifier. Each extra level lengthens both the input and the output of FlatGR, only the backbone decoding of FusedGR, and only the light decoding of Deco. From $K = 2$ to 8, Deco therefore slows down 2.9×, against 7.3× for FusedGR and 7.9× for FlatGR, and its speedup over its teacher grows from 2.7× to 6.8×. Extra levels never reach the backbone, which makes deeper SID hierarchies, such as PLUM’s [10], cheaper to serve.

History length. A longer history lengthens the prefill for both Deco and its teacher: from $T = 20$ to $T = 100$, Deco’s speedup over the teacher falls from 4.7–5.1× to 2.8–3.3× but stays at 5.5–6.5× over FlatGR (Table 2). Excluding prefill, it decodes 6.7–7.7× faster than its teacher (4.8× with copying), while maintaining accuracy on both long-history datasets. Concept caching is therefore especially valuable for long histories.

Backbone size and LLMs. On BPC-23, we scale the backbone from 4M to 69M parameters (S, M, L; trained from scratch) and on to Qwen2.5 with 0.5B, 1.5B and 3B parameters (Figure 5), keeping the same two-layer decoder. From L onward, Deco stays ahead of its teacher, and the margin widens with scale (0.0147 vs. 0.0138 NDCG@10 at 3B), so a fixed-size concept does not cap what a larger backbone contributes. From S to 3B, FusedGR slows down 258× (73k to 281 users/s) as each beam step uses the larger backbone, whereas Deco, whose backbone runs once, slows down 12× (52k to 4.5k). All of Deco’s added cost is in prefill: decoding stays at 58–59k users/s across backbones. The speedup grows from 4.8× at L to 9.7×, 14× and 16× at 0.5B, 1.5B and 3B. Deco is slower only with backbone S, whose two layers of width 128 are lighter than the decoder, as Eqs. (2) and (7) predict. Shrinking FusedGR’s backbone is no substitute for delegating the search: FusedGR with the 20M backbone is both slower and less accurate than Deco with the 69M backbone L. With a 22× larger 1.5B backbone, Deco is as fast as the 69M FusedGR and 11% more accurate in NDCG@10 (Figure 9, Appendix A). Deco thus scales more efficiently and transfers to pretrained LLMs. Larger backbones improve retrieval quality without increasing the cost of candidate search.

RQ4: Is a Compact Concept Enough?

Richer access to the history gives no consistent gain. In Table 5, we replace the concept with richer views of the history: all $T$ item states as a decoder prefix, the same states read by cross-attention in every decoder layer, as in the encoder–decoder access of TIGER and OneRec, and for Deco-flat, cross-attention to all $TK$ SID-token states. NDCG@10 moves by less than 5%, more often down than up, and Recall@100 gains at most about 3%, while latency grows by up to 20% and the per-user state from $O(1)$ to $O(T)$ or $O(TK)$. On these low-repeat datasets, once the backbone has condensed the history into an expectation of the next item, the individual items add little that the decoder can use, so a decoder that never sees the history is both cheaper and accurate enough to serve.

Decoder input	Input size	Sports N@10	Sports R@100	BPC-23 N@10	BPC-23 R@100	ms, b128 ↓
Last state	O(1)	0.0191 ± 0.0007	0.1301 ± 0.0023	0.0132 ± 0.0003	0.0905 ± 0.0005	3.32
All item states, as prefix	O(T)	0.0185 ± 0.0004	0.1293 ± 0.0016	0.0134 ± 0.0003	0.0911 ± 0.0007	3.40
All item states, cross-attn.	O(T)	0.0186 ± 0.0005	0.1285 ± 0.0033	0.0132 ± 0.0003	0.0913 ± 0.0009	3.97
Last state	O(1)	0.0160 ± 0.0009	0.1170 ± 0.0034	0.0144 ± 0.0001	0.0946 ± 0.0008	5.85
All item states, as prefix	O(T)	0.0155 ± 0.0017	0.1198 ± 0.0052	0.0143 ± 0.0001	0.0948 ± 0.0005	5.91
All item states, cross-attn.	O(T)	0.0153 ± 0.0006	0.1179 ± 0.0062	0.0144 ± 0.0002	0.0951 ± 0.0013	6.50
All SID-token states, cross-attn.	O(TK)	0.0154 ± 0.0002	0.1207 ± 0.0040	0.0146 ± 0.0001	0.0962 ± 0.0006	6.59

Repeat consumption. Without a pointer (light bars in Figure 6), the fused models trail FlatGR on repeat-heavy Yambda by a wide margin: 0.093 NDCG@10 for FusedGR and 0.079 for Deco against 0.153. More than 90% of the gap comes from repeat targets, while on new targets all four generators and SASRec+ are close (0.036–0.047 NDCG@10). The flat history exposes the exact codes of every past track for FlatGR’s attention to copy; fused tokens mix them, and a compact concept encodes what kind of track comes next rather than which past track to replay. The item-level SASRec+ shows the same split: it matches every generator on new targets but falls far behind FlatGR on repeats, the source of most of FlatGR’s lead, so on repeat-heavy data like Yambda, comparing SID generators with item-ID models largely measures copying ability.

With the pointer of §3.5 (dark bars), every model reaches about 0.90 Recall@100 and 0.48–0.50 NDCG@10 on repeat targets, while metrics on new targets change by at most 0.012. Deco’s NDCG@10 more than doubles, from 0.079 to 0.171, and the four models end up within 0.011 Recall@100 of each other, with Deco the highest at 0.421. The gate learns when to point without being told: it is computed from the history alone, yet its mean, 0.27, is close to the share of repeat targets, 0.29. It averages 0.63 before repeat targets and 0.12 before new ones (AUC 0.90), and on repeat targets 86% of the target’s probability comes from the pointer; all full-history generators learn nearly the same gate. The concept and pointer thus divide the work like semantic and episodic memory: one decides what kind of item comes next, the other retrieves the exact items the user returns to, and neither needs the backbone during search.

Sizing and Training the Light Decoder

A small decoder suffices. Decoders with 1–4 layers of width 128–512 (3.5M to 21.6M parameters; Appendix A) stay within about 3% of the default in NDCG@10 and Recall@100 on Sports and BPC-23, and a single layer cuts latency by 20–28%: once the concept is given, spelling it out is a small job. Our default is thus conservative; a smaller decoder offers a further latency–quality trade-off.

KD adds a consistent gain. Distillation raises N@10 over Deco-CE on every dataset (Table 2), and Figure 7 shows why. With cross-entropy alone, the decoder overfits on Sports (validation N@10 peaks at 0.023 after 1.2k steps, then declines) and plateaus early at 0.094 on MovieLens. With distillation, training is stable and reaches a higher level, 0.027 on Sports and 0.101–0.103 on MovieLens. KD alone nearly matches CE+KD. As teacher-informed label smoothing (see Eq. (10)), distillation should help most when labels are scarce; indeed, it adds 16–23% N@10 on the small Amazon-2014 datasets but 4–11% on the four larger ones. We also train a full-backbone FusedGR student with the same KD objective, but its accuracy gains are smaller and it remains less accurate than Deco (Appendix A). What matters for serving is that KD makes a far cheaper decoder at least as accurate as the model it replaces.

Teacher initialization is optional. A student initialized from the teacher’s backbone and fuser reaches comparable quality in fewer steps (Figure 7, green curves), which can shorten training; our main comparisons use independently initialized students.

Conclusion

Generative recommenders that run beam search in the backbone often spend most of their computation spelling identifiers rather than modeling user preferences. Deco separates these tasks: the backbone reads the history once into a next-item concept and delegates the entire beam search to a light decoder. Across seven datasets, it matches or exceeds its full-backbone teacher’s accuracy and serves 3.2–7.3× more users under a 50 ms p95 latency budget. Taking the backbone out of candidate expansion makes generative retrieval affordable even with LLM-scale backbones and allows thousand-candidate pools, common before reranking in dense retrieval. Refreshing the cached concept before requests arrive removes the backbone from online generation. Together, these results show that the user-modeling backbone can scale without increasing the cost of candidate search.

References

Additional Results

Accuracy against throughput per dataset. Figure 8 places every model of Table 2 in the accuracy–throughput plane. The arrows connect each full-backbone teacher with its concept-decoder student. On all seven datasets both arrows point to the right and upwards: concept decoding moves a model to higher throughput and higher Recall@100 at the same time. The two other variants fall short in different directions. FlatGR-match is fast but loses accuracy, and Deco-CE is as fast as Deco but less accurate.

Quality–throughput trajectories. Figure 9 combines both panels of Figure 5. As the backbone grows, FusedGR moves almost horizontally to the left, buying small accuracy gains with large losses in throughput. Deco climbs higher and slows down far less ($12\times$ against $258\times$ from S to 3B), so every larger backbone gives a more accurate model at moderate cost.

Per-user serving state. Figure 10 compares the state that must be kept per user to skip history encoding at request time: the KV cache of a full-backbone model grows with the history and its tokenization, whereas the Deco concept stays at 1 KiB.

Candidate budget. Table 6 complements Figure 3. A beam wider than $k$ adds little: about 3–4% to Recall@100 and 2% to Recall@300, and widening it from 300 to 1000 barely changes Recall@100.

Beam width	Recall@100			Recall@300
	$B = 100$	$B = 300$	$B = 1000$	$B = 300$	$B = 1000$
FlatGR	ref.	+2.9%	+2.9%	ref.	+2.0%
FusedGR	ref.	+3.6%	+3.6%	ref.	+1.7%
Deco	ref.	+3.7%	+3.8%	ref.	+1.9%

Decoder size. Table 7 details the decoder-size study of §4.2.5. Across a sixfold range of decoder parameters, NDCG@10 and Recall@100 stay within about 3% of the default, while latency grows with depth and width.

Layers × width	Params	Sports N@10	Sports R@100	BPC-23 N@10	BPC-23 R@100	Latency ↓
1 × 128	3.5M	0.0192 ±.0008	0.1297 ±.0011	0.0131 ±.0002	0.0899 ±.0007	×0.72
2 × 128	3.8M	0.0194 ±.0011	0.1308 ±.0021	0.0132 ±.0000	0.0901 ±.0005	×0.84
1 × 256	7.6M	0.0186 ±.0008	0.1276 ±.0018	0.0133 ±.0001	0.0909 ±.0003	×0.80
2 × 256	8.6M	0.0191 ±.0007	0.1301 ±.0023	0.0132 ±.0003	0.0905 ±.0005	ref.
3 × 256	9.7M	0.0194 ±.0012	0.1306 ±.0017	0.0133 ±.0002	0.0905 ±.0006	×1.20
4 × 256	10.7M	0.0191 ±.0010	0.1300 ±.0001	0.0133 ±.0002	0.0899 ±.0008	×1.39
2 × 512	21.6M	0.0186 ±.0016	0.1260 ±.0020	0.0131 ±.0002	0.0899 ±.0004	×1.46

Distillation without the concept decoder. To separate the concept decoder from distillation, we also distill the FusedGR teacher into a FusedGR student with the objective and loss weighting of Deco (FusedGR + KD, Table 8). Its NDCG@10 gain over the teacher

Model	Sports N@10	Sports R@100	BPC-23 N@10	BPC-23 R@100
FusedGR (teacher)	0.0160 ±.0003	0.1134 ±.0008	0.0129 ±.0001	0.0886 ±.0003
FusedGR + KD	0.0172 ±.0014	0.1196 ±.0005	0.0130 ±.0004	0.0896 ±.0005
Deco-CE	0.0164 ±.0004	0.1164 ±.0035	0.0127 ±.0001	0.0894 ±.0007
Deco	0.0191 ±.0007	0.1301 ±.0023	0.0132 ±.0003	0.0905 ±.0005

is less than half that of Deco: 7.5% vs. 19.4% on Sports and 0.8% vs. 2.3% on BPC-23. Since the concept decoder alone (Deco-CE) stays close to FusedGR, the two reinforce each other: the teacher’s graded preferences are worth more to the light decoder than to a full-backbone student.

Tied and Untied Code Tables

Code vectors play up to three roles: they describe history items (through the fuser or as flat input tokens), they condition the decoder on the generated prefix, and they score the next code as output class vectors. All main experiments keep separate tables for these roles, the more general choice: a code can have different learned representations for reading and writing, and extra item features can enter the input side without changing the output classes. In Table 9, we tie them into one table per level, from which the history fuser reads RMS-normalized rows. Tying raises mean Recall@100 for FlatGR, FusedGR and Deco on both datasets, while the effect on Deco-CE is mixed. The Content column shows what tying changes. First-level codes are clusters of similar items, yet in the untied fused models the ten nearest neighbors of a code among the history-side vectors barely overlap with its ten nearest clusters, and these vectors keep the norm of their random initialization. The output class vectors, in contrast, recover the content neighborhoods well above chance (8–11% on Sports, 39–41% on Yambda), and tying shares them with the history side; FlatGR inputs partially recover these neighborhoods even with untied tables. The main comparison does not depend on this choice: Deco is the most accurate of the four models with both tied and untied tables.

Model	Sports Untied	Sports Tied	Sports Content, %	Yambda Untied	Yambda Tied	Yambda Content, %
FlatGR	$0.1053 \pm.0036$	$0.1128 \pm.0030$	$6.1 \to10.4$	$0.4144 \pm.0022$	$0.4162 \pm.0012$	$25.9 \to41.3$
FusedGR	$0.1134 \pm.0008$	$0.1212 \pm.0021$	$0.7 \to10.6$	$0.4125 \pm.0025$	$0.4183 \pm.0015$	$1.5 \to46.1$
Deco-CE	$0.1164 \pm.0035$	$0.1129 \pm.0044$	$0.7 \to9.9$	$0.4099 \pm.0016$	$0.4128 \pm.0012$	$1.5 \to43.7$
Deco	$0.1301 \pm.0023$	$0.1317 \pm.0004$	$0.7 \to12.6$	$0.4209 \pm.0007$	$0.4254 \pm.0005$	$1.6 \to45.7$

Further Implementation Details

Architectures. Table 10 summarizes how each model reads the history and where its beam search runs. The scratch backbones S, M and L have widths 128, 384 and 640, feed-forward sizes 512, 1536 and 2560, and 2, 4 and 8 layers with 2, 6 and 10 heads. L is the default backbone of all other experiments. The Qwen2.5 backbones read the same fused item tokens as input embeddings. The fuser embeds each code in 64 dimensions (160 on Yambda and Books-23). The light decoder (2 layers, width 256, 4 heads, $M = 2$) is the same for every backbone; the decoder-size study of §4.2.5 varies its depth from 1 to 4 layers and its width from 128 to 512 (Table 7).

Training. Models are trained for at most 5k optimizer steps on the Amazon-2014 datasets, 20k on MovieLens, 30k on BPC-23 and Books-23 and 60k on Yambda, with early stopping on validation NDCG@10. Scratch-model learning rates are selected per family and dataset on validation, in the range of $10^{-4}$–$10^{-3}$, with backbone dropout 0.3. The distillation weights in (9) are set once from the initial values $\bar{\ell}_{\mathrm{ce}}$ and $\bar{\ell}_{\mathrm{kd}}$ of the two terms on a fixed sample, $w_{\mathrm{ce}} = 1/(3\bar{\ell}_{\mathrm{ce}})$ and $w_{\mathrm{kd}} = 2/(3\bar{\ell}_{\mathrm{kd}})$, so only the temperature is set per dataset: $\tau= 1$, and $\tau= 2$ on Beauty and MovieLens. The Qwen2.5 backbones use family-specific recipes selected on validation.

Timing. All efficiency results use one H100 80GB GPU and beam width 100 unless stated otherwise. Our six models and the scaling studies use one serving pipeline for every model family. Weights are in bf16. Each user’s history keys and values form one beam-shared history cache; attention over this prefix runs once for all beams and is merged with each beam’s own suffix, as in shared-prefix attention for LLM serving [17, 55]. Normalization, feed-forward and beam-step operations are compiled with torch.compile, and the prefill and the beam search are captured as two CUDA graphs per batch shape. Compilation, graph capture and warm-up are not timed. Each point runs 10 warm-up and 100 timed batches of test histories (5 and 50 above batch 64) and is repeated three times in a shuffled order. Latency covers the backbone prefill and the full beam search for a batch, including trie masking; data loading and the mapping from SIDs to items are excluded. Throughput is the batch size divided by the median batch latency. The batch-size sweep doubles the batch from 1 until throughput levels off, and the peak is reported at the smallest batch within 3% of the highest throughput. For the p95 budget in Figure 2, we report the highest throughput among the batches whose p95 latency stays within 50 ms; batch 128 is the largest batch at which every full-backbone baseline meets it on every dataset, at 88–97% of its peak throughput. Under this budget, Deco serves 3.2–7.3× more users than FusedGR on all seven datasets (6.5–8.6× more than FlatGR). Table 3 times every model through its own retrieval path in eager PyTorch (bf16 autocast, batch 128).

Model	Tokens per item	Input length	Decoder	Training
FlatGR	$K$	$TK$	backbone	CE
FlatGR-match	$K$	$T$	backbone	CE
FusedGR	1	$T$	backbone	CE
Deco-flat	$K$	$TK$	light decoder	CE+KD
Deco-CE	1	$T$	light decoder	CE
Deco	1	$T$	light decoder	CE+KD