Preference-Drift-Aware Subsequence Learning and Hierarchical Context Fusion for Long-Sequence Generative Recommendation

Introduction

Recommender systems are shifting from conventional multi-stage discriminative paradigms to generative recommendation systems [1]. While discriminative methods rely on query–candidate matching for next-item prediction, generative methods cast recommendation as sequence generation, autoregressively producing the next-item representation conditioned on a user’s interaction history. P5 [6] first unified diverse recommendation tasks as conditional text generation under a single language-modeling objective. To improve the semantic structure and generation efficiency of item identifiers, TIGER [20] assigns each item a discrete semantic identifier (SID) via residual quantization and trains a sequence-to-sequence Transformer to generate the next item’s SID. However, this two-stage design decouples tokenization from the downstream objective, leaving the tokenizer unsupervised and potentially yielding suboptimal identifiers. DIGER [4] enables recommendation-aware codebook updates via differentiable indexing and mitigates codebook collapse through annealed Gumbel noise. Nevertheless, these methods incur quadratic time and memory costs with respect to sequence length, limiting their ability to model complete user histories.

To address the high computational complexity of full-sequence modeling, existing long-sequence generative recommendation methods can be broadly categorized into two lines of work. (1) Efficient full-sequence modeling methods improve scalability through optimized interaction operators and attention mechanisms while preserving broad historical coverage. For example, HSTU [25] achieves efficient sequence modeling by replacing softmax attention with point-wise activation and temporal gating. RankMixer [29] improves the efficient modeling of feature interactions. However, coupling temporal signals with item semantics may cause interference. FuXi-Linear [23] addresses this issue by decoupling temporal dynamics and positional information into dedicated channels. (2) Target-aware context retrieval methods reduce the input sequence length via interest compression or item semantic retrieval. For example, CAUSE [16] and DualGR [24] compress long-term historical behaviors into a small set of interest representations based on item categories and interest disentanglement, respectively. To avoid discarding fine-grained information relevant to the current target, GLASS [2] performs coarse-grained prediction on recent items via a hierarchical SID mechanism. Then it uses the resulting prediction as a dynamic key to retrieve a semantically related subset of items.

However, our assumption-validation experiments reveal that both categories of methods struggle to balance accuracy and efficiency. (1) Efficient full-sequence modeling methods incur costs that scale with sequence length but yield limited or even negative accuracy gains (Table 1). (2) The performance of target-aware retrieval methods declines as sequence length increases. Although increasing the retrieval size initially improves performance, further expansion beyond a certain point degrades performance (Figure 1). Overall, their performance limitations can be attributed to the former’s homogeneous treatment of historical interactions and the latter’s reliance on retrieval based on a single similarity measure. Consequently, neither can effectively model the stage-wise evolution of user interests or the dependencies among subsequences.

To address these problems, we propose DRIFT, a preference-Drift-awaRe subsequence learning and hIerarchical context Fusion meThod for long-sequence generative recommendation. Unlike efficient full-sequence modeling with uniform aggregation and target-aware context retrieval relying on single-similarity filtering, our core idea is twofold. (1) We adaptively partition the long history into preference-consistent subsequences using preference drift information. (2) We model local-global contextual dependencies with linear complexity, thereby balancing accuracy and efficiency. Specifically, we first leverage multidimensional preference-drift information—including semantic shifts, temporal intervals, and interest distributions—to learn differentiable soft boundaries that partition a long interaction history into a few preference subsequences. Items are aggregated via linear attention weighted by item soft memberships, yielding preference-consistent subsequence representations that effectively filter historical noise irrelevant to the corresponding subsequence preference. Secondly, we devise a hierarchical context fusion mechanism. Recent items interact with the subsequence representations through cross-attention, which incorporates context from preference-consistent subsequences while filtering out preference-inconsistent noise. A gating mechanism then adaptively fuses the local recent context with the global subsequence context, enabling the target-item representation to capture both recent and long-term preferences. Across three public datasets, DRIFT improves the performance of three backbones, achieving up to a 21.6% gain in Recall@20 and a 97.4% reduction in training time, demonstrating superior accuracy and efficiency in long-sequence generative recommendation.

Assumption Validation

In this section, we study how sequence length affects long-sequence generative recommendation through two experiments: (1) the impact on efficient full-sequence modeling’s accuracy and efficiency; and (2) the impact of sequence lengths and retrieval sizes on target-aware context retrieval.

To assess the accuracy and efficiency of efficient full-sequence modeling methods, we evaluate HSTU [23] and FuXi-Linear [23] on the KuaiRec and Taobao MM datasets across input sequence lengths of $\{50, 100, 500, 1000\}$, while keeping other experimental settings identical to ensure a fair comparison. As shown in Table 1, increasing the sequence length yields diminishing or even negative accuracy gains while substantially increasing training time. On KuaiRec, both methods achieve their best or near-best NDCG@5 with short sequences, whereas longer histories introduce fluctuations or degradation; on Taobao MM, NDCG@5 remains nearly unchanged across sequence lengths. In contrast, when the sequence length increases from 50 to 1,000, HSTU’s training time grows by up to 6.9×, while that of FuXi-Linear increases by up to 33.9×. Experimental results show that, as sequence length increases, the growing number of target-irrelevant interactions degrades recommendation accuracy and computational efficiency.

Dataset	Len.	HSTU N@5	HSTU Time (s)	HSTU #P (M)	Fuxi-Linear N@5	Fuxi-Linear Time (s)	Fuxi-Linear #P (M)
KuaiRec	50	0.0676	1.62	1.04	0.0541	1.55	1.08
KuaiRec	100	0.0695	1.60	1.05	0.0526	1.58	1.09
KuaiRec	500	0.0652	2.05	1.09	0.0498	3.87	1.14
KuaiRec	1,000	0.0689	3.06	1.14	0.0536	16.02	1.20
Taobao MM	50	0.1338	3.09	201.34	0.1340	17.98	100.71
Taobao MM	100	0.1337	3.99	201.34	0.1340	16.52	100.72
Taobao MM	500	0.1338	10.95	201.38	0.1339	36.56	100.77
Taobao MM	1,000	0.1338	21.44	201.43	0.1339	609.47	100.84

To assess how sequence length and retrieval size influence target-aware context retrieval, we evaluate GLASS [2] under sequence lengths $\{100, 500, 1000\}$ and retrieval sizes $\{0, 10, 30, 50, 70, 90\}$. Appendix details experimental settings. As shown in Figure 1, both factors affect performance, though their impact varies across datasets. For sequence length, GLASS attains its best Recall@5 at length 100 (0.0831 on KuaiRec, 0.0931 on Taobao MM); increasing the length to 1,000 reduces Recall@5 by 7.3% and 33.7%, respectively. These results suggest that excessively long sequences introduce outdated or target-irrelevant interactions, thereby degrading performance. For retrieval size, without item retrieval, Recall@5 is 0.0761 (KuaiRec) and 0.0442 (Taobao MM). On KuaiRec, performance peaks at 0.0777 with 30 items—only a marginal gain over no retrieval—while 10 items lower Recall@5 to 0.0674. On Taobao MM, retrieval yields substantial gains: Recall@5 reaches 0.1415 with 10 items, up from 0.0442 without retrieval; further increasing the retrieval size leads to a slight decline (0.1344–0.1349). These results indicate that retrieving too few items may omit useful contextual information, whereas retrieving too many may introduce semantically similar but target-misaligned noise. Overall, relying on a single similarity metric prevents GLASS from effectively filtering such preference-inconsistent noise.

Methodology

We first introduce the notation and formally define generative recommendation. We then present DRIFT, which consists of a preference-drift-aware subsequence learning module and a hierarchical context fusion module, followed by the training procedure and computational complexity analysis. The overall framework is illustrated in Figure 2.

Preliminaries

Let the chronologically ordered interaction sequence of user $u$ be $\mathcal{I}_u = \{(i_1,t_1),(i_2,t_2),\ldots,(i_L,t_L)\}$, where $i_j$ denotes the item involved in the $j$-th interaction and $t_j$ is the corresponding timestamp. In generative recommendation, each item $i$ is represented as a sequence of $M$ discrete tokens: $\mathbf{z}_i = \left(z_i^{(1)},\ldots,z_i^{(M)}\right)$, where $z_i^{(m)} \in\mathcal{V}$ and $\mathcal{V}$ denotes the shared token vocabulary. Here, $M$ is the number of tokens used to represent an item. These tokens can be learned by embedding models such as BGE [3] and Qwen3 [26]. Generative recommendation formulates next-item prediction as the autoregressive generation of the target item’s token sequence:

We further divide the user interaction sequence $\mathcal{I}_u$ into a recent interaction sequence $\mathcal{R}_u = \{i_{L-r+1},\ldots,i_L\}$, and a historical interaction sequence $\mathcal{H}_u = \{i_1,\ldots,i_{L-r}\}$.

Preference-drift-aware Subsequence Learning

We propose a differentiable subsequence partitioning method driven by multidimensional preference-drift information. It segments each user’s historical interaction sequence into preference-consistent subsequences, with the number of subsequences adaptively determined for each user.

Preference-Drift-Based Boundary Scoring. Given the limited ability of a single drift metric to characterize preference drift, we designed a drift characterization operator $\phi$ that jointly models drift across three dimensions: semantic, temporal, and categorical. For user $u$’s interaction with item $i_j$ at time $t_j$, we define $\phi_{i_j} = \left[1-\cos(\mathbf{z}_{i_j},\mathbf{z}_{i_{j-1}});\mathbf{z}_{i_j}-\mathbf{z}_{i_{j-1}};\log(1+\Delta t_j);\operatorname{JSD}(P_{j-w:j}\|P_{j:j+w})\right]$. These four terms capture semantic dissimilarity, feature variation, interaction interval, and categorical distribution shift across the local windows around $i_j$, respectively. Subsequently, we employ a lightweight multilayer perceptron (MLP) to map the drift representation to a boundary score:

where $\sigma(\cdot)$ denotes the sigmoid function, and $\mathbb{1}[\cdot]$ masks invalid positions. The boundary score $b_{j>1} \in[0,1]$ quantifies the likelihood that user $u$ enters a new preference stage at position $j$. A higher score indicates that the corresponding position is more likely to mark a boundary between two adjacent preference stages. Since the first position in the sequence has no preceding interaction, we set $b_1=0$.

To accommodate the heterogeneity of preference-drift trajectories across users, we aggregate the position-wise boundary scores $\{b_j\}$ into three user-level statistics: the average boundary strength $\bar{b}_u$, the boundary variance $v_u$, and the normalized history length $\ell_u$. Based on these statistics, we adaptively estimate the number of subsequences:

where $\theta_0$ and $d_u$ are the drift constants. The value $\kappa_u \in[1, K]$ grows monotonically with boundary density, drift-point concentration, and history length, enabling personalized adaptation of the subsequence number. We accumulate boundary scores into monotonically increasing soft coordinates and normalize them to the range $[1, \kappa_u]$:

Subsequence Learning with Soft Memberships

We design a soft-membership-based linear attention mechanism to learn preference-consistent subsequence representations. Specifically, each candidate subsequence $k$ is anchored to a fixed center $\mu_k = k$ and equipped with a soft validity gate $m_k = \sigma((\kappa_u - k + 0.5)/\tau_m)$, which smoothly suppresses stages beyond the user-specific horizon $\kappa_u$. We set $\tau_m = 0.3$. The membership of item $i_j$ in subsequence $k$ is then defined by a gated Gaussian kernel over the soft coordinate $r_j$:

To mitigate over-smoothing while preserving end-to-end differentiability, we further sharpen the assignments using the Gumbel-Softmax reparameterization:

where $g_{j,k}, g_{j,v} \sim\operatorname{Gumbel}(0,1)$, and $\tau_a$ and $\tau_g$ denote the temperature parameters of the Gaussian kernel and the Gumbel-Softmax, respectively. Here, $\tau_a = 0.5$, $\tau_g = 1.0$.

Building upon the soft membership weights $y_{j,k}$, we define subsequence linear attention. These weights confine the representation learning of each subsequence to its own members, thereby avoiding cross-subsequence interference. Each subsequence $k$ is associated with a learnable query $p_k$. Through the shared projections $q_k = W_Qp_k$, $k_j = W_Kz_j$, and $v_j = W_Vz_j$, we adopt the feature map $\phi(x) = \operatorname{elu}(x) + 1$ to compute the attention weight for head $h$:

The subsequence representation $s_k$ can be calculated as

where the $h$-th head output is $o_k^{(h)} = \sum_{j=1}^{L-r}\beta_{j,k}^{(h)}v_j^{(h)}$, and LN denotes the layer normalization operation.

Hierarchical Context Fusion

The historical subsequences $S = [s_1;\ldots;s_K] \in\mathbb{R}^{K \times d}$ encode the user’s long-term preferences as they evolve, whereas the recent interaction window $R_u$ captures short-term preferences. Naively concatenating the two may allow irrelevant historical subsequences to interfere with short-term preferences; conversely, focusing solely on recent interactions overlooks important long-term preferences. To address this issue, we propose a hierarchical context fusion mechanism that adaptively integrates recent and long-term preferences.

Recent Preference Context Fusion. We employ an autoregressive encoder (e.g., T5 [19] or HSTU [25]) to learn recent item representations, i.e.,

where $f_A$ represents an attention method. In each encoder block, we use the cross-attention to capture the correlation between recent items and subsequence representations:

The resulting correlation weights enable each recent item to attend selectively to subsequence contexts that are consistent with its underlying preference, thereby mitigating the noise introduced by subsequences that are semantically similar but preference-inconsistent. Consequently, the hidden state $\mathbf{R}^* = \operatorname{LN}(\mathbf{R}^{(N)})$ encodes recent preference context.

Local-Global Context Fusion. We further perform adaptive context fusion at the decoder level. The decoder generates the token representations $\mathbf{z}_{\mathrm{dec}}$ of the target item in an autoregressive manner. At each decoding step, we design a gated cross-attention mechanism to fuse the recent context and global subsequence context. Specifically, this mechanism comprises two cross-attention operations. The first captures local dependencies with the recent context by attending to $\mathbf{R}^*$, and the second captures global dependencies across the subsequence context by attending to $S$, i.e.,

The resulting contextual representations are then jointly used to estimate a gate that adaptively combines recent and long-term preferences. The gate $g$ dynamically controls the contribution of each context: smaller gate values favor the recent context, whereas larger values assign greater importance to the global subsequence context. Formally,

where $\odot$ denotes element-wise multiplication.

Model Training

Our method is applicable to target-aware context retrieval methods (e.g., GLASS) and efficient full-sequence modeling methods (e.g., HSTU), unified by a loss framework:

where $\mathcal{L}_{\mathrm{task}}$ is the paradigm-specific recommendation loss and $\mathcal{L}_{\mathrm{sub}}$ is a generic subsequence structure constraint.

where $B$ is the training batch size, and the validity gate $m_{u,k}$. $\mathrm{sem}_{u,k}$ minimizes the cosine distance to the subsequence centroid. $\mathrm{time}_{u,k}$ penalizes large gaps between adjacent items assigned to the same subsequence. $\mathrm{cat}_{u,k}$ encourages items within the subsequence to share similar categories. $\lambda_t$ and $\lambda_c$ are weighting coefficients.

Take GLASS ([2]) as an example, we design $\mathcal{L}_{\mathrm{task}}$ to consist of SID generation loss, recent context recommendation loss, and subsequence recommendation loss:

SID generation loss is the autoregressive cross-entropy over the target SID tokens, given by $\mathcal{L}_{\mathrm{CE}}=-\sum_m\log P(z_m\mid z_{<m},S_u)$. $\mathcal{L}_{\mathrm{rec}}$ uses InfoNCE to align the enhanced hidden state $z_u=\operatorname{Pool}(\mathbf{R}^{*})$ with the target item embedding, while $\mathcal{L}_{\mathrm{sub-rec}}$ applies the same objective to the subsequence representation $z_{\mathrm{sub}}=\operatorname{Pool}(\mathbf{H}_{\mathrm{sub}})$.

Time Complexity

Let $L$ denote the sequence length, $L_h=L-r$ the length of the long-term history, $r$ the number of recent items, $K$ the number of subsequences, and $d$ the embedding dimension, where $K,r\ll L$. Our method comprises two modules: preference-drift-aware subsequence learning and hierarchical Context Fusion. The overall complexity is $O(L_hKd+r^2d+rKd)\simeq O(LKd)$. Hence, our method scales linearly with the historical sequence length $L$.

Experiments

In this section, we conduct extensive experiments to validate our methods. Due to space constraints, additional details and results are in the Appendix.

Experimental Settings

Description of Datasets and Baselines. We conduct experiments on three public datasets: KuaiRec ([5]), ML-20M ([8]), and Taobao MM ([22]). For KuaiRec and Taobao MM, the maximum sequence length is set to 1000 with 50 recent items; for ML-20M, it is set to 200 with 10 recent items. We follow the leave-one-out protocol and use the last two interactions of each user for validation and testing. We compare our method with the following baselines: two sequential recommendation methods (GRU4Rec ([9]) and SASRec ([12])), two efficient full-sequence modeling methods (HSTU ([25]) and Fuxi-Linear ([23])), and three generative recommendation methods based on SIDs (TIGER ([20]), DIGER ([4]), and GLASS ([2])), where GLASS also belongs to the target-aware context retrieval methods.

Implementation Details. All methods are implemented in PyTorch and executed on NVIDIA A100 40GB. We evaluate performance using two widely adopted metrics, Recall@M and NDCG@M, with $M\in\{5,20\}$. All methods use a hidden dimension of 96, sampled softmax with 500 negatives, and are evaluated by NDCG. SID-based generative recommendation methods use a T5-style encoder-decoder backbone with 4 encoder/decoder layers, an FFN dimension of 1024, 8 attention heads, a key/value dimension of 32, and dropout of 0.1. Moreover, for the SID-based generative recommendation methods, items are represented by three-level SIDs, with codebook sizes of $[256,256,256]$ on KuaiRec and ML-20M, and $[64,128,128]$ on Taobao MM. In our method, the number $K$ of subsequences is selected from $\{10,20,30\}$. The search range of $\lambda_{\mathrm{sub}}$ in (13) is $\{0.0,0.02,0.05,0.1,0.3\}$; those of $\lambda_t$ and $\lambda_c$ in (14) are $\{0.0,0.25,0.5,0.75,1.0\}$; and those of $\lambda_{\mathrm{rec}}$ and $\lambda_{\mathrm{sub-rec}}$ in (15) are $\{0.1,0.3,0.5,0.7,1.0\}$. We use the AdamW ([18]) optimizer to optimize the recommender. Models are trained for at most 100 epochs, with an early-stopping patience of 50.

Performance Comparison

Recommendation Accuracy

We compare DRIFT against seven baselines across KuaiRec, ML-20M, and Taobao MM, and integrate it with HSTU, TIGER, and GLASS. The results are presented in Table 2. (1) DRIFT consistently improves accuracy across datasets and backbones. Specifically, on KuaiRec, DRIFT-HSTU performs best on all metrics, significantly improving NDCG@20 and Recall@20 over HSTU by 13.1% and 21.6%, respectively. On ML-20M, despite DIGER and TIGER leading the baselines, DRIFT variants top all four metrics, substantially improving every backbone with gains of up to 74.3% over TIGER and 90.4% over GLASS. On Taobao MM, DRIFT-GLASS ranks first on three of four metrics. (2) DRIFT outperforms its backbones across HSTU, TIGER, and GLASS in all comparisons. It learns differentiable boundaries from preference-drift signals to partition long histories into preference-consistent subsequences, then integrates recent behavior and global preferences through linear attention, cross-attention, and gated fusion. This design filters irrelevant history while preserving cross-subsequence dependencies. Its larger gains on KuaiRec and ML-20M suggest greater benefits under stronger preference drift or noisier histories, while the moderate gains on Taobao MM reflect differences in data distributions and backbone strength.

Method	KuaiRec N@5	KuaiRec N@20	KuaiRec R@5	KuaiRec R@20	ML-20M N@5	ML-20M N@20	ML-20M R@5	ML-20M R@20	Taobao MM N@5	Taobao MM N@20	Taobao MM R@5	Taobao MM R@20
GRU4Rec	0.0687	0.0888	0.0892	0.1603	0.0089	0.0194	0.0155	0.0534	0.1344	0.1348	0.1350	0.1364
SASRec	0.0709	0.0924	0.0939	0.1703	0.0073	0.0155	0.0121	0.0419	0.1446	0.1453	0.1455	0.1477
HSTU	0.0689	0.0878	0.0865	0.1540	0.0027	0.0059	0.0044	0.0158	0.1338	0.1440	0.1439	0.1443
Fuxi-Linear	0.0536	0.0670	0.0680	0.1342	0.0042	0.0110	0.0083	0.0327	0.1339	0.1341	0.1341	0.1346
TIGER	0.0568	0.0716	0.0735	0.1239	0.0207	0.0396	0.0319	0.1004	0.0012	0.0016	0.0012	0.0025
DIGER	0.0208	0.0229	0.0537	0.0631	0.0233	0.0353	0.0356	0.0778	0.0285	0.0517	0.0456	0.1286
GLASS	0.0573	0.0737	0.0777	0.1346	0.0189	0.0359	0.0303	0.0922	0.1249	0.1288	0.1365	0.1500
DRIFT-HSTU	0.0753†	0.0993†	0.1006†	0.1872†	0.0275	0.0480	0.0391	0.1092	0.1442	0.1445	0.1447	0.1457
DRIFT-TIGER	0.0618	0.0800	0.0848	0.1493	0.0349†	0.0615	0.0556	0.1504	0.1136	0.1184	0.1261	0.1425
DRIFT-GLASS	0.0639	0.0798	0.0881	0.1532	0.0345	0.0638†	0.0577†	0.1608†	0.1439	0.1454†	0.1472†	0.1524†

Computational Efficiency

By adaptively partitioning long interaction histories into a small number of preference-coherent subsequences and applying linear-complexity attention to them, DRIFT effectively filters out preference-inconsistent noise, substantially reducing training costs while accommodating much longer input sequences. As shown in Table 3, DRIFT-GLASS supports sequences up to 20$\times$ longer than those handled by GLASS (increasing the maximum sequence length from 50 to 1,000), while achieving 3.9$\times$ and 2.7$\times$ faster per-epoch training on KuaiRec and Taobao MM, respectively. DRIFT-Tiger delivers up to 80.4$\times$ training speedup and 22.4$\times$ inference speedup over Tiger on Taobao MM (and 38.6$\times$ and 7.6$\times$ on KuaiRec). Meanwhile, DRIFT-HSTU cuts per-epoch training time by 37.6%–42.1% and GPU memory by 39.6%–61.1% compared with HSTU. These results confirm that modeling compact, preferenceconsistent subsequences effectively mitigates the efficiency bottleneck in long-sequence recommendation.

Method	KuaiRec Train.	KuaiRec Infer.	KuaiRec VRAM	Taobao MM Train.	Taobao MM Infer.	Taobao MM VRAM
TIGER	301.30	221.40	23.10	2743.10	1871.50	23.10
+DRIFT	7.80	29.20	17.66	34.10	83.40	16.95
Δ (%)	-97.41	-86.81	-23.55	-98.76	-95.54	-26.62
GLASS	33.60	31.30	9.13	95.50	60.90	9.13
+DRIFT	8.70	31.10	17.75	34.90	84.80	17.04
Δ (%)	-74.11	-0.64	+94.41	-63.46	+39.24	+86.64
HSTU	3.06	6.80	3.50	21.44	13.13	12.07
+DRIFT	1.91	6.70	1.36	12.41	7.41	7.29
Δ (%)	-37.58	-1.47	-61.14	-42.12	-43.56	-39.60

Ablation Experiments

Effectiveness of Preference-drift-aware Subsequence Learning. To validate the preference drift-aware subsequence learning module, we compare DRIFT with GLASS and two DRIFT variants: DRIFT-Uni, which uses uniform segmentation, and DRIFT-KM, which employs K-means-based segmentation. As shown in Figure 3, all subsequence compression-based methods outperform the retrieval-based GLASS method, indicating that compressing long sequences preserves broader historical information while mitigating context loss and noise. Among these methods, DRIFT achieves the best performance, with Recall@5 scores of 0.0881 on KuaiRec and 0.0577 on ML-20M. Unlike uniform segmentation, which overlooks the irregular evolution of user preferences, and K-means segmentation, which relies solely on representational similarity, DRIFT adaptively learns soft boundaries based on preference drift signals. This enables the construction of subsequences with greater preference consistency and effectively reduces noise in user histories, thereby demonstrating the effectiveness of preference drift-aware subsequence learning.

Effectiveness of Hierarchical Context Fusion. As shown in Table 4, DRIFT consistently outperforms all variants on both datasets, validating the effectiveness of our hierarchical context fusion mechanism. Specifically, (1) removing either w/o Recent-CA or w/o Global-CA consistently degrades performance, demonstrating that both local enhancement of recent preferences and long-term global context are essential; (2) directly concatenating subsequence representations, as in Subseq-Prefix and Concat-Fuse, leads to a catastrophic performance collapse on Taobao MM, with N@5 dropping to approximately $10^{-3}$ or lower, and also causes substantial degradation on KuaiRec. These results suggest that subsequence representations (encoding long-term preferences) and recent-item representations (capturing short-term transitions) are inherently heterogeneous. Naive concatenation introduces noise arising from semantic-space misalignment, severely disrupting the decoder representations. In contrast, our gated fusion mechanism aligns contextual information through cross-attention and mitigates conflicts between heterogeneous representations via adaptive weighting. It therefore effectively captures both recent and long-term preferences, achieving the best performance on both datasets.

Method	KuaiRec N@5	KuaiRec R@5	Taobao MM N@5	Taobao MM R@5
Ours	0.0639	0.0881	0.1439	0.1472
w/o Recent-CA	-5.32%	-3.86%	-2.08%	-0.75%
Subseq-Prefix	-5.79%	-5.22%	-99.31%	-99.12%
w/o Global-CA	-4.38%	-5.45%	-2.02%	-1.29%
Concat-Fuse	-7.82%	-5.45%	-99.79%	-99.66%

Sensitivity Analysis

Impact of Training Losses. We investigate the impact of $\lambda_{\mathrm{rec}}$, $\lambda_{\mathrm{sub-rec}}$, and $\lambda_{\mathrm{sub}}$ on the performance of our methods using the KuaiRec dataset. As shown in Figure 4, DRIFT-TIGER achieves the best performance with $\lambda_{\mathrm{rec}} = 0.1$, $\lambda_{\mathrm{sub-rec}} \approx0.3$, and $\lambda_{\mathrm{sub}} \approx0.02$, while DRIFT-GLASS favors approximately 0.5, 0.1, and 0.1, respectively. Excessive structural regularization degrades performance. Moreover, all three losses are beneficial. Removing $\mathcal{L}_{\mathrm{rec}}$ causes the largest performance drop, while excluding $\mathcal{L}_{\mathrm{sub-rec}}$ or $\mathcal{L}_{\mathrm{sub}}$ also consistently reduces Recall@5. Overall, jointly optimizing these losses yields the best performance.

Impact of Sequence Length and Subsequence Number. As shown in Figure 5, increasing the sequence length improves Recall@5 on KuaiRec while maintaining stable performance on Taobao MM, indicating that our method captures long-term preferences and suppresses historical noise. Both datasets peak at 20 subsequences and vary only slightly across settings. Too few subsequences may mix distinct preferences, whereas too many may fragment coherent behaviors. Overall, our method is robust to both hyperparameters.

Related Work

Generative Recommendation. SID-based recommenders represent items as discrete token sequences for constrained generation. TIGER [20] first quantizes item representations and then trains a Transformer to generate SIDs. To address the mismatch caused by this separate training, ETEGRec [16] and UniSID [11] jointly optimize tokenization and recommendation, DIGER [4] further propagates recommendation gradients to the codebook, and BLOGER [1] uses bilevel optimization to balance the two objectives. This line thus evolves from content-based tokenization to recommendation-aware SID learning. Another line of research redesigns identifier generation for efficiency. SE-TRec [15] and RPG [10] replace ordered SIDs with unordered tokens, enabling parallel prediction and constrained decoding. Despite advances in tokenization and generation, these methods largely rely on dense self-attention over user histories, whose quadratic cost necessitates truncation and limits long-term preference modeling.

Long-sequence Generative Recommendation. Existing methods mainly fall into two categories: efficient full-sequence modeling and context reduction. The former redesigns interaction operators to preserve complete histories. HSTU [25], RankMixer [29], OneTrans [27], and Make It Long, Keep It Fast [7] improve the scalability of long-sequence encoding through efficient architectures and training strategies. FuXi-Linear [23] further decouples temporal dynamics from semantic interactions to reduce their interference. Although these methods retain broad behavioral coverage, they still process every event, including substantial irrelevant or redundant information. Context-reduction methods instead compress or retrieve historical behaviors. CAUSE [17], GEMs [28], DualGR [24], and HiCoGen [14] summarize histories using categories, temporal windows, disentangled interests, or hierarchical clustering. While efficient, such target-agnostic compression may discard infrequent but decisive behaviors. GLASS [2] alleviates this issue by using a predicted SID to retrieve intent-relevant historical items, but inaccurate coarse predictions can exclude useful evidence. Thus, full-sequence methods preserve information at the cost of redundant computation, whereas context reduction improves efficiency at the risk of information loss.

Conclusions

We propose DRIFT to address the accuracy–efficiency trade-off and the ignore of preference drift modeling in long-sequence generative recommendation. Its core innovations include: (1) preference-drift-aware subsequence learning, which leverages multidimensional drift information to adaptively segment user histories into preference-consistent subsequences, enabling contextual aggregation and noise filtering with linear computation complexity; (2) hierarchical context fusion, which enriches recent-interaction representations through cross-attention and adaptively integrates local-global subsequence information via a gating mechanism, thereby effectively capturing both short- and long-term user preferences. Extensive experimental results demonstrate that our method outperforms existing baselines in computational efficiency and recommendation accuracy.

Dataset Description

We evaluate our method on three public datasets spanning diverse recommendation domains, scales, and interaction densities: KuaiRec[5], a short-video dataset collected from Kuaishou; ML-20M[8], a widely used MovieLens benchmark; and Taobao MM[22], a large-scale multimodal ecommerce dataset. Their statistics are summarized in Table 5. These datasets represent complementary evaluation settings. KuaiRec is relatively dense (83.72% sparsity), with a small item catalog and exceptionally long user histories (1,746 interactions per user on average), making it well-suited for evaluating long-range sequential modeling. ML-20M is a medium-scale, sparse dataset (99.46% sparsity) with moderate sequence lengths and is a standard benchmark for sequential recommendation. In contrast, Taobao MM is large-scale and highly sparse (99.89% sparsity), with over 870K items and long behavior sequences (an average of 905 interactions). Its rich visual and textual features further enable evaluation at an industrial scale in a multimodal setting.

Dataset	KuaiRec	ML-20M	Taobao MM
#Users	7,176	138,493	34,535
#Items	10,728	26,744	874,417
#Interactions	12,530,806	20,000,263	32,814,555
Sparsity	83.72%	99.46%	99.89%
Avg. Seq. Len.	1,746.2	144.4	905.5

Preprocessing. For each dataset, we chronologically order user interactions and adopt the standard leave-one-out protocol: the last interaction is used for testing, the penultimate one for validation, and the remainder for training. Following common practice [12, 21], we remove users with fewer than five interactions and truncate histories to a predefined maximum length. For Taobao MM, whose raw sequences contain up to 1,000 events, we additionally impose a minimum history length to ensure sufficient context.

Semantic ID Construction. To construct semantic identifiers (SIDs) for generative retrieval, we encode each item into a dense representation and quantize it using a residual-quantized variational autoencoder (RQ-VAE) [20]. For KuaiRec and ML-20M, we encode textual content, including titles, tags, and categorical metadata, with a pretrained text embedding model BGE [3]. For Taobao MM, we directly use the provided pretrained multimodal item embeddings. Three-level residual quantization then maps each item to a compact hierarchical SID.

Assumption Validation

Experiment Setting

To evaluate the impact on efficient full-sequence modeling’s accuracy and efficiency, we compare HSTU [25] and FuXi-Linear [23] on KuaiRec and Taobao MM with sequence lengths in $\{100, 500, 1000\}$. Both models use a single layer with a hidden dimension of 96. HSTU [25] employs one attention head, whereas FuXi-Linear [23] comprises a 16-dimensional linear temporal channel with 8 heads and base 2, a 16-dimensional attention channel, and a 32-dimensional positional channel, with a dropout rate of 0.5. These two methods are trained for up to 100 epochs using AdamW [18], early stopping with a patience of 50, 500 sampled negatives, and a softmax temperature of 0.05. The learning rate is set to $3 \times10^{-4}$ for HSTU on KuaiRec and $1 \times10^{-4}$ otherwise, with a weight decay of 0.01 on Taobao MM; FuXi-Linear uses a learning rate of $1 \times10^{-4}$. Experiments use a fixed random seed of 2025, and the best checkpoint is selected according to validation NDCG@20.

To investigate the effects of sequence length and retrieval size on target-aware context retrieval, we evaluate GLASS [2] on KuaiRec and Taobao MM by varying the user-history length $L \in\{100, 500, 1000\}$ and the number of retrieved target-similar items $N \in\{0, 10, 30, 50, 70, 90\}$. Setting $N = 0$ turns off the retrieval branch, reducing GLASS to a generator conditioned solely on user history. Three hierarchical codewords represent each item learned by a residual-quantized autoencoder, with codebook sizes of $[256, 256, 256]$ for KuaiRec and $[64, 128, 128]$ for Taobao MM. GLASS adopts a T5-style encoder–decoder architecture [19] with four encoder and four decoder layers, a hidden size of 96, a feed-forward dimension of 1024, eight attention heads with a per-head dimension of 32, and a dropout rate of 0.1. The model is trained using AdamW with a learning rate of $1 \times10^{-4}$ and a weight decay of 0.01 for up to 100 epochs. Early stopping is applied based on validation NDCG@20 with a patience of 50 epochs, and predictions are generated autoregressively using beam search with a beam width of 20. All other components and hyperparameters are fixed across experimental settings.

Interest Drift Analysis in Long Behavior Sequences

We investigate whether long user behavior sequences comprise locally coherent, yet temporally evolving, interest subsequences in the Taobao MM dataset. As shown in Figure 6, both Layer-1 SID and RQVAE representations exhibit pronounced high-similarity bands along the diagonal and distinct blockwise patterns, indicating strong within-stage continuity and clear transitions across stages. For instance, User 7 demonstrates stronger continuity between adjacent interactions, whereas User 15 shows greater alignment between the target item and historical interactions. The aggregate results in Table 6 further corroborate these observations. Across 4,997 valid users, adjacent items in the original sequences achieve a mean similarity of 0.630, exceeding those in shuffled sequences and randomly sampled item pairs by 0.392 and 0.518, respectively. Moreover, local coherence gradually weakens as the temporal span increases: the within-window similarity decreases from 0.630 at $w = 2$ to 0.444 at $w = 20$, while the diagonal-band gap narrows from 0.355 at $d = 1$ to 0.145 at $d = 20$. A similar pattern emerges at the subsequence level, where adjacent subsequences are consistently more similar than non-adjacent ones; for example, their similarities are 0.684 and 0.373, respectively, when $L = 5$. In addition, the target item is more closely aligned with recent interactions than with earlier ones, yielding similarities of 0.275 and 0.197, respectively. Collectively, these findings suggest that long behavior sequences consist of coherent interest stages, undergo semantic transitions across stages, and exhibit a pronounced recency effect. This observation motivates the explicit modeling of dynamic interest subsequences and their contextual dependencies.

Metric	Observed	Reference	Diff.
Adjacent-item similarity
Adjacent vs. shuffled	0.630	0.238	+0.392
Adjacent vs. random pairs	0.630	0.112	+0.518
Within-window similarity
$w = 2$	0.630	0.238	+0.392
$w = 5$	0.561	0.238	+0.323
$w = 20$	0.444	0.238	+0.206
Subsequence similarity
Adjacent vs. non-adjacent, $L = 5$	0.684	0.373	+0.311
Adjacent vs. non-adjacent, $L = 10$	0.697	0.436	+0.261
Adjacent vs. non-adjacent, $L = 20$	0.711	0.502	+0.209
Diagonal-band structure
Band gap, $d = 1$	0.355	$\approx0$	+0.355
Band gap, $d = 5$	0.245	$\approx0$	+0.245
Band gap, $d = 20$	0.145	$\approx0$	+0.145
Target–history association
Target vs. complete history	0.218	0.114	+0.104
Target vs. recent/earliest five	0.275	0.197	+0.078

Model Training

The training loss of our method consists of $\mathcal{L}_{\mathrm{task}}$ and $\mathcal{L}_{\mathrm{sub}}$:

where $\mathcal{L}_{\mathrm{task}}$ is the paradigm-specific recommendation loss and $\mathcal{L}_{\mathrm{sub}}$ is a generic subsequence structure loss.

Recommendation Loss

Our method is applicable to target-aware context retrieval methods and efficient full-sequence modeling methods. Because these paradigms employ different recommendation objectives, we denote their backbone-specific loss by $\mathcal{L}_{\mathrm{task}}$. For DRIFT-GLASS, the recommendation loss consists of an SID generation objective and two complementary sequence-level recommendation objectives:

where $\lambda_{\mathrm{rec}}$ and $\lambda_{\mathrm{sub-rec}}$ balance the two auxiliary objectives.

SID Generation Loss. The primary objective $\mathcal{L}_{\mathrm{CE}}$ is the autoregressive cross-entropy over the target SID tokens. Let $q_{i,m} \in\{0, 1\}$ indicate whether the $m$-th target token of training instance $i$ is a non-padding token, and let $N_{\mathrm{tok}} = \sum_{i=1}^{B}\sum_{m=1}^{M}q_{i,m}$. The loss is defined as

where $\mathbf{R}_i^{*}$ denotes the hidden state with recent context and $\mathbf{S}_i$ contains the learned historical subsequence representations. This loss carries unit weight and provides the primary token-level supervision for SID generation.

Sequence-Level Recommendation Losses. Although $\mathcal{L}_{\mathrm{CE}}$ supervises individual SID tokens, it does not directly enforce sequence-level alignment between a user representation and the corresponding target item. We therefore introduce two in-batch softmax objectives that provide complementary sequence-level discriminative supervision.

Let $\tilde{\mathbf{z}}_{u_i} = \mathbf{z}_{u_i}/\lVert\mathbf{z}_{u_i}\rVert_2$ and $\tilde{\mathbf{e}}_i = \mathbf{e}_i/\lVert\mathbf{e}_i\rVert_2$ denote the normalized user and target-item representations, respectively. The recent-context recommendation loss is

where $\mathbf{z}_{u_i} = \operatorname{Pool}(\mathbf{R}_i^{*})$ is the pooled enhanced encoder representation, $\mathbf{e}_i$ is the item representation of the target item, and $\tau$ is the temperature. The remaining $B - 1$ target representations in the mini-batch serve as in-batch negatives.

Similarly, let $\mathbf{z}_{\mathrm{sub},i} = \operatorname{Pool}(\mathbf{S}_i)$ denote the representation obtained by pooling only the valid subsequence slots. Its normalized form is $\tilde{\mathbf{z}}_{\mathrm{sub},i} = \mathbf{z}_{\mathrm{sub},i}/\lVert\mathbf{z}_{\mathrm{sub},i}\rVert_2$. The subsequence recommendation loss is

Both auxiliary objectives share the same temperature and target item representation. In our implementation, the target representation is retrieved from the shared SID embedding table using the first-tier SID code, i.e., $e_i = \operatorname{Embed}(z_i^{(1)})$. By requiring the aggregated subsequence representation to predict the target independently, $\mathcal{L}_{\mathrm{sub-rec}}$ discourages the subsequence branch from degenerating into an inactive pathway.

Subsequence Structure Loss

The recommendation objectives supervise next-item prediction, but do not explicitly constrain the internal coherence of the learned subsequences. We therefore introduce $\mathcal{L}_{\mathrm{sub}}$ to encourage each subsequence to contain semantically, temporally, and categorically consistent interactions.

Let $a_{u,j,k} \in[0,1]$ denote the gated soft assignment of the $j$-th historical interaction of user $u$ to subsequence $k$, where $\sum_{k=1}^{K} a_{u,j,k}=1$. We use

as the validity gate that softly suppresses slots beyond the user-adaptive subsequence budget $\kappa_u$. Let $A_{u,k}=\sum_j a_{u,j,k}$ denote the soft mass assigned to subsequence $k$. The structure constraint is

where $K$ is the maximum number of candidate subsequences, while $\lambda_t$ and $\lambda_c$ control the temporal and categorical constraints. Importantly, (22) averages over users but sums over subsequence slots. Because $\sum_k m_{u,k}\approx\kappa_u$, the effective regularization budget adapts to the number of subsequences required by each user.

Semantic Coherence. We first define the assignment-weighted semantic centroid as

where $x_{u,j}$ is the semantic representation of the $j$-th historical item. The semantic constraint is

Minimizing this term pulls co-assigned interactions toward a shared semantic prototype, thereby discouraging a subsequence from mixing unrelated preferences.

Temporal Coherence. Let $\Delta t_{u,j}=t_{u,j}-t_{u,j-1}$ denote the interval between two chronologically adjacent interactions. We define

The pairwise weight $a_{u,j,k}a_{u,j-1,k}$ restricts the penalty to adjacent interactions that are jointly assigned to the same subsequence. Meanwhile, the logarithmic transformation limits the influence of exceptionally long inactivity intervals. Minimizing this term, therefore, encourages preference boundaries to align with genuine temporal discontinuities.

Category Coherence. Because curated category labels are not consistently available across datasets, we use the first-tier SID code as a learning-free proxy for the item category. Let $y_{u,j}^{(1)}$ be the one-hot first-tier SID code of the $j$-th item and define its assignment-weighted distribution within subsequence $k$ as

The corresponding constraint is

Thus, the weighted cross-entropy is exactly the entropy of the first-tier SID distribution within the subsequence. Minimizing this entropy encourages each subsequence to exhibit a concentrated coarse-semantic distribution rather than mixing heterogeneous preference categories. All three constraints operate on the differentiable soft assignments $a_{u,j,k}$. Consequently, their gradients propagate through the soft partitioning process to both the boundary predictor and the adaptive subsequence budget $\kappa_u$, enabling end-to-end learning without subsequence boundary annotations.

Experiments

Implementation Details.

All models are implemented in PyTorch and trained on four NVIDIA A100 (40GB) GPUs. We use Recall@$M$ and NDCG@$M$ with $M\in\{5,20\}$ as the primary evaluation metrics. All methods use a hidden dimension of $d=96$.

Sequence Modeling Baselines. GRU4Rec ([9]), SASRec ([12]), HSTU ([25]), and FuXi-Linear ([23]) are trained with sampled softmax using 500 negative items per positive instance. HSTU and FuXi-Linear use a pointwise activation block with 1 block, 1 head, linear and attention dimensions of 16, and a dropout rate of 0.5. FuXi-Linear further decouples a positional channel of dimension 32 and a temporal channel with 8 heads and base 2. SASRec uses 1 Transformer block, 1 head, and a dropout rate of 0.2. GRU4Rec uses a single GRU layer with hidden size 96 and dropout rate 0.2. These four methods take as input the 1,000 most recent interactions from each of the KuaiRec and Taobao-MM datasets, along with the 200 most recent interactions from the ML-20M dataset. These methods are optimized with AdamW ([18]) at $1\times10^{-4}$, except GRU4Rec which uses Adam ([13]) at $3\times10^{-4}$. Weight decay is set to 0.01, except for GRU4Rec, where it is $1\times10^{-5}$. Models are trained for at most 100 epochs with an early stopping patience of 50. HSTU and FuXi-Linear use a softmax temperature of 0.05.

SID-based Generative Recommendations. TIGER ([20]), DIGER ([4]), and GLASS ([2]) adopt a T5-style encoder–decoder backbone with 4 encoder layers, 4 decoder layers, 8 attention heads, key/value dimension 32, FFN dimension 1024, ReLU activation, and dropout rate 0.1. Items are encoded by three-level SIDs, with codebook sizes $[256, 256, 256]$ for KuaiRec and ML-20M, and $[64, 128, 128]$ for Taobao MM. During inference, the next item is generated via beam search with beam size 20 over the SID vocabulary, and the top-$k$ results are evaluated for $k \in\{5, 20\}$. The model context length is fixed to 50 item tokens. The historical retrieval range $H$ is set to 1000 on KuaiRec and Taobao MM, and 200 on ML-20M; the recent window size $r$ is 50 on KuaiRec/Taobao MM and 10 on ML-20M. All models are optimized with AdamW ([18]) using a learning rate of $1 \times10^{-4}$, trained for up to 100 epochs with early stopping patience 50.

Hyperparameters of Our Method. The number of subsequences $K$ is selected from $\{10, 20, 30\}$. The search space of $\lambda_{\mathrm{sub}}$ in (16) is $\{0.0, 0.02, 0.05, 0.1, 0.3\}$; those of $\lambda_t$ and $\lambda_c$ in (22) are $\{0.0, 0.25, 0.5, 0.75, 1.0\}$; and those of $\lambda_{\mathrm{rec}}$ and $\lambda_{\mathrm{sub-rec}}$ in (17) are $\{0.1, 0.3, 0.5, 0.7, 1.0\}$. The in-batch softmax recommendation losses $L_{\mathrm{rec}}$ and $L_{\mathrm{sub-rec}}$ share a temperature $\tau_{\mathrm{rec}} = 0.07$. The subsequence structure loss $L_{\mathrm{sub}}$ is normalized only by the batch size $B$. The default search initialization is $\lambda_{\mathrm{sub}} = 0.05$, $\lambda_t = \lambda_c = 0.5$, and $\lambda_{\mathrm{rec}} = \lambda_{\mathrm{sub-rec}} = 0.5$. Our DRIFT is optimized with AdamW ([18]) and trained for up to 100 epochs with early stopping patience 50.

Impact of Subsequence Constraint Weights $(\lambda_t, \lambda_c)$

We analyze the sensitivity of DRIFT-GLASS to the internal weights $\lambda_t$ and $\lambda_c$ in $L_{\mathrm{sub}}$ ((22)). Specifically, we sweep each weight over $\{0, 0.25, 0.5, 0.75, 1.0\}$ while fixing the other to the default value of 0.5. As shown in Figure 7, DRIFT-GLASS remains highly robust to both hyperparameters. On KuaiRec, Recall@20 varies only within $[0.147, 0.154]$ when tuning $\lambda_t$ and within $[0.152, 0.155]$ when tuning $\lambda_c$. On TaobaoMM, the fluctuation is even smaller, with an absolute variation no greater than 0.003. The default setting $\lambda_t = \lambda_c = 0.5$ consistently achieves near-optimal performance across datasets. Moreover, completely disabling either constraint term (i.e., setting the corresponding weight to 0) leads to performance degradation. These results indicate that temporal coherence and category consistency provide complementary but non-critical regularization effects. The observed insensitivity further suggests that the semantic coherence term $\mathrm{sem}$, whose implicit weight is 1.0, plays the dominant role in $L_{\mathrm{sub}}$.