WorksComputer Science
SIDRec: End-to-End Semantic ID Representation and Modeling for Recommendation
Abstract
Semantic IDs (SIDs) have emerged as a promising approach for discriminative recommendation by encoding continuous item rep- resentations into compact discrete identifiers. However, existing SID-based methods face two fundamental limitations: semantic mis- match between content semantics and collaborative preferences, and objective mismatch between SID construction and downstream recommendation optimization. To address these challenges, we propose SIDRec, an end-to-end recommendation-aware SID learn- ing framework that jointly models content and collaborative in- formation while aligning SID construction with recommendation objectives. Specifically, SIDRec introduces a self-updating fusion mechanism that explicitly integrates pretrained content represen- tations with collaborative embeddings initialized from a trained recommendation model, enabling preference-aware semantic mod- eling. Furthermore, SIDRec employs an MMoE-inspired joint opti- mization mechanism with shared experts and task-specific gates, allowing recommendation supervision to directly guide differen- tiable quantization and produce task-adaptive discrete representa- tions. Extensive experiments on three public benchmark datasets demonstrate that SIDRec consistently outperforms all competing methods across both AUC and GAUC, achieving up to 5.15% relative improvements over the strongest baselines. Further ablation stud- ies validate the effectiveness and complementarity of collaborative semantic modeling and end-to-end SID optimization.
CCS Concepts
• Information systems → Recommender systems.
ACM Reference Format:
Zhuohang Jiang, Yongsen Pan, Pangjing Wu, Zihuai Zhao, Zheng Hu, and Yuxin Chen. 2018. SIDRec: End-to-End Semantic ID Representation and Modeling for Recommendation. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 10 pages. https://doi.org/XXXXXXX.XXXXXXX
Introduction
Modern recommender systems typically rely on effective item representations to capture user preferences and distinguish among candidate items. To address this, most existing recommendation models represent each item with an independent identifier learned from historical interactions [4, 31]. Despite their effectiveness for identifying item-specific patterns, such representations provide limited structure for sharing information across related items, particularly for sparse or newly introduced items. Recently, Semantic IDs (SIDs) have emerged as a promising alternative by representing items with sequences of discrete semantic codes. Specifically, items can be assigned shared codes (e.g., ‘125’ and ‘129’ share codes ‘1’ and ‘2’), enabling structured knowledge sharing and better generalization than independent item identifiers [17, 19].
Despite these advantages, learning SIDs from content semantics (e.g., what an item is) may not fully capture the collaborative preference patterns (e.g., how users interact with it) in downstream recommendations [23, 25]. For example, two products in the same category with similar descriptions and visual appearances may attract different user groups because of differences in brand preferences, price sensitivity, or intended usage scenarios [8, 10, 15]. Conversely, as illustrated in Figure 1, items with distinct content characteristics may exhibit similar interaction patterns because they are consumed by similar groups of users. These suggest that item representations for recommendations require both content semantics and collaborative preferences.
Figure 1. Semantic and collaborative signals capture distinct relationships. Items with different semantics (e.g. mice and USB flash drives) can exhibit similar collaborative patterns.
To improve SIDs for recommendations, recent studies have explored incorporating collaborative information into SID learning. For example, DIGER [3] and DAS [25] adopt exploratory learning and contrastive learning, respectively, to align semantic SIDs
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Conference acronym ’XX, Woodstock, NY
© 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/XXXXXXX.XXXXXXX and collaborative features. Meanwhile, DIG [22] embeds the SID tokenizer with a ranking model and integrates their losses during training. However, the collaborative information is primarily introduced through alignment objectives, rather than explicitly fused with content representations. Since quantization is the bottleneck that determines which item representations are preserved in SIDs, when constructing SIDs, the quantization process may not fully carry these two representations. Moreover, the optimization paradigm for SID learning further compounds such concerns. In conventional two-stage training, even when collaborative signals are incorporated into SID construction, the resulting SIDs remain fixed during downstream adaptation (i.e., recommendation objectives) [11, 23]. As a result, recommendation models can only learn to use SIDs, but cannot update how they are constructed to better support downstream objectives.
These limitations motivate explicitly fusing content and collaborative item representations, and end-to-end SID learning with downstream recommendation objectives, as illustrated in Figure 2. However, two closely related challenges arise in SID learning. First, semantic mismatch arises since content semantics and collaborative preferences characterize item representations from fundamentally different perspectives. Second, objective mismatch arises since preserving item representations does not necessarily adapt to recommendation objectives. To address these challenges, we propose SIDRec, an end-to-end SID learning framework that integrates semantic and collaborative information for recommendations. To mitigate semantic mismatch, SIDRec introduces a self-updating fusion, which initializes the collaborative embedding table from a trained ranking model and updates it during training. This allows collaborative representations to evolve with recommendation objectives, capturing interaction patterns that may not be reflected in content semantics alone, while retaining the pretrained semantic embedding of SIDs.
Figure 2. Comparison of existing paradigm with our SIDRec. (a) Existing SIDs paradigm mainly rely on content semantics and are optimized independently of downstream recommendation objectives, leading to semantic and objective mismatches. (b) Our SIDRec addresses these issues through content-collaborative semantic modeling and end-to-end SID learning.
While the self-updating fusion ensures incorporating content and collaborative signals in SIDs, it does not by itself resolve the objective mismatch. To achieve end-to-end SID adaptation to recommendation objectives, SIDRec draws inspiration from Multi-Gate Mixture of Experts (MMoE) structures [12] to jointly optimize two objectives. Specifically, by sharing experts across the two tasks and using a separate gate for each task, it enables gradients from recommendation objectives to seamlessly propagate through the quantization process of SID learning. In this manner, the learned SIDs serve not merely as static semantic identifiers, but as recommendation-aware representations tailored to recommendation objectives.
Notably, we conduct extensive experiments on both public benchmarks and real-world industrial datasets. The results demonstrate that SIDRec consistently improves recommendation performance, while comprehensive ablation studies verify the individual contributions of content-collaborative fusion and end-to-end SID learning. Specifically, SIDRec consistently outperforms all competing methods, achieving up to 5.15% relative improvement over the strongest baselines. Furthermore, removing either component degrades recommendation performance, with their both removal resulting in relative drops of 7.19% in AUC and 7.11% in GAUC, demonstrating the contributions of both components.
The main contributions are summarized as follows:
We identify existing limitations of SIDs for recommendations, including semantic mismatch between content semantics and collaborative preferences, and objective mismatch between SID construction and downstream objectives.
We propose SIDRec, an end-to-end recommendation-aware SID learning framework that addresses these two mismatches by seamlessly integrating content and collaborative information via self-updating, and allowing recommendation objectives to seamlessly optimize SID construction.
We conduct comprehensive experiments on public and industrial recommendation datasets, demonstrating the effectiveness of SIDRec and validating the contributions of SID modeling and corresponding end-to-end learning.
Related Work
Semantic IDs for Recommendation
Conventional recommendation models typically rely on hash-based item identifiers (HIDs), whose embeddings are learned from historical interactions [4, 31]. However, these independent identifiers offer limited knowledge sharing across related items, particularly for sparse or newly introduced items [19, 33]. SIDs encode items as sequences of discrete codes, allowing related items to share prefixes rather than unrelated atomic identifiers [17, 19]. In generative recommendation, TIGER [17] constructs hierarchical codes through residual quantization for autoregressive item generation. LC-Rec [28] integrates collaborative semantics into language-model adaptation, and OneRec [2] unifies generative retrieval and ranking. Penha et al. [15] further show that task-specific SIDs do not necessarily transfer between search and recommendation, highlighting the influence of the underlying embedding space. For discriminative ranking, Singh et al. [19] use content-derived SIDs to generalize to new and long-tail items, while Zheng et al. [29] improve embedding stability with prefix-based parameterization. SID-Coord [6] balances SID generalization and hashed-ID memorization, and QARM [11] learns downstream embeddings for quantized multimodal representations in retrieval and ranking. However, prefix sharing also introduces collisions, which AdaSID [14] addresses through adaptive overlap regulation. In these ranking applications, learning token embeddings does not necessarily update the code assignments, which are commonly determined before downstream optimization.
Learnable Semantic IDs
Despite their advantages, conventional SIDs are often constructed offline from content representations and remain fixed during downstream recommendation training [17, 19]. This paradigm may overlook collaborative preferences and prevents recommendation objectives from directly refining SID assignments. Recent studies incorporate interaction evidence into SID construction in different ways. TokenRec [16] quantizes collaborative embeddings, whereas LETTER [23] regularizes content-based codes through collaborative contrastive learning and code diversity. DAS [25] jointly optimizes quantization and multi-view collaborative alignment, while CSG4Rec [10] uses collaborative similarity to organize quantized item and user representations for generative sequential recommendation. These approaches distinguish direct use of interaction-derived inputs from supervision that shapes code-space geometry. However, preserving collaborative neighborhoods does not guarantee that quantization boundaries favor candidate-level click discrimination. Another research direction concerns how tokenizers interact with recommendation losses. ETEGRec [9] alternates tokenizer and generative-model training, while DIGER [3] employs differentiable assignments and uncertainty-controlled exploration under generative objectives. DIG [22] embeds a tokenizer inside a discriminative ranker, allowing ranking feedback to influence codebook boundaries while also supporting retrieval. In contrast, SIDRec explicitly integrates content and collaborative representations through self-updating fusion to mitigate semantic mismatch. It further enables recommendation objectives to guide differentiable SID learning through joint optimization, addressing the objective mismatch between SID construction and downstream recommendation.
Methodology
In this section, we present SIDRec to enable end-to-end SID learning, which integrates both semantic and collaborative information for recommendations, as illustrated in Figure 3.
Figure 3. Overview of the SIDRec framework. It first integrates collaborative representations from a trained ranking model with pretrained multimodal embeddings for SID construction. The framework then adapts a pretrained SID tokenizer through downstream CTR learning. Candidate and history representations built from the learned SIDs are used for click prediction and SID prediction, which are optimized jointly.
Notation. For a user , let denote the interaction history before candidate item . The model predicts the click probability for the label . Each item has a fixed multimodal embedding and a collaborative embedding . We use and for the tokenizer and ranking dimensions. Each item is represented by an -level SID , where is the token at level . We use in SIDRec.
Self-Updating Collaborative–Multimodal Fusion
Existing SID construction often relies on item content and does not directly model the interaction patterns captured by collaborative representations. SIDRec combines these two sources before quantization. We initialize the collaborative embedding table from a trained ranking model, where is the item set and is its -th row. The table continues to be updated during joint training, while the multimodal embeddings remain fixed.
We project both representations into a shared space and control the collaborative contribution with a learnable sigmoid gate. The fused representation is
where is the sigmoid function, is a trainable scalar, and and are projection matrices. The result is passed to the SID tokenizer. This step makes both item content and interaction-derived preferences available for SID learning. It is expected to help the resulting codes retain preference-relevant item differences for CTR prediction.
End-to-End SID Adaptation
In conventional SID-enhanced ranking, item codes are often learned offline and then fixed, so ranking feedback cannot refine their assignments. SIDRec starts from a pretrained tokenizer and updates its encoder and codebooks together with the ranking model.
The SID encoder first produces . The residual quantizer then selects a codeword at each level to represent the remaining information. Starting from , the selected codeword and the next residual are
where is the codebook at level and is its selected row. The corresponding row index is . The quantized representation is decoded as .
To pass CTR gradients to the tokenizer, we use straight-through quantization. Each item uses a hard SID code in the forward pass, while gradients follow Gumbel-perturbed soft codeword probabilities. Let denote this assignment. Its SID embedding is
where is the trainable SID embedding table at level . This gives the embedding of the selected code during the forward pass and carries CTR gradients back to the tokenizer.
The candidate branch projects together with the SID embeddings into a candidate representation . The history branch uses the same token embeddings to model interactions over time. Let contain the SID embeddings of in temporal order and let denote positional embeddings. The causal Transformer applies masked self-attention and feed-forward transformations to obtain
where contains the contextual token states. Let denote the state for level of history item . We obtain its item-level representation by and a candidate-independent history summary by , with pooling over valid history positions.
For CTR prediction, we use DIN attention to summarize history in relation to the candidate item. With , the candidate-aware user representation is
where Attn denotes DIN target attention, which scores each history item using the candidate, history representation, their difference, and their elementwise product. The resulting captures the historical interests relevant to the current candidate. Together, the history and candidate branches let the CTR task update SID construction using preference signals from both sides of the interaction.
Joint Optimization
SIDRec learns click prediction together with an auxiliary SID prediction task. The CTR task uses candidate-aware interests, while SID prediction uses the history summary to predict the codes of the positive target item. We employ a Multi-Gate Mixture of Experts (MMoE) to jointly optimize these two objectives by sharing four experts across the two tasks and using a separate gate for each task.
The CTR input combines , , and their elementwise product . The SID input contains with the remaining dimensions padded. For , task-specific gating gives
where is the task-specific gating matrix, contains the expert weights, and is the representation passed to task .
The CTR tower maps its task representation to the predicted click probability
where is the CTR prediction tower. The SID tower predicts a distribution over the codes at each quantization level
where is conditioned on the observed history.
The CTR loss is computed over all labeled training examples
where denotes the total number of training examples and BCE is binary cross-entropy.
The SID prediction loss measures how well the history representation predicts the positive target’s SID at every level
where denotes the number of positive examples and is the probability assigned to the observed target code at level . These codes are supplied by the current tokenizer and used as fixed labels within each update. The quantization objective also reconstructs the multimodal input to preserve item information during SID adaptation. For reconstructed items, it is
where includes the codebook and commitment penalties of residual quantization. The SID learning objective combines target-code prediction with quantization regularization
where and control their relative contributions.
Finally, CTR prediction and SID learning are optimized through a unified objective
This objective lets click prediction guide SID adaptation through the differentiable tokenizer while the SID learning terms support target-code prediction and multimodal reconstruction. The shared experts also allow both tasks to learn from the same history representations.
Experiments
In this section, we evaluate SIDRec on public datasets and in real-world industrial scenarios to assess its effectiveness for fine ranking. Our experiments address three research questions.
RQ1. How does SIDRec perform on public benchmarks and across user activity levels?
RQ2. How do collaborative information and end-to-end optimization affect ranking performance and the learned SID structure?
RQ3. How does SIDRec perform across industrial domains and prediction tasks?
| Statistic | Electronics | ML-20M | Beauty |
| Users | 192,403 | 138,287 | 22,363 |
| Items | 63,001 | 20,720 | 12,101 |
| Interactions | 1,689,188 | 9,995,410 | 198,502 |
| Average history | 8.78 | 72.28 | 8.88 |
| Maximum history | 431 | 3,177 | 204 |
| Sparsity | 99.986% | 99.651% | 99.927% |
Table 1. Statistics of the processed public datasets. History lengths are measured before truncation.
Experimental Settings
Datasets
Public datasets. We use Amazon Electronics, Amazon Beauty, and MovieLens-20M, covering product and movie recommendation. The Amazon datasets use the 5-core review data. For MovieLens-20M, ratings of at least 4 are treated as positive interactions. Table 1 reports the processed dataset statistics. Each item has a fixed 4,096-dimensional semantic representation.
The public preprocessing pipeline sorts each user’s interactions by time. For eligible users, the last interaction is held out for testing and the preceding one for validation. Earlier interactions form history–target training pairs. We evaluate candidate-level binary prediction with sampled negatives. All compared methods use the same data splits and evaluation candidate sets.
Industrial datasets. We also evaluate SIDRec on two domains of a large-scale recommendation platform, denoted as Domain 1 and Domain 2. Each domain includes three prediction targets, namely exposure (Show), clicks (CTR), and conversions (CVR). These domains provide real-world interaction data for evaluating SIDRec across multiple industrial prediction objectives.
Baselines
We compare SIDRec against fifteen representative CTR models, grouped according to their primary modeling mechanisms.
Linear and factorized models. LR models additive feature effects. FM [18] captures pairwise interactions through factorized parameters.
Feature interaction networks. DeepFM [4] combines factorization machines with a deep network. DCNv2 [21] combines explicit feature crosses with nonlinear transformations. FinalMLP [13] uses two MLP streams with feature selection and bilinear fusion. GDCN [20] introduces gated cross layers to filter high-order interactions. WuKong [26] stacks factorization-machine layers to capture high-order interactions. RankMixer [32] adopts efficient token mixing to scale feature interactions. QNN [7] models feature interactions using quadratic neural transformations.
Personalized networks. PPNet [1] uses personalized gates to modulate deep-network parameters.
Interest and behavior models. DIN [31] uses target-aware attention to aggregate historical behaviors. DIEN [30] models the temporal evolution of user interests. TransAct [24] uses a Transformer to capture short-term preferences from real-time user actions.
Unified sequence–interaction models. OneTrans [27] jointly models feature interactions and behavior sequences in a single Transformer. HyFormer [5] integrates long-sequence modeling and heterogeneous feature interactions in a hybrid Transformer.
Evaluation Metrics
For public datasets, we report AUC and GAUC. AUC measures discrimination over all evaluated examples. GAUC averages per-user AUC scores with weights proportional to each user’s number of examples. Users with only one label class are excluded from GAUC. Higher AUC and GAUC indicate better performance. Industrial evaluation reports AUC and the platform’s user-level metrics, UAUC and WUAUC.
Relative gains in the public comparison are computed against the corresponding baseline for each metric. Component ablation reports relative percentage drops (%), while industrial gains are expressed in percentage points (pp).
Implementation Details
We evaluate SIDRec across diverse public benchmarks, industrial scenarios, and ranking objectives.
General backbone. On both public benchmarks and industrial scenarios, SIDRec uses a causal Transformer to encode historical SID tokens and a DIN attention module to extract candidate-aware interests. The candidate representation combines the tokenizer latent with embeddings of its SID codes. Four shared MMoE experts with task-specific gates support CTR prediction and auxiliary prediction of the positive target’s SID. The SID tokenizer combines semantic–collaborative fusion, an MLP encoder, residual quantization, and a reconstruction decoder. During CTR training, the pretrained semantic tables remain fixed, while the collaborative embedding, fusion projections, tokenizer, and ranking network are finetuned.
General reference configuration. The code configures three residual codebooks, each containing 256 codewords, with a latent dimension of 256. The history encoder has two Transformer layers, a hidden size of 128, four attention heads, a feed-forward width of 512, and dropout of 0.1. Histories are truncated to the most recent 50 items. The DIN attention width is 64, and the expert and prediction-tower hidden widths are 128. The optimizer is AdamW, with reference learning rates of for the ranker and for the tokenizer during joint training. Checkpoint selection uses validation GAUC. The reference CTR protocol samples 10 negatives per positive for training and 100 for evaluation. Its MovieLens configuration caps training at 1,000,000 history–target pairs and leaves evaluation uncapped. The reported public results use seed 2026.
Industrial setup. In real-world industrial scenarios, we evaluate SIDRec against the corresponding industrial baseline for each domain and each of the Show, CTR, and CVR targets. Performance is assessed through offline changes in prediction metrics.
Performance on Public Datasets (RQ1)
We first evaluate SIDRec against representative CTR baselines on three public datasets and examine its performance across different user activity levels.
Overall Comparison
Table 2 compares SIDRec with fifteen baselines. The results reveal two main patterns.
Overall ranking quality. SIDRec achieves the highest AUC and GAUC on all three datasets and ranks first by the mean of the six per-metric ranks. TransAct is the strongest baseline for all six metrics and ranks second overall, followed by OneTrans and HyFormer in third and fourth place. Relative AUC gains over TransAct are 3.29%, 0.32%, and 4.27% on Electronics, MovieLens-20M, and Beauty, respectively. The corresponding GAUC gains are 3.19%, 0.43%, and 5.15%. These improvements indicate stronger global discrimination and per-user ranking quality.
Dataset differences. The largest relative gains in both metrics occur on Beauty, followed by Electronics and MovieLens-20M. Both Amazon datasets have shorter average interaction histories than MovieLens-20M. On MovieLens-20M, SIDRec improves AUC from 0.97989 to 0.98302 and GAUC from 0.97573 to 0.97990.
These results show that SIDRec outperforms all fifteen CTR baselines in both AUC and GAUC across the three public benchmarks.
User Activity
We group Beauty users by their historical interaction lengths to examine how performance varies with user activity. Table 3 shows two main patterns.
History-length trend. AUC and GAUC increase across the five groups, from 0.8121 and 0.7973 for users with 3–5 interactions to 0.9189 and 0.8846 for users with more than 50 interactions. This trend indicates that SIDRec effectively leverages richer interaction histories to model user interests.
User coverage. Users with 3–10 interactions account for 84.1% of the evaluated users, whereas the longest-history group contains 144 users. The concentration of users in the short-history groups highlights the importance of effective ranking for users with limited interactions.
These results show that SIDRec achieves effective ranking across different activity levels, including the short-history users who constitute the majority of the Beauty dataset.
Analysis of SID Learning (RQ2)
We then conduct ablation studies to assess the contributions of collaborative fusion and end-to-end optimization, followed by analyses of how collaborative signals affect SID assignments and their structural consistency.
Component Ablation
We conduct an ablation study on Electronics to investigate the contributions of collaborative fusion and end-to-end SID optimization. Specifically, w/o Collaborative View removes collaborative information from SID construction, w/o End-to-end SID freezes SID construction during ranking training, and w/o Both removes both components. Table 4 reports these ablation results.
Effectiveness of end-to-end optimization. Removing end-to-end SID optimization produces relative AUC and GAUC drops of 1.97% and 1.96%, respectively. These results show the contribution of end-to-end SID optimization to ranking performance.
Contribution of collaborative fusion. Removing collaborative information leads to relative AUC and GAUC drops of 0.39% and 0.44%, respectively. These results show that
| Category | Model | Electronics AUC | Δ | Electronics GAUC | Δ | MovieLens-20M AUC | Δ | MovieLens-20M GAUC | Δ | Beauty AUC | Δ | Beauty GAUC | Δ | Rank |
| Linear | LR | 0.75960 | +15.79% | 0.75912 | +15.60% | 0.94158 | +4.40% | 0.94143 | +4.09% | 0.66963 | +23.02% | 0.66640 | +20.97% | 16 |
| FM | 0.76155 | +15.49% | 0.76176 | +15.20% | 0.92802 | +5.93% | 0.93990 | +4.26% | 0.72163 | +14.15% | 0.70956 | +13.61% | 11 | |
| Interaction | DeepFM | 0.76340 | +15.21% | 0.75985 | +15.49% | 0.94452 | +4.08% | 0.94517 | +3.67% | 0.71900 | +14.57% | 0.70124 | +14.96% | 13 |
| DCNv2 | 0.76428 | +15.08% | 0.76084 | +15.34% | 0.94725 | +3.78% | 0.94884 | +3.27% | 0.72021 | +14.38% | 0.70647 | +14.11% | 8 | |
| FinalMLP | 0.76283 | +15.30% | 0.76035 | +15.42% | 0.94484 | +4.04% | 0.94480 | +3.72% | 0.71699 | +14.89% | 0.70083 | +15.03% | 14 | |
| GDCN | 0.76418 | +15.10% | 0.76132 | +15.27% | 0.94685 | +3.82% | 0.94694 | +3.48% | 0.72321 | +13.90% | 0.70681 | +14.05% | 8 | |
| WuKong | 0.76471 | +15.02% | 0.76262 | +15.07% | 0.94832 | +3.66% | 0.94903 | +3.25% | 0.70024 | +17.64% | 0.68648 | +17.43% | 10 | |
| RankMixer | 0.82956 | +6.02% | 0.82776 | +6.02% | 0.97298 | +1.03% | 0.96928 | +1.10% | 0.76518 | +7.66% | 0.74149 | +8.72% | 5 | |
| QNN | 0.75839 | +15.97% | 0.75718 | +15.90% | 0.94641 | +3.87% | 0.94461 | +3.74% | 0.68115 | +20.94% | 0.67123 | +20.10% | 15 | |
| Personalized Interest | PPNet | 0.76228 | +15.38% | 0.75994 | +15.48% | 0.94589 | +3.93% | 0.94810 | +3.35% | 0.71630 | +15.00% | 0.70215 | +14.81% | 11 |
| DIN | 0.81627 | +7.75% | 0.81516 | +7.66% | 0.97206 | +1.13% | 0.96797 | +1.23% | 0.74149 | +11.10% | 0.72598 | +11.04% | 7 | |
| DIEN | 0.83576 | +5.24% | 0.83408 | +5.22% | 0.97318 | +1.01% | 0.96791 | +1.24% | 0.75811 | +8.66% | 0.73620 | +9.50% | 5 | |
| TransAct | 0.85154 | +3.29% | 0.85045 | +3.19% | 0.97989 | +0.32% | 0.97573 | +0.43% | 0.79001 | +4.27% | 0.76663 | +5.15% | 2 | |
| Unified | OneTrans | 0.84125 | +4.55% | 0.84002 | +4.47% | 0.97605 | +0.71% | 0.97269 | +0.74% | 0.77914 | +5.73% | 0.75543 | +6.71% | 3 |
| HyFormer | 0.83535 | +5.29% | 0.83556 | +5.03% | 0.97534 | +0.79% | 0.97125 | +0.89% | 0.77842 | +5.82% | 0.75616 | +6.61% | 4 | |
| Ours | SIDRec | 0.87954 | 0.87758 | 0.98302 | 0.97990 | 0.82376 | 0.80614 | 1 |
Table 2. Performance comparison on three public datasets. Each column reports the relative improvement (%) of SIDRec over the corresponding baseline. Darker green cells indicate larger gains within each metric, and all gain values are printed in black. Bold denotes the best result and underlining denotes the strongest baseline. Rank is determined by the mean of the six per-metric ranks. Equal mean ranks share the same rank.
| History Length | Users (%) | AUC | GAUC |
| 3–5 | 11,383 (50.9%) | 0.8121 | 0.7973 |
| 6–10 | 7,433 (33.2%) | 0.8213 | 0.8044 |
| 11–20 | 2,528 (11.3%) | 0.8533 | 0.8281 |
| 21–50 | 875 (3.9%) | 0.8849 | 0.8597 |
| 51+ | 144 (0.6%) | 0.9189 | 0.8846 |
| Overall | 22,363 (100%) | 0.8238 | 0.8061 |
Table 3. Performance of SIDRec-CTR across different user activity levels on the Beauty dataset.
collaborative fusion improves the discriminative capability of the learned SID representations.
Impact of joint removal. Removing both components produces the largest relative drops, with AUC decreasing by 7.19% and GAUC by 7.11%. This larger degradation supports the combined contribution of collaborative fusion and end-to-end SID optimization.
Overall, the ablation results show that both collaborative fusion and end-to-end SID optimization contribute to SIDRec’s ranking performance.
Collaborative Sensitivity
We examine how collaborative inputs influence the discrete item identifiers. We randomly shuffle collaborative embeddings across items and recompute the first-level SID assignments while keeping model parameters fixed. As shown in Figure 4, 51.0%, 93.0%, and 53.0% of items change their assignments on Electronics, MovieLens-20M, and Beauty, respectively. MovieLens-20M exhibits the highest reassignment rate. These results demonstrate that item-level collaborative information directly influences discrete SID assignments across all three datasets.
Figure 4. First-level SID assignments after shuffling collaborative embeddings across items with model parameters fixed.
| Variant | AUC | Drop | GAUC | Drop |
| SIDRec | 0.87954 | – | 0.87758 | – |
| w/o Collaborative View | 0.87610 | 0.39%↓ | 0.87372 | 0.44%↓ |
| w/o End-to-end SID | 0.86223 | 1.97%↓ | 0.86040 | 1.96%↓ |
| w/o Both | 0.81627 | 7.19%↓ | 0.81516 | 7.11%↓ |
Table 4. Component ablation on Electronics.
SID Consistency
Table 5 examines latent representation stability and the structural consistency of first-level SID assignments.
| Metric | Electronics | ML-20M | Beauty |
| Representation Stability | |||
| Latent Cosine (Changed) | 0.985 | 0.954 | 0.989 |
| Latent Cosine (Unchanged) | 0.990 | 0.979 | 0.993 |
| Structural Consistency | |||
| Codebook Cosine (Random) | 0.84 (0.67) | 0.87 (0.80) | 0.89 (0.69) |
| Token Cosine (Random) | 0.34 (0.00) | 0.25 (0.08) | 0.26 (0.02) |
| Partition ARI | 0.710 | 0.374 | 0.601 |
| Partition NMI | 0.852 | 0.675 | 0.835 |
Table 5. Semantic consistency of SID assignments under collaborative perturbation.
Latent representations. Items with changed assignments retain average latent cosine similarities of 0.985, 0.954, and 0.989 across the three datasets. The corresponding values for unchanged items are 0.990, 0.979, and 0.993. These high similarities show that the latent representations remain closely aligned as discrete SID assignments change.
Code migration. Similarities between the original and reassigned codewords exceed the random-slot baselines on all datasets. Token similarities also exceed their baselines, reaching 0.34, 0.25, and 0.26 compared with 0.00, 0.08, and 0.02. This pattern indicates that SID reassignments favor related codewords rather than arbitrary alternatives.
Partition agreement. ARI ranges from 0.374 to 0.710, and NMI ranges from 0.675 to 0.852. MovieLens-20M has the lowest agreement under both measures, consistent with its higher reassignment rate. These results characterize how SID partitions adapt to changes in collaborative signals across datasets.
These ablation and SID analyses show that collaborative information influences discrete item representations and that end-to-end SID optimization improves ranking performance.
SID Assignment Dynamics During Training
To investigate the stability of learned Semantic IDs (SIDs), we analyze their assignment dynamics throughout training across three datasets, as shown in Figure 5. Across all datasets, SID reassignment rates decrease as training progresses. By the final epoch, the reassignment rates of complete SIDs reach 18.2%, 20.9%, and 15.1% on Electronics, MovieLens-20M, and Beauty, respectively. The corresponding first-level (L1) rates decrease to 2.3%, 4.2%, and 3.1%, remaining below those at deeper levels (L2 and L3). These results show that SID assignments become progressively more stable during end-to-end training, with the first-level assignments exhibiting the lowest change rates.
Figure 5. SID assignment dynamics during training across three datasets.
Industrial Evaluation (RQ3)
Finally, we validate SIDRec in real-world recommendation scenarios on our company’s large-scale recommendation platform. We conduct offline evaluations across two industrial domains and three prediction tasks, including Show, CTR, and CVR. These experiments aim to verify the practical effectiveness and generalizability of SIDRec across diverse industrial ranking objectives.
Table 6 reports offline gains over the corresponding baselines. UAUC and WUAUC improve in all six domain–task pairs, and AUC improves in five pairs. The three prediction objectives exhibit the following improvements.
| Domain | Task | AUC | UAUC | WUAUC |
| Domain 1 | Show | +0.50 | +0.62 | +1.13 |
| Domain 1 | CTR | +0.36 | +1.15 | +1.05 |
| Domain 1 | CVR | +1.08 | +2.57 | +2.48 |
| Domain 2 | Show | +0.42 | +0.60 | +1.11 |
| Domain 2 | CTR | +0.08 | +0.06 | +0.06 |
| Domain 2 | CVR | +0.00 | +1.97 | +2.08 |
Table 6. Offline gains over industrial baselines in percentage points. Darker green indicates larger gains within each metric. The largest gain within each domain and metric is bold.
Exposure prediction. Show gains are consistent across the two domains. AUC improves by 0.50 and 0.42 pp, UAUC by 0.62 and 0.60 pp, and WUAUC by 1.13 and 1.11 pp. These closely matched improvements demonstrate SIDRec’s effectiveness for exposure prediction in both industrial settings.
Click prediction. Domain 1 improves by 0.36 pp in AUC, 1.15 pp in UAUC, and 1.05 pp in WUAUC. Domain 2 improves by 0.08, 0.06, and 0.06 pp, respectively. The gains across all three metrics demonstrate consistent improvements in click prediction across both domains.
Conversion prediction. CVR achieves the largest UAUC and WUAUC gains within each domain. Domain 1 improves by 1.08 pp in AUC, 2.57 pp in UAUC, and 2.48 pp in WUAUC. Domain 2 records a change of +0.00 pp in AUC, with improvements of 1.97 pp in UAUC and 2.08 pp in WUAUC. These results show the strongest user-level improvements for conversion prediction.
In summary, SIDRec consistently improves prediction quality across both industrial domains and all three tasks, with gains in UAUC and WUAUC across all six domain–task pairs and AUC improvements in five. The particularly strong gains in conversion prediction highlight its effectiveness in capturing user-level behavioral patterns, demonstrating the practical value of integrating semantic and collaborative information into industrial recommendation models.
Conclusion
In this paper, we proposed SIDRec, a recommendation-aware semantic ID learning framework for discriminative recommendation. To address the semantic and objective mismatches in conventional SID-based methods, SIDRec integrates content and collaborative representations into SID construction and enables end-to-end optimization under downstream recommendation supervision. By aligning discrete representation learning with recommendation objectives, SIDRec captures preference-relevant information and produces more discriminative semantic identifiers. Extensive experiments on three public benchmarks demonstrate that SIDRec consistently outperforms all competing methods across both AUC and GAUC, achieving up to 5.15% relative improvements over the strongest baselines. Further ablation studies confirm the effectiveness of incorporating collaborative semantics and jointly optimizing SID assignments with recommendation objectives. Overall, these findings highlight the importance of recommendation-aware semantic ID learning and establish SIDRec as an effective framework for discriminative recommendation.
References
- [1]Jianxin Chang, Chenbin Zhang, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, and Kun Gai. 2023. PEPNet: Parameter and Embedding Personalized Network for Infusing with Personalized Prior Information. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3795–3804.arxiv.org/abs/2302.01115
- [2]Jiaxin Deng, Shiyao Wang, Kuo Cai, Leijian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou. 2025. Onerec: Unifying retrieve and rank with generative recommender and iterative preference alignment. arXiv preprint arXiv:2502.18965 (2025).
- [3]Junchen Fu, Xuri Ge, Alexandros Karatzoglou, Ioannis Arapakis, Suzan Verberne, Jeomon M Jose, and Zhaochun Ren. 2026. Differentiable semantic id for generative recommendation. arXiv preprint arXiv:2601.19711 (2026).
- [4]Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. 1725–1731.arxiv.org/abs/1703.04247
- [5]Yunwen Huang, Shiyong Hong, Xijun Xiao, Jinqiu Jin, Xuanyuan Luo, Zhe Wang, Zheng Chai, Shikang Wu, Yuchao Zheng, and Jingjian Lin. 2026. HyFormer: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. 590–599.
- [6]Guowen Li, Yuepeng Zhang, Shunyu Zhang, Yi Zhang, Xiaozhe Jiang, Yi Wang, and Jingwei Zhuo. 2026. SID-Coord: Coordinating Semantic IDs for ID-based Ranking in Short-Vided Search. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. 4737–4741. doi:10.1145/3805712.3808436DOI
- [7]Honghao Li, Yiwen Zhang, Yi Zhang, Lei Sang, and Jieming Zhu. 2025. Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Prediction. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. 1365–1375.
- [8]Fake Lin, Binbin Hu, Zhi Zheng, Xi Zhu, Ziqi Liu, Zhiqian Zhang, Jun Zhou, and Tong Xu. 2026. Token-level Collaborative Alignment for LLM-based Generative Recommendation. In Proceedings of the ACM Web Conference 2026. 6909–6919.
- [9]Enze Liu, Bowen Zheng, Cheng Ling, Lantao Hu, Han Li, and Wayne Xin Zhao. 2025. Generative recommender with end-to-end learnable item tokenization. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 729–739.DOI
- [10]Wei Liu, Bo Yang, Qianyang Zhu, Ruijie Xiao, Sirui Huang, and Qing Li. 2026. CSG4Rec: Collaborative-Similarity Guided Semantic IDs Tokenization for Generative Sequential Recommendation. Information Fusion (2026), 104853.DOI
- [11]Xinch en Luo, Jiangxia Cao, Tianyun Sun, Jinkai Yu, Rui Huang, Wei Yuan, Hezheng Lin, Yichen Zheng, Shiyao Wang, Qigen Hu, et al. 2025. Qarm: Quantitative alignment multi-model recommendation at kuaishou. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management. 5915–5922.arxiv.org/abs/2411.11739
- [12]Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1930–1939.DOI
- [13]Kelong Mao, Jieming Zhu, Liangcai Su, Guohao Cai, Yuru Li, and Zhenhua Dong. 2023. FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction. In Proceedings of the AAAI Conference on Artificial Intelligence. 4552–4560.
- [14]Yongsen Pan, Yuxin Chen, Zheng Hu, Xu Yuan, Daoyuan Wang, Yuting Yin, Songhao Ni, Hongyang Wang, Jun Wang, Fuji Ren, et al. 2026. Beyond Static Collision Handling: Adaptive Semantic ID Learning for Multimodal Recommendation at Industrial Scale. arXiv preprint arXiv:2604.23522 (2026).arxiv.org/abs/2604.23522
- [15]Gustavo Penha, Eduardo D’Amico, Marco De Nadai, Enrico Palumbo, Alexandre Tamborrino, Ali Vardasbi, Max Lefarov, Shawn Lin, Timothy Heath, Francesco Fabbri, et al. 2025. Semantic ids for joint generative search and recommendation. In Proceedings of the Nineteenth ACM Conference on Recommender Systems. 1296–1301.
- [16]Haohao Qu, Wenqi Fan, Zhih uai Zhao, and Qing Li. 2025. Tokenrec: Learning to tokenize id for llm-based generative recommendations. IEEE Transactions on Knowledge and Data Engineering (2025).DOI
- [17]Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al. 2023. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems 36 (2023), 10299–10315.arxiv.org/abs/2305.05065
- [18]Steffen Rendle. 2010. Factorization Machines. In 2010 IEEE International Conference on Data Mining. 995–1000.DOI
- [19]Anima Singh, Trung Vu, Nikhil Mehta, Raghunandan Keshavan, Maheswaran Sathiamoorthy, Yilin Zheng, Lichan Hong, Lukasz Heldt, Li Wei, Devansh Tandon, et al. 2024. Better generalization with semantic ids: A case study in ranking for recommendations. In Proceedings of the 18th ACM Conference on Recommender Systems. 1039–1044.arxiv.org/abs/2306.08121
- [20]Fangye Wang, Hansu Gu, Dongsheng Li, Tun Lu, Peng Zhang, and Ning Gu. 2023. Towards Deeper, Lighter and Interpretable Cross Network for CTR Prediction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2525–2533.arxiv.org/abs/2311.04635
- [21]Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. In Proceedings of the Web Conference 2021. 1785–1797.DOI
- [22]Shuli Wang. 2026. Discrimination Is Generation: Unifying Ranking and Retrieval from a Tokenizer Perspective. arXiv preprint arXiv:2605.14853 (2026).arxiv.org/abs/2605.14853
- [23]Wenj ie Wang, Honghui Bao, Xinyu Lin, Jizhi Zhang, Yongqi Li, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. 2024. Learnable item tokenization for generative recommendation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2400–2409.arxiv.org/abs/2405.07314
- [24]Xue Xia, Peng Eksombatchai, Nikhil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and Andrew Zhai. 2023. TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5249–5259.DOI
- [25]Wencai Ye, Mingjie Sun, Shaoyun Shi, Peng Wang, Wenjin Wu, and Peng Jiang. 2025. Dual-aligned semantic ids empowered industrial recommender system. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management. 6217–6224.
- [26]Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, et al. 2024. Wukong: Towards a scaling law for large-scale recommendation. arXiv preprint arXiv:2403.02545 (2024).arxiv.org/abs/2403.02545
- [27]Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, and Aixin Sun. 2026. OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender. In Proceedings of the ACM Web Conference 2026. 8162–8170.arxiv.org/abs/2510.26104
- [28]Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, Ming Chen, and Ji-Rong Wen. 2024. Adapting large language models by integrating collaborative semantics for recommendation. In 2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 1435–1448.arxiv.org/abs/2311.09049
- [29]Carolina Zheng, Minhui Huang, Dmitrii Pedchenko, Kaushik Rangadurai, Siyu Wang, Fan Xia, Gaby Nahum, Jie Lei, Yang Yang, Tao Liu, et al. 2025. Enhancing embedding representation stability in recommendation systems with semantic id. In Proceedings of the Nineteenth ACM Conference on Recommender Systems. 954–957.arxiv.org/abs/2504.02137
- [30]Guoru Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep Interest Evolution Network for Click-Through Rate Prediction. In Proceedings of the AAAI Conference on Artificial Intelligence. 5941–5948.arxiv.org/abs/1809.03672
- [31]Guorui Zhou, Xiaoqiang Zhu, Chenyu Song, Ying Fan, Han Zhu, Xiao Yao, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1059–1068.arxiv.org/abs/1706.06978
- [32]Jie Zhu, Zhifang Fan, Xiaoxie Zhu, Yuchen Jiang, Hangyu Wang, Xintian Han, Haoran Ding, Xinmin Wang, Wenlin Zhao, Zhen Gong, Huizhi Yang, Zheng Chai, Zhe Chen, Yuchao Zheng, Qiwei Chen, Feng Zhang, Xun Zhou, Peng Xu, Xiao Yang, Di Wu, and Zuotao Liu. 2025. RankMixer: Scaling Up Ranking Models in Industrial Recommenders. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management. 6309–6316.
- [33]Yongchun Zhu, Ruobing Xie, Fuzhen Zhuang, Kaikai Ge, Ying Sun, Xu Zhang, Leyu Lin, and Juan Cao. 2021. Learning to Warm Up Cold Item Embeddings for Cold-start Recommendation with Meta Scaling and Shifting Networks. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1167–1176.DOI




