WorksComputer Science
Beyond Independent Predictions: Learning the Browsing Journey for Sequence-Level Re-ranking
Abstract
Re-ranking aims to present users with an ordered list that maximizes sequence-level utility. Industrial deployments widely adopt the Generator–Evaluator framework, where the generator proposes candidate sequences, and the evaluator scores them to select the best one for display. The fidelity of the evaluator's utility estimates thus directly governs recommendation quality. However, existing evaluators predominantly follow the Independent Behavior Modeling (IBM) paradigm, which simplifies the continuous user browsing journey into independent per-item predictions (e.g., CTR). This oversimplification ignores two physical realities: (i) a click can occur only after the item has been reached and impressed, and (ii) items compete for finite user attention. As a result, IBM produces systematically biased evaluations and suboptimal sequence selection. To address this, we propose a Joint Behavior Modeling (JBM) perspective that explicitly models the display impression click process to capture causal and competitive dependencies within a sequence. Building on JBM, we develop the Impression-Click Evaluator (ICE), a plug-and-play framework that (i) predicts a browsing-depth distribution to estimate per-position impression probabilities, (ii) leverages predicted impressions to suppress spurious context from unexposed items during click modeling, and (iii) introduces a click distribution matching objective to capture intra-sequence dependencies. Extensive offline experiments across diverse backbones, together with a 14-day online A/B test in the Taobao App (+0.70% clicks, +3.67% revenue) and full-traffic deployment, validate the effectiveness and robustness of our approach.
*Equal contribution.
†Corresponding author.
CCS Concepts
• Information systems → Recommender systems; Learning to rank; Evaluation of retrieval results.
ACM Reference Format:
Jiayi Zhang, Jingxin Hai, Zhenjia Ma, Hongbin Pei, Dagui Chen, Han Zhu, and Jian Xu. 2026. Beyond Independent Predictions: Learning the Browsing Journey for Sequence-Level Re-ranking. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 7–11, 2026, Rome, Italy. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3799682.3840169
Introduction
Modern industrial recommender systems [10, 14, 29] typically follow a multi-stage cascade architecture: retrieval [7, 17, 18, 24], ranking [9, 10, 15, 33], and re-ranking [2, 6, 27, 40]. As the final decision stage, re-ranking directly determines the ordered list displayed to users and must account for complex sequence-level effects such as inter-item interactions, rather than evaluating items in isolation. To this end, industrial practice has converged on the Generator–Evaluator (G–E) framework [11, 34, 35, 36], where the generator proposes multiple candidate sequences and the evaluator estimates their sequence-level utility to select the optimal one for display.
Accordingly, the performance of the G–E framework is fundamentally bounded by the evaluator’s ability to perform reliable sequence-level evaluation [38]. Sequence-level utility is not a static property intrinsic to items; rather, it is dynamically realized through the user browsing journey along the displayed sequence. However, as shown in Figure 1(A), existing evaluators predominantly follow the Independent Behavior Modeling (IBM) paradigm, which approximates the overall sequence response as a collection of independent per-item interactions (e.g., clicks). Such oversimplification ignores two fundamental characteristics of real-world user behavior.
Figure 1. Comparison of modeling paradigms for the evaluator in sequence-level re-ranking. (A) IBM treats user behaviors along the sequence as isolated events, oversimplifying the physical browsing process and causing systematic tail-utility overestimation. (B) JBM (ours) models the browsing journey by explicitly estimating impression probabilities to bridge the display-to-impression gap, while capturing intra-sequence dependencies via joint click modeling, yielding high-fidelity utility estimation.
This work is licensed under a Creative Commons Attribution 4.0 International License. CIKM ’26, Rome, Italy. © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2539-5/2026/11 https://doi.org/10.1145/3799682.3840169 First, there exists a universal display-to-impression gap (C1) driven by browsing drop-off. While the system displays a fixed-length sequence (e.g., 20 items), users often disengage early due to dwindling attention or unfavorable content, leaving tail items unseen. This gap impairs the evaluator in two ways: (i) it systematically overestimates tail-item utility, favoring sequences that look promising offline but fail to deliver online gains; (ii) it introduces spurious context, as unexposed items are mistakenly treated as valid neighbors for those actually seen, contaminating the contextual representations and degrading evaluation reliability.
Second, user interactions exhibit complex intra-sequence behavioral dependencies (C2). On one hand, a causal dependency arises as any interaction is strictly predicated on the item being successfully reached and impressed. On the other hand, items compete for a user’s finite attention and intent satisfaction, leading to a competitive dependency: once a user clicks an item and finds it satisfactory, the marginal utility of subsequent similar items drops significantly even if their individual quality is high. Our empirical analysis (Figure 4) confirms that IBM-style models struggle in such “multi-click” scenarios, as they treat clicks as isolated events and fail to capture how items shadow each other across the sequence.

Figure 4. GAUC on Single-click (top of each bar) and Multi-click (bottom) groups across six backbones, comparing vanilla (blue) and ICE-augmented (green) models. ICE consistently elevates overall performance (horizontal line) and narrows the performance gap between groups.
In this work, we focus exclusively on advancing the evaluator design, taking the generator as given. To address the above issues, we propose the Joint Behavior Modeling (JBM) paradigm. As illustrated in Figure 1(B), JBM reformulates the sequence-level evaluation by explicitly estimating impression probabilities to bridge the display-to-impression gap and modeling the joint distribution of interactions to capture intra-sequence behavioral dependencies. Building on JBM, we develop ICE (Impression-Click Evaluator), a plug-and-play framework with three core components: (i) browsing-depth modeling predicts positional impression (exposure) probabilities to capture early disengagement; (ii) impression-aware dynamic masking suppresses context noise from unexposed items during click modeling; (iii) click distribution matching captures competitive click patterns within a sequence. Extensive experiments on a large-scale industrial dataset, together with comprehensive ablation and rationality analyses, consistently validate ICE’s effectiveness across diverse evaluator backbones. A 14-day online A/B test in the Taobao App further yields +0.70% clicks and +3.67% commercial revenue, leading to successful large-scale full-traffic deployment for hundreds of millions of daily users.
The main contributions of this work are summarized as follows:
We formally identify the systematic bias in current sequence-level evaluation and propose JBM as a principle paradigm to ensure physical consistency and capture behavioral dependencies along the browsing journey.
We develop ICE, a backbone-agnostic, plug-and-play solution under the JBM paradigm that seamlessly enhances diverse industrial evaluator architectures.
We validate JBM and ICE through comprehensive offline and online evaluations, with successful full-traffic deployment further demonstrating their real-world impact.
Related Work
Re-ranking and the G–E Framework. Re-ranking refines an initially ranked list to maximize sequence-level utility under multiple objectives and constraints. Early work followed the Learning-to-Rank (LTR) framework, progressing from point-wise [9, 10, 15, 21, 33] and pair-wise [3, 4, 19, 39] to list-wise [5, 6] methods. Since items presented together mutually influence user behavior [8, 13, 20, 28], context-aware models (e.g., PRM [27]) and sequential encoders [1, 2, 40] are introduced to capture inter-item interactions. Nevertheless, they suffer from the evaluation-before-reranking dilemma, where items are scored in a context that is subsequently reshaped by the re-sorting process itself. Recent generative re-rankers [12, 16, 23, 30, 37] mitigate this mismatch via end-to-end sequence generation, but face deployment challenges in controllability and latency. As a practical alternative, the G–E framework has emerged as the industrial standard [11, 34, 35, 36, 38].
Evaluator Design. Within G–E, prior work has predominantly focused on the generator [31, 36], leaving the evaluator—the final decision-maker—significantly under-explored. Existing evaluators, despite diverse architectures (e.g., RNNs [11, 32], Transformers [34, 35]), largely follow the IBM paradigm, overlooking the display-to-impression gap and intra-sequence behavioral dependencies.
Impression Modeling. Exposure bias has been widely studied at the ranking stage [22, 25, 26], but remain under-explored at the sequence level. A recent work [38] models personalized consumption boundaries to mitigate estimation biases, yet still relies on factorized per-item modeling and ignores the competitive nature of user interactions. In contrast, our JBM paradigm jointly models the entire chain, rather than treating user responses as a collection of isolated per-item events.
Joint Behavior Modeling Paradigm
We formalize sequence-level evaluation in the G–E framework, contrast IBM with our proposed JBM paradigm, and instantiate JBM under the impression-click chain to guide the design in Sec. 4.
Problem Formulation
Let denote a candidate sequence of items proposed by the generator, and represent the target user with associated context. Sequence-level utility emerges from user behaviors along the displayed sequence, i.e., the browsing journey from the first to the last position. Let denote the collection of user behavioral events (e.g., impression, click) induced by user on , where corresponds to the event at position . The sequence-level utility can be defined as the expected utility:
where maps behavioral outcomes to utility (e.g., total clicks or revenue). The fidelity of thus hinges entirely on how faithfully is modeled, which is governed by the two characteristics highlighted in § 1: the display-to-impression gap (C1) and intra-sequence behavioral dependencies (C2).
From IBM to JBM
IBM paradigm. Most existing evaluators [11, 22, 25, 26, 32, 34, 35, 38] factorize as a product of per-position conditionals:
where each is treated as an independent response conditioned only on its item and the static sequence context . This factorization implicitly assumes uniform exposure across all positions (ignoring C1) and neglects how earlier behaviors shape later ones (ignoring C2).
JBM paradigm. To overcome these mismatches, we advocate the JBM paradigm, which models the joint distribution of all behavioral events across the sequence:
where each behavior is conditioned on all preceding behaviors . The key distinction from Eq. (2) is the conditioning on , which enables JBM to model how engagement at position depends on prior user activity (addressing C1) and explicitly captures how earlier interactions shape subsequent ones (addressing C2), within a unified probabilistic framework.
JBM with the Impression-Click Chain
In general recommendation systems, impressions and clicks are the most fundamental user behaviors that constitute the browsing journey. In this section, we instantiate JBM by explicitly modeling the joint distribution of the impression-click chain to perform more reliable sequence-level evaluation.
Accordingly, we represent the behavior sequence as a pair of binary event vectors , where denote the occurrence of exposure and click at position . To maintain physical consistency with the browsing process, we apply the chain rule to factorize Eq. (3) as:
The first factor accounts for the dynamic impression decay (C1), while the second factor models inter-dependent clicks conditioned on actual impressions (C2).
To clarify our notation, we distinguish between empirical rates observed in dataset for offline measurement and per-sequence probabilities predicted by the evaluator. Specifically, Impression-Through Rate (ITR) quantifies the proportion of displayed items that are actually impressed:
Click-Through Rate (CTR) follows the standard industrial definition over impressed items:
To measure the end-to-end conversion from display to click, we define the Impression-Through Click-Through Rate (ITCTR):
On the modeling side, our evaluator predicts per-position probabilities with and with . The predicted end-to-end click probability vector follows as:
i.e., for each position .
Impression-Click Evaluator Framework
As shown in Figure 2, Impression-Click Evaluator (ICE) is a plug-and-play framework that instantiates the JBM paradigm. To implement Eq. (4), ICE adopts a multi-task architecture where the impression and click branches share a common embedding layer and are physically coupled through impression-aware context denoising. We detail the three modular components below.
Figure 2. Overview of ICE. Starting from a shared embedding layer, the architecture bifurcates into the impression and click modeling branches to realize the JBM paradigm in Eq. (4). The two branches are coupled through three modular components: (i) browsing-depth modeling for position-dependent impression probabilities; (ii) impression-aware masking for context denoising; (iii) sequence-level click distribution matching for behavioral competition.
Browsing-depth Impression Modeling
To bridge the display-to-impression gap (C1), ICE models position-dependent impression probabilities. In feed interfaces, visibility follows a monotonic prior: later items are visible only if earlier positions are reached, implying .
To capture this physical process, the Impression Branch processes the shared representations to estimate the user’s browsing depth as a latent categorical variable , representing the last position reached before disengagement. It predicts a categorical distribution , from which the impression probability for each position is derived as the cumulative distribution:
which guarantees the strict monotonicity and effectively corrects the overestimation of tail items.
Impression-aware Dynamic Masking
Even though the evaluator takes the full sequence as input, unexposed items should not contribute contextual signals for click modeling. To suppress such spurious context (C1), ICE utilizes the predicted as confidence signals to modulate contextual interactions within the Click Branch. This module is backbone-agnostic and supports two implementations:
Transformer-based backbones (Attention Bias). We define an impression-confidence matrix where , transformed into a stable additive bias . For any self-attention layer with , , and , the attention is computed as:
This bias effectively suppresses attention weights involving unexposed items. We treat as a detached (stop-gradient) signal to prevent click modeling from biasing the impression modeling task.
DNN-style backbones (Feature Gating). For backbones without attention mechanisms, ICE implements masking via feature gating. Let be the contextual representation of position . We modulate it using a gating function :
where controls the suppression intensity. Similarly, is treated as a stop-gradient signal.
Sequence-level Click Distribution Matching
To capture behavioral dependencies (C2), ICE augments point-wise click supervision with a sequence-level click distribution matching objective. This forces the evaluator to model how items compete for the user’s limited attention across the entire sequence.
For each sequence , let denote the click logits. The predicted click-allocation distribution is obtained via a temperature-scaled softmax: . The target distribution is constructed by normalizing the observed clicks: if , ; otherwise, to provide a non-informative target for zero-click sequences. We then match the two distributions via KL divergence:
Unlike point-wise objectives that score each item independently, the softmax normalization in imposes a finite probability budget across the sequence: a higher predicted score at one position structurally suppresses others, directly capturing the competitive nature of clicks that point-wise losses cannot express.
Training and Inference
ICE is trained end-to-end in a multi-task fashion, optimizing three objectives simultaneously:
Impression Modeling: The Impression Branch is supervised by the observed labels over all positions via binary cross-entropy (BCE) loss: .
Click Modeling: Following the impression-click chain (Sec. 3.3), the Click Branch takes impression-aware masked representations (Sec. 4.2) and is trained on impressed items () via a masked BCE loss: .
Combined with the sequence-level click distribution matching loss from Eq. (12), the overall objective is: .
During inference, the evaluator scores each candidate sequence by instantiating the expected utility in Eq. (1).
where is an item-specific reward configurable to different deployment objectives: setting yields expected total clicks, while setting to the predicted purchase value or bid price targets expected GMV or ad revenue, respectively. The candidate sequence with the highest is selected for display.
Experiments
We organize experiments around four research questions:
Q1 (Overall Performance): Is ICE a plug-and-play booster that consistently improves diverse evaluator backbones?
Q2 (Ablation Study): How does each ICE component contribute?
Q3 (Mechanism Analysis): Why do these components work?
Q4 (Online Test): Does ICE deliver real-world gains in production?
Experiment Settings
Baselines
To verify the generality of ICE, we evaluate it by integrating with a range of representative evaluator backbones from prior work, all of which fall under the IBM paradigm and span both item-wise scorers and context-aware sequential models:
DNN / DCN [9, 33]: MLP-based models that predict item-level utility with static contextual features.
PRM [27]: A personalized re-ranking model that utilizes self-attention to capture intra-list item interactions.
BiGRNN / Transformer-based Evaluators [32, 35]: Advanced sequential evaluators that employ bidirectional GRUs or multi-head attention to model global dependencies within the sequence.
NLGR-E [34]: A recent evaluator that models relationships in the feature space to capture sequence-level dependencies.
Datasets
We conduct offline experiments on a large-scale industrial E-commerce Re-ranking Dataset, comprising hundreds of millions of user interaction logs. Each instance records a user’s browsing journey on a displayed sequence of length , including binary labels for impressions (impr) and clicks (click). We employ standard industrial time-based splitting: the first 6 days for training and the 7th day for testing, ensuring no temporal leakage. Table 1 summarizes the key statistics, illustrating a clear conversion funnel from initial display to final click. Notably, the significant decay between displayed and impressed items justifies the necessity of explicit impression modeling.
| Dataset | # User | # Sequence | # Disp. | # Impr. | # Click |
| E-com. | 48.38M | 141.79M | 2,835.71M | 1,652.35M | 35.45M |
Table 1. Statistics of the E-commerce Re-ranking Dataset. Note: "Disp.", "Impr.", and "Click" represent the total number of items that are displayed, impressed and clicked, respectively. "M" denotes millions.
Evaluation Metrics
We evaluate ICE on three perspectives:
(1) Click Prediction Metrics. We report the group-AUC over impressed items, denoted as GAUC, computed from predicted conditional click probabilities . Following standard practice, GAUC is computed by grouping instances by user and then taking a weighted average across groups. To evaluate the end-to-end click probability on displayed items, we additionally report GAUC, computed from
for all displayed positions. For baseline models that lack impression modeling, we apply the plug-in assumption .
(2) Impression Modeling Metrics. In feed browsing, impressions are highly imbalanced across positions and exhibit a strict cascade, making GAUC less informative as they can be inflated by position priors. We thus employ two complementary metrics:
Position-AUC (PAUC). We compute the AUC metric within each position and take a weighted average:
where is the number of impressed samples at position .
Browsing-depth Error (MAE). We calculate the Mean Absolute Error between the true exit position and the predicted depth :
where denotes the expected browsing depth calculated by .
(3) Sequence-level Evaluation Metrics. Since the evaluator’s primary task is candidate sequence selection rather than item re-ordering, item-level metrics (e.g., GAUC) are insufficient for assessing the overall utility of the displayed sequence. We thus propose SeqAUC, a sequence-level metric that measures the consistency between the ground-truth utility and the evaluator’s predicted utility defined in (13).
In this work, we instantiate the ground-truth utility as the observed total clicks: , and the evaluator’s score as the predicted expected total clicks: . Let denote the set of all ordered pairs from the test set. SeqAUC is computed as:
A higher SeqAUC indicates a superior capability of the evaluator to identify candidate sequences that yield greater overall utility. Moreover, SeqAUC can be readily extended to other objectives by redefining and .
Overall Comparison Across Backbones (Q1)
To verify the effectiveness of ICE, we integrate it with various backbones and report performance on impression modeling (PAUC, MAE), click prediction (GAUC, GAUC), and sequence-level evaluation (SeqAUC). All experiments use a unified setup with , , and to align the loss scales.
As shown in Table 2, ICE yields consistent and significant gains across all backbones and metrics. Notably, the GAUC on displayed items exhibits a substantial and robust jump (e.g., +11.38% for the Transformer backbone). This gain is primarily attributed to the explicit impression modeling, which effectively corrects the systematic overestimation of utility at tail positions. These improvements directly translate into higher SeqAUC, demonstrating that ICE-enhanced evaluators are significantly more capable of identifying sequences with higher actual utility.
| Impression Modeling | Click Prediction | Seq. Value Estim. | |||
| PAUC (%) | MAE | GAUC (%) | GAUC (%) | SeqAUC (%) | |
| DNN | 50.00 0.00 | 3.92 0.00 | 65.09 0.49 | 64.51 0.12 | 82.11 0.56 |
| +ICE | 60.48 2.15 | 1.17 0.06 | 65.83 0.14 | 72.48 0.39 | 84.87 0.15 |
| DCN | 50.00 0.00 | 3.92 0.00 | 65.24 0.14 | 65.01 0.07 | 82.44 0.06 |
| +ICE | 59.38 1.38 | 1.51 0.32 | 65.82 0.51 | 72.30 0.39 | 84.33 0.45 |
| PRM | 50.00 0.00 | 3.92 0.00 | 63.72 1.90 | 62.37 0.25 | 81.22 0.57 |
| +ICE | 73.24 1.48 | 1.03 0.01 | 65.84 0.04 | 73.94 0.34 | 85.65 0.11 |
| BiGRNN | 50.00 0.00 | 3.92 0.00 | 65.05 0.15 | 63.00 0.18 | 81.38 0.25 |
| +ICE | 71.44 0.38 | 1.04 0.03 | 65.90 0.11 | 73.92 0.36 | 85.56 0.03 |
| Trans. | 50.00 0.00 | 3.92 0.00 | 64.07 1.06 | 62.53 0.18 | 81.31 0.16 |
| +ICE | 73.84 0.54 | 1.04 0.01 | 65.86 0.07 | 73.91 0.15 | 85.84 0.07 |
| NLGR-E | 50.00 0.00 | 3.92 0.00 | 65.15 0.06 | 62.90 0.08 | 81.55 0.08 |
| +ICE | 70.01 0.68 | 1.04 0.01 | 65.84 0.10 | 73.92 0.04 | 85.50 0.05 |
Table 2. Overall performance comparison across diverse backbones (Q1). For baselines without impression modeling, PAUC is naturally 50.00% and MAE is constant under the assumption. Results are averaged over 5 runs ( std).
Furthermore, GAUC on impressed items also shows steady improvement. This suggests that by capturing sequence-level dependencies and suppressing noise from unexposed items, ICE also benefits conventional conditional click prediction. These gains hold across architectures from simple MLPs (e.g., DNN) to complex sequential encoders (e.g., Transformer), confirming that ICE serves as a robust, backbone-agnostic, plug-and-play framework.
Ablation Study (Q2)
In this section, we perform a component-wise ablation study using PRM [27] as the representative backbone to quantify the contribution of each ICE module. All variants are trained under identical settings, differing only in the targeted component:
w/o Browsing-depth Modeling (Comp-I): Removes the impression branch and assumes for all positions. As a result, impression-aware masking degenerates to an all-one mask.
w/o Impression-aware Dynamic Masking (Comp-II): Disables the dynamic masking in click modeling while maintaining the impression prediction module.
w/o Sequence-level Click Distribution Matching (Comp-III): Removes the sequence-level KL divergence objective and trains the click branch solely with point-wise BCE loss.
As shown in Table 3, the complete ICE framework achieves the best performance across all metrics. Removing Comp-I leads to the most substantial degradation in GAUC and SeqAUC, underscoring that precise impression estimation is the cornerstone of sequence evaluation. Meanwhile, disabling Comp-II or Comp-III further compromises click prediction and overall sequence selection quality, confirming that each component contributes a distinct and complementary improvement.
| PAUC | GAUC | GAUC | SeqAUC | |
| PRM + ICE | 73.24 | 65.84 | 73.94 | 85.65 |
| w/o Im. Model | 50.00 | 64.53 | 67.34 | 83.29 |
| w/o Im. Mask | 72.12 | 65.74 | 70.54 | 85.58 |
| w/o Clk. Dist. | 70.69 | 65.00 | 70.22 | 85.50 |
Table 3. Component-wise ablation of ICE on the PRM backbone (Q2). Results are reported in percentage (%).
Mechanism Analysis (Q3)
To further understand the sources of ICE’s performance gains, we conduct mechanism analyses on its three core components. Specifically, we examine whether browsing-depth modeling captures personalized exposure, why impression-aware masking helps click prediction, and what sequence-level click distribution matching captures beyond point-wise CTR training. Throughout this section, we analyze impression-related components using several impression probability schemes as references:
Constant: Assuming for all positions.
Exponential Decay: A heuristic decay ().
Positional Empirical: , the average impression rate at each position over the training set, representing a non-personalized position prior.
ICE Predicted: impression probabilities derived from ICE.
Oracle Impression: ground-truth impression labels, serving as the upper bound in masking analysis.
Analysis of Browsing-depth Impression Modeling
We first evaluate whether ICE can accurately model the browsing drop-off and reconcile the display-to-impression gap (C1). Figure 3 plots the average across positions for each scheme. The Constant assumption () yields a systematic overestimation at later positions. While heuristic-based schemes (Decay, Empir.) capture the general monotonic decay of visibility, they rely on static priors and fail to account for user-specific browsing intentions.

Figure 3. Average impression probability per position across different schemes.
As evidenced in Table 4(a), these non-personalized schemes yield a PAUC of exactly 50.00%, indicating a complete lack of discriminative power within each position. In contrast, ICE achieves a substantial PAUC of 73.24% and the lowest MAE (1.03). These results validate ICE’s superior ability to capture personalized browsing boundaries by learning context-dependent exit points, thereby providing a reliable foundation for sequence evaluation.
| Impr. Schemes | PAUC | MAE | GAUC | GAUC |
| Const. | 50.00 | 3.92 | 65.74 | 70.54 |
| Decay | 50.00 | 2.40 | 65.78 | 72.30 |
| Empir. | 50.00 | 1.19 | 65.79 | 73.34 |
| Pred. (ICE) | 73.24 | 1.03 | 65.84 | 73.94 |
| Oracle (GT) | 100.0 | 0.00 | 74.19 | 82.44 |
Table 4. Mechanism analysis of impression modeling accuracy and its de-noising effect on click prediction (PRM backbone). (a) evaluates the precision of different impression schemes. (b) assesses how these signals, when used as context masks, improve click modeling. Note: GAUC is consistently calculated using the ICE-predicted across all rows to isolate the gains from masking.
Analysis of Impression-aware Masking
We next investigate how the masking mechanism facilitates click prediction. Specifically, we compare click prediction performance under masks derived from different impression sources: (i) Const. (no masking); (ii) heuristic schemes (Decay, Empir.); (iii) the ICE predicted mask; and (iv) the Oracle mask (ground-truth, GT), which represents the theoretical upper bound for context denoising. To isolate the denoising effect, we hold other components fixed and vary only the mask source.
Table 4(b) provides two key insights. First, the Oracle mask yields a striking 8.45% lift in GAUC compared to the Constant scheme, confirming that filtering unexposed items is essential for pruning irrelevant context and recovering true interaction patterns. Second, click prediction performance correlates directly with mask quality (Pred. (ICE) > Empir. > Decay), demonstrating that more accurate impression estimation produces more effective denoising signals, which in turn yields better click prediction.
Analysis of Sequence-level Click Distribution Modeling
Finally, we analyze how sequence-level distribution matching (Comp-III) handles attention competition. We hypothesize that sequences with multiple clicks exhibit stronger intra-sequence behavioral dependencies, as user attention is limited and shared across items. To verify this, we partition the test set into two groups: Single-click (exactly 1 click) and Multi-click (more than 1 click).
Figure 4 presents range-bars per backbone. We observe three key findings: (i) ICE-augmented models consistently outperform their vanilla counterparts across all architectures. (ii) ICE bars are markedly narrower, indicating reduced performance discrepancy between
simple and complex sequences and more robust modeling of user browsing journeys. (iii) ICE delivers larger lifts on the harder Multi-click group, where competitive dependencies are most pronounced. These trends confirm that ICE captures attention competition, addressing the behavioral dependencies in C2.
Online Experiment (Q4)
We deploy ICE in the Guess You Like feed of the Taobao App, a flagship recommendation scenario serving hundreds of millions of daily users. The production system follows a G–E re-ranking framework with a Transformer-based baseline evaluator. We integrate ICE into this evaluator while keeping the generator fixed.
We conduct an online A/B test with 5% traffic allocation over 14 days, tracking core metrics alongside 14-day Lifetime Value (LT-14d) as a guardrail metric to monitor user retention. Results show that ICE improves impressions by +0.58%, clicks by +0.70%, and commercial revenue by +3.67%. Notably, LT-14d also increases by +0.23%, indicating that the gains are sustained without compromising the long-term user experience. Regarding efficiency, the average response time (RT) increases by less than 0.5ms, which is negligible compared to the total system latency. Given these consistent improvements, ICE has been fully deployed to the online system, validating the JBM paradigm at industrial scale.
Conclusion
This paper introduces the JBM paradigm for evaluator learning in G–E re-ranking. Unlike the prevailing IBM that performs independent item-level predictions, JBM explicitly models how sequence utility is realized through interdependent user behaviors. We instantiate JBM with ICE, a plug-and-play framework combining browsing-depth impression modeling, impression-aware masking (C1), and sequence-level click distribution matching (C2). Across six evaluator backbones, ICE consistently improves impression, click, and sequence-level evaluation. A 14-day Taobao A/B test further yields +0.70% clicks and +3.67% revenue with negligible latency overhead, followed by full-traffic deployment. Future work will extend JBM to richer behavioral signals (e.g., dwell time) and leverage ICE as a reward signal for generative re-rankers.
GenAI Usage Disclosure
We used Claude for language polishing and grammar checking of the manuscript. All technical contributions, including the problem formulation, methodology design, experimental setup, and result analysis, were entirely conceived and conducted by the authors without GenAI assistance.
References
- [1]Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. 2018. Learning a deep listwise context model for ranking refinement. In The 41st international ACM SIGIR conference on research & development in information retrieval. 135–144.DOI
- [2]Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier, Ed H. Chi, Edan Xiyang Luo, Alan Mackey, and Ofer Meshi. 2018. Seq2Slate: Re-ranking and slate optimization with RNNs. arXiv preprint arXiv:1810.02019 (2018).arxiv.org/abs/1810.02019
- [3]Christopher Burges, Robert Ragno, and Quoc Le. 2006. Learning to rank with nonsmooth cost functions. Advances in neural information processing systems 19 (2006).DOI
- [4]Chris Burges, Tal Shaked, Erinn Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning. 89–96.DOI
- [5]Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11, 23-581 (2010), 81.
- [6]Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning. 129–136.
- [7]Rihan Chen, Bin Liu, Han Zhu, Yaoxuan Wang, Qi Li, Buting Ma, Qingbo Hua, Jun Jiang, Yunlong Xu, Hongbo Deng, et al. 2022. Approximate nearest neighbor search under neural similarity metric for large-scale recommendation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 3013–3022.arxiv.org/abs/2202.10226
- [8]Sirui Chen, Xiao Zhang, Xu Chen, Zhiyu Li, Yuan Wang, Quan Lin, and Jun Xu. 2023. Reinforcement re-ranking with 2d grid-based recommendation panels. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region. 282–287.DOI
- [9]Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7–10.arxiv.org/abs/1606.07792
- [10]Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on Recommender systems. 191–198.DOI
- [11]Yufei Feng, Yun Gong, Fei Sun, Junfeng Ge, and Wenwu Ou. 2021. Revisit recommender system in the permutation perspective. arXiv preprint arXiv:2102.12057 (2021).arxiv.org/abs/2102.12057
- [12]Yufei Feng, Binbin Hu, Yu Gong, Fei Sun, Qingwen Liu, and Wenwu Ou. 2021. GRN: Generative Rank Network for Context-wise Recommendation. arXiv preprint arXiv:2104.00860 (2021).arxiv.org/abs/2104.00860
- [13]Maurizio Ferrara Diacono, Nicolò Felicioni, and Paolo Cremonesi. 2022. Offline evaluation of recommender systems in a user interface with multiple carousels. Frontiers in big Data 5 (2022), 910030.
- [14]Carlos A Gomez-Uribe and Neil Hunt. 2015. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS) 6, 4 (2015), 1–19.
- [15]Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiangqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247 (2017).arxiv.org/abs/1703.04247
- [16]Ruidong Han, Bin Yin, Shangyu Chen, He Jiang, Fei Jiang, Xiang Li, Chi Ma, Mincong Huang, Xiaoguang Li, Chunzhen Jing, et al. 2025. Mtgr: Industrial-scale generative recommendation framework in metauran. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management. 5731–5738.DOI
- [17]Junjie Huang, Jizheng Chen, Jianghao Lin, Jiarui Qin, Ziming Feng, Weinan Zhang, and Yong Yu. 2025. A comprehensive survey on retrieval methods in recommender systems. ACM Transactions on Information Systems 44, 1 (2025), 1–43.arxiv.org/abs/2407.21022
- [18]Junguang Jiang, Yanwen Huang, Bin Liu, Xiaoyu Kong, Xinhang Li, Ziru Xu, Han Zhu, Jian Xu, and Bo Zheng. 2025. Large Language Model as Universal Retriever in Industrial-Scale Recommender System. arXiv preprint arXiv:2502.03041 (2025).arxiv.org/abs/2502.03041
- [19]Thorsten Joachims. 2006. Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 217–226.DOI
- [20]Jan Kislinger. 2025. Full-Page Recommender: A Modular Framework for Multi-CarouseI Recommendations. In Proceedings of the Nineteenth ACM Conference on Recommender Systems. 1479–1484.DOI
- [21]Ping Li, Qiang Wu, and Christopher Burges. 2007. Mcrank: Learning to rank using multiple classification and gradient boosting. Advances in neural information processing systems 20 (2007).
- [22]Dawen Liang, Laurent Charlin, James McInerney, and David M Blei. 2016. Modeling user exposure in recommendation. In Proceedings of the 25th international conference on World Wide Web. 951–961.arxiv.org/abs/1510.07025
- [23]Shuchang Liu, Qingpeng Cai, Zhankui He, Bowen Sun, Julian McAuley, Dong Zheng, Peng Jiang, and Kun Gai. 2023. Generative flow network for listwise recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1524–1534.DOI
- [24]Ze Liu, Jin Zhang, Defu Lian, Chao Feng, Jie Wang, and Enhong Chen. 2025. Learning Deep Tree-Based Retriever for Efficient Recommendation: Theory and Method. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025).DOI
- [25]Masoud Mansoury, Bamshad Mobasher, and Herke van Hoof. 2022. Exposure-aware recommendation using contextual bandits. arXiv preprint arXiv:2209.01665 (2022).arxiv.org/abs/2209.01665
- [26]Masoud Mansoury, Bamshad Mobasher, and Herke van Hoof. 2024. Mitigating exposure bias in online learning to rank recommendation: A novel reward model for cascading bandits. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 1638–1648.arxiv.org/abs/2408.04332
- [27]Changhua Pei, Yi Zhang, Yongfeng Zhang, Fei Sun, Xiao Lin, Hanxiao Sun, Jian Wu, Peng Jiang, Junfeng Ge, Wenwu Ou, et al. 2019. Personalized re-ranking for recommendation. In Proceedings of the 13th ACM conference on recommender systems. 3–11.arxiv.org/abs/1904.06813
- [28]Amira Raj and Michael D Ekstrand. 2023. Towards measuring fairness in grid layout in recommender systems. arXiv preprint arXiv:2309.10271 (2023).
- [29]Shaina Raza, Mizanur Rahman, Safiullah Kamawal, Armin Toroghi, Ananya Raval, Farshad Navah, and Amirmohammad Kazemeini. 2026. A comprehensive review of recommender systems: Transitioning from theory to practice. Computer Science Review 59 (2026), 100849.arxiv.org/abs/2407.13699
- [30]Yuxin Ren, Qiya Yang, Yichun Wu, Wei Xu, Yalong Wang, and Zhiqiang Zhang. 2024. Non-autoregressive generative models for reranking recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5625–5634.arxiv.org/abs/2402.06871
- [31]Xiaowen Shi, Fan Yang, Ze Wang, Xiaoxu Wu, Muzhi Guan, Guogang Liao, Wang Yongkang, Xingxing Wang, and Dong Wang. 2023. PIER: Permutation-Level Interest-Based End-to-End Re-ranking Framework in E-commerce. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4823–4831.arxiv.org/abs/2302.03487
- [32]Fan Wang, Xiaomin Fang, Lihang Liu, Yaxue Chen, Jiucheng Tao, Zhiming Peng, Chiang Jin, and Hao Tian. 2019. Sequential evaluation and generation framework for combinatorial recommender system. arXiv preprint arXiv:1902.00245 (2019).arxiv.org/abs/1902.00245
- [33]Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. In Proceedings of the ADKDD’17. 1–7.arxiv.org/abs/1708.05123
- [34]Shuli Wang, Xue Wei, Senjie Kou, Chi Wang, Wenshuai Chen, Qi Tang, Yinhua Zhu, Xiong Xiao, and Xingxing Wang. 2025. NLGR: Utilizing Neighbor Lists for Generative Rerank in Personalized Recommendation Systems. In Companion Proceedings of the ACM on Web Conference 2025. 530–537.DOI
- [35]Yunxia Xi, Weiwen Liu, Xinyi Dai, Ruiming Tang, Weinan Zhang, Qing Liu, Xi- uqiang He, and Yong Yu. 2021. Context-aware reranking with utility maximization for recommendation. arXiv preprint arXiv:2110.09059 (2021).
- [36]Hailan Yang, Zhenyu Qi, Shuchang Liu, Xiaoyu Yang, Xiaobei Wang, Xiang Li, Lantao Hu, Han Li, and Kun Gai. 2025. Comprehensive list generation for multi-generator reranking. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2298–2308.DOI
- [37]Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Yuan Cao, Leon Gao, Zhao- jie Gong, Fangda Gu, Michael He, et al. 2024. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations. arXiv preprint arXiv:2402.17152 (2024).arxiv.org/abs/2402.17152
- [38]Kaike Zhang, Xiaobei Wang, Xiaoyu Yang, Shuchang Liu, Hailan Yang, Xiang Li, Fei Sun, and Qi Cao. 2025. From generation to consumption: Personalized list value estimation for re-ranking. arXiv preprint arXiv:2508.02242 (2025).arxiv.org/abs/2508.02242
- [39]Zhaohui Zheng, Keke Chen, Gordon Sun, and Hongyuan Zha. 2007. A regression framework for learning ranking functions using relative relevance judgments. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 287–294.DOI
- [40]Tao Zhang, Wenwu Ou, and Zhirong Wang. 2018. Globally optimized mutual influence aware ranking in e-commerce search. arXiv preprint arXiv:1805.08524 (2018).

