WorksComputer Science
Do Search-Augmented Recommenders Use the Latest Search Episode?
Abstract
When a user returns to a recommendation feed after searching, does the latest search episode help rank the next item? Aggregate ranking gains leave open whether search-augmented recommenders use the latest search at prediction time or benefit from broader search state and representation learning. FocalSwap is a counterfactual audit that masks or replaces the latest episode or the full search history while holding the target, candidates, recommendation history, and trained scorer fixed. A planted-signal test and GenSAR's native search-item prediction task verify that FocalSwap detects known search dependence. In proxy analyses, KuaiSAR's routing gain falls below the shuffled threshold, and Qilin's clearest matched-search advantage appears at high taxonomy overlap. The primary comparisons for SESRec and UniSAR do not establish a positive ranking gain from the latest search episode alone. For the full search history, the estimated contribution depends on the control. On 66,139 shared instances across three UniSAR checkpoints, neutral masking consistently lowers mean reciprocal rank (RR). Replacing the history with observed searches matched on category and activity profile changes 44.1 to 44.8 percent of target ranks, yet the original history has little average advantage. In the primary assignment, matched replacements yield mean RR 0.002770 to 0.006604 higher than neutral masking across checkpoints. The adjusted intervals for this difference lie above zero under the primary assignment and all five alternative assignments per checkpoint. Search audits should specify which inputs changed and what replaced them, and report both how often ranks move and how much the original search helps.
ACM Reference Format: Jiandong Ding. 2027. Do Search-Augmented Recommenders Use the Latest Search Episode? In Proceedings of the Twentieth ACM International Conference on Web Search and Data Mining (WSDM ’27), February 15–19, 2027, Hong Kong, Hong Kong. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3862322.3863570
Introduction
When someone searches for an item and then opens a recommendation feed, the search they just made seems like an obvious clue to what they want next. Yet the ranking gains of search-augmented recommendation (SAR) leave open where those gains come from [31, 35, 42]. Search can help train item and user representations, earlier searches can shape the model’s view of the user’s interests, and the latest episode can provide a clue at prediction time. The question arises across architectures, from joint graph learning [49] and intent transfer [46] to generative retrieval [32].
To see what the latest episode contributes at prediction time, we change only that episode and ask how the same recommendation would rank. The target, candidates, recommendation history, and trained scorer all stay fixed. Changing the entire search history asks a broader question, and in either case the replacement matters. A neutral mask removes search content and can introduce an unfamiliar input. A replacement from real search logs preserves selected properties, such as category and search profile, while supplying different content.
FocalSwap makes these comparisons for each evaluation instance (Figure 1). It measures how often the target moves, how much the original search improves reciprocal rank (RR) relative to a control, and how often the original search wins against a matched alternative, giving ties half credit. We call the episode closest to the recommendation the focal search.

Figure 1. FocalSwap fixes the recommendation history (shown as for a search-centered window), target, candidates, and scorer. It masks or replaces either the most recent episode or the full search history, as indicated by the dashed columns. The primary comparisons do not establish a latest-episode gain in SESRec or UniSAR, and the full-history estimate in UniSAR depends on the control.
The primary comparisons in SESRec and UniSAR leave a ranking gain from the latest episode unestablished (Figure 4(a)). For the full search history in UniSAR, the choice of control changes the answer. On the same 66,139 instances, masking yields a positive contribution at all three checkpoints. Matched replacements yield smaller estimates, even though they change the target rank in nearly half the instances. The scorer responds to the search input, and the comparison determines how much benefit we attribute to the original searches.
Figure 4. Search contribution under the original replacement assignment. In (a), the intervals cross zero for masking (16,788 SESRec and 157,192 UniSAR instances) and UniSAR matched replacement (87,044 instances). In (b) and (c), all controls use the same 66,139 UniSAR instances. Masking gives positive estimates, and the paired mask-minus-matched intervals are above zero for all three seeds. User-cluster intervals are 95% for masking and profile-prioritized replacement, and Bonferroni-adjusted 97.5% for matched replacement and the paired gaps.
KuaiSAR supplies same-user search and recommendation histories with anonymous tokens [38], and Qilin provides real query and item text [3]. The behavioral and lexical analyses ask when history and search offer complementary ranking signals. A planted signal and GenSAR’s native query task check that the audit can detect dependence on a particular search.
This work is licensed under a Creative Commons Attribution 4.0 International License.
WSDM ’27, Hong Kong, Hong Kong
© 2027 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-3117-4/2027/02
https://doi.org/10.1145/3862322.3863570
Our contributions are to distinguish the latest search episode from the accumulated search history as possible sources of ranking gains, introduce an audit that detects known search signals, and show how the reference input changes attribution in trained recommenders. Table 1 previews the results. For each audit, we report what changed, what replaced it, and how the ranking responded.
| Setting | Question | Finding |
| SESRec; UniSAR seeds 1–3 | Latest episode | The primary comparisons do not establish a positive ranking contribution. |
| UniSAR seeds 1–3 | Full history | Masking yields a positive contribution on 66,139 shared instances. Matched replacement gives smaller estimates. |
| Planted signal | Calibration | The audit detects routing opportunity, contribution, and specificity at full signal strength in all three runs. |
| GenSAR | Calibration | The original query raises mean RR@30 over matched replacement on 8,958 search-item instances. |
| KuaiSAR and Qilin | Auxiliary analyses | KuaiSAR falls below the shuffled threshold. Qilin clears the 95th-percentile threshold with BM25. Its strongest matched-search result is a 0.084 preference lift on 6.1% of instances. |
Table 1. Where each evaluation contributes evidence.
Related Work
We relate SAR models, attribution audits, and search-rich datasets to distinguish system-level utility from evidence about the latest search episode.
Search-augmented and joint search–recommendation models. SAR builds on the broader implicit-feedback and sequential recommendation line, where ranking objectives and sequence encoders such as BPR, SASRec, and BERT4Rec define standard recommendation backbones [19, 25, 37]. HPGR models session structure and attends selectively to preference-relevant history [4]. Early joint systems model search and recommendation behavior together [1, 45, 49]. Closer to our checkpoint analysis, SESRec learns a search representation for recommendation [35], UnifiedSSR unifies sequential search and recommendation [42], and UniSAR models transitions between the two [31]. In parallel, work on shared representations links search and recommendation by modeling integrated behavior, transferring intent, or disentangling signals [6, 30, 44, 46]. This line also addresses sparse search histories [33], different interest horizons [50], and multiple recommendation scenarios [22, 41].
Generative models bring the two tasks together through identifiers, intent, and queries [21, 32, 43, 47], shared training [24, 48], or a unified task formulation [7, 15]. Across these designs, search can shape representations during training, accumulate in user state, or enter the scorer at prediction time. FocalSwap compares the search evidence attached to one recommendation instance: changing the latest episode tests focal attribution, while changing all search inputs tests broader search-state contribution. GenSAR’s native current-query task provides a separate calibration of known query use [32].
Evaluation audits and signal attribution. Recommendation evaluation has a long history of re-examining whether reported gains survive stronger baselines, controlled comparisons, or reproducibility checks [12, 13, 29]. Behavioral testing systems make a related distinction between aggregate metrics and targeted capability checks [5, 26]. Diagnostic resources similarly examine components behind ranking outcomes: SIDInspctor evaluates Semantic-ID mappings separately from downstream scores [8], while SIDSCope traces their effects on candidate retrieval and item resolution [9]. Model-interpretability research exposes a complementary risk in input attribution. Amnesic probing distinguishes information encoded in a representation from information used by the model through a behavioral intervention [11]. Feature-attribution studies further show that the baseline chosen to represent missingness can change an explanation and that test-time feature removal can create out-of-distribution inputs [16, 36]. FocalSwap brings these concerns to temporal SAR attribution: it fixes the complete recommendation instance, separates latest-episode from accumulated-search claims, and compares neutral masking with matched observed search sequences on identical instances. A separate line of research uses logged feedback to estimate policy value [2, 10, 14, 39] or correct for exposure and selection bias [18, 27, 28]. Related causal methods target particular sources of confounding. DIFL learns duration-invariant features for watch-time prediction [17], while IV4Rec uses search behavior as an instrument to identify causal preferences [34]. In FocalSwap, the question is how a fixed scorer ranks the same target when given a different search.
Intent, transfer, and search-rich datasets. Search-rich datasets expose intent through behavior tokens, query text, or richer semantic representations. KuaiSAR provides same-user search and recommendation behavior but anonymizes tokens [38]. JDSearch and KuaiSearch expose real-query-derived tokens and retrieval context [20, 23], and query representation learning offers another path for using explicit search signals in recommendation [40]. Qilin adds real query and item text in a search/recommendation behavior setting [3]. The available inputs determine which aspects of search each dataset can test.
FocalSwap Protocol
Consider a validation instance with a planted search signal (§ 5.1). The target ranks first with the original search and recommendation history (). It ranks third with either a neutral mask or a matched search (). With the same candidates and scorer in every view, the audit records a rank change, a positive contribution, and a win over the replacement. History alone gives of 0.333, while search alone gives 1.000.
We link search windows to later recommendations and report the coverage of those links. An eligible instance has the scorer’s recommendation history , a shared search input , and candidates . Its nonempty evaluated positives satisfy , where denotes the post-search positives. The latest non-padding episode in is focal. Under view , let be the best positive rank (starting at 1):
The target and candidate sets stay fixed across views. SESRec and UniSAR use midpoint ranks for exact candidate-score ties. Eligibility also requires that the target cannot leak through the history or focal search fields supplied to the scorer.
combines history and the original search, uses history alone, and uses the focal search alone. Intervention maps to , with denoting matched replacement. We average over the nonempty set where applies. Matching requires a valid replacement.
We put a neutral ID or matched observed content in either the latest non-padding slot or all non-padding search slots. This measures how the fixed scorer reranks the same candidates.
What the Audit Measures
The intervention analysis tracks three quantities. The behavioral and lexical settings also let us ask whether one can improve ranking by routing between history-only and search-only predictions.
First, rank response measures how often the best positive rank changes. The indicator 1 equals one when the ranks differ:
Second, signed ranking contribution measures whether the original search input improves average RR relative to the named control:
Frequent positive and negative rank changes can yield a small signed mean, so response rate and contribution answer different questions. On the same eligible instances, subtracting two contribution estimates cancels the original RR. The paired gap is the mean RR under the second control minus the first.
Third, replacement preference compares to its matched . Checkpoint swaps compare complete inputs (), and proxy swaps compare search-only predictions (). The comparison uses RR for . The shared-candidate diagnostic uses the target score. For matched instances, ties receive half credit:
A tie-adjusted preference of 0.500 means that wins and losses balance. Table 3(b) reports the statistic for replacing the latest episode or the full history. Evidence for focal attribution requires both positive signed contribution and a preference for the original over matched replacements under the calibration and uncertainty procedures below.
Routing with history and search. In the behavioral and lexical settings, we can also ask whether history and search make complementary predictions.
A per-instance oracle selects whichever of History-only and Search-only yields higher RR. Its margin over History-only bounds the gain from routing between those views, but selecting the better score can create headroom even after shuffling. A shuffled-search null calibrates this selection effect. The observed margin is
In each of null draws, we hold the history predictions fixed and shuffle or replace the search predictions. Let be the same margin in draw . Routing opportunity clears the 95th-percentile null threshold when
The recovery probe measures how much of that oracle margin a routing rule recovers on held-out instances using features that exclude the target outcome.
Checkpoint intervals and Qilin swap intervals resample whole users. The KuaiSAR swap intervals resample instances. In Qilin, we apply Holm correction to one-sided user-cluster sign-flip tests across the ten swap buckets. For each primary checkpoint endpoint (matched contribution and the paired gap), Bonferroni-adjusted 97.5% intervals cover its two scopes.
Choosing the reference input. A neutral ID removes search content and may be rare or absent in the scorer’s usual inputs. We therefore also use replacements that preserve selected category or profile properties. The same-scorer comparisons apply to proxy scorers, controlled interfaces, and trained checkpoints. For released implementations, the native task and performance determine which comparisons enter the checkpoint analysis.
Evaluation Settings
KuaiSAR Behavioral Evidence
We construct search-centered KuaiSAR windows from anonymized search and recommendation interactions, linking pre-search recommendation history, focal search evidence, and post-search positives. We remove targets from the scored history and search fields. The fixed candidates include the post-search positives, bounded history and search items, and train-split popularity negatives, capped at 80 (mean 48.8 on held-out instances).
A deterministic ranker scores the history, original search, and replacement search using transitions learned on the training split. It counts distinct evidence–target pairs separately for the pre-search history and focal search in each training instance. Let count transitions from evidence item to target , and let . For evidence set and candidate , scores
The maximum selects the strongest evidence-to-target association, and the denominator reduces the weight of frequent evidence items.
Unseen pairs and empty evidence score zero. Equal scores use deterministic hash tie-breaking.
KuaiSAR analyzed instances are selected deterministically by sorting eligible examples by MD5(episode_id) within each split and taking 10,000 validation and 10,000 test instances.
Qilin Lexical Evidence
Qilin ranks each logged recommendation slate with BM25 and taxonomy overlap. The search branch uses the query and clicked-item taxonomy paths from that search, both of which change with a replacement. The history branch builds a BM25 pseudo-query and taxonomy features from up to 20 clicks in the recommendation request’s history, which can include focal-search clicks. For each distinct path, matching nonempty levels add 0.04 to the search score or 0.03 to the history score. RR uses the first ranked clicked item. Score ties follow the logged slate order.
The Qilin construction records both match coverage and match quality. Same-user and taxonomy-near swaps are available for nearly all instances. For taxonomy matching, we count distinct taxonomy paths among the donor’s clicked items whose top-level category also appears among the focal search’s clicked items. The bucket contains instances with at least such donor paths. A random-search placebo gives each bucket a reference on the same instances. This setup lets the audit distinguish a broad lexical effect from a local taxonomy-overlap effect on fixed targets and candidates.
Table 2 shows how many episodes enter the proxy analyses after the fixed-instance eligibility checks. Figure 2 compares the two settings, and Figure 3 shows the Qilin replacement results by matching rule.
| Dataset | Users | Valid episodes | Proxy episodes |
| KuaiSAR | 13,159 | 96,994 | 20,000 |
| Qilin | 1,544 | 6,135 | 6,135 |
Table 2. Behavioral and lexical evaluation scale. Valid episodes satisfy the fixed-instance criteria; analyzed episodes enter the proxy analyses.
Figure 2. Search and history in the proxy evaluations. Panel (a) compares observed oracle gains with the shuffled or counterfactual null thresholds. Panel (b) shows KuaiSAR’s tie-adjusted preference above 0.500 for same-source replacements, with 95% instance-bootstrap intervals. Panel (c) compares KuaiSAR routers with the history baseline and oracle.
Figure 3. Qilin’s matched-search signal concentrates at high taxonomy overlap. Panel (a) gives the tie-adjusted preference for the original search. Panel (b) subtracts the preference against a random search on the same instances. Bars are pointwise 95% confidence intervals. Open markers denote below 20% coverage. Only the at-least-four-path bucket passes Holm correction.
Counterfactuals and Calibration
For KuaiSAR, nuisance matching (L1) picks a search from the same split and source, with similar feedback count, pre-history length, and bridge length. Category matching (L2) samples a search from the same split that shares at least one search category. In Qilin, we use same-user, time-near, and taxonomy-based replacements, and compare them with random search on the same instances. The Qilin routing null draws uniformly from these four replacement predictions for each instance. Recovery is measured against History-only within the same candidate pool. Qilin’s router is an offline diagnostic using features from the logged replacement pool.
Checkpoints from Released Code
The recommendation audits use SESRec and UniSAR checkpoints trained with the released implementations on KuaiSAR-compatible inputs. UniSAR seeds 1–3 provide three independent checkpoints. tested on 157,192 recommendation instances. SESRec uses 16,788 eligible test windows. Each model ranks one target against 99 model-specific negatives, fixed across interventions. The recommendation context stays as supplied to the checkpoints, including interactions after the focal search when present. Full-input RR is 0.0423 for SESRec and 0.3417, 0.3542, and 0.3403 for the three UniSAR seeds. The KuaiSAR proxy uses the separate candidate sets capped at 80 items.
SESRec takes one search session as input, so intervening on it also changes the full search input. Its replacement shares at least one search-item category. UniSAR’s matched replacement also uses search-profile buckets. GenSAR’s recommendation checkpoint scored 0.1857 in test NDCG@10, below the prespecified inclusion threshold of 0.1914 (95% of the published score). We use its native search-item prediction task (PerSrc) to check whether the audit detects use of the current query.
Positive Calibration
Before auditing recommendation checkpoints, we verify that FocalSwap detects input dependence when it is known to be present. One score construction plants a focal signal; a released scorer explicitly conditions its native task on the current query. Together they calibrate the recommendation-side analysis.
Planted Focal Signal
An auxiliary score-sensitivity check adds a boost only when the focal search has positive feedback and does not directly conflict with pre-search history. With no injection, oracle gain is 0 and the injected-versus-passive preference is 0.500. At the smallest positive signal strength (oracle gain 0.003), preference reaches 0.645 and a held-out two-layer MLP router recovers the injected gain. At the largest signal strength, oracle gain reaches 0.042 and recovery persists. This check isolates score sensitivity and routing. The following full-chain control adds matched donors and the shuffled-search opportunity null.
The separate full-chain control keeps Qilin inputs, candidates, and donors fixed and generates synthetic target labels with a fixed teacher. The teacher standardizes history and focal-search scores within each candidate set, then adds the history score, the weighted focal-search score, and fixed noise. The highest-scoring candidate becomes the target and stays fixed across all audit views. For each injection weight (0, 0.5, or 1.0), we train the student scorer with three seeds. With zero signal weight, Full-minus-masked RR ranges from −0.0070 to −0.0041: neutral masking changes ranking even without a planted focal signal. None of these zero-signal runs passes the positive contribution gate. At full signal weight (1), the signed RR contribution under masking is 0.052–0.054, preference for the original search is 0.639–0.651, and placebo-adjusted specificity is 0.147–0.163; every interval clears its null boundary. The planted control therefore establishes routing opportunity, signed contribution, and focal specificity for all three full-strength runs. On its test split, the full-strength run used for the worked example gives an oracle gain of 0.128 against a shuffled p95 of 0.068, a contribution of +0.054 RR, and a tie-adjusted preference of 0.647.
Released-Code Current-Query Signal
GenSAR provides a second positive calibration on its native search-item prediction task [32]. On 8,958 leakage-filtered validation instances, RR@30 uses the 30 beam results and assigns zero when they omit the target. Masking the current query reduces RR@30 by 0.035 (95% CI [0.029, 0.041]). A matched query reduces it by 0.106 (95% CI [0.099, 0.112]), with a preference of 0.635 for the original. Replacing only the most recent prior search session reduces RR@30 by 0.002 and gives a preference of 0.518. These results recover current-query dependence at the preselected epoch-3 checkpoint.
Behavioral and Lexical Evidence
The proxy analyses ask where search provides a complementary ranking signal. History-only and search-only predictions let us test whether there is room to improve by routing between them and how much of that improvement a router can recover (Figure 2).
Coverage and Opportunity Separate
KuaiSAR offers broad episode coverage under the target-leakage checks, but the gain from choosing the better prediction is 0.037, below the shuffled-search 95th percentile of 0.070. The proxy falls short of the routing-opportunity threshold.
Qilin’s BM25 scorer clears its 95th-percentile null threshold by 0.0005 RR (). Across 30 Monte Carlo null seeds, the observed oracle gain exceeds the 95th percentile in each run. It stays below the pooled 99th percentile, and the result does not replicate with TF-IDF. The opportunity is small and specific to BM25.
Matched-Swap Signal Is Local
Within the same split and data source in KuaiSAR, the tie-adjusted preference is 0.501 for replacements with no shared category and 0.510 for replacements with a shared category. The latter is +0.010 over symmetry, a small advantage on that comparison.
In Qilin we compare each taxonomy-overlap bucket with random-search replacement on the same instances (Figure 3). Only the strictest bucket, with at least four qualifying paths, passes Holm correction. It covers 6.1% of instances, with a tie-adjusted preference of 0.566 and a lift of +0.084 over random replacement ().
KuaiSAR’s routers do not improve on history alone. Qilin’s small BM25 opportunity also produces no held-out routing gain, including in the at-least-four-path bucket.
Checkpoint Evidence
Figure 4 shows what happens when we change the latest episode and when we change the full search history. Table 3 gives the estimates and interval endpoints.
| Checkpoint | Latest episode: Mask (C0), 95% | Latest episode: Matched (C2), 97.5% | Full history on 66,139 shared instances: Mask, 95% | Full history on 66,139 shared instances: Matched, 97.5% | Full history on 66,139 shared instances: Mask − matched, 97.5% |
| SESRec | +0.138 [−0.743, +0.936] | n/a | n/a | n/a | n/a |
| UniSAR seed 1 | −0.213 [−0.495, +0.072] | −0.408 [−0.822, +0.009] | +5.427 [+4.183, +6.689] | −1.177 [−2.074, −0.302] | +6.604 [+5.238, +7.951] |
| UniSAR seed 2 | +0.089 [−0.191, +0.376] | −0.163 [−0.567, +0.253] | +2.016 [+0.856, +3.138] | −0.755 [−1.647, +0.131] | +2.770 [+1.582, +3.962] |
| UniSAR seed 3 | −0.160 [−0.426, +0.102] | +0.043 [−0.361, +0.462] | +3.197 [+2.040, +4.331] | −0.354 [−1.171, +0.469] | +3.551 [+2.272, +4.819] |
| Checkpoint | Latest episode | Full history | Profile-prioritized (C1), full history | ||
| SESRec (L2) | 0.4966 [0.4868, 0.5065] | n/a | n/a | ||
| UniSAR seed 1 | 0.4977 [0.4960, 0.4994] | 0.4899 [0.4870, 0.4928] | −1.600 [−2.516, −0.691] | ||
| UniSAR seed 2 | 0.4982 [0.4965, 0.4999] | 0.4959 [0.4929, 0.4989] | −0.068 [−0.959, +0.830] | ||
| UniSAR seed 3 | 0.4993 [0.4975, 0.5013] | 0.4957 [0.4928, 0.4986] | −0.132 [−0.986, +0.722] | ||
| Checkpoint | Matched range | Mask − matched range | Paired 97.5% CIs above zero | ||
| UniSAR seed 1 | −1.679 to −0.953 | +6.380 to +7.106 | 5/5 | ||
| UniSAR seed 2 | −0.837 to −0.316 | +2.331 to +2.852 | 5/5 | ||
| UniSAR seed 3 | −0.534 to +0.141 | +3.057 to +3.732 | 5/5 |
Table 3. Recommendation checkpoint results. Panel (a) uses the original assignment. Panel (b) reports tie-adjusted preference (C2 for UniSAR, L2 for SESRec) and the profile-prioritized contribution. Panel (c) varies the matched assignment. RR contributions and gaps are in units. Brackets give user-cluster intervals. Bold contribution intervals exclude zero. SESRec takes one search session and uses a category-only match. Its matched contribution is reported in the text, and its preference appears once in (b).
What replaces the search. The mask (C0) puts a checkpoint-compatible neutral ID in the selected slots. The profile-prioritized control (C1) prefers similar item counts, query lengths, and frequencies, with broader fallback pools and no category requirement. The matched control (C2) uses both category and profile. In UniSAR, a matched search is available for 87,044 of 157,192 instances (55.4%) at the latest slot. Replacing the full history requires a match at every slot, which leaves 66,139 instances (42.1%). For that comparison, we apply all three controls to these same instances.
The Latest Episode
The primary comparisons leave a positive contribution from the latest episode unestablished. Masking and UniSAR’s matched replacement give intervals that cross zero (Figure 4(a)). Each intervention changes just the latest non-padding slot.
Wins and losses against the original episode nearly balance. The preference intervals for UniSAR seeds 1 and 2 favor the replacement, while seed 3 and SESRec include the 0.500 symmetry reference (Table 3(b)).
Comparisons on the same instances tell a similar story. On SESRec’s 8,631 category-matchable instances from 2,134 users, the estimates in RR are −0.183 [−1.259, +0.835] for masking and −0.213 [−1.135, +0.783] for replacement. The paired gap is +0.029 [−1.112, +1.127]. The mask interval is 95%, and the replacement and paired intervals are 97.5%. On UniSAR’s 87,044 matchable instances, the mask gives a negative estimate for seed 1 (−0.486 [−0.931, −0.036] × RR). The seed 2 and 3 intervals include zero.
The Full Search History
The full-history estimate depends on what replaces the original search. All three mask intervals are above zero on the same matchable instances. With matched replacement, the original search gives lower mean RR for seed 1, and the intervals for seeds 2 and 3 include zero. The paired gap is positive for every seed, from +2.770 × to +6.604 × RR (Figure 4(b–Figure 4c)). The tie-adjusted preference for the original history ranges from 0.4899 to 0.4959 (Table 3(b)).
Replacing the full search history with matched searches changes the target’s rank in 44.1–44.8% of instances, even though the original history shows little average advantage over the replacements. Among the ranks that change, 47.7–49.1% worsen. Masking moves more ranks (59.7–61.7%), and worsening is more common among those changes (53.3–55.3%). When the rank moves, the median absolute shift is one position with matched replacement and two with masking. The 90th percentiles are four and nine to ten positions, respectively.
When we prioritize profile without requiring a category match, the estimates range from to RR. The seed 1 interval is below zero, and the other two cross zero. The contrast with masking thus extends to a second replacement rule.
Scope and Sensitivity
We build the matched control by assigning observed search IDs to one another. The map covers 45,895 IDs with content. Each mapped ID receives a different donor ID, and each donor ID maps from only one original ID. Matching uses category and profile buckets for item count, query length, and frequency. It uses no user, target, candidate, label, rank, or model score. Masking the full history gives positive contributions both among instances with matches and among those without.
We varied the offsets within matching buckets to build five alternative assignments before scoring them. Each assignment uses the same checkpoints and instances, with 10,000 user-cluster resamples. The 97.5% intervals account for the two scopes within each assignment and checkpoint. Masking attributes more benefit to the original history in all fifteen comparisons, with paired gaps from to RR and all intervals above zero (Table 3(c)). Matched replacement again leaves a positive contribution unestablished for the full history and for the latest episode on the 87,044 matchable instances. The same users recur across assignments.
The profile-prioritized rule changes the category mix. On the same 66,139 instances, 10.6% of non-padding search slots share a top-level category with their replacement. Across histories, the mean fraction of original categories retained is 47.4%. The original mask-versus-match contrast also varies with history length. In the 4–30-slot strata, all 12 seed-by-stratum estimates are positive and 11 pointwise 97.5% intervals exclude zero. In the 1–3-slot strata, all six include zero.
Earlier searches and intervention scope. Replacing the latest search with the nearest earlier distinct search from the same user (C3) gives a positive contribution for UniSAR seed 3 on 155,678 instances (+0.238 [+0.017, +0.464] RR, pointwise 95% CI). The other two intervals include zero. Full-history C3 rotates search IDs within each instance to change all slots. Its three intervals include zero. We also intervene at the 2, 4, or 8 slots, with 155,678, 141,172, or 113,555 eligible instances, respectively. With pointwise 95% intervals, masking eight slots gives a positive contribution only for seed 1. Masking all slots gives positive contributions at all three checkpoints. Profile-prioritized replacements leave a positive contribution unestablished at these scopes.
Comparison on shared candidates. To compare the models on a shared evaluation set, we retrained both with episode-level IDs and a transductive context vocabulary, then evaluated them on the same 3,249 episodes with 100 candidates each. Full-input RR ranges from 0.0637 to 0.1138 for SESRec and from 0.1705 to 0.1928 for UniSAR. Both models receive a single search session in this setting. Masking that session gives a contribution interval above zero in two of three SESRec runs and none of the UniSAR runs. When we replace the session with a category-matched search, all six intervals for preference based on the target item’s score include 0.500. Both comparisons use pointwise 95% user-cluster confidence intervals.
Discussion
Why the reference input matters. The two contribution estimates start from the same original prediction. Their difference tells us how much higher RR is with the matched history than with the mask. A matched search may retain category or activity information the scorer uses, and a neutral ID may disturb a scorer that rarely sees it. Either mechanism could account for part of the gap. The mask measures what changes when search content is removed. The matched comparison asks whether the original history improves ranking over observed searches with similar measured properties.
What to report in a search audit.
(1) Scope. Name the inputs that change, such as the latest episode or the full search history.
(2) Reference. Describe the mask or the matching rule. Compare controls on the same eligible instances and candidate sets when evaluating their difference.
(3) Ranking results. Report how often the target moves, the signed RR difference, and the preference for the original over matched searches. Give the uncertainty and eligible sample size.
Conclusion
The primary comparisons in SESRec and UniSAR leave a ranking benefit from the latest search episode unestablished. In UniSAR, estimates for the full search history depend on the reference input, even though both masking and matched replacement often move the target. FocalSwap can detect known search signals in the planted and GenSAR calibrations and helps locate where evidence for a benefit comes from. A search audit should report the inputs it changes, the replacements it uses, and how the ranking responds.
Reproducibility
The FocalSwap repository (https://github.com/jdding/focalswap) provides the audit code, examples, and aggregate results, with scripts to regenerate selected tables and figures. Rerunning the checkpoint evaluations requires the original datasets, model implementations, and compatible checkpoints.
Ethical Considerations
We report aggregate analyses of public benchmark logs and do not redistribute user-level data. Search data may reveal sensitive intent; findings concern the audited datasets, checkpoints, and interfaces.
References
- [1]Moumita Bhattacharya, Vito Ostuni, and Sudarshan Lamkhede. 2024. Joint Modeling of Search and Recommendations Via an Unified Contextual Recommender (UniCoRn). In Proceedings of the 18th ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 793–795. doi:10.1145/3640457.3688034DOI
- [2]Léon Bettou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portule, Dipankar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising. Journal of Machine Learning Research 14, 101 (2013), 3207–3260. https://www.jmlr.org/papers/v14/bettou13a.html
- [3]Jia Chen, Qian Dong, Haitao Li, Xiaohui He, Yan Gao, Shaosheng Cao, Yi Wu, Ping Yang, Chen Xu, Yao Hu, Qingyao Ai, and Yiqun Liu. 2025. Qilin: A Multimodal Information Retrieval Dataset with APP-level User Sessions. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 3670–3680. doi:10.1145/3726302.3730279
- [4]Zerui Chen, Heng Chang, Tianying Liu, Chuantian Zhou, Yi Cao, Jiandong Ding, Ming Liu, and Bing Qin. 2026. Beyond the Flat Sequence: Hierarchical and Preference-Aware Generative Recommendations. In Proceedings of the ACM Web Conference 2026. Association for Computing Machinery, New York, NY, USA, 7999–8007. doi:10.1145/37794904.3792790
- [5]Patrick John Chia, Jacopo Tagliabue, Federico Bianchi, Chloe He, and Brian Ko. 2022. Beyond NDCG: Behavioral Testing of Recommender Systems with RecList. In Companion Proceedings of the Web Conference 2022. Association for Computing Machinery, New York, NY, USA, 99–104. doi:10.1145/3487553.3524215DOI
- [6]Jiajun Cui, Xu Chen, Shuai Xiao, Chen Ju, Jinsong Lan, Tingting Li, Jianyong Wang, and Wei Zhang. 2026. Disentangling Representations from Search Behaviors for Recommendation via Counterfactual Learning. ACM Transactions on Information Systems 44, 5 (2026), 11:1–11:27. doi:10.1145/3808226
- [7]Marco De Nadai, Edoardo D’Amico, Max Lefarov, Alexandre Tamborrino, Divita Vohra, Mark VanMiddlesworth, Shawn Lin, Jacqueline Wood, Jan Stypka, Eliza Klyce, Keshi Dai, Matthew N. K. Smith, Timothy Christopher Heath, Martin D. Gould, Yves Raimond, Sandeep Ghael, Tony Jebara, Andreas Damianou, Vladan Radosavljevic, Paul N. Bennett, Mounia Lalmas, and Praveen Chandan. 2026. A Unified Model for Personalization: Language-Steerable Generative Recommendation, Search, and User Understanding. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. Association for Computing Machinery, New York, NY, USA, 779–790. doi:10.1145/3770855.3817955
- [8]Jiandong Ding, Heng Chang, Huijie Qin, and Tianying Liu. 2026. SIDInspector: A Mapping-First Diagnostic Resource for Semantic-ID Tokenizers. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 8890–8894. doi:10.1145/3799682.3840174DOI
- [9]Jiandong Ding, Huijie Qin, Tiandeng Wu, and Yi Cao. 2026. SIDScope: A Diagnostic Resource for Semantic-ID Interfaces in Generative Recommendation. arXiv:2608.18779v2arxiv.org/abs/2608.18779
- [10]Miraislaw Doudik, Dumitru Erhan, John Langford, and Lihong Li. 2014. Doubly Robust Policy Evaluation and Optimization. Statist. Sci. 29, 4 (2014), 485–511. doi:10.1214/14-STS500
- [11]Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics 9 (2021), 160–175. doi:10.1162/tacl_a_00359DOI
- [12]Maurizio Ferrari Dacrema, Simone Bοglio, Paolo Cremonesi, and Dietmar Jannach. 2021. A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research. ACM Transactions on Information Systems 39, 2 (2021), 20:1–20:49. doi:10.1145/3434185DOI
- [13]Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. In Proceedings of the 13th ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 101–109. doi:10.1145/3298689.3347058
- [14]Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. 2018. Offline A/B Testing for Recommender Systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, NY, USA, 198–206. doi:10.1145/3159652.3159687
- [15]Guo Xun. 2026. OnceFeed: A Unified Generative Framework for Feed Content Enhancement and Query Generation. arXiv:2606.07972v2arxiv.org/abs/2606.07972
- [16]Peter Hase, Harry Xie, and Mohit Bansal. 2021. The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations. In Advances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., Online, 3650–3666. https://proceedings.neurips.cc/paper/2021/hash/1def1713ebf17722cbe300c1cfc88558-Abstract.htmlarxiv.org/abs/2106.00786
- [17]Chenghou Jin, Yixin Ren, Hongxu Ma, Yewei Xia, Yi Guan, Hao Zhang, Jiandong Ding, Jihong Guan, and Shuigeng Zhou. 2026. Invariant Feature Learning for Counterfactual Watch-time Prediction in Video Recommendation. Proceedings of the AAAI Conference on Artificial Intelligence 40, 17 (2026), 14964–14972. doi:10.1609/aaai.v40i17.38518DOI
- [18]Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased Learning-to-Rank with Biased Feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, NY, USA, 781–789. doi:10.1145/3018661.3018699DOI
- [19]Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Recommendation. In Proceedings of the 2018 IEEE International Conference on Data Mining. IEEE Computer Society, Los Alamitos, CA, USA, 197–206. doi:10.1109/ICDM.2018.00035DOI
- [20]Yupeng Li, Ben Chen, Mingyue Cheng, Qingpeng Cai, Zhidin Liu, Xuxin Zhang, Zhentao Song, Chenyie Lei, Qi Liu, and Peng Jiang. 2026. KuaiSearch: An E-Commerce Search Dataset with Authentic Queries and Product Texts for Recall, Ranking, and Relevance. arXiv:2602.11518v4
- [21]Dongliang Liao, Chenxing Wang, and Yawen Zeng. 2026. Bridging Explicit and Implicit Intent: Unified Interest Generative Method for Joint Search-Recommendation Modeling. In Proceedings of the ACM Web Conference 2026. Association for Computing Machinery, New York, NY, USA, 6103–6113. doi:10.1145/3774904.3792253DOI
- [22]Jinhan Liu, Qiyu Chen, Junjie Xu, Junjie Li, Baoli Li, and Sulong Xu. 2024. A Unified Search and Recommendation Framework Based on Multi-Scenario Learning for Ranking in E-commerce. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 2890–2894. doi:10.1145/3626772.3661356
- [23]Jingnan Liu, Zhicheng Dou, Guoyu Tang, and Sulong Xu. 2023. JDSearch: A Personalized Product Search Dataset with Real Queries and Full Interactions. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 2945–2952. doi:10.1145/3539618.3591900DOI
- [24]Gustavo Penha, Ali Varadshi, Enrico Palumbo, Marco De Nadai, and Hugues Bouchard. 2024. Bridging Search and Recommendation in Generative Retrieval: Does One Task Help the Other? In Proceedings of the 18th ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 340–349. doi:10.1145/3640457.3688123
- [25]Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (Proceedings of Machine Learning Research, Vol. R7). PMLR, 460–469. https://proceedings.mlr.press/r7/rendle09a.html
- [26]Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 4902–4912. doi:10.18653/v1/2020.acl-main.442
- [27]Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. 2020. Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback. In Proceedings of the 13th International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, NY, USA, 501–509. doi:10.1145/3336191.3371783DOI
- [28]Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as Treatments: Debiasing Learning and Evaluation. In Proceedings of the 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48). PMLR, New York, New York, USA, 1670–1679. https://proceedings.mlr.press/v48/schnabel16.htmlarxiv.org/abs/1602.05352
- [29]Faisal Shehzad, Maurizio Ferrari Dacrema, and Dietmar Jannach. 2025. A Worrying Reproducibility Study of Intent-Aware Recommendation Models. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 3155–3164. doi:10.1145/3726302.3730307
- [30]Teng Shi, Weicong Qin, Weijie Yu, Xiao Zhang, Ming He, Jianping Fan, and Jun Xu. 2026. Mining Information Retrieval's Latent Cross Reasoning for Search Enhanced Recommendation. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 1612–1622. doi:10.1145/3805712.3809678
- [31]Teng Shi, Zihua Si, Jun Xu, Xiao Zhang, Xiaoxue Zang, Kai Zheng, Dewei Leng, Yanan Niu, and Yang Song. 2024. UniSAR: Modeling User Transition Behaviors between Search and Recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 1029–1039. doi:10.1145/3626772.3657811DOI
- [32]Teng Shi, Jun Xu, Xiao Zhang, Xiaoxue Zang, Kai Zheng, Yang Song, and Enyun Yu. 2025. GenSAR: Unifying Balanced Search and Recommendation with Generative Retrieval. In Proceedings of the Nineteenth ACM Conference on Recommender Systems. Association for Computing Machinery, New York, NY, USA, 124–134. doi:10.1145/3705328.3748071DOI
- [33]Teng Shi, Weijie Yu, Xiao Zhang, Ming He, Jianping Fan, and Jun Xu. 2025. Benefit from Rich: Tackling Search Interaction Sparsity in Search Enhanced Recommendation. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 2685–2695. doi:10.1145/3746252.3761314DOI
- [34]Zihua Si, Xueran Han, Xiao Zhang, Jun Xu, Yue Yin, Yang Song, and Ji-Rong Wen. 2022. A Model-Agnostic Causal Learning Framework for Recommendation using Search Data. In Proceedings of the ACM Web Conference 2022. Association for Computing Machinery, New York, NY, USA, 224–233. doi:10.1145/3485447.3511951
- [35]Zhiha Si, Zhongxiang Sun, Xiao Zhang, Jun Xu, Xiaoxue Zang, Yang Song, Kun Gai, and Ji-Rong Wen. 2023. When Search Meets Recommendation: Learning Disentangled Search Representation for Recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 1313–1323. doi:10.1145/3539618.3591786
- [36]Pranav Pasumarthi, Scott Lundberg, and Su-In Lee. 2020. Visualizing the Impact of Feature Attribution Baselines. Distill 5, 1 (2020), e22. doi:10.23915/distill.00022
- [37]Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenshu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 1441–1450. doi:10.1145/3357384.3357895DOI
- [38]Zhongxiang Sun, Zihua Si, Xiaoxue Zang, Dewei Leng, Yanan Niu, Yang Song, Xiao Zhang, and Jun Xu. 2023. KUAISAR: A Unified Search and Recommendation Dataset. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 5407–5411. doi:10.1145/3583780.3615123DOI
- [39]Adith Swaminathan and Thorsten Joachims. 2015. Counterfactual Risk Minimization: Learning from Logged Bandit Feedback. In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 37). PMLR, Lille, France, 814–823. https://proceedings.mlr.press/v37/swaminathan15.htmlarxiv.org/abs/1502.02362
- [40]Yuening Wang, Man Chen, Yaochen Hu, Wei Guo, Yingxue Zhang, Huifeng Guo, Yong Liu, and Mark Coates. 2024. Enhancing Click-through Rate Prediction in Recommendation Domain with Search Query Representation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 2462–2471. doi:10.1145/3672763.3679849DOI
- [41]Zhilan Xia, Chuanqiu Xu, Tao Zhang, and Chengfu Huo. 2025. SuperRS: Multi Scenario Reciprocal-Aware Dual MoE for Unified Recommendation-Search Ranking. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 4284–4288. doi:10.1145/3726302.3731949
- [42]Jiayi Xie, Shang Liu, Gao Cong, and Zhenzhong Chen. 2024. UnifiedSSR: A Unified Framework of Sequential Search and Recommendation. In Proceedings of the ACM Web Conference 2024. Association for Computing Machinery, New York, NY, USA, 3410–3419. doi:10.1145/3589334.3645427DOI
- [43]Huimin Yan, Longfei Xu, Junjie Sun, Ni Ou, Wei Luo, Xing Tan, Ran Cheng, Kaikun Liu, and Xiangxiang Chu. 2026. IntSR: A Deployed Generative System for Integrated Search and Recommendation at Amap. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 8671–8678. doi:10.1145/3799682.3840093DOI
- [44]Jing Yao, Zhicheng Dou, Ruobing Xie, Yanxiong Lu, Zhiping Wang, and Ji-Rong Wen. 2021. USER: A Unified Information Search and Recommendation Model based on Integrated Behavior Sequence. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, New York, NY, USA, 2373–2382. doi:10.1145/3459637.3482489DOI
- [45]Hamed Zamani and W. Bruce Croft. 2020. Learning a Joint Search and Recommendation Model from User-Item Interactions. In Proceedings of the 13th International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, NY, USA, 717–725. doi:10.1145/3336191.3371818
- [46]Yuting Zhang, Yiqing Wu, Ruidong Han, Ying Sun, Yongchun Zhu, Xiang Li, Wei Lin, Fuzhen Huang, Zhanlin Han, and Yongjun Xu. 2024. Unified Dual-Intent Translation for Joint Modeling of Search and Recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, USA, 6291–6300. doi:10.1145/3637528.3671519DOI
- [47]Jujia Zhao, Wenjie Wang, Chen Xu, Xiuying Chen, Zhaochun Ren, and Suzan Verberne. 2026. Unifying Search and Recommendation with Dual-View Representation Learning in a Generative Paradigm. ACM Transactions on Information Systems 44, 8 (2026), 1–27. doi:10.1145/3841470
- [48]Jujia Zhao, Zihan Wang, Shuaquan Pan, Suzan Verberne, and Zhaochun Ren. 2026. Unifying Search and Recommendation in LLMs via Gradient Multi-Subspace Tuning. In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 2486–2497. doi:10.1145/3805712.3809719DOI
- [49]Kai Zhao, Yukun Zheng, Tao Zhuang, Xiang Li, and Xiaoyi Zeng. 2022. Joint Learning of E-commerce Search and Recommendation with a Unified Graph Neural Network. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, NY, USA, 1461–1469. doi:10.1145/3488560.3498414DOI
- [50]Wenhao Zhu, Yuxin Li, Shuo Wang, and Hao Wang. 2026. Dual-Horizon Interest Model for Unified Search and Recommendation. Proceedings of the AAAI Conference on Artificial Intelligence 40, 34 (2026), 29159–29167. doi:10.1609/aaai.v40i34.40154


