WorksComputer Science
FAIDER: Fairness-Aware Item-Level Decoding for Large Language Model Recommenders
Abstract
Large language model (LLM) recommenders can produce semantically different recommendations when a user's protected attribute changes, even if the interaction history remains fixed. Controlling this disparity is challenging when model weights and full-vocabulary logits are inaccessible. We propose FAIDER (Fairness-Aware Item-level DEcoding for Recommendation), which incorporates fairness into list construction by selecting complete item candidates rather than manipulating token-level logits or repeatedly repairing finished outputs. The central idea is to separate the operational criterion for a violation from how strongly generation is constrained. Conformal-inspired empirical calibration fixes a cross-group semantic-disparity threshold, while online decoding applies either hard filtering or a soft penalty adjusted through dual-inspired feedback to target a prescribed violation rate. Separately, we establish a pathwise guarantee: under a uniform enforceability assumption, the soft decoder's empirical violation rate exceeds its target by at most . Experiments across recommendation benchmarks and LLM backbones show that Soft FAIDER improves both measured semantic fairness and ranking quality over the evaluated prompt-repair baselines. It also generally achieves a more favorable fairness-utility balance than the hard variant, indicating that stricter per-recommendation constraints do not necessarily yield lower measured disparity.
Introduction
Large language models (LLMs) make it possible to express recommendation as a language-based decision process. Text-to-text frameworks such as P5 unify recommendation tasks within a generative interface [1], while zero-shot rankers show that pretrained LLMs can rank candidates from interaction histories without recommendation-specific training [2]. The broader generative recommendation literature uses this interface to connect behavioral context, item semantics, and natural-language requests [3]. Yet sensitivity to contextual language also creates a fairness concern: demographic information in a user profile can affect recommendation behavior even when the interaction history is held fixed [4, 5].
Fairness in recommendation encompasses distinct objectives, including provider exposure, consumer utility, and individual-level consistency [6, 7, 8]. Language-based systems introduce additional targets, such as fairness in personalized exposure [9]. We focus on minimal-attribute-change semantic stability: changing a protected attribute while holding a user’s interaction history fixed should not substantially alter the semantic content of the recommended items. This operational criterion concerns sensitivity to an input perturbation; it does not identify a structural causal counterfactual in the sense of Kusner et al. [10]. Nor should similarity between recommendation lists be equated with equal user benefit, a distinction highlighted by consumer-fairness evaluation [11]. We therefore assess semantic disparity alongside recommendation quality.
The available interventions depend on model access. Tuning-based frameworks adapt pretrained models to recommendation data [12], and UP5 learns continuous prompts to address recommendation fairness [13]. Such approaches require a trainable component. We instead study a fixed recommender that exposes sampled item continuations and their log-probabilities, but not model weights or full-vocabulary logits. The design problem is to control recommendation behavior through the item proposals and scores available at inference time.
One response is to revise prompts after inspecting generated outputs. FACTER combines conformal thresholding with prompt updates informed by previous fairness violations [14]. More broadly, iterative self-refinement uses feedback on an initial response to guide subsequent revisions without additional model training [15]. These approaches motivate inference-time intervention, but their feedback operates on outputs that have already been generated. We ask a complementary question: can a fairness criterion guide which items are committed to a recommendation list as the list is being constructed?
Decoding-time control offers a relevant starting point. [16, 17, 18] guide generation through hidden-state updates or adjustments to token distributions. Their control rules, however, are not directly available through an interface restricted to sampled item continuations and their likelihoods. Moreover, the semantic disparity of a recommendation list is evaluated over items, rather than isolated tokens. This motivates an item-level decision rule: evaluate each proposed item’s effect on the partial list, select a continuation, and condition subsequent generation on that commitment.
We propose FAIDER (Fairness-Aware Item-Level DEcoding for Recommendation), built around a separation between the criterion for a violation and the strength of its enforcement. Offline calibration fixes an empirical threshold on embedding-based cross-group disparity, a decoding surrogate for the paired attribute-change metric. During generation, the hard decoder filters candidates against this threshold and abstains if no feasible beam remains. The soft decoder instead balances candidate likelihood against disparity through a penalty whose weight is updated from observed violations. A violation thus changes how strongly the decoder intervenes, rather than changing the cutoff that determines whether a violation occurred. Keeping these roles separate provides a stable operational criterion while allowing the fairness–utility trade-off to adapt.
This separation also exposes a statistical issue. Split conformal prediction gives calibrated thresholds a coverage interpretation when calibration and test scores are exchangeable [19]; related ideas extend to risk control and set-valued language generation [20, 21]. Our procedure uses the same calibration set to supply reference outputs and threshold-selection scores, while keeping the test set separate. This reference reuse does not automatically preserve the score exchangeability required by the standard argument; we therefore use the quantile as a conformal-inspired empirical cutoff, without claiming finite-sample test coverage. Adaptive decoding introduces an additional change to the output-selection mechanism. Motivated by feedback-based approaches to adaptive conformal inference and online risk control [22, 23], we analyze realized violations separately from offline calibration. Under an explicit uniform enforceability assumption on candidate pools and the selection rule, we establish a pathwise upper bound on the soft decoder’s excess empirical violation rate for arbitrary query sequences. The guarantee is conditional: selection alone cannot ensure compliance when acceptable continuations are unavailable.
Experiments across recommendation benchmarks and LLM backbones show that Soft FAIDER improves measured semantic fairness and ranking quality over the evaluated prompt-repair baselines. Its generally more favorable fairness–utility balance than the hard variant also shows that stricter per-recommendation constraints do not necessarily yield lower measured disparity. Our contributions are:
Item-level generation control. We introduce a fairness-aware decoder for fixed LLM recommenders using item proposals and candidate likelihoods, without full-vocabulary logits, LLM retraining, or iterative prompt repair.
Separated calibration and enforcement. We combine a fixed empirical disparity threshold with hard and soft decoders, and establish a conditional pathwise violation-rate bound for the soft decoder through dual-inspired online feedback.
Empirical characterization of the trade-off. We compare fairness and ranking quality across recommendation benchmarks and LLM backbones, examining when soft penalties offer a more favorable balance than strict filtering.
Related Work
Fairness objectives and evaluation. Surveys organize fair recommendation by the stakeholders protected, the quantities compared, and the stage at which intervention occurs [8, 7, 24, 5]. Exposure-based approaches address the allocation of attention to ranked items [6], whereas consumer-fairness evaluation considers whether recommendations serve users’ preferences equitably [11]. These objectives are not interchangeable with similarity between generated lists. Likewise, counterfactual fairness is defined relative to a causal model [10], rather than an attribute edit alone. Our focus is narrower: semantic stability under protected-attribute changes, assessed together with ranking quality. This criterion complements, rather than replaces, exposure-based, preference-aware, and causal notions of fairness.
Fairness interventions for LLM recommendation. UP5 learns continuous prefix prompts through adversarial training to suppress sensitive information in user representations [13]. COFFEE instead targets measure-specific counterfactual fairness in personalized explanation generation [9], addressing the quality of generated text rather than the semantic composition of item lists. FACTER is closer to our setting: it combines adaptive conformal thresholding with prompts updated from observed fairness violations, without retraining the recommender [14]. Iterative output refinement also appears in general-purpose methods such as Self-Refine [15], although its objective is not recommendation fairness. FAIDER uses a different intervention point. It neither learns prefix prompts nor revises completed lists through corrective prompting. Its soft variant uses observed list-level violations to adjust the penalty applied during item selection, while the calibrated disparity cutoff remains fixed.
Controlled generation and the decision unit. PPLM steers generation by applying attribute gradients to hidden states [16], while FUDGE uses predictions of future attributes to modify next-token probabilities [17]. DExperts combines expert and anti-expert language-model scores [18], and GeDi derives token-level guidance from class-conditional language models [25]. These approaches demonstrate that a fixed base model can be steered at inference time, but their control rules operate through internal states or token distributions. FAIDER shares the idea of generation-time guidance while changing its unit of action: it scores complete item proposals through their effect on the partial recommendation list, commits one, and conditions subsequent proposals on that choice. This differs from re-ranking a completed list and requires candidate likelihoods, not an accessible full-vocabulary distribution or modifiable hidden states.
Conformal calibration for generated outputs. Classical conformal prediction provides marginal coverage under exchangeability and appropriate score construction [26, 19]. Conformal risk control extends this principle to expected bounded monotone losses [20]. In language generation, Conformal Language Modeling calibrates sampling and filtering procedures to construct sets containing an acceptable response with a prescribed probability [21]. Such set-level guarantees differ from controlling the disparity of a single emitted recommendation list. Weighted conformal methods can accommodate covariate shift when the required likelihood-ratio information is available [27], but do not by themselves justify arbitrary changes to output selection. In FAIDER, the same calibration examples supply both reference outputs and scores for threshold selection. We consequently use the quantile as a conformal-inspired empirical cutoff, not a finite-sample coverage guarantee. Adaptive selection of a single output is an additional reason that classical coverage results cannot simply be transferred to the decoder.
Online feedback and violation control. Online methods complement marginal coverage with retrospective guarantees based on observed outcomes. Adaptive conformal inference adjusts calibration in response to coverage errors [22], while online risk control extends feedback-based regulation to broader losses [23]. Work on decaying step sizes further examines the relationship between long-run coverage and pointwise behavior [28]. Soft FAIDER uses a related feedback structure but updates a dual-inspired penalty weight while keeping the disparity threshold fixed. Its guarantee concerns the realized frequency of threshold violations and requires uniform enforceability of the candidate-selection rule. Neither offline calibration nor the existence of a feasible recommendation alone establishes that assumption. The resulting claim is conditional online control of recommendation generation, rather than an unconditional extension of conformal coverage to adaptively selected lists.
Problem Setup
Let denote a user context, such as an interaction history, a protected attribute, and a recommendation represented as a ranked list of item names. A fixed LLM recommender defines . We assume access to sampled item continuations and their log-probabilities, but not to model weights or full-vocabulary logits. The selection rules below require candidate log-probabilities. A sample-only interface would require a different scoring rule.
We write for a fixed text-embedding map into , applied to the textual representations of user contexts and item lists. Context embeddings are compared with other contexts, and list embeddings with other lists.
Definition 1 (Minimal-attribute-change fairness). A recommender satisfies minimal-attribute-change fairness at tolerance if, for protected-attribute changes with context fixed, .
This criterion expresses the evaluation objective: semantic stability when a protected attribute changes but the user context remains fixed. In the experiments, CFR [14] aggregates the corresponding embedding distances (§ 6).
Method: FAIDER
FAIDER separates threshold selection from generation control. It has two stages: offline calibration and online decoding. Offline calibration constructs cross-group references and fixes an empirical disparity threshold. Online decoding then uses that threshold through either hard feasibility checks or an adaptive soft penalty during list construction.
Offline Calibration with Cross-Group References
Let be the calibration set, disjoint from the test set. For each calibration example, let denote its reference item. We define cross-group neighborhood and semantic disparity as below. Cross-group neighborhood. For any , the set of calibration examples with similar contexts but a different protected attribute is
Semantic disparity. For a candidate output , its disparity relative to cross-group references is
The maximum measures the largest semantic discrepancy within the selected cross-group neighborhood. It compares different users with similar contexts, rather than paired attribute changes for the same context. If the neighborhood is empty, the score is undefined. Deployment therefore requires a documented fallback, such as abstention or a broader similarity threshold. Our analysis assumes nonempty neighborhoods.
Empirical quantile threshold. Using the same reference bank, we compute for the calibration examples and select their upper empirical quantile at nominal level as . The reference bank and remain fixed during test-time decoding. Although excludes each example from its own neighborhood, calibration outputs can still contribute to one another’s scores.
The quantile construction is motivated by split conformal calibration [19]. Under exchangeability of calibration and new scores, its corrected order statistic , with and when , yields marginal exceedance probability at most . In our reference-reuse protocol, however, calibration outputs help construct the scores being calibrated, whereas a new output does not enter that reference bank. Calibration/test separation alone therefore does not establish the score exchangeability required by this argument. We treat the selected quantile as a conformal-inspired empirical cutoff, without claiming distribution-free finite-sample test coverage for this construction. Adaptive decoding also changes the output-selection mechanism. The online bound in § 5 is established separately, conditional on enforceability, and does not rely on offline coverage.
Note that the item is the semantically correct unit. A recommendation’s unit of meaning is the item (“Blade Runner”), not the token. Our disparity is defined on item-list embeddings precisely because that is where semantic meaning lives. Controlling at the item level therefore matches the granularity the fairness notion is about. Instead, token-level control would regulate a list-level quantity at a level where it may not even be defined.1
Online Decoding
Given , FAIDER offers two operating points over the same item-proposal loop:
Hard FAIDER: return a list only if its final disparity satisfies ; abstain when no feasible beam remains.
Soft FAIDER: balance likelihood and disparity through an adaptive penalty, using completed-list violations to regulate its strength.
Algorithm 1 Hard FAIDER (constrained beam search over items)
Input: beam size , threshold , max items , proposer ProposeItems, embedder Emb
1: {(item list, logprob) pairs} 2: for to do 3: 4: for all do 5: for all do 6: 7: if then 8: Add to 9: end if 10: end for 11: end for 12: if then 13: return ABSTAIN (infeasible beam) 14: end if 15: top- of by logprob 16: end for 17: return best list in if final ; otherwise ABSTAIN
Hard FAIDER: Constrained Item Selection
Hard FAIDER maintains a beam of partial recommendation lists. At each slot, it proposes next items, removes extensions whose disparity exceeds , and retains the top- feasible extensions by cumulative log-probability (Algorithm 1). If no extension remains, the decoder abstains. A final score check ensures that every returned list satisfies the operational constraint . Any optional fallback output must be identified separately and is not certified by this check.
Prefix filtering is a search heuristic, not a feasibility-complete procedure. List-level embedding disparity need not be monotone as items are appended, so a rejected prefix could have led to a feasible complete list. The guarantee concerns returned lists, not the existence or discovery of a feasible list for every query, nor the evaluation criterion in Definition 1.
Soft FAIDER with Dual-Inspired Feedback
Soft FAIDER permits individual threshold violations while targeting their frequency across queries. It samples candidate items at each slot and selects the one with the highest fairness-penalized score. The penalty weight is held fixed while constructing query ’s list and updated once after that list is complete.
Constrained objective. The following population-level objective motivates the controller. For a decoding policy producing , write for expectation over both queries and policy randomness:
This objective specifies the desired trade-off; the implemented decoder is an approximation, and the proved result concerns empirical violation frequency rather than an unconditional population-risk guarantee.
Lagrangian and per-slot selection rule. Introducing a multiplier gives . We use this formulation to motivate two
Algorithm 2 Soft FAIDER with dual-inspired feedback (streaming)
Input: $p_\theta$, embedder Emb, neighborhoods $\mathcal{N}$, threshold $\delta$, target rate $\alpha$, step $\eta > 0$, init $\lambda_1 \ge 0$, items $K$, proposals $M$
1: for each incoming query $(x_t,a_t)$, $t = 1,2,\ldots$ do
2: Build $\mathcal{N}(x_t,a_t)$; $L \leftarrow [\ ]$
3: for $k = 1$ to $K$ do
4: Sample proposals $\{c_m\}_{m=1}^{M} \sim p_\theta(\cdot \mid x_t,a_t,\operatorname{FormatList}(L))$
5: for $m = 1$ to $M$ do
6: $r_m \leftarrow \Delta(x_t,a_t;\operatorname{Join}(L \oplus [c_m]))$ // raw disparity, same units as $\delta$
7: end for
8: $c^\star \leftarrow \arg\max_m [\log p_\theta(c_m \mid x_t,a_t,L) - \lambda_t r_m]$; $L \leftarrow L \oplus [c^\star]$
9: end for
10: $y_t \leftarrow \operatorname{Join}(L)$; $\operatorname{err}_t \leftarrow 1\{\Delta(x_t,a_t;y_t) > \delta\}$
11: $\lambda_{t+1} \leftarrow \max(0,\lambda_t+\eta(\operatorname{err}_t-\alpha))$ // feedback update, {xref}`Eq. (5)`
12: emit $y_t$
13: end for
approximations: replacing the violation indicator with raw disparity during selection, and making greedy item-level decisions over a finite proposal pool. For partial list and proposals , the resulting rule is
Here appends an item, and converts the list to its textual representation. The surrogate provides a graded penalty, but its minimization is not equivalent to minimizing threshold violations. Likewise, greedy selection does not solve the full-list Lagrangian maximization. Fixed- Soft FAIDER uses (4) with a constant penalty. The adaptive variant uses the feedback rule below:
Online feedback update. After constructing the complete list for query , we evaluate its threshold violation and update the penalty:
with step size . A violation increases by . A non-violation decreases it by , subject to projection onto the nonnegative half-line. The projection ensures nonnegativity, not complementary slackness or optimality of the policy. Because (4) is a surrogate greedy rule, we interpret (5) as dual-inspired feedback, rather than exact dual subgradient optimization. The update is also motivated by adaptive conformal inference [22], but adjusts enforcement strength while keeping the disparity cutoff fixed. Selection uses raw disparity, whereas feedback uses the completed list’s binary violation. Algorithm 2 summarizes the procedure. The conditional analysis in § 5 bounds the empirical frequency of this binary event directly, without requiring an exact Lagrangian optimizer.
Implementation conventions. We penalize raw disparity on the same scale as , avoiding a step-dependent rescaling of ’s effect. Both threshold calibration and violation feedback use complete lists. Prefix scores guide selection only. We initialize through a warm-up on the calibration set before test evaluation. The reference bank and stay fixed throughout the evaluated stream.
Theoretical Analysis
We analyze how completed-list feedback regulates the empirical frequency of disparity violations. The argument separates a deterministic property of the update from a sufficient condition linking penalty strength to feasible outputs. It uses the cumulative-error and telescoping perspective of adaptive conformal inference [22], but here the updated variable controls decoding strength while the reference bank and threshold remain fixed.
Throughout, let , , and be fixed, and let be the completed list’s violation indicator. Write
for the empirical violation frequency over the first queries.
Feedback and Penalty Growth
Lemma 1 (Pathwise feedback bound). For any realized binary violation sequence, the iterates of (5) satisfy, for every ,
Proof. Since , the update gives . Rearranging and summing over yields
Dividing by proves (6). □
This inequality requires neither enforceability nor a distributional assumption on the queries. It identifies penalty growth as the quantity governing excess violations: suffices for , while a uniformly bounded penalty yields an upper bound on the excess rate. The next step is to identify a sufficient condition for such boundedness.
Conditional Finite-Horizon Control
Assumption 1 (Uniform enforceability). There exists a finite , independent of the horizon , such that along the entire query and proposal sequence,
This condition links penalty strength to final-list feasibility. It is not ensured by the feedback update alone. The existence of a feasible complete list is insufficient: a finite proposal pool may omit the required continuations, and greedy selection using raw disparity need not reach that list. With randomized proposals, the results below hold pathwise on realizations satisfying Assumption 1. An almost-sure conclusion would additionally require this condition to hold almost surely.
Lemma 2 (Bounded penalty). Under Assumption 1, the iterates satisfy for every , where
Proof. Projection ensures nonnegativity. For the upper bound, proceed by induction, starting from . If , then implies . If , Assumption 1 gives , and hence . Both cases preserve the bound. □】【。
Proposition 1 (Finite-horizon empirical violation control). Under Assumption 1, for every ,
In particular, .
Proof. Apply Lemma 1 and substitute from Lemma 2. The second inequality follows from .
Thus, enforceability supplies the boundedness needed to convert the general feedback inequality into finite-horizon rate control. Note that the result is one-sided: the violation frequency may remain below , rather than converging to it. Its finite-horizon usefulness depends on the constants. An unknown or large can make the bound loose.
Remark. The results concern empirical exceedances of the decoding surrogate’s threshold, not the paired attribute-change metric in Definition 1, recommendation utility, or population risk. They require neither offline conformal coverage nor exact Lagrangian optimization. No exchangeability assumption is needed, so the pathwise argument permits nonstationary or adversarial queries. The conclusion nevertheless requires uniform enforceability throughout the sequence, including after distribution shifts.
Experiments
Setup
Datasets and evaluation protocol. Following [14], we evaluate on MovieLens-1M [29] and Amazon Movies & TV [30], with gender as the protected attribute. We split users into disjoint calibration and test sets in a 70/30 ratio. Held-out items provide the ground truth for recommendation evaluation. Calibration users supply both the cross-group reference outputs and the scores used to select . Test users are excluded from reference-bank construction and threshold selection.
Candidate-set implementation. Algorithm 2 presents the sampling-based formulation, in which the LLM proposes candidate items. The reported experiments use a simplified candidate-set implementation: recommendations are constructed from supplied candidates instead of next-item proposals sampled from the LLM. Fairness-adjusted item selection operates over these supplied candidates. The results therefore evaluate control within a candidate set, not the LLM proposal-generation step in Algorithm 2.
Decoding settings. We use beam width for Hard FAIDER. Unless otherwise noted, the neighborhood similarity threshold is , the feedback step size is , and the target violation rate is . The soft penalty is initialized through a warm-up on the calibration set before test evaluation. For CFR evaluation, we use decoding temperature zero when comparing recommendations under protected-attribute changes.
Baselines. We compare with three methods. Zero-Shot [2] uses the LLM without fairness control. UP5 [13] uses soft prompt tuning to address recommendation fairness. FACTOR [14] combines conformal thresholding with iterative prompt repair, providing a comparison with an intervention that operates through corrective prompting. We evaluate both FAIDER variants: hard disparity constraints and a soft penalty adjusted through dual-inspired feedback.
Models and embeddings. The recommendation backbones are Qwen3-8B [31] and Llama3.1-8B-Instruct [32]. For embedding-based comparisons, we use the SentenceTransformer model paraphrase-mpnet-base-v2 [33], fine-tuned on recommendation-specific data. Experiments run on NVIDIA A100 GPUs with 80 GB of memory.
Metrics. Recommendation quality is measured using Recall@10 and normalized discounted cumulative gain at rank 10 (NDCG@10). Recall@10 measures the fraction of held-out relevant items retrieved in the top ten recommendations. NDCG@10 additionally accounts for their ranking positions. We measure counterfactual disparity using CFR [14], the mean embedding shift between recommendation lists when a protected attribute changes while the user context remains fixed. Lower CFR indicates greater semantic stability under this attribute change. Higher Recall@10 and NDCG@10 and lower CFR are preferred.
Main Results
Table 1 compares recommendation quality and counterfactual disparity across the two datasets and two LLM backbones. We have the following observations.
| Dataset | Method | Qwen3-8B Recall@10 | Qwen3-8B NDCG@10 | Qwen3-8B CFR | Llama3.1-8B-Instruct Recall@10 | Llama3.1-8B-Instruct NDCG@10 | Llama3.1-8B-Instruct CFR |
| MovieLens | Zero-shot | 0.402 | 0.458 | 0.742 | 0.358 | 0.402 | 0.701 |
| MovieLens | UP5 | 0.381 | 0.427 | 0.613 | 0.336 | 0.374 | 0.556 |
| MovieLens | FACTER | 0.389 | 0.445 | 0.591 | 0.338 | 0.382 | 0.440 |
| MovieLens | Hard FAIDER | 0.382 | 0.432 | 0.389 | 0.333 | 0.371 | 0.398 |
| MovieLens | Soft FAIDER | 0.390 | 0.446 | 0.474 | 0.352 | 0.397 | 0.243 |
| Amazon | Zero-shot | 0.317 | 0.351 | 0.814 | 0.296 | 0.312 | 0.768 |
| Amazon | UP5 | 0.294 | 0.328 | 0.721 | 0.261 | 0.289 | 0.672 |
| Amazon | FACTER | 0.301 | 0.339 | 0.634 | 0.277 | 0.279 | 0.581 |
| Amazon | Hard FAIDER | 0.297 | 0.326 | 0.574 | 0.266 | 0.283 | 0.509 |
| Amazon | Soft FAIDER | 0.318 | 0.347 | 0.491 | 0.297 | 0.306 | 0.411 |
Table 1. Recommendation quality and counterfactual disparity on MovieLens and Amazon. FAIDER results use the candidate-set implementation. The best and second-best reported values for each dataset and backbone are shown in bold and underlined, respectively.
Counterfactual fairness. Both variants have lower reported CFR than all three baselines in every dataset–backbone setting. Soft FAIDER achieves the lowest CFR in three of the four settings; Hard FAIDER achieves the lowest value on MovieLens with Qwen3-8B. Relative to FACTER, the soft variant reduces CFR by 19.8%–44.8%. The improvement across both backbones and datasets supports the usefulness of incorporating a fairness signal into item selection within the evaluated candidate-set setting.
Recommendation quality. Soft FAIDER has higher Recall@10 and NDCG@10 than FACTER in all four settings and the second-highest NDCG@10 among the compared methods in each setting. Its NDCG@10 remains close to Zero-Shot, while Recall@10 is lower on MovieLens and numerically similar on Amazon. The main benefit is therefore the joint improvement in measured fairness and ranking quality relative to prompt repair. Sometimes the soft FAIDER even achieves slightly better Recall than the naive baseline. This indicates that the LLM biasedly picks stereotypical items that are not what the user actually liked, and our method can help it.
Comparison between Soft and Hard variants. The soft variant has higher Recall@10 and NDCG@10 than the hard variant throughout, and improves all three reported metrics in three settings. On MovieLens with Qwen3-8B, hard filtering instead achieves lower CFR (0.389 versus 0.474), at the cost of lower ranking quality. Thus, stricter enforcement does not consistently minimize the evaluation metric. This is consistent with the distinction between the reference-based decoding surrogate in Equation (2) and the paired attribute-change criterion in Definition 1: enforcing the former does not directly optimize the latter.
Sensitivity Analysis
Figure 1 examines how neighborhood similarity and fixed penalty strength affect ranking quality and counterfactual disparity on MovieLens.

Figure 1. (a) Neighborhood threshold . (b) Fixed penalty strength . Sensitivity analysis on MovieLens. Each panel reports NDCG@10 and CFR as the indicated parameter varies; the right panel uses a constant penalty rather than online feedback adaptation.
Neighborhood similarity threshold . Increasing restricts the reference neighborhood to cross-group users with more similar contexts. Across the tested range from 0.80 to 0.95, CFR decreases from 0.598 to 0.474, while NDCG@10 decreases from 0.454 to 0.446. Stricter context matching is therefore associated with lower measured disparity and a modest ranking-quality cost in this experiment. Unlike the penalty weight, this parameter changes which cross-group comparisons define the control score. The observed trend does not imply that narrower neighborhoods are uniformly preferable.
Penalty strength . We further evaluate the soft decoder with a fixed penalty strength to isolate the effect of . As shown in the figure, increasing from 0 to 4 consistently reduces CFR, accompanied by a decrease in NDCG@10. In particular, CFR decreases substantially as increases, while the reduction in recommendation quality becomes more pronounced at larger penalty strengths. These results demonstrate a clear fairness–utility trade-off: stronger fairness penalties improve counterfactual fairness but may compromise ranking performance. Unlike the fixed-penalty variant, the dual-controlled variant dynamically adjusts in response to observed fairness violations to target a specified violation rate .
Conclusion
We presented FAIDER, an item-level framework for fairness-aware LLM recommendation that separates a fixed empirical disparity threshold from the strength of decoding-time control. It supports hard filtering and adaptive soft penalties without retraining the base recommender. Our analysis connects empirical violation control to penalty growth, yielding a conditional guarantee under uniform enforceability. Experiments suggest that soft control generally offers a more favorable balance between semantic stability and ranking quality than hard filtering. This finding highlights the distinction between satisfying a reference-based constraint and achieving the fairness objective used in evaluation. The broader implication is that decoding-time fairness should be assessed through its joint effects on semantic stability and recommendation quality, rather than through constraint satisfaction alone.
References
- [1]Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. “Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)”. In: Proceedings of the 16th ACM Conference on Recommender Systems. 2022, pp. 299–315. DOI: 10.1145/3523227.3546767.DOI
- [2]Yupeng Hou et al. “Large language models are zero-shot rankers for recommender systems”. In: European Conference on Information Retrieval. Springer. 2024, pp. 364–381.arxiv.org/abs/2305.08845
- [3]Lei Li, Yongfeng Zhang, Dugang Liu, and Li Chen. “Large Language Models for Generative Recommendation: A Survey and Visionary Discussions”. In: Proceedings of LREC-COLING. 2024, pp. 10146–10159. URL: https://aclanthology.org/2024.lrec-main.886/.arxiv.org/abs/2309.01157
- [4]Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. “Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation”. In: Proceedings of the 17th ACM Conference on Recommender Systems. 2023, pp. 993–999.DOI
- [5]Song-Duo Ma, Chu Yun Chen, Bang-An Li, Pin-Yu Chen, Shau-Yung Hsu, and Yun-Nung Chen. “Rethinking Fairness in LLM-Based Recommender Systems: A Survey”. In: Findings of the Association for Computational Linguistics: EMNLP 2026. 2026, pp. 25277–25296. DOI: 10.18653/2026.findings-emnlp.1197.arxiv.org/abs/2606.28340
- [6]Ashudeep Singh and Thorsten Joachims. “Fairness of Exposure in Rankings”. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2018. URL: https://www.kdd.org/kdd2018/accepted-papers/view/fairness-of-exposure-in-rankings.DOI
- [7]Bjørnar Vassøy and Helge Langseth. “Consumer-Side Fairness in Recommender Systems: A Systematic Survey of Methods and Evaluation”. In: Artificial Intelligence Review 57.101 (2024). DOI: 10.1007/s10462-023-10663-5.DOI
- [8]Yashar Deldjoo, Dietmar Jannach, Alejandro Bellogin, Alessandro Difonzo, and Dario Zanzonelli. “Fairness in Recommender Systems: Research Landscape and Future Directions”. In: User Modeling and User-Adapted Interaction 34 (2024), pp. 59–108. DOI: 10.1007/s11257-023-09364-z.DOI
- [9]Nan Wang et al. “COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023, pp. 13258–13275. DOI: 10.18653/v1/2023.emnlp-main.819.DOI
- [10]Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. “Counterfactual Fairness”. In: Advances in Neural Information Processing Systems. 2017.arxiv.org/abs/1703.06856
- [11]Yashar Deldjoo and Tommaso Di Noia. “CFaiRLLM: Consumer Fairness Evaluation in Large-Language Model Recommender System”. In: ACM Transactions on Intelligent Systems and Technology 16.6 (2025), pp. 1–31. DOI: 10.1145/3725853.DOI
- [12]Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. “TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation”. In: Proceedings of the 17th ACM Conference on Recommender Systems. 2023, pp. 1007–1014. DOI: 10.1145/3604915.3608857.DOI
- [13]Wenyue Hua, Yingqiang Ge, Shuyuan Xu, Jianchao Ji, Zelong Li, and Yongfeng Zhang. “Up5: Unbiased foundation model for fairness-aware recommendation”. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, pp. 1899–1912.arxiv.org/abs/2305.12090
- [14]Arya Fayyazi, Mehdi Kamal, and Massoud Pedram. “FA CTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for Enabling Fair LLM-Based Recommender Systems”. In: Forty-second International Conference on Machine Learning. 2025.DOI
- [15]Aman Madaan et al. “Self-Refine: Iterative Refinement with Self-Feedback”. In: Advances in Neural Information Processing Systems. Vol. 36. 2023, pp. 46534–46594. DOI: 10.52202/075280–2019.arxiv.org/abs/2303.17651
- [16]Sumanth Datathri et al. “Plug and Play Language Models: A Simple Approach to Controlled Text Generation”. In: International Conference on Learning Representations. 2020.
- [17]Kevin Yang and Dan Klein. “FUDGE: Controlled text generation with future discriminators”. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021, pp. 3511–3535.
- [18]Alisa Liu et al. “DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts”. In: Proceedings of ACL-IJCNLP. 2021, pp. 6691–6706. DOI: 10.18653/v1/2021.acl-long.522.DOI
- [19]Anastasios N. Angelopoulos and Stephen Bates. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification”. In: Foundations and Trends in Machine Learning 16.4 (2023), pp. 494–591. DOI: 10.1561/2200000101.
- [20]Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. “Conformal Risk Control”. In: International Conference on Learning Representations. 2024. URL: https://proceedings.iclr.cc/paper_files/paper/2024/hash/f3549ef9b5ff520a7e41f3cc306ab2b-Abstract-Conference.html.
- [21]Victor Quach et al. “Conformal Language Modeling”. In: International Conference on Learning Representations. 2024. URL: https://proceedings.iclr.cc/paper_files/paper/2024/hash/31421b112e5f7faf4fc577b74e45dab2-Abstract-Conference.html.arxiv.org/abs/2306.10193
- [22]Isaac Gibbs and Emmanuel Candes. “Adaptive conformal inference under distribution shift”. In: Advances in Neural Information Processing Systems 34 (2021), pp. 1660–1672.arxiv.org/abs/2106.00170
- [23]Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. “Achieving Risk Control in Online Learning Settings”. In: Transactions on Machine Learning Research (2023). URL: https://openreview.net/forum?id=5Y04GWvoJu.arxiv.org/abs/2205.09095
- [24]Di Jin et al. “A Survey on Fairness-Aware Recommender Systems”. In: Information Fusion 100 (2023), p. 101906. DOI: 10.1016/j.inffus.2023.101906.DOI
- [25]Ben Krause et al. “GeDi: Generative Discriminator Guided Sequence Generation”. In: Findings of EMNLP. 2021, pp. 4929–4952. DOI: 10.18653/v1/2021.findings-emnlp.424.
- [26]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005. DOI: 10.1007/b106715.DOI
- [27]Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candes, and Aaditya Ramdas. “Conformal Prediction Under Covariate Shift”. In: Advances in Neural Information Processing Systems. Vol. 32. 2019. URL: https://proceedings.neurips.cc/paper/2019/hash/8fb21ee7a2207526da55a679f0332de2-Abstract.html.arxiv.org/abs/1904.06019
- [28]Anastasios Nikolas Angelopoulos, Rina Barber, and Stephen Bates. “Online Conformal Prediction with Decaying Step Sizes”. In: Proceedings of the 41st International Conference on Machine Learning. Vol. 235. PMLR. 2024, pp. 1616–1630. URL: https://proceedings.mlr.press/v235/angelopoulos24a.html.arxiv.org/abs/2402.01139
- [29]F Maxwell Harper and Joseph A Konstan. “The movielens datasets: History and context”. In: Acm transactions on interactive intelligent systems (tiis) 5.4 (2015), pp. 1–19.
- [30]Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. “Image-based recommendations on styles and substitutes”. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 2015, pp. 43–52.arxiv.org/abs/1506.04757
- [31]An Yang et al. “Qwen3 technical report”. In: arXiv preprint arXiv:2505.09388 (2025).arxiv.org/abs/2505.09388
- [32]Aaron Grattafiori et al. “The llama 3 herd of models”. In: arXiv preprint arXiv:2407.21783 (2024).arxiv.org/abs/2407.21783
- [33]Nils Reimers and Iryna Gurevych. “Sentence-bert: Sentence embeddings using siamese bert-networks”. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019, pp. 3982–3992.arxiv.org/abs/1908.10084