betaFeedback

WorksComputer Science

FAIDER: Fairness-Aware Item-Level Decoding for Large Language Model Recommenders

Abstract

Large language model (LLM) recommenders can produce semantically different recommendations when a user's protected attribute changes, even if the interaction history remains fixed. Controlling this disparity is challenging when model weights and full-vocabulary logits are inaccessible. We propose FAIDER (Fairness-Aware Item-level DEcoding for Recommendation), which incorporates fairness into list construction by selecting complete item candidates rather than manipulating token-level logits or repeatedly repairing finished outputs. The central idea is to separate the operational criterion for a violation from how strongly generation is constrained. Conformal-inspired empirical calibration fixes a cross-group semantic-disparity threshold, while online decoding applies either hard filtering or a soft penalty adjusted through dual-inspired feedback to target a prescribed violation rate. Separately, we establish a pathwise guarantee: under a uniform enforceability assumption, the soft decoder's empirical violation rate exceeds its target by at most O(1/T)O(1/T). Experiments across recommendation benchmarks and LLM backbones show that Soft FAIDER improves both measured semantic fairness and ranking quality over the evaluated prompt-repair baselines. It also generally achieves a more favorable fairness-utility balance than the hard variant, indicating that stricter per-recommendation constraints do not necessarily yield lower measured disparity.

Introduction

Large language models (LLMs) make it possible to express recommendation as a language-based decision process. Text-to-text frameworks such as P5 unify recommendation tasks within a generative interface [1], while zero-shot rankers show that pretrained LLMs can rank candidates from interaction histories without recommendation-specific training [2]. The broader generative recommendation literature uses this interface to connect behavioral context, item semantics, and natural-language requests [3]. Yet sensitivity to contextual language also creates a fairness concern: demographic information in a user profile can affect recommendation behavior even when the interaction history is held fixed [4, 5].

Fairness in recommendation encompasses distinct objectives, including provider exposure, consumer utility, and individual-level consistency [6, 7, 8]. Language-based systems introduce additional targets, such as fairness in personalized exposure [9]. We focus on minimal-attribute-change semantic stability: changing a protected attribute while holding a user’s interaction history fixed should not substantially alter the semantic content of the recommended items. This operational criterion concerns sensitivity to an input perturbation; it does not identify a structural causal counterfactual in the sense of Kusner et al. [10]. Nor should similarity between recommendation lists be equated with equal user benefit, a distinction highlighted by consumer-fairness evaluation [11]. We therefore assess semantic disparity alongside recommendation quality.

The available interventions depend on model access. Tuning-based frameworks adapt pretrained models to recommendation data [12], and UP5 learns continuous prompts to address recommendation fairness [13]. Such approaches require a trainable component. We instead study a fixed recommender that exposes sampled item continuations and their log-probabilities, but not model weights or full-vocabulary logits. The design problem is to control recommendation behavior through the item proposals and scores available at inference time.

One response is to revise prompts after inspecting generated outputs. FACTER combines conformal thresholding with prompt updates informed by previous fairness violations [14]. More broadly, iterative self-refinement uses feedback on an initial response to guide subsequent revisions without additional model training [15]. These approaches motivate inference-time intervention, but their feedback operates on outputs that have already been generated. We ask a complementary question: can a fairness criterion guide which items are committed to a recommendation list as the list is being constructed?

Decoding-time control offers a relevant starting point. [16, 17, 18] guide generation through hidden-state updates or adjustments to token distributions. Their control rules, however, are not directly available through an interface restricted to sampled item continuations and their likelihoods. Moreover, the semantic disparity of a recommendation list is evaluated over items, rather than isolated tokens. This motivates an item-level decision rule: evaluate each proposed item’s effect on the partial list, select a continuation, and condition subsequent generation on that commitment.

We propose FAIDER (Fairness-Aware Item-Level DEcoding for Recommendation), built around a separation between the criterion for a violation and the strength of its enforcement. Offline calibration fixes an empirical threshold on embedding-based cross-group disparity, a decoding surrogate for the paired attribute-change metric. During generation, the hard decoder filters candidates against this threshold and abstains if no feasible beam remains. The soft decoder instead balances candidate likelihood against disparity through a penalty whose weight is updated from observed violations. A violation thus changes how strongly the decoder intervenes, rather than changing the cutoff that determines whether a violation occurred. Keeping these roles separate provides a stable operational criterion while allowing the fairness–utility trade-off to adapt.

This separation also exposes a statistical issue. Split conformal prediction gives calibrated thresholds a coverage interpretation when calibration and test scores are exchangeable [19]; related ideas extend to risk control and set-valued language generation [20, 21]. Our procedure uses the same calibration set to supply reference outputs and threshold-selection scores, while keeping the test set separate. This reference reuse does not automatically preserve the score exchangeability required by the standard argument; we therefore use the quantile as a conformal-inspired empirical cutoff, without claiming finite-sample test coverage. Adaptive decoding introduces an additional change to the output-selection mechanism. Motivated by feedback-based approaches to adaptive conformal inference and online risk control [22, 23], we analyze realized violations separately from offline calibration. Under an explicit uniform enforceability assumption on candidate pools and the selection rule, we establish a pathwise O(1/T)O(1/T) upper bound on the soft decoder’s excess empirical violation rate for arbitrary query sequences. The guarantee is conditional: selection alone cannot ensure compliance when acceptable continuations are unavailable.

Experiments across recommendation benchmarks and LLM backbones show that Soft FAIDER improves measured semantic fairness and ranking quality over the evaluated prompt-repair baselines. Its generally more favorable fairness–utility balance than the hard variant also shows that stricter per-recommendation constraints do not necessarily yield lower measured disparity. Our contributions are:

  • Item-level generation control. We introduce a fairness-aware decoder for fixed LLM recommenders using item proposals and candidate likelihoods, without full-vocabulary logits, LLM retraining, or iterative prompt repair.

  • Separated calibration and enforcement. We combine a fixed empirical disparity threshold with hard and soft decoders, and establish a conditional pathwise violation-rate bound for the soft decoder through dual-inspired online feedback.

  • Empirical characterization of the trade-off. We compare fairness and ranking quality across recommendation benchmarks and LLM backbones, examining when soft penalties offer a more favorable balance than strict filtering.

Related Work

Fairness objectives and evaluation. Surveys organize fair recommendation by the stakeholders protected, the quantities compared, and the stage at which intervention occurs [8, 7, 24, 5]. Exposure-based approaches address the allocation of attention to ranked items [6], whereas consumer-fairness evaluation considers whether recommendations serve users’ preferences equitably [11]. These objectives are not interchangeable with similarity between generated lists. Likewise, counterfactual fairness is defined relative to a causal model [10], rather than an attribute edit alone. Our focus is narrower: semantic stability under protected-attribute changes, assessed together with ranking quality. This criterion complements, rather than replaces, exposure-based, preference-aware, and causal notions of fairness.

Fairness interventions for LLM recommendation. UP5 learns continuous prefix prompts through adversarial training to suppress sensitive information in user representations [13]. COFFEE instead targets measure-specific counterfactual fairness in personalized explanation generation [9], addressing the quality of generated text rather than the semantic composition of item lists. FACTER is closer to our setting: it combines adaptive conformal thresholding with prompts updated from observed fairness violations, without retraining the recommender [14]. Iterative output refinement also appears in general-purpose methods such as Self-Refine [15], although its objective is not recommendation fairness. FAIDER uses a different intervention point. It neither learns prefix prompts nor revises completed lists through corrective prompting. Its soft variant uses observed list-level violations to adjust the penalty applied during item selection, while the calibrated disparity cutoff remains fixed.

Controlled generation and the decision unit. PPLM steers generation by applying attribute gradients to hidden states [16], while FUDGE uses predictions of future attributes to modify next-token probabilities [17]. DExperts combines expert and anti-expert language-model scores [18], and GeDi derives token-level guidance from class-conditional language models [25]. These approaches demonstrate that a fixed base model can be steered at inference time, but their control rules operate through internal states or token distributions. FAIDER shares the idea of generation-time guidance while changing its unit of action: it scores complete item proposals through their effect on the partial recommendation list, commits one, and conditions subsequent proposals on that choice. This differs from re-ranking a completed list and requires candidate likelihoods, not an accessible full-vocabulary distribution or modifiable hidden states.

Conformal calibration for generated outputs. Classical conformal prediction provides marginal coverage under exchangeability and appropriate score construction [26, 19]. Conformal risk control extends this principle to expected bounded monotone losses [20]. In language generation, Conformal Language Modeling calibrates sampling and filtering procedures to construct sets containing an acceptable response with a prescribed probability [21]. Such set-level guarantees differ from controlling the disparity of a single emitted recommendation list. Weighted conformal methods can accommodate covariate shift when the required likelihood-ratio information is available [27], but do not by themselves justify arbitrary changes to output selection. In FAIDER, the same calibration examples supply both reference outputs and scores for threshold selection. We consequently use the quantile as a conformal-inspired empirical cutoff, not a finite-sample coverage guarantee. Adaptive selection of a single output is an additional reason that classical coverage results cannot simply be transferred to the decoder.

Online feedback and violation control. Online methods complement marginal coverage with retrospective guarantees based on observed outcomes. Adaptive conformal inference adjusts calibration in response to coverage errors [22], while online risk control extends feedback-based regulation to broader losses [23]. Work on decaying step sizes further examines the relationship between long-run coverage and pointwise behavior [28]. Soft FAIDER uses a related feedback structure but updates a dual-inspired penalty weight while keeping the disparity threshold fixed. Its guarantee concerns the realized frequency of threshold violations and requires uniform enforceability of the candidate-selection rule. Neither offline calibration nor the existence of a feasible recommendation alone establishes that assumption. The resulting claim is conditional online control of recommendation generation, rather than an unconditional extension of conformal coverage to adaptively selected lists.

Problem Setup

Let x∈Xx \in\mathcal{X} denote a user context, such as an interaction history, a∈Aa \in\mathcal{A} a protected attribute, and y∈Yy \in\mathcal{Y} a recommendation represented as a ranked list of item names. A fixed LLM recommender defines pθ(y∣x,a)p_\theta(y \mid x, a). We assume access to sampled item continuations and their log-probabilities, but not to model weights or full-vocabulary logits. The selection rules below require candidate log-probabilities. A sample-only interface would require a different scoring rule.

We write Emb⁡(⋅)\operatorname{Emb}(\cdot) for a fixed text-embedding map into Rm\mathbb{R}^{m}, applied to the textual representations of user contexts and item lists. Context embeddings are compared with other contexts, and list embeddings with other lists.

Definition 1 (Minimal-attribute-change fairness). A recommender satisfies minimal-attribute-change fairness at tolerance εfair\varepsilon_{\mathrm{fair}} if, for protected-attribute changes a→a′a \to a' with context fixed, ∥Emb⁡(y(x,a))−Emb⁡(y(x,a′))∥2≤εfair\lVert\operatorname{Emb}(y(x,a)) - \operatorname{Emb}(y(x,a')) \rVert_{2} \le\varepsilon_{\mathrm{fair}}.

This criterion expresses the evaluation objective: semantic stability when a protected attribute changes but the user context remains fixed. In the experiments, CFR [14] aggregates the corresponding embedding distances (§ 6).

Method: FAIDER

FAIDER separates threshold selection from generation control. It has two stages: offline calibration and online decoding. Offline calibration constructs cross-group references and fixes an empirical disparity threshold. Online decoding then uses that threshold through either hard feasibility checks or an adaptive soft penalty during list construction.

Offline Calibration with Cross-Group References

Let Dcal={(xi,ai)}i=1n\mathcal{D}_{\mathrm{cal}} = \{(x_i,a_i)\}_{i=1}^{n} be the calibration set, disjoint from the test set. For each calibration example, let yiy_i denote its reference item. We define cross-group neighborhood and semantic disparity as below. Cross-group neighborhood. For any (x,a)(x,a), the set of calibration examples with similar contexts but a different protected attribute is

N(x,a)={j:cos⁡(Emb⁡(x),Emb⁡(xj))≥τρ ∧ aj≠a}.(1)\mathcal{N}(x,a)=\{j:\cos(\operatorname{Emb}(x),\operatorname{Emb}(x_j))\ge\tau_{\rho}\ \wedge\ a_j\ne a\}. \tag*{(1)}

Semantic disparity. For a candidate output yy, its disparity relative to cross-group references is

Δ(x,a;y)=max⁡j∈N(x,a)∥Emb⁡(y)−Emb⁡(yj)∥2.(2)\Delta(x,a;y)=\max_{j\in\mathcal{N}(x,a)}\left\lVert\operatorname{Emb}(y)-\operatorname{Emb}(y_j)\right\rVert_2. \tag*{(2)}

The maximum measures the largest semantic discrepancy within the selected cross-group neighborhood. It compares different users with similar contexts, rather than paired attribute changes for the same context. If the neighborhood is empty, the score is undefined. Deployment therefore requires a documented fallback, such as abstention or a broader similarity threshold. Our analysis assumes nonempty neighborhoods.

Empirical quantile threshold. Using the same reference bank, we compute Δi=Δ(xi,ai;y^i)\Delta_i=\Delta(x_i,a_i;\hat{y}_i) for the calibration examples and select their upper empirical quantile at nominal level 1−α1-\alpha as δ\delta. The reference bank and δ\delta remain fixed during test-time decoding. Although aj≠aia_j\ne a_i excludes each example from its own neighborhood, calibration outputs can still contribute to one another’s scores.

The quantile construction is motivated by split conformal calibration [19]. Under exchangeability of calibration and new scores, its corrected order statistic δ=Δ(k)\delta=\Delta_{(k)}, with k=⌈(n+1)(1−α)⌉k=\lceil(n+1)(1-\alpha)\rceil and δ=+∞\delta=+\infty when k>nk>n, yields marginal exceedance probability at most α\alpha. In our reference-reuse protocol, however, calibration outputs help construct the scores being calibrated, whereas a new output does not enter that reference bank. Calibration/test separation alone therefore does not establish the score exchangeability required by this argument. We treat the selected quantile as a conformal-inspired empirical cutoff, without claiming distribution-free finite-sample test coverage for this construction. Adaptive decoding also changes the output-selection mechanism. The online bound in § 5 is established separately, conditional on enforceability, and does not rely on offline coverage.

Note that the item is the semantically correct unit. A recommendation’s unit of meaning is the item (“Blade Runner”), not the token. Our disparity Δ\Delta is defined on item-list embeddings precisely because that is where semantic meaning lives. Controlling at the item level therefore matches the granularity the fairness notion is about. Instead, token-level control would regulate a list-level quantity at a level where it may not even be defined.1

Online Decoding

Given δ\delta, FAIDER offers two operating points over the same item-proposal loop:

  • Hard FAIDER: return a list only if its final disparity satisfies Δ≤δ\Delta\le\delta; abstain when no feasible beam remains.

  • Soft FAIDER: balance likelihood and disparity through an adaptive penalty, using completed-list violations to regulate its strength.

Algorithm 1 Hard FAIDER (constrained beam search over items)

Input: beam size BB, threshold δ\delta, max items KK, proposer ProposeItems, embedder Emb

1: B0←{([ ],0)}\mathcal{B}_0 \leftarrow\{([\,],0)\} {(item list, logprob) pairs} 2: for k=1k = 1 to KK do 3: C←∅\mathcal{C} \leftarrow\varnothing 4: for all (L,S)∈Bk−1(L,S) \in\mathcal{B}_{k-1} do 5: for all c∈ProposeItems⁡(x,a,L)c \in\operatorname{ProposeItems}(x,a,L) do 6: L′←L⊕[c]; Δ^←Δ(x,a;Join⁡(L′))L' \leftarrow L \oplus[c];\ \hat{\Delta} \leftarrow\Delta(x,a;\operatorname{Join}(L')) 7: if Δ^≤δ\hat{\Delta} \le\delta then 8: Add (L′,S+log⁡pθ(c∣x,a,L))(L', S + \log p_{\theta}(c \mid x,a,L)) to C\mathcal{C} 9: end if 10: end for 11: end for 12: if C=∅\mathcal{C} = \varnothing then 13: return ABSTAIN (infeasible beam) 14: end if 15: Bk←\mathcal{B}_k \leftarrow top-BB of C\mathcal{C} by logprob 16: end for 17: return best list in BK\mathcal{B}_K if final Δ≤δ\Delta\le\delta; otherwise ABSTAIN

Hard FAIDER: Constrained Item Selection

Hard FAIDER maintains a beam of partial recommendation lists. At each slot, it proposes next items, removes extensions whose disparity exceeds δ\delta, and retains the top-BB feasible extensions by cumulative log-probability (Algorithm 1). If no extension remains, the decoder abstains. A final score check ensures that every returned list satisfies the operational constraint Δ≤δ\Delta\le\delta. Any optional fallback output must be identified separately and is not certified by this check.

Prefix filtering is a search heuristic, not a feasibility-complete procedure. List-level embedding disparity need not be monotone as items are appended, so a rejected prefix could have led to a feasible complete list. The guarantee concerns returned lists, not the existence or discovery of a feasible list for every query, nor the evaluation criterion in Definition 1.

Soft FAIDER with Dual-Inspired Feedback

Soft FAIDER permits individual threshold violations while targeting their frequency across queries. It samples MM candidate items at each slot and selects the one with the highest fairness-penalized score. The penalty weight λt\lambda_t is held fixed while constructing query tt’s list and updated once after that list is complete.

Constrained objective. The following population-level objective motivates the controller. For a decoding policy π\pi producing y∼π(⋅∣x,a)y \sim\pi(\cdot\mid x,a), write Eπ\mathbb{E}_{\pi} for expectation over both queries and policy randomness:

max⁡π Eπ[log⁡pθ(y∣x,a)]s.t.Eπ[1{Δ(x,a;y)>δ}]≤α.(3)\max_{\pi}\ \mathbb{E}_{\pi}[\log p_{\theta}(y \mid x,a)] \quad\text{s.t.}\quad \mathbb{E}_{\pi}[\mathbf{1}\{\Delta(x,a;y) > \delta\}] \le\alpha. \tag*{(3)}

This objective specifies the desired trade-off; the implemented decoder is an approximation, and the proved result concerns empirical violation frequency rather than an unconditional population-risk guarantee.

Lagrangian and per-slot selection rule. Introducing a multiplier λ≥0\lambda\ge0 gives L(π,λ)=Eπ[log⁡pθ(y∣x,a)]−λ(Eπ[1{Δ>δ}]−α)\mathcal{L}(\pi,\lambda) = \mathbb{E}_{\pi}[\log p_{\theta}(y \mid x,a)] - \lambda(\mathbb{E}_{\pi}[\mathbf{1}\{\Delta> \delta\}] - \alpha). We use this formulation to motivate two

Algorithm 2  Soft FAIDER with dual-inspired feedback (streaming)
Input: $p_\theta$, embedder Emb, neighborhoods $\mathcal{N}$, threshold $\delta$, target rate $\alpha$, step $\eta > 0$, init $\lambda_1 \ge 0$, items $K$, proposals $M$
1: for each incoming query $(x_t,a_t)$, $t = 1,2,\ldots$ do
2:   Build $\mathcal{N}(x_t,a_t)$; $L \leftarrow [\ ]$
3:   for $k = 1$ to $K$ do
4:     Sample proposals $\{c_m\}_{m=1}^{M} \sim p_\theta(\cdot \mid x_t,a_t,\operatorname{FormatList}(L))$
5:     for $m = 1$ to $M$ do
6:       $r_m \leftarrow \Delta(x_t,a_t;\operatorname{Join}(L \oplus [c_m]))$                 // raw disparity, same units as $\delta$
7:     end for
8:     $c^\star \leftarrow \arg\max_m [\log p_\theta(c_m \mid x_t,a_t,L) - \lambda_t r_m]$;   $L \leftarrow L \oplus [c^\star]$
9:   end for
10:  $y_t \leftarrow \operatorname{Join}(L)$;   $\operatorname{err}_t \leftarrow 1\{\Delta(x_t,a_t;y_t) > \delta\}$
11:  $\lambda_{t+1} \leftarrow \max(0,\lambda_t+\eta(\operatorname{err}_t-\alpha))$                 // feedback update, {xref}`Eq. (5)`
12:  emit $y_t$
13: end for

approximations: replacing the violation indicator with raw disparity during selection, and making greedy item-level decisions over a finite proposal pool. For partial list LL and proposals {cm}m=1M\{c_m\}_{m=1}^{M}, the resulting rule is

c⋆=arg⁡max⁡m[log⁡pθ(cm∣x,a,L)−λΔ(x,a;Join⁡(L⊕[cm]))].(4)c^\star= \arg\max_m \left[\log p_\theta(c_m \mid x,a,L) - \lambda\Delta(x,a;\operatorname{Join}(L \oplus[c_m]))\right]. \tag*{(4)}

Here ⊕\oplus appends an item, and Join⁡\operatorname{Join} converts the list to its textual representation. The surrogate provides a graded penalty, but its minimization is not equivalent to minimizing threshold violations. Likewise, greedy selection does not solve the full-list Lagrangian maximization. Fixed-λ\lambda Soft FAIDER uses (4) with a constant penalty. The adaptive variant uses the feedback rule below:

Online feedback update. After constructing the complete list yty_t for query (xt,at)(x_t,a_t), we evaluate its threshold violation and update the penalty:

λt+1=[λt+η(err⁡t−α)]+,err⁡t=1{Δ(xt,at;yt)>δ},(5)\lambda_{t+1} = [\lambda_t+\eta(\operatorname{err}_t-\alpha)]_+, \qquad\operatorname{err}_t = 1\{\Delta(x_t,a_t;y_t) > \delta\}, \tag*{(5)}

with step size η>0\eta> 0. A violation increases λ\lambda by η(1−α)\eta(1-\alpha). A non-violation decreases it by ηα\eta\alpha, subject to projection onto the nonnegative half-line. The projection ensures nonnegativity, not complementary slackness or optimality of the policy. Because (4) is a surrogate greedy rule, we interpret (5) as dual-inspired feedback, rather than exact dual subgradient optimization. The update is also motivated by adaptive conformal inference [22], but adjusts enforcement strength while keeping the disparity cutoff fixed. Selection uses raw disparity, whereas feedback uses the completed list’s binary violation. Algorithm 2 summarizes the procedure. The conditional analysis in § 5 bounds the empirical frequency of this binary event directly, without requiring an exact Lagrangian optimizer.

Implementation conventions. We penalize raw disparity on the same scale as δ\delta, avoiding a step-dependent rescaling of λ\lambda’s effect. Both threshold calibration and violation feedback use complete lists. Prefix scores guide selection only. We initialize λ1\lambda_1 through a warm-up on the calibration set before test evaluation. The reference bank and δ\delta stay fixed throughout the evaluated stream.

Theoretical Analysis

We analyze how completed-list feedback regulates the empirical frequency of disparity violations. The argument separates a deterministic property of the update from a sufficient condition linking penalty strength to feasible outputs. It uses the cumulative-error and telescoping perspective of adaptive conformal inference [22], but here the updated variable controls decoding strength while the reference bank and threshold δ\delta remain fixed.

Throughout, let α∈(0,1)\alpha\in(0,1), η>0\eta> 0, and λ1≥0\lambda_1 \ge0 be fixed, and let err⁡t=1{Δ(xt,at;yt)>δ}\operatorname{err}_t = 1\{\Delta(x_t,a_t;y_t) > \delta\} be the completed list’s violation indicator. Write

v^T=1T∑t=1Terr⁡t\widehat{v}_T = \frac{1}{T}\sum_{t=1}^{T}\operatorname{err}_t

for the empirical violation frequency over the first TT queries.

Feedback and Penalty Growth

Lemma 1 (Pathwise feedback bound). For any realized binary violation sequence, the iterates of (5) satisfy, for every T≥1T \ge1,

v^T≤α+λT+1−λ1ηT.(6)\widehat{v}_T \le\alpha+ \frac{\lambda_{T+1}-\lambda_1}{\eta T}. \tag*{(6)}

Proof. Since [z]+≥z[z]_+ \ge z, the update gives λt+1≥λt+η(err⁡t−α)\lambda_{t+1} \ge\lambda_t + \eta(\operatorname{err}_t-\alpha). Rearranging and summing over t=1,…,Tt=1,\ldots,T yields

∑t=1T(err⁡t−α)≤1η∑t=1T(λt+1−λt)=λT+1−λ1η.\sum_{t=1}^{T}(\operatorname{err}_t-\alpha) \le\frac{1}{\eta}\sum_{t=1}^{T}(\lambda_{t+1}-\lambda_t) = \frac{\lambda_{T+1}-\lambda_1}{\eta}.

Dividing by TT proves (6). □

This inequality requires neither enforceability nor a distributional assumption on the queries. It identifies penalty growth as the quantity governing excess violations: λT+1=o(T)\lambda_{T+1}=o(T) suffices for lim sup⁡T→∞v^T≤α\limsup_{T\to\infty}\widehat{v}_T \le\alpha, while a uniformly bounded penalty yields an O(1/T)O(1/T) upper bound on the excess rate. The next step is to identify a sufficient condition for such boundedness.

Conditional Finite-Horizon Control

Assumption 1 (Uniform enforceability). There exists a finite λ⋆≥0\lambda^\star\ge0, independent of the horizon TT, such that along the entire query and proposal sequence,

λt≥λ⋆  ⟹  err⁡t=0for every t≥1.\lambda_t \ge\lambda^\star\implies\operatorname{err}_t = 0 \qquad\text{for every } t \ge1.

This condition links penalty strength to final-list feasibility. It is not ensured by the feedback update alone. The existence of a feasible complete list is insufficient: a finite proposal pool may omit the required continuations, and greedy selection using raw disparity need not reach that list. With randomized proposals, the results below hold pathwise on realizations satisfying Assumption 1. An almost-sure conclusion would additionally require this condition to hold almost surely.

Lemma 2 (Bounded penalty). Under Assumption 1, the iterates satisfy 0≤λt≤λmax⁡0 \le\lambda_t \le\lambda_{\max} for every t≥1t \ge1, where

λmax⁡=max⁡{λ1,λ⋆+η}.\lambda_{\max} = \max\{\lambda_1,\lambda^\star+\eta\}.

Proof. Projection ensures nonnegativity. For the upper bound, proceed by induction, starting from λ1≤λmax⁡\lambda_1 \le\lambda_{\max}. If λt<λ⋆\lambda_t < \lambda^\star, then err⁡t−α≤1\operatorname{err}_t-\alpha\le1 implies λt+1≤λt+η<λ⋆+η≤λmax⁡\lambda_{t+1} \le\lambda_t+\eta< \lambda^\star+\eta\le\lambda_{\max}. If λt≥λ⋆\lambda_t \ge\lambda^\star, Assumption 1 gives err⁡t=0\operatorname{err}_t=0, and hence λt+1=max⁡{0,λt−ηα}≤λt≤λmax⁡\lambda_{t+1}=\max\{0,\lambda_t-\eta\alpha\} \le\lambda_t \le\lambda_{\max}. Both cases preserve the bound. □】【。

Proposition 1 (Finite-horizon empirical violation control). Under Assumption 1, for every T≥1T \ge1,

v^T≤α+λmax⁡−λ1ηT≤α+λmax⁡ηT.(7)\widehat{v}_T \le\alpha+ \frac{\lambda_{\max}-\lambda_1}{\eta T} \le\alpha+ \frac{\lambda_{\max}}{\eta T}. \tag*{(7)}

In particular, lim sup⁡T→∞v^T≤α\limsup_{T \to\infty} \widehat{v}_T \le\alpha.

Proof. Apply Lemma 1 and substitute λT+1≤λmax⁡\lambda_{T+1} \le\lambda_{\max} from Lemma 2. The second inequality follows from λ1≥0\lambda_1 \ge0.

Thus, enforceability supplies the boundedness needed to convert the general feedback inequality into finite-horizon rate control. Note that the result is one-sided: the violation frequency may remain below α\alpha, rather than converging to it. Its finite-horizon usefulness depends on the constants. An unknown or large λ∗\lambda^* can make the bound loose.

Remark. The results concern empirical exceedances of the decoding surrogate’s threshold, not the paired attribute-change metric in Definition 1, recommendation utility, or population risk. They require neither offline conformal coverage nor exact Lagrangian optimization. No exchangeability assumption is needed, so the pathwise argument permits nonstationary or adversarial queries. The O(1/T)O(1/T) conclusion nevertheless requires uniform enforceability throughout the sequence, including after distribution shifts.

Experiments

Setup

Datasets and evaluation protocol. Following [14], we evaluate on MovieLens-1M [29] and Amazon Movies & TV [30], with gender as the protected attribute. We split users into disjoint calibration and test sets in a 70/30 ratio. Held-out items provide the ground truth for recommendation evaluation. Calibration users supply both the cross-group reference outputs and the scores used to select δ\delta. Test users are excluded from reference-bank construction and threshold selection.

Candidate-set implementation. Algorithm 2 presents the sampling-based formulation, in which the LLM proposes candidate items. The reported experiments use a simplified candidate-set implementation: recommendations are constructed from supplied candidates instead of next-item proposals sampled from the LLM. Fairness-adjusted item selection operates over these supplied candidates. The results therefore evaluate control within a candidate set, not the LLM proposal-generation step in Algorithm 2.

Decoding settings. We use beam width B=3B = 3 for Hard FAIDER. Unless otherwise noted, the neighborhood similarity threshold is τρ=0.95\tau_\rho= 0.95, the feedback step size is η=0.05\eta= 0.05, and the target violation rate is α=0.05\alpha= 0.05. The soft penalty is initialized through a warm-up on the calibration set before test evaluation. For CFR evaluation, we use decoding temperature zero when comparing recommendations under protected-attribute changes.

Baselines. We compare with three methods. Zero-Shot [2] uses the LLM without fairness control. UP5 [13] uses soft prompt tuning to address recommendation fairness. FACTOR [14] combines conformal thresholding with iterative prompt repair, providing a comparison with an intervention that operates through corrective prompting. We evaluate both FAIDER variants: hard disparity constraints and a soft penalty adjusted through dual-inspired feedback.

Models and embeddings. The recommendation backbones are Qwen3-8B [31] and Llama3.1-8B-Instruct [32]. For embedding-based comparisons, we use the SentenceTransformer model paraphrase-mpnet-base-v2 [33], fine-tuned on recommendation-specific data. Experiments run on NVIDIA A100 GPUs with 80 GB of memory.

Metrics. Recommendation quality is measured using Recall@10 and normalized discounted cumulative gain at rank 10 (NDCG@10). Recall@10 measures the fraction of held-out relevant items retrieved in the top ten recommendations. NDCG@10 additionally accounts for their ranking positions. We measure counterfactual disparity using CFR [14], the mean embedding shift between recommendation lists when a protected attribute changes while the user context remains fixed. Lower CFR indicates greater semantic stability under this attribute change. Higher Recall@10 and NDCG@10 and lower CFR are preferred.

Main Results

Table 1 compares recommendation quality and counterfactual disparity across the two datasets and two LLM backbones. We have the following observations.

DatasetMethodQwen3-8B Recall@10 ↑\uparrowQwen3-8B NDCG@10 ↑\uparrowQwen3-8B CFR ↓\downarrowLlama3.1-8B-Instruct Recall@10 ↑\uparrowLlama3.1-8B-Instruct NDCG@10 ↑\uparrowLlama3.1-8B-Instruct CFR ↓\downarrow
MovieLensZero-shot0.4020.4580.7420.3580.4020.701
MovieLensUP50.3810.4270.6130.3360.3740.556
MovieLensFACTER0.3890.4450.5910.3380.3820.440
MovieLensHard FAIDER0.3820.4320.3890.3330.3710.398
MovieLensSoft FAIDER0.3900.4460.4740.3520.3970.243
AmazonZero-shot0.3170.3510.8140.2960.3120.768
AmazonUP50.2940.3280.7210.2610.2890.672
AmazonFACTER0.3010.3390.6340.2770.2790.581
AmazonHard FAIDER0.2970.3260.5740.2660.2830.509
AmazonSoft FAIDER0.3180.3470.4910.2970.3060.411

Table 1. Recommendation quality and counterfactual disparity on MovieLens and Amazon. FAIDER results use the candidate-set implementation. The best and second-best reported values for each dataset and backbone are shown in bold and underlined, respectively.

Counterfactual fairness. Both variants have lower reported CFR than all three baselines in every dataset–backbone setting. Soft FAIDER achieves the lowest CFR in three of the four settings; Hard FAIDER achieves the lowest value on MovieLens with Qwen3-8B. Relative to FACTER, the soft variant reduces CFR by 19.8%–44.8%. The improvement across both backbones and datasets supports the usefulness of incorporating a fairness signal into item selection within the evaluated candidate-set setting.

Recommendation quality. Soft FAIDER has higher Recall@10 and NDCG@10 than FACTER in all four settings and the second-highest NDCG@10 among the compared methods in each setting. Its NDCG@10 remains close to Zero-Shot, while Recall@10 is lower on MovieLens and numerically similar on Amazon. The main benefit is therefore the joint improvement in measured fairness and ranking quality relative to prompt repair. Sometimes the soft FAIDER even achieves slightly better Recall than the naive baseline. This indicates that the LLM biasedly picks stereotypical items that are not what the user actually liked, and our method can help it.

Comparison between Soft and Hard variants. The soft variant has higher Recall@10 and NDCG@10 than the hard variant throughout, and improves all three reported metrics in three settings. On MovieLens with Qwen3-8B, hard filtering instead achieves lower CFR (0.389 versus 0.474), at the cost of lower ranking quality. Thus, stricter enforcement does not consistently minimize the evaluation metric. This is consistent with the distinction between the reference-based decoding surrogate in Equation (2) and the paired attribute-change criterion in Definition 1: enforcing the former does not directly optimize the latter.

Sensitivity Analysis

Figure 1 examines how neighborhood similarity and fixed penalty strength affect ranking quality and counterfactual disparity on MovieLens.

Sensitivity analysis plots for neighborhood threshold and fixed penalty strength

Figure 1. (a) Neighborhood threshold τρ\tau_{\rho}. (b) Fixed penalty strength λ\lambda. Sensitivity analysis on MovieLens. Each panel reports NDCG@10 and CFR as the indicated parameter varies; the right panel uses a constant penalty rather than online feedback adaptation.

Neighborhood similarity threshold τρ\tau_{\rho}. Increasing τρ\tau_{\rho} restricts the reference neighborhood to cross-group users with more similar contexts. Across the tested range from 0.80 to 0.95, CFR decreases from 0.598 to 0.474, while NDCG@10 decreases from 0.454 to 0.446. Stricter context matching is therefore associated with lower measured disparity and a modest ranking-quality cost in this experiment. Unlike the penalty weight, this parameter changes which cross-group comparisons define the control score. The observed trend does not imply that narrower neighborhoods are uniformly preferable.

Penalty strength λ\lambda. We further evaluate the soft decoder with a fixed penalty strength to isolate the effect of λ\lambda. As shown in the figure, increasing λ\lambda from 0 to 4 consistently reduces CFR, accompanied by a decrease in NDCG@10. In particular, CFR decreases substantially as λ\lambda increases, while the reduction in recommendation quality becomes more pronounced at larger penalty strengths. These results demonstrate a clear fairness–utility trade-off: stronger fairness penalties improve counterfactual fairness but may compromise ranking performance. Unlike the fixed-penalty variant, the dual-controlled variant dynamically adjusts λ\lambda in response to observed fairness violations to target a specified violation rate α\alpha.

Conclusion

We presented FAIDER, an item-level framework for fairness-aware LLM recommendation that separates a fixed empirical disparity threshold from the strength of decoding-time control. It supports hard filtering and adaptive soft penalties without retraining the base recommender. Our analysis connects empirical violation control to penalty growth, yielding a conditional guarantee under uniform enforceability. Experiments suggest that soft control generally offers a more favorable balance between semantic stability and ranking quality than hard filtering. This finding highlights the distinction between satisfying a reference-based constraint and achieving the fairness objective used in evaluation. The broader implication is that decoding-time fairness should be assessed through its joint effects on semantic stability and recommendation quality, rather than through constraint satisfaction alone.

References

  1. [1]Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. “Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)”. In: Proceedings of the 16th ACM Conference on Recommender Systems. 2022, pp. 299–315. DOI: 10.1145/3523227.3546767.DOI
  2. [2]Yupeng Hou et al. “Large language models are zero-shot rankers for recommender systems”. In: European Conference on Information Retrieval. Springer. 2024, pp. 364–381.arxiv.org/abs/2305.08845
  3. [3]Lei Li, Yongfeng Zhang, Dugang Liu, and Li Chen. “Large Language Models for Generative Recommendation: A Survey and Visionary Discussions”. In: Proceedings of LREC-COLING. 2024, pp. 10146–10159. URL: https://aclanthology.org/2024.lrec-main.886/.arxiv.org/abs/2309.01157
  4. [4]Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. “Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation”. In: Proceedings of the 17th ACM Conference on Recommender Systems. 2023, pp. 993–999.DOI
  5. [5]Song-Duo Ma, Chu Yun Chen, Bang-An Li, Pin-Yu Chen, Shau-Yung Hsu, and Yun-Nung Chen. “Rethinking Fairness in LLM-Based Recommender Systems: A Survey”. In: Findings of the Association for Computational Linguistics: EMNLP 2026. 2026, pp. 25277–25296. DOI: 10.18653/2026.findings-emnlp.1197.arxiv.org/abs/2606.28340
  6. [6]Ashudeep Singh and Thorsten Joachims. “Fairness of Exposure in Rankings”. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2018. URL: https://www.kdd.org/kdd2018/accepted-papers/view/fairness-of-exposure-in-rankings.DOI
  7. [7]Bjørnar Vassøy and Helge Langseth. “Consumer-Side Fairness in Recommender Systems: A Systematic Survey of Methods and Evaluation”. In: Artificial Intelligence Review 57.101 (2024). DOI: 10.1007/s10462-023-10663-5.DOI
  8. [8]Yashar Deldjoo, Dietmar Jannach, Alejandro Bellogin, Alessandro Difonzo, and Dario Zanzonelli. “Fairness in Recommender Systems: Research Landscape and Future Directions”. In: User Modeling and User-Adapted Interaction 34 (2024), pp. 59–108. DOI: 10.1007/s11257-023-09364-z.DOI
  9. [9]Nan Wang et al. “COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023, pp. 13258–13275. DOI: 10.18653/v1/2023.emnlp-main.819.DOI
  10. [10]Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. “Counterfactual Fairness”. In: Advances in Neural Information Processing Systems. 2017.arxiv.org/abs/1703.06856
  11. [11]Yashar Deldjoo and Tommaso Di Noia. “CFaiRLLM: Consumer Fairness Evaluation in Large-Language Model Recommender System”. In: ACM Transactions on Intelligent Systems and Technology 16.6 (2025), pp. 1–31. DOI: 10.1145/3725853.DOI
  12. [12]Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. “TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation”. In: Proceedings of the 17th ACM Conference on Recommender Systems. 2023, pp. 1007–1014. DOI: 10.1145/3604915.3608857.DOI
  13. [13]Wenyue Hua, Yingqiang Ge, Shuyuan Xu, Jianchao Ji, Zelong Li, and Yongfeng Zhang. “Up5: Unbiased foundation model for fairness-aware recommendation”. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, pp. 1899–1912.arxiv.org/abs/2305.12090
  14. [14]Arya Fayyazi, Mehdi Kamal, and Massoud Pedram. “FA CTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for Enabling Fair LLM-Based Recommender Systems”. In: Forty-second International Conference on Machine Learning. 2025.DOI
  15. [15]Aman Madaan et al. “Self-Refine: Iterative Refinement with Self-Feedback”. In: Advances in Neural Information Processing Systems. Vol. 36. 2023, pp. 46534–46594. DOI: 10.52202/075280–2019.arxiv.org/abs/2303.17651
  16. [16]Sumanth Datathri et al. “Plug and Play Language Models: A Simple Approach to Controlled Text Generation”. In: International Conference on Learning Representations. 2020.
  17. [17]Kevin Yang and Dan Klein. “FUDGE: Controlled text generation with future discriminators”. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021, pp. 3511–3535.
  18. [18]Alisa Liu et al. “DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts”. In: Proceedings of ACL-IJCNLP. 2021, pp. 6691–6706. DOI: 10.18653/v1/2021.acl-long.522.DOI
  19. [19]Anastasios N. Angelopoulos and Stephen Bates. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification”. In: Foundations and Trends in Machine Learning 16.4 (2023), pp. 494–591. DOI: 10.1561/2200000101.
  20. [20]Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. “Conformal Risk Control”. In: International Conference on Learning Representations. 2024. URL: https://proceedings.iclr.cc/paper_files/paper/2024/hash/f3549ef9b5ff520a7e41f3cc306ab2b-Abstract-Conference.html.
  21. [21]Victor Quach et al. “Conformal Language Modeling”. In: International Conference on Learning Representations. 2024. URL: https://proceedings.iclr.cc/paper_files/paper/2024/hash/31421b112e5f7faf4fc577b74e45dab2-Abstract-Conference.html.arxiv.org/abs/2306.10193
  22. [22]Isaac Gibbs and Emmanuel Candes. “Adaptive conformal inference under distribution shift”. In: Advances in Neural Information Processing Systems 34 (2021), pp. 1660–1672.arxiv.org/abs/2106.00170
  23. [23]Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. “Achieving Risk Control in Online Learning Settings”. In: Transactions on Machine Learning Research (2023). URL: https://openreview.net/forum?id=5Y04GWvoJu.arxiv.org/abs/2205.09095
  24. [24]Di Jin et al. “A Survey on Fairness-Aware Recommender Systems”. In: Information Fusion 100 (2023), p. 101906. DOI: 10.1016/j.inffus.2023.101906.DOI
  25. [25]Ben Krause et al. “GeDi: Generative Discriminator Guided Sequence Generation”. In: Findings of EMNLP. 2021, pp. 4929–4952. DOI: 10.18653/v1/2021.findings-emnlp.424.
  26. [26]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005. DOI: 10.1007/b106715.DOI
  27. [27]Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candes, and Aaditya Ramdas. “Conformal Prediction Under Covariate Shift”. In: Advances in Neural Information Processing Systems. Vol. 32. 2019. URL: https://proceedings.neurips.cc/paper/2019/hash/8fb21ee7a2207526da55a679f0332de2-Abstract.html.arxiv.org/abs/1904.06019
  28. [28]Anastasios Nikolas Angelopoulos, Rina Barber, and Stephen Bates. “Online Conformal Prediction with Decaying Step Sizes”. In: Proceedings of the 41st International Conference on Machine Learning. Vol. 235. PMLR. 2024, pp. 1616–1630. URL: https://proceedings.mlr.press/v235/angelopoulos24a.html.arxiv.org/abs/2402.01139
  29. [29]F Maxwell Harper and Joseph A Konstan. “The movielens datasets: History and context”. In: Acm transactions on interactive intelligent systems (tiis) 5.4 (2015), pp. 1–19.
  30. [30]Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. “Image-based recommendations on styles and substitutes”. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 2015, pp. 43–52.arxiv.org/abs/1506.04757
  31. [31]An Yang et al. “Qwen3 technical report”. In: arXiv preprint arXiv:2505.09388 (2025).arxiv.org/abs/2505.09388
  32. [32]Aaron Grattafiori et al. “The llama 3 herd of models”. In: arXiv preprint arXiv:2407.21783 (2024).arxiv.org/abs/2407.21783
  33. [33]Nils Reimers and Iryna Gurevych. “Sentence-bert: Sentence embeddings using siamese bert-networks”. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019, pp. 3982–3992.arxiv.org/abs/1908.10084

Paper details

vlnc:W_6sdxesfa
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents