betaFeedback

WorksComputer Science

TUGB: A Two-stage Uncertainty-aware Generative Auto-Bidding Framework

Abstract

Generative modeling, particularly Decision Transformers, shows promise in auto-bidding strategies. However, it struggles with limited exploration in static offline datasets, vulnerability to dynamic environmental shifts, and a reliance on heuristic return-to-go (RTG) specifications that may yield suboptimal performance on unseen traffic. To address these challenges, we propose the Two-stage Uncertainty-aware Generative Auto-Bidding (TUGB) framework. TUGB integrates generative sequence modeling with Reinforcement Learning (RL). It employs a two-stage pipeline where a pre-trained Generative Bidding Model is fine-tuned via an ensemble-based Uncertainty-aware Environment Model. This approach quantifies epistemic uncertainty to penalize high-risk trajectories and evolves the policy from RTG-conditioning to direct bidding optimization. Extensive experiments demonstrate that TUGB significantly outperforms SOTA baselines. Notably, full-scale deployment on JD Search Ad achieved a 2.32% increase in cost and 3.57% in target cost, validating its industrial efficacy, while its successful adaptation to subsequent mechanism upgrades and JD Food Delivery Ad further validates TUGB's superior generalization capability.

Introduction

In modern online advertising systems, such as search engines, social media platforms, and e-commerce platforms, auto-bidding has emerged as the primary tool for campaign management [1, 7, 10, 36]. It allows advertisers to go beyond manually managing bids for enormous ad impressions. Instead, advertisers simply define high-level business objectives (e.g., maximize conversions, target Return on Investment (tROI) or target a cost-per-acquisition (tCPA)) and set constraints like budget, cost per click, etc. For each ad request, the auto-bidding agent dynamically computes a bid by combining these objectives with real-time contextual signals and user characteristics. This bid is then submitted to the real-time auction, determining whether the ad is displayed and the price paid based on market competition. Therefore, auto-bidding serves as a critical bridge in the ad system: it translates an advertiser’s macroscopic business target into impression level, per-auction bidding actions, aiming to maximize the advertiser’s outcome within a complex, varying bidding environment.

Recently, generative modeling has emerged as a powerful paradigm for auto-bidding, with the Decision Transformer (DT) and its variants serving as representative frameworks [5, 12, 15, 21, 23]. These methods cast auto-bidding as a sequence generation task, where decision-making is conditioned on a Return-to-Go (RTG) and state trajectories. By specifying a target RTG, the model generates bidding actions based on learned historical patterns. However, the effectiveness of DT-based methods is often compromised by the non-stationary nature of real-world auction environments. First, current DT-based methods are trained on offline datasets collected via existing online policies, which inherently limits their exploration capabilities and prevents the model from reliably extrapolating to policies that outperform the data source [25, 29, 37]. Second, dynamic shifts in auction environments introduce epistemic uncertainty, as the model struggles to generalize to unseen market states. This uncertainty creates a critical risk: the agent may overestimate the value of these out-of-distribution (OOD) states, leading to aggressive bidding decisions that undermine campaign performance [13]. Last but not least, determining an optimal RTG is non-trivial in complex auctions; an improper RTG setting—especially on unseen traffic where the maximum potential return is unknown and can lead to suboptimal or even detrimental actions [3, 28]. While recent approaches like GAS and GAVE attempt to enhance exploration [12, 23], they still rely heavily on precise RTG settings and lack the uncertainty-awareness to mitigate overestimation in policy optimization.

To address these challenges, we propose a Two-stage Uncertainty-aware Generative Auto-Bidding Framework (TUGB). TUGB comprises two key components: a Generative Bidding Model (GBM) designed to model bidding distributions and drive decision-making, and an Uncertainty-aware Environment Model (UEM) that facilitates the optimization of the GBM while quantifying environmental uncertainty. Through an Uncertainty-aware Bidding Optimization fine-tuning stage, the framework empowers the agent to actively explore beyond static offline datasets while leveraging the UEM to quantify epistemic uncertainty and penalize high-risk trajectories. This design effectively evolves the bidding policy from heuristic Return-to-Go (RTG) conditioning to risk-aware optimal auto-bidding decision-maker, enabling dynamic adaptation without oracle dependencies. TUGB has been fully deployed on the JD Search Ad, achieving a 2.32% increase in cost and a 3.57% increase in target cost, validating its industrial efficacy. Furthermore, TUGB has been adapted to the subsequent ad mechanism upgrade and JD Food Delivery Ad. These results demonstrate that TUGB not only significantly boosts cost and target cost but also exhibits superior ability to adapt different bidding target and data sparse scenario.

The main contributions of this paper are summarized as follows:

  • TUGB Framework: We propose TUGB, a two-stage framework bridging generative modeling with uncertainty-aware reinforcement learning for auto-bidding. TUGB addresses three critical challenges in generative auto-bidding: the high financial risk of exploration, epistemic uncertainty arising from non-stationary market dynamics, and the reliance on heuristic RTG targets. By leveraging an Uncertainty-aware Environment Model (UEM) to explicitly quantify uncertainty and penalize high-risk trajectories, TUGB facilitates safe exploration beyond offline data, enabling the generative bidding model to transform from a passive imitator into a risk-aware optimal auto-bidding decision-maker.

  • Industrial Deployment and Generalization: TUGB has been fully deployed on JD.com’s search advertising system, achieving 2.32% in cost and 3.57% in target cost while maintaining strict constraint satisfaction. Beyond static deployment, the framework demonstrates robust cross-scenario transferability: (i) it rapidly adapts to system mechanism upgrades via reward function changing without architectural modifications; (ii) it generalizes to JD Food Delivery through few-shot fine-tuning, leveraging the pre-trained policy prior to overcome distributional shifts.

Preliminaries

In this section, we provide the foundational background necessary to understand our proposed framework. We formalize the auto-bidding task as a constrained optimization problem. We then briefly introduce generative decision-making methods, with a specific emphasis on their practical applications in auto-bidding scenario and their RL-integrated variants.

Auto-Bidding Problem Formulation

In automated bidding systems, advertisers typically establish long-term performance objectives—such as a target Cost Per Acquisition (tCPA) or target Return on Investment (ROI) subject to a specific budget constraint over a defined period. To achieve these goals, the bidding agent continuously interacts with the ad auction mechanism, making sequential bidding decisions for each arriving impression.

For example, we consider an advertiser aiming to deliver an ad campaign on JD.com (a leading e-commerce platform). The advertiser specifies a target cost per acquisition (tCPA) and allocates a total budget BB for the campaign. In an evaluation period, the campaign involves a sequence of NN requests. For each request ii, let xi∈{0,1}x_i \in\{0,1\} denote the binary allocation variable, where xi=1x_i = 1 indicates winning the impression at cost cic_i, and xi=0x_i = 0 otherwise. The final conversion derived from impression ii is denoted as viv_i. In the example, vi=1v_i = 1 if the request ii brings a deal else vi=0v_i = 0. Thus, the expected total number of conversions is given by ∑i=1Nxivi\sum_{i=1}^{N} x_i v_i. The advertiser’s optimization problem is formulated as follows:

max⁡x∑i=1Nxivis.t.∑i=1Nxici≤B∑i=1Nxici∑i=1Nxivi≤tCPA(1)\begin{aligned} \max_{\mathbf{x}} \quad& \sum_{i=1}^{N} x_i v_i \\ \text{s.t.} \quad& \sum_{i=1}^{N} x_i c_i \le B \\ & \frac{\sum_{i=1}^{N} x_i c_i}{\sum_{i=1}^{N} x_i v_i} \le tCPA \tag*{(1)} \end{aligned}

To solve this constrained optimization problem, the Lagrange duality method [1, 16, 36] is typically introduced to relax the global constraints. By constructing the dual problem and applying the complementary slackness conditions, the optimal allocation rule can be analyzed locally. Specifically, the condition for winning an impression (xi=1x_i = 1) is determined by comparing the incurred cost cic_i against a threshold derived from the value viv_i, weighted by the dual variables. In the context of a real-time bidding system (assuming a second-price auction mechanism [2, 9]), this threshold theoretically corresponds to the optimal bid b∗b^*. Previous studies [16] have derived the optimal bidding strategy for this constrained optimization problem. The closed-form optimal bid is given by:

b∗=λ0vi+tCPAλ1vi(2)b^* = \lambda_0 v_i + tCPA\lambda_1 v_i \tag*{(2)}

However, it is crucial to note that explicitly calculating this optimal bid requires precise knowledge of the optimal dual parameters λ1,λ2\lambda_1, \lambda_2 and stable estimation of viv_i. In real-world online advertising systems, the auction environment and traffic distribution vary rapidly, making static parameter estimation infeasible. Consequently, the auto-bidding problem is typically formulated as a sequential decision-making problem (e.g., a Markov Decision Process) to adaptively learn optimal policies in dynamic environments [16].

Formally, we define the bidding process as a tuple (S,A,P,R,C)(S,A,P,R,C). At each time step tt, the bidding agent observes a state st∈Ss_t \in S, which typically includes the current ad request features, remaining budget BtB_t, and remaining time. The bidding agent makes an action at∼πθ(⋅∣st)a_t \sim\pi_\theta(\cdot|s_t), representing the bid price btb_t. The environment returns a reward rtr_t (e.g., conversion value) and a cost ctc_t based on the auction dynamics PP. The optimization objective is to maximize the expected cumulative reward:

max⁡πθJ(πθ)=Eτ∼πθ[∑t=1Tr(st,at)](3)\max_{\pi_\theta}\quad J(\pi_\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=1}^{T}r(s_t,a_t)\right] \tag*{(3)}

where τ=(s1,a1,…,sT,aT)\tau= (s_1, a_1, \ldots, s_T, a_T) denotes the trajectory of the bidding episode. This formulation allows the agent to learn a policy πθ\pi_\theta that dynamically takes an action with maximal long-term reward without relying on static dual parameter assumptions. Previous work like USCB [16], MBAB [6] have demonstrate their capability in auto bidding scenario.

Generative Bidding

Generative decision making reformulates sequential control as conditional sequence modeling rather than explicit value estimation [24]. A representative approach is the Decision Transformer (DT) [5, 35], which models offline trajectories and predicts actions autoregressively conditioned on historical states, actions, and a target Return-to-Go (RTG):

π(at∣st,RTGt,s<t,a<t),RTGt=∑k=tTrk.\pi(a_t \mid s_t, \mathrm{RTG}_t, s_{<t}, a_{<t}), \qquad\mathrm{RTG}_t = \sum_{k=t}^{T} r_k.

This formulation is attractive for auto-bidding because it captures long-term dependencies in logged auction trajectories and avoids unstable value bootstrapping.

Recent generative bidding methods mainly follow two lines. Diffusion-based methods such as AIGB and CBD [15, 22] model state-action trajectories through denoising, while DT-based methods such as GAS and GAVE [12, 23] improve autoregressive bidding with mechanisms such as MCTS and value expansion. We focus on the DT-based line because of its efficient inference and direct fit to sequential bidding logs. However, existing DT-style bidders remain limited by offline imitation: they explore only within the support of logged policies, lack explicit epistemic uncertainty estimation for non-stationary markets [8, 20], and depend heavily on hand-specified RTG targets that are difficult to set on unseen traffic [3, 28].

Fine-tuning generative models with reinforcement learning has become an effective way to move beyond imitation, as shown by RLHF/RLVR-style post-training and GRPO [4, 14, 27, 30, 32, 34]. In auto-bidding, however, direct online exploration is financially risky and auction trajectories are more stochastic than natural language sequences. Thus, a practical generative bidding framework must combine RL-based policy improvement with uncertainty-aware safeguards against unreliable simulated or out-of-distribution trajectories.

Two-stage Uncertainty-aware Generative Auto-Bidding Framework

To overcome the limitations of static offline datasets and epistemic uncertainty in auto-bidding, we present Two-stage Uncertainty-aware Generative Bidding framework (TUGB). As shown in Figure 1, TUGB integrates sequence modeling with uncertainty-aware reinforcement learning in two stages. Stage 1 jointly pre-trains a Generative Bidding Model (GBM) as a behavioral prior and an ensemble-based Uncertainty-aware Environment Model (UEM) as a simulator of auction dynamics. Stage 2 fine-tunes the GBM on UEM, where ensemble disagreement penalizes unreliable trajectories and prevents the policy from exploiting out-of-distribution simulation errors. This design is intended to retain the benefits of supervised pre-training while supporting risk-aware policy improvement beyond the logged data.

Overview of the TUGB Framework

Figure 1. Overview of the TUGB Framework. The pipeline consists of two phases: (1) Offline Supervised Pre-training, where the Generative Bidding Model (GBM) learns a behavioral prior, and the Uncertainty-aware Environment Model (UEM) learns the auction dynamics from the real-world delivery logs; (2) Uncertainty-aware Bidding Optimization, where the agent interacts with the UEM to evolve beyond the offline dataset, guided by uncertainty-penalized rewards.

Stage 1: Offline Supervised Pre-training

In this part, we introduce how the generative bidding model (GBM) and uncertainty environment model (UEM) are pretrained. Stage 1 learns a behavioral prior and a risk-aware simulator from real-world delivery logs.

Generative Bidding Model

Unlike traditional reinforcement learning policies that map isolated states to actions, we treat the bidding process as a sequence modeling problem. We employ a Transformer-based architecture to capture the long-term dependencies within historical bidding trajectories τ=(s0,a0,…,sT,aT)\tau= (s_0, a_0, \ldots, s_T, a_T). Standard Decision Transformers (DT) condition the policy on a Return-to-Go (RTG) token to guide generation. However, explicit RTG conditioning in varying auction environments is difficult [3, 28], leading to suboptimal performance. Therefore, in TUGB, we alleviate the RTG dependency during the pre-training phase. The Transformer is trained via Behavior Cloning (BC) to minimize the negative log-likelihood of the expert actions given the historical context:

LBC(θ)=−Eτ∼D[∑t=1Tlog⁡πθ(at∣s≤t,a<t)](4)\mathcal{L}_{\mathrm{BC}}(\theta) = -\mathbb{E}_{\tau\sim\mathcal{D}}\left[\sum_{t=1}^{T}\log\pi_\theta(a_t|s_{\leq t},a_{<t})\right] \tag*{(4)}

where s≤ts_{\leq t} denotes the sequence of states up to time tt. This design allows the backbone to focus purely on learning the intrinsic transition dynamics and mimicking the diverse bidding behaviors present in the offline dataset, providing a generalized initialization for the subsequent fine-tuning stage.

Environment Model

Enhancing the decision-making capability of the Transformer necessitates a high-fidelity environment model, particularly in real-time bidding where exploring new actions entails significant financial risk. However, accurately modeling the auction landscape is complicated by the inherent epistemic uncertainty of the ad system [8, 20]. Epistemic uncertainty arises from the non-stationary nature of market dynamics—such as shifting competitor strategies or budget exhaustion, reflecting the model’s ignorance regarding Out-of-Distribution (OOD) states. To effectively address this dual challenge, we employ a Probabilistic Ensemble Environment Model to explicitly quantify the epistemic component, thereby enabling risk-aware policy updates that prevent the agent from exploiting simulation errors in unseen market conditions. In addition to uncertainty quantification, employing an ensemble architecture reduces the generalization error of the environment model, leading to more precise state predictions.

We employ an Ensemble model architecture to quantify the epistemic uncertainty. Specifically, we capture epistemic uncertainty by calculating the divergence among the different sub-models. Based on this formulation, the epistemic uncertainty is calculated as follows: Let {fθm}m=1M\{f_{\theta_m}\}_{m=1}^{M} denote the ensemble of MM probabilistic models. For a given input xx, the mm-th model outputs a Gaussian distribution parameterized by a mean μθm(x)\mu_{\theta_m}(x) and a variance σθm2(x)\sigma^2_{\theta_m}(x).

The epistemic uncertainty, reflecting the model’s lack of knowledge due to environmental dynamics, is quantified as the variance of the means across the ensemble:

σepistemic(x)=1M∑m=1M(μθm(x)−μˉ(x))2(5)\sigma_{\mathrm{epistemic}}(x) = \frac{1}{M}\sum_{m=1}^{M}\left(\mu_{\theta_m}(x)-\bar{\mu}(x)\right)^2 \tag*{(5)}

where μˉ(x)=1M∑m=1Mμθm(x)\bar{\mu}(x)=\frac{1}{M}\sum_{m=1}^{M}\mu_{\theta_m}(x) is the predictive mean of the ensemble.

In auto-bidding scenarios, state variables (such as remaining budget and current CPA) often exhibit high variance and non-linear jumps, making the prediction of absolute states st+1s_{t+1} numerically unstable. To mitigate this, we adopt a residual learning paradigm. The model predicts the state residual Δst=st+1−st\Delta s_t=s_{t+1}-s_t rather than the absolute next state. This approach simplifies the learning landscape by focusing on the local dynamics of change, thereby enhancing the model’s sensitivity to action perturbations and improving generalization in continuous spaces.

Stage 2: Uncertainty-aware Bidding Optimization

Building upon the behavioral prior and environment simulator established in the pre-training stage, we proceed to the fine-tuning phase. The objective is to transcend the limitations of supervised imitation and improve decision-making capabilities of the bidding agent.

Reinforcement Learning (RL) is widely adopted to refine the reasoning and planning abilities of generative models [4, 14, 27, 34]. Leveraging our uncertainty-aware environment model, we sample simulated delivery trajectories with varying actions. In the bidding scenario, the optimization objective is explicitly defined; thus, we can evaluate these trajectories using a rule-based reward formulation. For instance, in strict tCPA bidding, the reward R(τ)R(\tau) for a delivery trajectory τ\tau could be formalized as [33]:

R(τ)=min⁡(tCPAβCPAβ,1)∗∑τv(6)R(\tau)=\min\left(\frac{tCPA^{\beta}}{CPA^{\beta}},1\right)*\sum_{\tau}v \tag*{(6)}

where β\beta is a fixed parameter that take penalty on the tCPA constraint.

Regarding policy optimization, we employ Group Relative Policy Optimization (GRPO) [32], a state-of-the-art algorithm in the domain of RL with Verifiable Rewards (RLVR). Unlike standard PPO [31], GRPO computes advantages by comparing multiple outputs within a group rather than relying on an additional value network. This approach not only alleviates the burden of training a critic, but also enhances the precision of policy updates by using the group mean as a robust baseline.

To mitigate epistemic uncertainty arising from traffic distribution shifts, complex market dynamics, or imperfect modeling of delayed conversions, we employ Uncertainty-aware Environment Model (UEM) to quantify the uncertainty and integrate it as a penalty term directly into the reward signal. This approach alleviates the impact of simulator bias, acting as a critical safety constraint that prevents the policy from chasing unreliable rewards in under-explored regions. Specifically, the uncertainty-adjusted reward R^i\hat{R}_i and the group-relative advantage AiA_i for trajectory τi\tau_i are formulated as:

R^i=R(τi)−λ⋅σ(τi)(7)\hat{R}_i=R(\tau_i)-\lambda\cdot\sigma(\tau_i) \tag*{(7)}

Algorithm 1 Two-Stage Uncertainty-Aware Generative Auto-Bidding Framework

1: Stage 1: Offline Supervised Pre-Training 2: for step t1=1t_1 = 1 to NpreN_{\mathrm{pre}} do 3: Sample batch B∼DB \sim D 4: LBC(θ)=−E(τ,a)∼B[log⁡πθ(a∣τ)]L_{\mathrm{BC}}(\theta) = -\mathbb{E}_{(\tau,a)\sim B}[\log\pi_\theta(a \mid\tau)] 5: θ←θ−η1∇θLBC\theta\leftarrow\theta- \eta_1\nabla_\theta L_{\mathrm{BC}} 6: for m=1m = 1 to MM do 7: LModel(ϕm)=E(τ,a,r,s′)∼B[∥s′−s^ϕm′(τ,a)∥2]L_{\mathrm{Model}}(\phi_m) = \mathbb{E}_{(\tau,a,r,s')\sim B}[\lVert s' - \hat{s}'_{\phi_m}(\tau,a)\rVert^2] 8: ϕm←ϕm−η1∇ϕmLModel\phi_m \leftarrow\phi_m - \eta_1\nabla_{\phi_m}L_{\mathrm{Model}} 9: end for 10: end for 11: Stage 2: Uncertainty-Aware Bidding Optimization 12: Initialize πref←πθ\pi_{\mathrm{ref}} \leftarrow\pi_\theta 13: for step t=1t = 1 to NRLN_{\mathrm{RL}} do 14: Sample batch of states S={τ}∼DS = \{\tau\} \sim D 15: for each state τ∈S\tau\in S do 16: Sample GG actions {ai}i=1G∼πθ(⋅∣τ)\{a_i\}_{i=1}^{G} \sim\pi_\theta(\cdot\mid\tau) 17: Generate trajectories τi\tau_i using the ensemble 18: for i=1i = 1 to GG do 19: Compute RimeanR_i^{\mathrm{mean}} and σi\sigma_i 20: R^i←Rimean−λσi\hat{R}_i \leftarrow R_i^{\mathrm{mean}} - \lambda\sigma_i 21: end for 22: Compute group statistics μR^\mu_{\hat{R}} and σR^\sigma_{\hat{R}} 23: Ai←(R^i−μR^)/(σR^+ϵ)A_i \leftarrow(\hat{R}_i - \mu_{\hat{R}})/(\sigma_{\hat{R}} + \epsilon) 24: end for 25: Compute loss L(θ)L(\theta) via GRPO, Eq. (9) 26: θ←θ+η∇θL(θ)\theta\leftarrow\theta+ \eta\nabla_\theta L(\theta) 27: end for 28: return πθ∗\pi_\theta^*

Ai=R^i−mean⁡({R^1,…,R^G})std⁡({R^1,…,R^G})+ϵ(8)A_i = \frac{\hat{R}_i-\operatorname{mean}(\{\hat{R}_1,\ldots,\hat{R}_G\})}{\operatorname{std}(\{\hat{R}_1,\ldots,\hat{R}_G\})+\epsilon} \tag*{(8)}

where σ(τi)\sigma(\tau_i) represents the epistemic uncertainty estimated by the ensemble environment model (e.g., variance of predicted returns), and λ\lambda is the penalty coefficient. The final objective function is defined as:

J(θ)=Es∼D[1G∑i=1G(min⁡(ρiAi,clip⁡(ρi,1−ϵ,1+ϵ)Ai)−βKKLDKL(πθ∥πref))](9)J(\theta) = \mathbb{E}_{s\sim D}\left[\frac{1}{G}\sum_{i=1}^{G}\left(\min\left(\rho_i A_i,\operatorname{clip}(\rho_i,1-\epsilon,1+\epsilon)A_i\right)-\beta K_{\mathrm{KL}}D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}})\right)\right] \tag*{(9)}

where ρi=πθ(τi∣s)πθold(τi∣s)\rho_i = \frac{\pi_\theta(\tau_i\mid s)}{\pi_{\theta_{\mathrm{old}}}(\tau_i\mid s)} is the probability ratio, and the KL term prevents the policy from deviating excessively from the reference model πref\pi_{\mathrm{ref}}. The pseudocode of total training pipeline of TUGB is shown in Algo 1.

It is noteworthy that in real-world production deployments, only the Generative Bidding Policy Model is required for inference. The uncertainty-aware environment model, while critical during the fine-tuning stage, does not participate in the online execution flow, thereby maintaining high inference efficiency and minimizing computational overhead.

Experiments

To validate the effectiveness and deployment viability of the TUGB framework, we conducted extensive evaluations involving both online production A/B tests within JD.com, a leading e-commerce advertising platform and large-scale offline benchmarks. Our experiments are designed to answer the following key research questions: RQ1: Does TUGB deliver substantial improvements in core business metrics compared to state-of-the-art industrial baselines in real-world online bidding scenarios? RQ2: Can TUGB maintain superior performance and stability across diverse marketing campaigns, fluctuating traffic patterns, and unseen competition dynamics? RQ3: How do the key components specifically the uncertainty aware environment model and the RTG-free mechanism contribute to the model’s auto-bidding performance?

Online Deployment

To empirically substantiate the effectiveness of our approach in real-world industrial production settings, we integrated the framework into large-scale real-world bidding system of JD.com, one of the world’s largest e-commerce platforms. The evaluation was conducted across two different scenarios:

  • JD Search Ad: The platform’s search-ad system. Here, the objective is typically constrained optimization, where advertisers define a target CPA (tCPA) or target ROI (tROI) to maximize conversions.

  • JD Food Delivery: A local services scenario characterized by high temporal dynamics (lunch/dinner peaks) and significantly sparser data compared to retail e-commerce.

In both scenarios, the bidding agent must maximize conversion volume while strictly adhering to the advertiser-specified constraints (e.g., Realized CPA ≤\leq tCPA).

RQ1: Performance on Large-Scale Ad System (JD Search Ad).

We constructed a large-scale offline dataset derived from online ad delivery logs from JD Search Ad. This dataset comprises over 1 million bidding trajectories and tens of millions of impression opportunities. We separate each delivery session into T=24T = 24 time steps. For each step tt, the state sts_t encompasses multi-dimensional features: (1) Resource Status: Remaining budget, Remaining time; (2) Market Feedback: Realized CPA, Realized CPC, Cost speed, Conversions; (3) Advertiser Context: tCPA, Product Category, Advertiser type; the action space ata_t is continuous, defined as a scalar bid ratio a∈[0,αmax⁡]a \in[0,\alpha_{\max}].

The training pipeline follows the two-stage paradigm described in § 3. The Decision Transformer backbone utilizes a context length of K=24K = 24. We employ the AdamW optimizer with a warm-up strategy. The ensemble consists of M=8M = 8 independent Transformer sub-models to capture epistemic uncertainty. The reward function RtR_t is designed to balance conversion maximization with constraint satisfaction.

Rt=(1−abs⁡(1−Real_CPAtCPA))∗conv(10)R_t = (1-\operatorname{abs}(1-\frac{\mathrm{Real\_CPA}}{\mathrm{tCPA}}))*\mathrm{conv} \tag*{(10)}

As for the parameter choice in Stage 2, the uncertainty penalty coefficient is set as 11 and the GRPO group size is set 1616.

We conducted a strictly controlled A/B test against USCB [16], the current production baseline, over a 7-day observation period.

Traffic was randomly split by delivery campaign ID buckets to ensure fairness. As shown in Table 2, TUGB improves the primary business metric Cost by +2.32% and Target Cost by +3.57%, while simultaneously improving the constraint satisfaction rate (CSR) by +0.55%. This simultaneous improvement in both order volume and CSR indicates that TUGB does not simply "spend more" to get conversions; rather, it intelligently reallocates budget from low-efficiency impressions to high-value opportunities, validating the efficacy of the learned value-driven policy.

ScenarioCostTarget Cost
JD Search Ad+2.32%<br>p=0.01p = 0.01<br>CI=[+0.56,+4.08]+3.57%<br>p=0.04p = 0.04<br>CI=[-0.18,+5.96]
Mechanism Update+2.26%<br>p=0.03p = 0.03<br>CI=[+0.22,+4.30]+2.24%<br>p=0.01p = 0.01<br>CI=[+0.54,+3.94]
JD Food Delivery+1.35%<br>p=0.03p = 0.03<br>CI=[+0.13,+2.57]–

Table 2. Summary of online deployment result.

To further examine the stability of TUGB, we report segment-level results across budget regimes and major product categories. Table 1 shows that TUGB delivers consistent improvements across all budget regimes. The gains are most pronounced in the mid-budget segment, with +3.55% in Cost and +3.22% in Target Cost, suggesting that TUGB is particularly effective for advertisers with moderate budgets.

TypeSegmentCostTarget Cost
Budget0–1000+1.84%+4.46%
Budget1000–10000+3.55%+3.22%
Budget10000++0.62%+0.66%

Table 1. Segment-level analysis on JD Search Ad.

Robustness and Generalization Capabilities

4.1.2 RQ2: Robustness and Generalization Capabilities. A critical requirement for industrial auto-bidding systems is the ability to adapt to non-stationary auction environments and transfer to data-sparse domains. We evaluate TUGB in two representative cases.

Case 1: Rapid Adaptation to Mechanism Iterations. Since we have deployed TUGB in online system, the JD Search Ad system introduced a bank-account-based pricing mechanism [26], which shifts the bidding objective toward optimizing future conversions. To adapt TUGB to this mechanism update, we simply modified the reward function to Rτ=(1−abs(1−RealCPAfuturetCPAfuture))∗convfutureR_{\tau} = \left(1 - abs\left(1 - \frac{RealCPA_{future}}{tCPA_{future}}\right)\right) * conv_{future} in the fine-tuning stage to align with the new bidding target, while reusing the pre-trained GBM and UEM from Stage 1. Without changing the model architecture or re-collecting a large-scale pre-training dataset, TUGB achieved +2.26% in Cost and +2.24% in Target Cost under the new mechanism, as shown in Table 2. This result demonstrates the practical adaptability of TUGB under mechanism changes.

Case 2: Cross-Domain Transfer to Data-Sparse Scenarios (JD Food Delivery). We further evaluated TUGB on JD Food Delivery. In this scenario, we reused the GBM pre-trained on JD Search Ad and trained a new UEM using JD Food Delivery data. This setting is more data-sparse: JD Food Delivery contains only about 100K trajectories, roughly 10 times fewer than the trajectories available in JD Search Ad. Despite the much smaller amount of target-domain data, TUGB achieves a +1.35% Cost improvement on JD Food Delivery, demonstrating the same efficiency of the pre-trained GBM prior when adapting to data-sparse domains. Notably, we initially neglected a domain-specific time-sensitivity feature, namely delivery time windows. This caused the transferred model to make suboptimal budget pacing decisions. After incorporating delivery time as a critical feature in the UEM, performance improved substantially.

This experience reveals a practical lesson: domain-specific features absent from the source domain must be explicitly identified and incorporated during transfer.

Overall, Table 2 answers RQ1 by confirming significant gains in the main JD Search Ad deployment and answers RQ2 by showing that TUGB remains effective under both a mechanism update and a data-sparse transfer scenario.

Offline Evaluation

The online deployment results above primarily answer RQ1 and RQ2 by measuring business impact and robustness in production. The goal of the offline evaluation is complementary: it provides a reproducible public-benchmark comparison under controlled budget constraints and, more importantly, isolates the contribution of the main TUGB components for RQ3.

Evaluation Setup

4.2.1 Evaluation Setup. To ensure reproducibility and facilitate fair comparison, we utilize AuctionNet [33], a large-scale public benchmark released by Alibaba that faithfully simulates the complexity of real-world ad auctions. The environment simulates a highly competitive market with 48 advertisers competing for impression opportunities across 21 delivery periods. Each period consists of 48 time steps involving approximately 500,000 impression requests. Our study utilizes two specific subsets: (1) AuctionNet-Standard, which contains comprehensive bidding trajectories representing typical market conditions; and (2) AuctionNet-Sparse, a challenging variant characterized by significantly reduced conversion rates (CVR), designed to test model robustness in data-sparse scenarios.

We benchmark TUGB against representative baselines from four categories. First, we include offline RL and production bidding methods, including IQL [18], CQL [19], BCQ [11], and USCB [16], where USCB is a constrained bidding algorithm deployed in our production system. Second, we compare with model-based RL methods, including MBPO [17] and MOPO [38], to evaluate whether TUGB brings benefits beyond standard model-based policy optimization and uncertainty-penalized rollouts. Third, we include recent generative bidding methods, including DiffBid [15], vanilla Decision Transformer [5], GAS [23], and GAVE [12]. Finally, we conduct ablation studies with TUGB-pretrain, TUGB-PPO, TUGB-PG, and TUGB-w/o-U to isolate the effects of supervised generative pre-training, GRPO-style optimization, vanilla policy-gradient optimization, and uncertainty-aware ensemble modeling.

DatasetBudgetUSCBCQLIQLBCQDiffBidDTGASGAVEMOPOMBPOTUGB-pretrainTUGB-PPOTUGB-PGTUGB-w/o-UTUGBImprove
AuctionNet50%861131641905419119320186671861911931952041.49%
75%135139232259100265287296128962592672812743022.03%
100%1571712813211523293593761691263373483673683801.06%
125%2202013553791933964094212151564034114144124220.24%
150%2812384014292344504614672561914674704784744833.43%
AuctionNet-Sparse50%11.512.816.517.79.914.818.419.612.49.716.918.319.918.820.44.08%
75%14.916.722.124.615.422.927.528.316.714.624.725.326.125.428.61.06%
100%17.522.230.031.119.529.636.137.225.615.831.032.636.134.337.40.54%
125%26.728.637.134.225.334.340.042.729.324.638.440.642.842.344.03.04%
150%31.335.843.137.930.844.546.547.436.431.642.542.347.644.748.21.69%

Table 3. Performance comparison against the sota baselines on AuctionNet and AuctionNet-Sparse under varying budget.

We adopt score (RR) defined in AuctionNet [33] as the decisive metric to evaluate bidding performance. All experiments are conducted on NVIDIA P40 GPUs using a fixed batch size of 200. The model parameters are optimized through AdamW with a learning rate of 1×10−51 \times10^{-5}. To ensure a fair comparison, all Transformer-based variants (DT, GAVE, and TUGB) share a unified backbone architecture consisting of 3 transformer layers with 16 attention heads.

RQ3: Component Contribution Analysis

Table 3 answers RQ3 by comparing the full TUGB model with both external baselines and internal variants. TUGB achieves the best reward under every budget setting on both AuctionNet and AuctionNet-Sparse, improving over the strongest non-TUGB baseline by up to 3.43% and 4.08%, respectively. The ablation results further identify the contribution of each core component.

First, the uncertainty-aware environment model is essential for stable policy improvement. TUGB-w/o-U consistently underperforms the full model, especially on AuctionNet-Sparse, indicating that a single environment model is more vulnerable to value overestimation under sparse feedback. The ensemble UEM mitigates this issue by penalizing uncertain rollouts and enabling safer exploration. This conclusion is also supported by an online ablation study in JD Search Ad. Replacing the ensemble UEM with a single environment model yields +1.32% Cost and +2.09% Target Cost, compared with +2.32% and +3.57% from the full TUGB model. The gap corresponds to approximately +1.00% in Cost and +1.48% in Target Cost, confirming that the ensemble-based uncertainty mechanism contributes additional production gains by reducing value overestimation and improving policy optimization robustness.

Second, the RTG-free design provides a stronger starting point than directly applying a standard Decision Transformer. TUGB-pretrain outperforms DT in most budget settings, suggesting that fixed RTG conditioning is unreliable in stochastic auto-bidding. However, TUGB further improves over TUGB-pretrain, showing that RTG-free pretraining and uncertainty-aware policy optimization are complementary.

We further examine whether the above conclusions are sensitive to key hyper-parameters. As shown in Table 4, TUGB remains stable around the deployment configuration: a moderate uncertainty penalty performs best, while larger MM and GG improve reward with diminishing returns and higher rollout cost. We therefore use λ=1\lambda= 1, M=8M = 8, and G=16G = 16 in deployment.

Param.Value / RR
λ\lambda0/3680.5/3761†/3802/344
MM2/3564/3668†/38016/384
GG2/3644/3688/37316†/380

Table 4. Sensitivity on AuctionNet with 100% budget. Entries denote parameter value/RR; † marks the default.

Conclusion and Future Work

We presented TUGB, which combines RTG-free generative pre-training with uncertainty-aware policy optimization. Production deployment and AuctionNet experiments demonstrate improved performance, robustness, and cross-domain transfer. Future work will extend TUGB to multi-objective bidding and more adaptive market dynamics.

GenAI Usage Disclosure

The authors used generative AI tools to assist with language polishing, conciseness editing, and formatting suggestions. All technical ideas, experimental design, results, analysis, and conclusions were created, verified, and approved by the authors.

References

References

  1. [1]Gagan Aggarwal, Ashwinkumar Badanidiyuru, Santiago R Balseiro, Kshipra Bhawalkar, Yuan Deng, Zhe Feng, Gagan Goel, Christopher Liaw, Haihao Lu, Mohammad Mahdian, et al. 2024. Auto-bidding and auctions in online advertising: A survey. ACM SIGecom Exchanges 22, 1 (2024), 159–183.
  2. [2]Gagan Aggarwal, Shan Muthukrishnan, David Pál, and Martin Pál. 2009. General auction mechanism for search advertising. In Proceedings of international conference on World wide web. 241–250.
  3. [3]David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. 2022. When does return-conditioned supervised learning work for offline reinforcement learning? Proceedings of Advances in Neural Information Processing Systems 35 (2022), 1542–1553.
  4. [4]Ali Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. 2025. MiniMax-MI: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv preprint arXiv:2506.13585 (2025).
  5. [5]Lily Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. Proceedings of Advances in neural information processing systems 34 (2021), 15084–15097.
  6. [6]Shuang Chen, Qixin Xu, Liang Zhang, Yongbo Jin, Wenhao Li, and Linjiao Mo. 2023. Model-based reinforcement learning for auto-bidding in display advertising. In Proceedings of International Conference on Autonomous Agents and Multiagent Systems. 1560–1568.
  7. [7]Ye Chen, Pavel Berkhin, Bo Anderson, and Nikhil R Devanur. 2011. Real-time bidding algorithms for performance-based display ad allocation. In Proceedings of ACM SIGKDD international conference on Knowledge discovery and data mining. 1307–1315.
  8. [8]Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. 2018. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In Proceedings of International conference on machine learning. PMLR, 1184–1193.
  9. [9]Benjamin Edelman, Michael Ostrovsky, and Michael Schwarz. 2007. Internet advertising and the generalized second-price auction: Selling billions of dollars worth of keywords. American economic review 97, 1 (2007), 242–259.
  10. [10]David S Evans. 2009. The online advertising industry: Economics, evolution, and privacy. Journal of economic perspectives 23, 3 (2009), 37–60.DOI
  11. [11]Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep reinforcement learning without exploration. In Proceedings of International conference on machine learning. PMLR, 2052–2062.
  12. [12]Jingtong Gao, Yewen Li, Shuai Mao, Peng Jiang, Nan Jiang, Yejing Wang, Qingpeng Cai, Fei Pan, Peng Jiang, Kun Gai, et al. 2025. Generative auto-bidding with value-guided explorations. In Proceedings of International ACM SIGIR Conference on Research and Development in Information Retrieval. 244–254.
  13. [13]Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. 2022. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Proceedings of Advances in Neural Information Processing Systems 35 (2022), 18267–18281.
  14. [14]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 8081 (2025), 633–638.
  15. [15]Jiayan Guo, Yusen Huo, Zhilin Zhang, Tianyu Wang, Chuan Yu, Jian Xu, Bo Zheng, and Yan Zhang. 2025. Generative auto-bidding with conditional diffusion modeling. In Proceedings of ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5038–5049.
  16. [16]Yue He, Xiujun Chen, Di Wu, Junwei Pan, Qing Tan, Chuan Yu, Jian Xu, and Xiaoqiang Zhu. 2021. A Unified Solution to Constrained Bidding in Online Display Advertising. In Proceedings of ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2993–3001.
  17. [17]Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. 2019. When to trust your model: Model-based policy optimization. Proceedings of Advances in neural information processing systems 32 (2019).
  18. [18]Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2022. Offline Reinforcement Learning with Implicit Q-Learning. In Proceedings of International Conference on Learning Representations.
  19. [19]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative q-learning for offline reinforcement learning. Proceedings of Advances in neural information processing systems 33 (2020), 1179–1191.arxiv.org/abs/2006.04779
  20. [20]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Proceedings of Advances in neural information processing systems 30 (2017).
  21. [21]Kaiyuan Li, Pengyu Wang, Yunshan Peng, Pengjia Yuan, Yanxiang Zeng, Rui Xiang, Yanhua Cheng, Xiaolong Liu, and Peng Jiang. 2025. EBaReT: Expert-guided Bag Reward Transformer for Auto Bidding. In Companion Proceedings of ACM on Web Conference. 1104–1108.
  22. [22]Yewen Li, Jingtong Gao, Nan Jiang, Shuai Mao, Ruyi An, Fei Pan, Xiangyu Zhao, Bo An, Qingpeng Cai, and Peng Jiang. 2025. Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Aligner. arXiv preprint arXiv:2509.03348 (2025).
  23. [23]Yewen Li, Shuai Mao, Jingtong Gao, Nan Jiang, Yunjian Xu, Qingpeng Cai, Fei Pan, Peng Jiang, and Bo An. 2025. Generative Auto-bidding with Post-training Search. In Companion Proceedings of ACM on Web Conference. 315–324.
  24. [24]Yinchuan Li, Xinyu Shao, Jianping Zhang, Haowei Wang, Leo Maxime Brunswick, Kaiwen Zhou, Jiqian Dong, Kaiyang Guo, Xiu Li, Zhitang Chen, et al. 2025. Generative models in decision making: A survey. arXiv preprint arXiv:2502.17100 (2025).
  25. [25]Ishita Mediratta, Qingfei You, Minqi Jiang, and Roberta Raileanu. 2024. The Generalization Gap in Offline Reinforcement Learning. In Proceedings of International Conference on Learning Representations.
  26. [26]Vahab S. Mirrokni, Renato Paes Leme, Pingzhong Tang, and Song Zuo. 2016. Dynamic Auctions with Bank Accounts. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. 387–393.
  27. [27]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Proceedings of Advances in neural information processing systems 35 (2022), 27730–27744.
  28. [28]Keiran Paster, Sheila McIlraith, and Jimmy Ba. 2022. You can’t count on luck: Why decision transformers and rvs fail in stochastic environments. Proceedings of Advances in neural information processing systems 35 (2022), 38966–38979.arxiv.org/abs/2205.15967
  29. [29]Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. 2023. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems 35, 8 (2023), 10237–10257.
  30. [30]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Proceedings of Advances in neural information processing systems 36 (2023), 53728–53741.arxiv.org/abs/2305.18290
  31. [31]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).arxiv.org/abs/1707.06347
  32. [32]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024).
  33. [33]Kefan Sun, Yusen Huo, Zhilin Zhang, Shuai Dou, Chuan Yu, Jian Xu, Zongqing Lu, and Bo Zheng. 2024. Auctionnet: A novel benchmark for decision-making in large-scale games. Proceedings of Advances in Neural Information Processing Systems 37 (2024), 94428–94452.
  34. [34]Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 (2025).
  35. [35]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Proceedings of Advances in neural information processing systems 30 (2017).
  36. [36]Jun Wang and Shuai Yuan. 2015. Real-time bidding: A new frontier of computational advertising research. In Proceedings of ACM International Conference on Web Search and Data Mining. 415–416.
  37. [37]Xiyao Wang, Wichayaporn Wongkamjan, Ruonan Jia, and Furong Huang. 2023. Live in the moment: Learning dynamics model adapted to evolving policy. In Proceedings of International Conference on Machine Learning. PMLR, 36470–36493.
  38. [38]Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. 2020. Mopo: Model-based offline policy optimization. Proceedings of Advances in Neural Information Processing Systems 33 (2020), 14129–14142.

Paper details

vlnc:W_cuq7zerd
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents