betaFeedback

WorksComputer Science

Reinforcement Learning for Controller Parameters in the Auto-Bidding System at Pinterest

Abstract

Auto-bidding systems in online advertising must balance competing objectives-maximizing budget delivery while minimizing cost per action-across heterogeneous campaigns under non-stationary market conditions. This trade-off evolves over campaign lifespans: early on, cost optimization dominates when budgets roll over; near deadlines, budget delivery becomes urgent. While PID controllers are widely used in production for their stability and interpretability, fixed tunings cannot adapt to diverse campaigns and shifting traffic patterns. We present RLBP (Reinforcement Learning for Bidding Parameters), a hybrid architecture deployed at Pinterest that uses RL as a meta-controller to select campaign-specific PID parameters daily while retaining PID for high-frequency bid adjustments. RLBP trains a multi-head Q-network using a relative-lift reward formulation normalized against the production baseline, and performs inference-time fusion that adapts the cost-delivery trade-off to campaign states without retraining. Through online A/B testing at Pinterest, RLBP achieves a statistically significant reduction in effective cost-per-click (eCPC) with no revenue degradation, and two months of post-rollout measurements against a holdout also show consistent cost reductions. These results demonstrate that meta-controller RL can successfully augment production control systems at scale.

Introduction

Pinterest is a visual discovery platform that serves approximately 619 million monthly active users and generated $4.2 billion in advertising revenue in 2025 [19]. Like every large advertising platform, its ads delivery system must solve an auto-bidding problem on behalf of each advertiser: given a budget, a lifetime schedule, and cost-efficiency targets such as cost-per-click (CPC) or return-on-spend (ROS), decide how much to bid for each user-level impression so that the campaign delivers its budget smoothly while satisfying its KPI constraints [2, 4]. Because these decisions are taken billions of times per day, even fractional improvements to the bidding policy translate into substantial advertiser value and platform revenue.

A long line of work formulates auto-bidding as a budget- and KPI-constrained optimization problem, whose optimal solution, under standard assumptions and primal–dual arguments, reduces to bidding a scalar multiple of the predicted impression value [2, 4, 7, 10, 24, 27]. The operational question is then how to update that scalar online, under non-stationary traffic and in the face of daily variation in user behavior, advertiser entry, and competitive pressure. In production, this online update is often implemented using a proportional–integral–derivative (PID) feedback controller that tracks the running error between realized performance and a pacing target and adjusts a bid multiplier to close the gap [1, 13, 20, 27, 28]. PID controllers are simple, interpretable, and widely used, which is why they remain the backbone of industrial pacing systems at LinkedIn, Yahoo, Alibaba, and elsewhere. Their weakness, however, is rigidity: a single fixed setup cannot be simultaneously appropriate for hundreds of thousands of heterogeneous campaigns and shifting market conditions.

Reinforcement learning (RL) is a natural way to replace such rules with a policy that adapts to campaign states. A large body of work formulates real-time bidding as a Markov decision process and trains a deep RL agent to emit bids directly [6, 9, 10, 12, 23, 24]. In production, however, a black-box policy that outputs bids using advertisers’ budgets faces two stubborn obstacles. First, *safety and explainability: neural policies can fail catastrophically (severe under- or over-delivery, cost performance degradation), and such failures are hard to diagnose and remediate immediately and can lead to unrecoverable revenue impact and terrible advertiser experience [14]. Second, off-policy learning: because of safety concerns, online exploration with random bids is often unacceptable, so policies are constrained to be trained from logged data, exposing the agent to the full difficulty of offline RL [15].

We present RLBP (Reinforcement Learning for Bidding Parameters), the auto-bidding pacing system deployed and evaluated with live traffic at Pinterest. The central design choice is to not use RL to produce bids directly. Instead, we retain the existing PID controller and use RL as a meta-controller that configures the controller. Specifically, at the beginning of each day and for each campaign, the RL agent observes the campaign state, produces a set of controller parameter multipliers and materializes these decisions as a lightweight lookup table for serving. This table is fixed for the next 24 hours. During online serving, the PID controller consumes the same table and continues to perform fine-grained bid adjustment at the serving cadence. We deliberately restrict the RL agent to operate on a small, bounded set of controller multipliers. This confines its decisions to modest adjustments around a validated production configuration, rather than allowing it to generate bids directly or select unconstrained controller settings. Beyond safety, this design is also lightweight in infrastructure cost, since only a flat lookup table—not a model artifact—is loaded at serving time. Related designs include Korenkevych et al. [14], who use offline RL to optimize the parameters of an existing base policy at Meta, and Yang et al. [26], who apply deep RL to tune the ranking utility function at Pinterest. RLBP shares the philosophy of augmenting an existing production bidder with a learned layer.

Our main contributions are listed below:

  • A hybrid RL–PID architecture for production auto-bidding. We learn campaign-specific PID parameter multipliers while retaining high-frequency feedback control. No model inference at serving time is required.

  • Multi-objective balancing without retraining. We train a multi-head QQ-network for separate goals, budget utilization and cost efficiency, and then fuse the values at inference time for policy induction. This design adapts the delivery–cost trade-off dynamically based on campaign states.

  • Relative-lift rewards for clearer learning guidance. Instead of using absolute values of the objective metrics (e.g. revenue or cost in dollars), we define rewards as the metric lift against the production baseline. This transformation not only normalizes different metrics to a comparable scale but also reshapes the reward distribution and provides a clearer learning signal for the agent to improve policy performance.

  • End-to-end production deployment and empirical validation at Pinterest. We design an exploration–training–serving pipeline that integrates seamlessly into Pinterest’s production bidding system with negligible infrastructure overhead. The online A/B test shows a statistically significant 0.25% eCPC reduction with no revenue degradation. The post-rollout measurements against holdout traffic demonstrate consistent cost reduction and click-through rate improvement.

The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 describes the formulation of the meta-controller MDP. Section 4 details the central techniques including the multi-head DQN, reward shaping, and adaptive fusion. Section 5 covers production integration, and Section 6 presents offline and online experiments, followed by concluding remarks.

Related Work

Auto-Bidding and Budget Pacing

Auto-bidding is the problem of bidding in real-time auctions on behalf of advertisers to maximize campaign value under budget and KPI constraints [2, 4, 17]. Classical analyses reduce the optimal bid to a scalar multiple of the predicted impression value, so the operational task reduces to online estimation of that scalar [10, 24, 27]. Production systems realize this in two ways. The feedback-control lineage drives the running spend error to a pacing target via online feedback—from probabilistic throttling on a participation probability at LinkedIn [1], to PI-style multiplicative bid control in Yahoo’s Mystique [20], Alibaba’s multivariable controller [27], and the RTB work of Zhang et al. [28]; Karlsson [13] surveys this line. The optimization lineage instead solves a parametric bid-shading program online via dual-based algorithms, such as those of Balseiro et al. [4] and Chen et al. [7]. Xu et al. [25] take a complementary approach that allocates delivery rate across traffic segments. Our work complements these pacing mechanisms by learning campaign-specific controller settings and adapting the delivery–cost trade-off over campaign lifecycles.

Reinforcement Learning for Bidding

Starting with Cai et al. [6]’s MDP formulation of real-time bidding, a substantial line of work has applied deep RL to produce bids directly. Wu et al. [24] introduce reward shaping tailored for budget-constrained bidding; Jin et al. [12] and Wen et al. [23] address the multi-agent setting; He et al. [10] unify constrained bidding under a DDPG-based controller; Guo et al. [9] reformulate bidding as generative trajectory modeling via diffusion. On the offline-RL side, Kumar et al. [15] provide the canonical CQL estimator, Mou et al. [18] propose an online RL framework with a safety-constrained exploration policy that bridges the offline–online gap, and Korenkevych et al. [14] deploy at Meta a hybrid agent whose neural network is trained offline to adjust an existing base policy and then discarded at serving time. In the Pinterest setting specifically, Yang et al. [26] apply deep RL to tune the ranking utility function, a different subsystem from pacing. Across most of this line, the learned policy ultimately determines the bid itself, which creates deployment friction in safety-sensitive production environments.

Hybrid RL and Classical Control

Outside advertising, combining RL with classical controllers has a long history in robotics and process control: a line of work uses RL to tune the gains of a PID controller rather than replace it, trading off adaptivity against stability [5, 16, 21]; the broader safe-RL and constrained-MDP literature studies explicit safety constraints [3, 8]. The closest work to ours is Korenkevych et al. [14], who augment Meta’s production bidder with offline RL. Our focus is daily, campaign-specific selection over a discrete set of PID gain multipliers around the production configuration. We further implement a multi-head QQ-network for separate business goals and use adaptive fusion to combine the values at inference time.

Overall, our work sits at the intersection of the three lines above: we retain the PID feedback-control structure, borrow offline training and IPS-based screening [11, 22] from the RL lineage, and validate the design through A/B tests on live Pinterest traffic.

Problem Formulation

This section formalizes the auto-bidding problem as a reinforcement learning task. We first describe how PID controllers are used in production bidding systems (§ 3.1), then present our meta-controller MDP formulation where the RL agent learns to select PID controller parameters (§ 3.2).

PID Controller for Auto Bidding

Using PID controllers for online bidding is prevalent in industrial applications [1, 20, 27, 28]. Based on campaign objectives, the controller can be designed to fulfill the budget smoothly or achieve desired cost performance. The core idea is to track the error between the current performance metric (e.g., spend rate, cost efficiency) and a target value, then adjust the bid accordingly to minimize this error over time.

Once the target is given, at time step ℓ\ell, the controller adjusts the next bid value bℓ+1b_{\ell+1} with

bℓ+1=f(bℓ,kp⋅errPℓ,ki⋅errIℓ,kd⋅errDℓ),(1)b_{\ell+1}=f\left(b_{\ell}, k_p\cdot errP_{\ell}, k_i\cdot errI_{\ell}, k_d\cdot errD_{\ell}\right), \tag*{(1)}

where the function ff encapsulates the update formula, which typically follows a conventional summation or multiplicative approach. The terms errPℓ,errIℓ,errDℓerrP_{\ell},errI_{\ell},errD_{\ell} represent the proportional, integral, and derivative error terms, respectively. The parameters kp,ki,kd∈R+k_p,k_i,k_d\in\mathbb{R}_{+} are known as the controller coefficients, which determine the relative influence of each error term on the bid adjustment.

Although the PID controller has been adopted for many decades, it remains a practical first-choice option for bidding systems due to its stability, interpretability, and effectiveness. However, a critical challenge is dynamically selecting appropriate parameters: a fixed set of parameters cannot simultaneously accommodate the diverse characteristics of different campaigns (e.g., budget size, lifetime, traffic patterns) or adapt to dynamic market conditions.

MDP for Controller-based Bidding

We adopt the standard Markov decision process (MDP) formulation throughout the paper. Specifically, an MDP is defined by a state space SS, a finite action space A\mathcal{A}, a transition kernel P(s′∣s,a)P(s' \mid s,a), a reward function r:S×A→Rr:S\times\mathcal{A}\to\mathbb{R}, and a discount factor γ∈[0,1)\gamma\in[0,1). A policy π(a∣s)\pi(a\mid s) specifies the probability of taking action a∈Aa\in\mathcal{A} at state s∈Ss\in S, and induces the action-value function

Qπ(s,a):=Eπ[∑t=0∞γtr(st,at) | s0=s, a0=a].Q^{\pi}(s,a):=\mathbb{E}^{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_t,a_t)\,\middle|\,s_0=s,\ a_0=a\right].

The objective is to learn a policy that maximizes the expected cumulative discounted reward, or equivalently, the corresponding QQ-values over the state-action space.

We formulate the online bidding process as an MDP, where the key design is to use RL as a meta-controller which only selects PID parameters (kp,ki,kd)(k_p,k_i,k_d) rather than directly producing bids. In our implementation, the agent takes one step per day. We use tt to index daily RL decisions and ℓ\ell to index the controller updates in Eq. (1). At the start of each day, the agent observes each campaign’s state and selects the corresponding parameters (kp,ki,kd)(k_p,k_i,k_d). The PID controller then uses these parameters to update the campaign’s bids according to Eq. (1) throughout that day. Practitioners can use a different inference cadence as needed.

State Space. The state st∈Ss_t\in S encodes the campaign’s status at the beginning of day tt, including the campaign ID embedding, lifetime remaining days, daily budget, PID controller targets (e.g., a campaign’s target spend curve), and previous-day aggregated metrics (CTR, CPC, revenue). These features collectively provide the agent with information about the campaign’s lifecycle stage, performance history, and operational constraints, enabling it to make informed decisions about appropriate controller tuning.

Action Space. Each action at=(mp,mi,md)∈Aa_t=(m_p,m_i,m_d)\in\mathcal{A} specifies a multiplicative adjustment to the default controller parameters (kp0,ki0,kd0)(k_p^0,k_i^0,k_d^0). Concretely, the deployed parameters are given by

(kp,ki,kd)=(mp⋅kp0, mi⋅ki0, md⋅kd0).(2)(k_p,k_i,k_d)=(m_p\cdot k_p^0,\ m_i\cdot k_i^0,\ m_d\cdot k_d^0). \tag*{(2)}

In our implementation, we optimize only the proportional (mp)(m_p) and integral multipliers (mi)(m_i), while keeping the derivative multiplier (md)(m_d) fixed at 1. This design choice is motivated by the fact that the derivative error signal errDterrD_t is fairly noisy, making derivative control less reliable for policy optimization. For each of the two tuned dimensions, the multiplier is chosen from the discrete set {1−α,1,1+α}\{1-\alpha,1,1+\alpha\}, where α∈(0,1)\alpha\in(0,1) controls the maximum deviation from the production baseline. For example, α=0.2\alpha=0.2 corresponds to allowing ±20%\pm20\% adjustments around the default setting. The resulting action space therefore contains 32=93^2=9 actions. Both the number of actions and the allowable deviation are intentionally kept small to limit online exploration to bounded changes around the validated configuration.

Transition Dynamics. After the agent selects action ata_t at the start of day tt, the PID controller uses the adjusted parameters from Eq. (2) to serve bids throughout the day. The next state st+1s_{t+1} is determined by the aggregated outcomes of these bids: total spend, clicks, revenue, and remaining budget. The transition P(st+1∣st,at)P(s_{t+1}\mid s_t,a_t) is stochastic due to the inherent randomness in user behavior, auction competition, and ad delivery dynamics. Importantly, the environment is non-stationary: market conditions, user engagement patterns, and competitor strategies evolve daily, making adaptive policies crucial for sustained performance.

Reward Function. The reward r(st,at)r(s_t,a_t) quantifies the campaign’s performance over day tt under the selected PID parameters. We aim to balance two primary objectives:

(1) Maximizing budget utilization (BU): Spending the campaign’s allocated budget to maximize revenue delivery;

(2) Minimizing cost per click (CPC): Achieving cost-efficient delivery to meet advertiser ROI expectations.

Note that these two objectives can easily conflict with each other. Higher bids can increase delivery at the expense of higher CPC, while lower bids may improve cost efficiency but leave part of the budget unspent. The intricacy here is to balance the two objectives based on campaign states, which we will detail later.

Discount Factor. In our deployment, we instantiate this RL framework with γ=0\gamma= 0, yielding a one-step contextual-bandit objective based on daily outcomes. This choice avoids bootstrapping errors in offline training, and the cross-day trade-off between cost efficiency and delivery is instead handled by the lifecycle-dependent fusion weight (see § 4.2). It is a deployment choice rather than a restriction of the meta-controller architecture: practitioners can use γ∈(0,1)\gamma\in(0, 1) and train the QQ-heads to estimate discounted returns for longer-term optimization, while retaining the state, action, and controller interface. We discuss this extension in § 8.

Proposed Methods

In this section, we present RLBP, an RL framework for ads bidding that jointly optimizes budget utilization and cost efficiency. We first describe the multi-head QQ-networks (§ 4.1), then explain how its predictions are fused (§ 4.2) and converted into a bidding-control policy (§ 4.3). Finally, we specify the relative-lift rewards used to train the agent (§ 4.4).

Multi-head QQ-Networks

Since we have multiple objectives for the agent to optimize, we define separate reward functions, e.g., reward for budget utilization and reward for cost reduction. A conventional approach is to combine these rewards into a scalar and train a QQ-network to predict the resulting value. However, in production ads bidding systems, the relative importance of different optimization objectives often changes over the lifetime of a campaign. Early in a campaign, since unspent budget can roll over to future days, we can prioritize cost performance. Near the final day of a campaign, spending the remaining budget becomes a more immediate concern. As a result, the desired trade-off between cost efficiency and revenue generation is inherently dynamic rather than fixed. Such a requirement of flexibility in objective weighting can be troublesome under a conventional RL paradigm where the reward is a fixed function. To obtain different policies, practitioners often need to redefine the reward and retrain the model, which is computationally expensive and operationally inconvenient in a production environment.

To address this issue, we propose a multi-head QQ-network. The model feeds a common state representation into multiple separate branches, each predicting QQ-values for one objective. As illustrated in Figure 1, we have a head for budget utilization that outputs Qbu(s,a)Q^{bu}(s, a) and a head for CPC that outputs Qcpc(s,a)Q^{cpc}(s, a). The outputs are fused only at inference to derive a policy. Depending on how the predicted QQ-values are fused, we can generate different policies from a single trained model.

Multi-head Q-network architecture

Figure 1. Multi-head QQ-network architecture.

Loss Functions

The total loss comprises temporal difference losses from all heads. In our case, given a training dataset {(si,ai,ribu,ricpc,si′)}i=1N\{(s_i, a_i, r_i^{bu}, r_i^{cpc}, s_i')\}_{i=1}^{N}, the per-objective losses and the total loss respectively are

Lbu(θ)=1N∑i=1N(Qθbu(si,ai)−ribu−γ⋅max⁡a∈AQθbu(si′,a))2L^{bu}(\theta) = \frac{1}{N}\sum_{i=1}^{N}\left(Q_{\theta}^{bu}(s_i,a_i)-r_i^{bu}-\gamma\cdot\max_{a\in\mathcal{A}}Q_{\theta}^{bu}(s_i',a)\right)^2
Lcpc(θ)=1N∑i=1N(Qθcpc(si,ai)−ricpc−γ⋅min⁡a∈AQθcpc(si′,a))2L^{cpc}(\theta) = \frac{1}{N}\sum_{i=1}^{N}\left(Q_{\theta}^{cpc}(s_i,a_i)-r_i^{cpc}-\gamma\cdot\min_{a\in\mathcal{A}}Q_{\theta}^{cpc}(s_i',a)\right)^2
Ltotal(θ)=Lbu(θ)+Lcpc(θ)L^{total}(\theta)=L^{bu}(\theta)+L^{cpc}(\theta)

where θ\theta denotes the model parameters; ribur_i^{bu} and ricpcr_i^{cpc} are the budget utilization and CPC rewards; NN is the total number of training examples. The CPC head uses min in its Bellman backup because lower CPC lift indicates better cost performance. In our deployment, with γ=0\gamma= 0, each QQ-head is trained to fit the corresponding daily rewards.

State Representation

The state representation is constructed from the state definition in § 3. Before being fed into the QQ-network, the campaign’s target spend curve is normalized into an hourly distribution, and budget features are log-transformed. Categorical features are embedded, while all vector and scalar numerical features are concatenated and batch-normalized. The final state representation is the concatenation of the categorical embeddings and normalized numerical feature block.

Inference-time Adaptive QQ-Value Fusion

We define the fused QQ-value as a fusion of the two heads:

Q^(s,a)=Qbu(s,a)−β(d)⋅Qcpc(s,a)\hat{Q}(s,a)=Q^{bu}(s,a)-\beta(d)\cdot Q^{cpc}(s,a)

where dd denotes the number of remaining days in the campaign’s lifetime, and β(d)≥0\beta(d)\geq0 is a cost-penalty weight that balances budget utilization against cost reduction. The negative sign reflects that we seek to maximize budget utilization while minimizing CPC. We design β(d)\beta(d) to decrease as dd approaches zero, progressively reducing the cost penalty and shifting the policy’s emphasis toward budget utilization as the campaign approaches its deadline. We compare alternative adaptive weighting functions in § 6.1.2. This inference-time adaptive fusion design provides two main advantages. First, it separates outcome prediction from business preferences by learning each objective separately, avoiding the difficulty of encoding a changing trade-off in a single reward function. Second, it offers practical flexibility in deployment, since the trade-off can be adjusted directly through the fusion weight β(d)\beta(d) as campaign conditions evolve. This makes RLBP better suited for real-world ad systems, where optimization priorities are often non-stationary and must be updated in a lightweight and interpretable manner.

Policy Induction

Given the fused scores in § 4.2, we construct a policy over the discrete joint action space.

Rather than using a greedy policy that always selects the highest-scoring action, we adopt a top-KK stochastic policy. Specifically, the policy retains the KK actions with the highest fused scores, forming the state-dependent candidate set A′(s)A'(s). A temperature-scaled softmax is applied over this reduced action set:

π(a∣s)=exp⁡(Q^(s,a)/τ)∑a′∈A′(s)exp⁡(Q^(s,a′)/τ)\pi(a|s) = \frac{\exp\left(\hat{Q}(s,a)/\tau\right)}{\sum_{a' \in A'(s)} \exp\left(\hat{Q}(s,a')/\tau\right)}

for a∈A′(s)a \in A'(s).

where Q^(s,a)\hat{Q}(s,a) is the fused score after numerical preprocessing and τ>0\tau> 0 is the temperature; the probability is zero outside A′(s)A'(s). The final bidding-control decision is then sampled from this distribution.

Reward Shaping

We now specify the daily rewards used to train the multi-head QQ-networks in § 4.1.

Learning directly from raw business metrics, especially CPC, presents significant challenges. Since the CPC difference between candidate actions is often small relative to its absolute value (for example, $1.01 versus $1.00 per click), training the QQ-network to accurately rank these actions can be difficult. Furthermore, CPC ranges vary widely across campaigns, and campaigns with larger absolute prediction errors can disproportionately dominate the loss function during training.

An additional challenge stems from the varying scales across distinct reward signals. For instance, CPC values may span from 0 to 100, whereas budget utilization remains strictly bounded within the interval [0,1][0, 1]. Such scale discrepancies generate imbalanced gradient signals backpropagated to the shared network layers during training, while simultaneously introducing complexity when tuning fusion weights at inference.

For policy selection, the key question is how candidate actions compare under the two objectives. We therefore shape the reward to the metric lift over the production baseline rather than the raw metric value. Formally, let v(s,a)v(s,a) denote the observed metric (e.g., CPC) under state ss and action aa, and let vprod(s)v^{prod}(s) denote the corresponding metric estimated from matched observations under the production baseline. CPC is total spend divided by total clicks, whereas BU is total spend divided by allocated budget. Instead of using v(s,a)v(s,a) directly as the target reward, we define the reward as:

r(s,a)=v(s,a)−vprod(s)vprod(s)r(s,a) = \frac{v(s,a) - v^{prod}(s)}{v^{prod}(s)}

Applied to BU and CPC, this definition yields the two training rewards rbur^{bu} and rcpcr^{cpc} used in § 4.1, so that each QQ-function learns the relative metric change of an action over the baseline system. Their outputs are then combined through the fusion rule to construct the action-selection policy.

This reward shaping provides two advantages. First, it centers targets around the existing controller and expresses deviations on a relative scale. Under squared-error training, normalization by the reference vprod(s)v^{prod}(s) weights squared prediction errors by 1/(vprod(s))21/(v^{prod}(s))^2, emphasizing relative discrepancies across campaigns. Figure 2 illustrates the relative action-gap distributions under the two reward formulations. In our offline study, we also evaluate their effect on policy outcomes; see Table 1. Second, the lift values of different metrics, such as CPC and BU, tend to lie on more comparable scales than their raw absolute values. This makes it easier to balance multiple objectives in the inference-time QQ-value fusion discussed in § 4.2, since the combination weights no longer need to compensate for large differences in metric magnitude.

Reward relative gap distribution comparison

Figure 2. Reward relative gap distribution comparison between absolute metric values vs. metric lifts. The left panel shows the relative difference between the best and second-best rewards constructed with raw CPC, while the right panel shows the corresponding relative difference by using CPC lift.

Reward variantRevenue lift (d=0d=0)CPC lift (d>0d>0)
Absolute reward0.08%-2.17%
Relative-lift reward2.77%-3.02%

Table 1. Reward formulation ablation. Each row reports IPS lift (%) over the production baseline. Revenue lift is evaluated on the final day of the campaign lifetime (d=0d=0), where the policy prioritizes budget utilization; CPC lift is evaluated on d>0d>0, where cost efficiency is the primary objective. Bold marks the production choice.

Overall, by shaping the reward as metric lift, RLBP learns baseline-relative action values rather than absolute metric values. This design aligns target scales across campaigns and simplifies multi-objective trade-off control at inference time.

Production Integration

This section describes the integration of RLBP into Pinterest’s production auction and bidding system. As illustrated in Figure 3, the pipeline comprises three stages: online exploration-and-exploitation, offline training, and daily serving.

End-to-end training-serving pipeline of RLBP

Figure 3. End-to-end training-serving pipeline of RLBP.

Online exploration-and-exploitation. Training the DQN requires logged tuples (st,at,rtbu,rtcpc,st+1)(s_t, a_t, r_t^{bu}, r_t^{cpc}, s_{t+1}) collected under a known logging policy. While serving the learned policy, we reserve a dedicated portion of campaign traffic and budget for uniform random exploration. Specifically, each campaign’s budget is split into two segments: primary portion is allocated to the RL policy for exploitation, while the remaining portion is used for the uniform random exploration, with assignments refreshed daily. Each segment uses distinct user buckets and separate budgets and bids. This separation allows exploratory actions to be evaluated within their allocated traffic and budget.

Offline training. A batch training job periodically aggregates the transition samples over a historical time window and retrains the QQ-network. The model learns separate QQ-functions for the target objectives using the reward formulation described in Section 4. Once training is complete, the resulting checkpoint is used to generate serving-time decisions for active campaigns.

Daily serving. A scheduled job runs RLBP once per day over all eligible campaigns: it computes QQ-values for every a∈Aa \in\mathcal{A}, fuses the two heads by the adaptive rule in § 4.2, selects an action under the top-KK stochastic policy of § 4.3, and materializes the chosen multipliers as a campaign-level lookup table (Figure 4). The production bidder then adjusts bids on the order of seconds, fetching the corresponding controller parameters from that table and applying them through the existing controller; it never loads the DQN or invokes the agent at request time, which minimizes inference latency.

Two-layer RL+PID architecture

Figure 4. Two-layer RL+PID architecture. The RL agent observes campaign state once daily and selects controller gains from a small, bounded set. The PID controller uses those fixed gains for all bid adjustments over the next 24 hours, retaining the existing feedback-control structure.

Experiments

In this section, we evaluate RLBP both offline and online. In the offline study, we evaluate the design choices and screen candidate configurations against the production baseline. In the online experiment, we validated the efficacy of RLBP through A/B testing on Pinterest’s production advertising platform.

Offline Study

IPS-based offline lift

We use 21 days of transition data for training and 7 days of held-out data for evaluation. Let π^(a∣s)\hat{\pi}(a \mid s) denote the known logging propensity. Suppose the held-out dataset contains NN examples, {(si,ai,ri,si′)}i=1N\{(s_i, a_i, r_i, s'_i)\}_{i=1}^{N}, where sis_i is the state, aia_i is the action, rir_i is the observed reward, and si′s'_i is the next state. For a policy π\pi, we can estimate its value with IPS:

IPS⁡(π)=1N∑i=1Nπ(ai∣si)π^(ai∣si)⋅ri\operatorname{IPS}(\pi) = \frac{1}{N}\sum_{i=1}^{N}\frac{\pi(a_i \mid s_i)}{\hat{\pi}(a_i \mid s_i)} \cdot r_i

The offline results below use uniformly randomized exploration logs, with π^(a∣s)=1/∣A∣\hat{\pi}(a \mid s) = 1/|\mathcal{A}|. After rollout (§ 7.1), the logging policy mixes the deployed policy with uniform exploration.

To measure the improvement, we compare the IPS of an offline-trained policy π\pi against the production policy πprod\pi_{\mathrm{prod}}, i.e.,

lift⁡(π)=IPS⁡(π)−IPS⁡(πprod)IPS⁡(πprod).\operatorname{lift}(\pi) = \frac{\operatorname{IPS}(\pi)-\operatorname{IPS}(\pi_{\mathrm{prod}})}{\operatorname{IPS}(\pi_{\mathrm{prod}})}.

In the following section, we present the IPS lift achieved by each technique. Because our framework incorporates dual reward functions—one designed to boost budget utilization and the other to minimize costs—we report the corresponding IPS lifts separately for each objective.

Ablation study

We isolate the contribution of two core techniques of RLBP: the relative-lift reward formulation (§ 4.4) and the choice of adaptive function β(d)\beta(d) in the inference-time QQ-fusion (§ 4.2). In each ablation, all other components are held fixed at RLBP’s deployed configuration and results are averaged across five training runs with different random seeds. Bold rows mark the production choice.

Reward Shaping. We compare training with absolute metric values against relative lift as the reward signal. Table 1 quantifies the final policy quality: relative lift achieves 2.77% revenue lift on the final campaign day and 3.02% CPC reduction on earlier days, compared to 0.08% and 2.17% for absolute rewards. The difference in revenue-lift is especially pronounced.

Adaptive function. We compare three forms of the fusion weight β(d)\beta(d) (Figure 5a): a step function switching abruptly at d=1d=1, a linear ramp β(d)∝d\beta(d) \propto d, and a sigmoid between them. All increase cost emphasis with remaining days, differing in their transition profiles.

Three adaptive functions and resulting learned policy behavior

Figure 5. (a) Three adaptive functions β(d)\beta(d) controlling cost emphasis as a function of lifetime remaining days. (b) Resulting learned policy behavior: fraction of campaigns assigned a cost-reduction action under each adaptive function.

fig:5b shows the consequence for policy behavior: greater cost emphasis yields more CPC-reducing actions, gradually under the linear schedule and abruptly under the step.

Table 2 reports the overall IPS lift on a cohort of campaigns in the last week of their lifetime, and Figure 6 breaks down CPC and revenue lift by lifetime remaining days across the same cohort. Day-level lifts vary across the lifecycle. The linear schedule is the only evaluated variant with both nonnegative revenue lift (+0.05%) and reduced CPC (−0.34%) in the point estimates (Table 2), so we adopt it in production. Sigmoid has higher revenue (+1.41%) alongside higher CPC (+0.49%).

Adaptive function β(d)\beta(d)Revenue liftCPC lift
Step-1.11%-1.23%
Linear+0.05%−0.34%
Sigmoid+1.41%+0.49%

Table 2. Adaptive function ablation. Each row reports IPS lift (%) on a cohort of campaigns tracked across their full last-seven-day lifecycle. Bold marks the production choice.

Offline IPS CPC and revenue lift by lifetime remaining days

Figure 6. Offline IPS CPC and revenue lift (%) by lifetime remaining days for different adaptive functions, evaluated on campaigns in the last seven days of their lifetime.

Online A/B Experiments

We evaluate RLBP in a seven-day live A/B test using the following deployed configuration: relative-lift reward, linear adaptive fusion, and top-KK softmax inference. All experiment arms share the same

serving infrastructure; the production baseline uses the default controller with no RL-driven adjustment. Against this baseline, we report revenue, clicks, and effective cost-per-click (eCPC). We report results at two levels of aggregation: all campaigns, which includes every treated campaign-day and reflects overall online impact, and lifetime-ended campaigns, restricted to campaigns whose lifetime completed during the experiment. The latter captures campaign-end behavior using observations within the test window.

6.2.1 Overall results. Table 3 summarizes online performance at two levels of aggregation. Across all treated campaign-days, RLBP achieves a statistically significant 0.25% reduction in eCPC (p<0.05p < 0.05) while keeping revenue approximately unchanged, indicating that the learned policy routes spend more cost-efficiently. For lifetime-ended campaigns, revenue and budget utilization are particularly relevant because the opportunity to spend the remaining budget ends with the campaign. In this cohort, we observe a +0.43% revenue lift and +0.25% click lift, alongside +0.18% eCPC lift. This pattern is consistent with the intended shift toward delivery near campaign end.

MetricAll campaignsLifetime-ended campaigns
Revenueflat+0.43%
Clicks+0.19%+0.25%
eCPC-0.25%+0.18%

Table 3.

Table 3: Online A/B results. Bold numbers are statistically significant (pp-value < 0.05).

6.2.2   Segmented analysis. We further break results down by budget level and historical CPC tier (Table 4). eCPC point estimates are negative across all reported segments, with the largest gains in the high-budget (−0.90% eCPC, +0.81% clicks) and high-CPC (−0.64% eCPC) segments where there is the most room for optimization.

Table 4: eCPC and click lifts in online A/B experiments, segmented by budget level and historical CPC tier. Bold numbers are statistically significant (pp-value < 0.05).

Budget levelHistorical CPC tier
MetricLowHighLowHigh
eCPC lift-0.66%-0.90%-0.29%-0.64%
Clicks lift+0.02%+0.81%+0.12%+0.10%

Table 4.

Post-Rollout Performance

Following the online A/B experiments in § 6.2, we measured RLBP’s performance over two full months after rollout at Pinterest against a small slice of holdout control traffic using default parameters (kp0,ki0,kd0)(k_p^0, k_i^0, k_d^0). This extends the evaluation beyond the seven-day A/B test to a larger campaign population.

Measurements. We divide lifetime-budget click-optimized campaigns into two groups for evaluation: campaign-days with seven or more remaining days of lifetime, and those with fewer. This division aligns with our linear adaptive fusion deployment, in which the weighting progressively transitions from cost efficiency to budget utilization starting when seven days remain (§ 4.2). Since the holdout control has much less traffic and budget, we compare BU (instead of absolute revenue), eCPC, and click-through rate (CTR). Within each group we sum spend, budget, clicks and impressions over the campaigns active on each day, form that day’s metrics, and average across days with equal weight. Due to the imbalanced traffic, no statistical test is conducted.

Results. The results for the two lifecycle groups are consistent with the linear adaptive fusion function (see Table 5). Where cost receives the greatest emphasis, RLBP achieves nearly 1% lower eCPC, with slightly lower BU than the control; where the weighting turns toward BU, delivery is above the control while eCPC remains lower. These patterns are consistent with both earlier offline and online evaluations: offline, the linear schedule was selected because it reduced CPC while yielding nonnegative revenue lift; online, the A/B test showed an eCPC reduction overall, with a positive revenue-lift point estimate among lifetime-ended campaigns. These figures are a directional reference on long-run behavior, with § 6.2 remaining the controlled estimate.

Table 5: Post-rollout performance of RLBP relative to the control holdout by two lifecycle stages. BU denotes budget utilization (delivered spend over budget).

Lifecycle stageBUeCPCCTR
≥ 7 days remaining-0.59%-0.98%+0.66%
< 7 days remaining+0.60%-0.72%+0.00%

Table 5.

Discussion

In this section, we discuss several directions for extending the RLBP framework in industrial systems.

Multi-step learning. As a first step toward industrial deployment, we instantiate RLBP with γ=0\gamma= 0. Nevertheless, this framework is not confined to a single-step setup. The meta-controller architecture, the reward design, and the multi-head QQ-network (trained with temporal-difference errors) with adaptive inference-time fusion all extend naturally to multi-step settings, enabling long-horizon optimization. One caveat in the multi-step setting is that each head bootstraps through its own greedy policy, so the fused QQ-value is generally not the optimal QQ-value of the combined reward. Fusion then serves as a practical heuristic, and developing fusion schemes with formal guarantees is a promising direction for future work.

Expanded action space. In this work, we limit RL actions to a small number of multipliers around the production default parameters mainly due to safety considerations. This narrows down the scope of exploration and simplifies deployment, at the cost of adjustment precision and range. A potential next step is to experiment with a finer action grid or continuous gain multipliers for more precise tuning, but this would require broader exploration coverage, additional safeguards, and more robust offline evaluation.

More frequent inference. Another direction of improvement is to increase the inference frequency. Currently, inference occurs once per day as an initial implementation. We are actively exploring shorter cadences, such as hourly or minute-level updates. Elevating the inference rate enables the state representation to track intraday fluctuations, traffic spikes, and shifting market dynamics with greater precision. This allows the RL agent to adjust decisions more dynamically, thereby reducing suboptimal performance driven by update delays. However, transitioning toward near-real-time serving renders offline lookup tables unusable, requiring online model deployment and leading to higher infrastructure costs.

Conclusion

We presented RLBP, an RL meta-controller that augments (rather than replaces) a production PID auto-bidder, providing a practical path for deploying RL in safety-sensitive advertising systems. RLBP introduces a multi-head DQN with inference-time adaptive fusion and a relative-lift reward formulation. Deployed at Pinterest, RLBP achieves statistically significant eCPC reductions with approximately unchanged revenue in online A/B experiments, demonstrating that RL can deliver measurable value on live advertising traffic at scale. Future directions built upon the current framework include extending to multi-step optimization for longer-lived campaigns, relaxing to continuous action spaces, and generalizing to additional campaign types and controller parameters.

Acknowledgments

We would like to express our sincere gratitude to all those who contributed to the development of our RLBP model and pipeline. We especially thank Akanksha Baid, Adam Fer, and Amol Goyal for their general support in delivering the project; Aniket Ketkar, Sameer Bhide, Jorge Alberto Bayona Romo, and Jen-An Lien for providing guidance on infrastructure implementation; Ludek Cigler and Christina Huang for sharing their experience with real-time bidding. Their expertise, dedication, and collaboration were essential to the success of this project.

References

References

  1. [1]Deepak Agarwal, Souvik Ghosh, Kai Wei, and Siyu You. 2014. Budget Pacing for Targeted Online Advertisements at LinkedIn. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, NY, USA, 1613–1619.DOI
  2. [2]Gagan Aggarwal, Ashwinkumar B. Badanidiyuru, Santiago R. Balseiro, Kshipra Bhawalkar, Yuan Deng, Zhe Feng, Gagan Goel, Christopher Liaw, Haihao Lu, Mohammad Mahdian, Jieming Mao, Aranynak Mehta, Vahab Mirrokni, Renato Paes Leme, Andres Perlroth, Georgios Piliouras, Jon Schneider, Ariel Schwartzman, Balasubramanian Sivan, Kelly Spendlove, Yifeng Teng, Di Wang, Hanrui Zhang, Mingfei Zhao, Wennan Zhu, and Song Zuo. 2024. Auto-bidding and Auctions in Online Advertising: A Survey. arXiv preprint arXiv:2408.07685.
  3. [3]Eitan Altman. 1999. Constrained Markov Decision Processes. CRC Press.
  4. [4]Santiago R. Balseiro, Kshipra Bhawalkar, Zhe Feng, Haihao Lu, Vahab Mirrokni, Balasubramanian Sivan, and Di Wang. 2024. A Field Guide for Pacing Budget and ROS Constraints. In Proceedings of the 41st International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 235). PMLR, Vienna, Austria, 2607–2638.arxiv.org/abs/2302.08530
  5. [5]Ghita Buigio and Dorin Sendrescu. 2025. Tuning of PID Controllers Using Reinforcement Learning for Nonlinear System Control. Processes 13, 3 (2025), 735.
  6. [6]Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, and Defeng Guo. 2017. Real-Time Bidding by Reinforcement Learning in Display Advertising. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (WSDM). ACM, New York, NY, USA, 661–670.
  7. [7]Qinyi Chen, Phuong Ha Nguyen, and Jordy Gligorijevic. 2024. Optimization-Based Budget Pacing in eBay Sponsored Search. In Companion Proceedings of the ACM Web Conference 2024 (WWW Companion). ACM, New York, NY, USA, 328–337.DOI
  8. [8]Javier García and Fernando Fernández. 2015. A Comprehensive Survey on Safe Reinforcement Learning. Journal of Machine Learning Research 16, 1 (2015), 1437–1480.DOI
  9. [9]Jianyan Guo, Yusen Huo, Zhilin Zhang, Tianyu Wang, Chuan Yu, Jian Xu, Yan Zhang, and Bo Zheng. 2024. AIGB: Generative Auto-Bidding via Conditional Diffusion Modeling. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, NY, USA, 5038–5049.
  10. [10]Yue He, Xijun Chen, Di Wu, Junwei Pan, Qing Tan, Chuan Yu, Jian Xu, and Xiaoqiang Zhu. 2021. A Unified Solution to Constrained Bidding in Online Display Advertising. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, NY, USA, 2993–3001.
  11. [11]Daniel G. Horvitz and Donovan J. Thompson. 1952. A Generalization of Sampling without Replacement from a Finite Universe. J. Amer. Statist. Assoc. 47, 260 (1952), 663–685.DOI
  12. [12]Junqi Jin, Chengru Song, Han Li, Kun Gai, Jun Wang, and Weinan Zhang. 2018. Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM). ACM, New York, NY, USA, 2193–2202.DOI
  13. [13]Niklas Karlsson. 2020. Feedback Control in Programmatic Advertising: The Frontier of Optimization in Real-Time Bidding. IEEE Control Systems Magazine 40, 5 (2020), 40–77.DOI
  14. [14]Dmytro Korenkevych, Frank Cheng, Artsiom Balakir, Alex Nikkulkov, Lingnan Gao, Zhihao Cen, Zuobing Xu, and Zheqing Zhu. 2024. Offline Reinforcement Learning for Optimizing Production Bidding Policies. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, NY, USA.arxiv.org/abs/2310.09426
  15. [15]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems 33 (NeurIPS). Curran Associates, Inc., Red Hook, NY, USA, 13 pages.arxiv.org/abs/2006.04779
  16. [16]Ayub I. Lakhan, Myisha A. Chowdhury, and Qiguang Lu. 2022. Stability-Preserving Automatic Tuning of PID Control with Reinforcement Learning. Complex Engineering Systems 2, 1 (2022).
  17. [17]Brendan Lucier, Sarath Pattathil, Aleksandrs Slivkins, and Mengxiao Zhang. 2024. Autbidders with Budget and ROI Constraints: Efficiency, Regret, and Pacing Dynamics. In Proceedings of the 37th Conference on Learning Theory (COLT).
  18. [18]Zhiyu Mou, Yusen Huo, Hongquan Bai, Mingzhu Xie, Chuan Yu, Jian Xu, and Bo Zheng. 2022. Sustainable Online Reinforcement Learning for Auto-Bidding. In Advances in Neural Information Processing Systems 35 (NeurIPS). Curran Associates, Inc., Red Hook, NY, USA, 14 pages.arxiv.org/abs/2210.07006
  19. [19]Pinterest Inc. 2026. Pinterest Announces Fourth Quarter and Full Year 2025 Results, Delivers 14% Revenue Growth and Record Users. https://s24.q4cdn.com/369458543/files/doc_earnings/2025/q4/earnings-result/Q425-PressRelease.pdf. Press release, February 12, 2026; 619M global monthly active users; $4.2B full-year 2025 revenue.
  20. [20]Rotem Stram, Rani Abboud, Alex Shftoff, Oren Somekh, Ariel Raviv, and Yair Koren. 2024. Mystique: A Budget Pacing System for Performance Optimization in Online Advertising. In Companion Proceedings of the ACM Web Conference 2024 (WWW Companion). ACM, New York, NY, USA, 433–442.
  21. [21]J. A. van Niekerk, J. D. Le Roux, and I. K. Craig. 2025. Reinforcement Learning Based Automatic Tuning of PID Controllers in Multivariable Grinding Mill Circuits. Control Engineering Practice 165 (2025), 106522.
  22. [22]Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudik. 2017. Optimal and Adaptive Off-Policy Evaluation in Contextual Bandits. In Proceedings of the 34th International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research). PMLR, Sydney, Australia, 3589–3597.arxiv.org/abs/1612.01205
  23. [23]Chao Wen, Miao Xu, Zhilin Zhang, Zhenzhe Zheng, Yuhui Wang, Xiangyu Liu, Xu Rong, Dong Xie, Xiaoyang Tan, Chuan Yu, Jian Xu, Fan Wu, Guihai Chen, Xiaoqiang Zhu, and Bo Zheng. 2022. Cooperative-Competitive Multi-Agent Framework for Auto-Bidding in Online Advertising. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM). ACM, New York, NY, USA, 1129–1139.DOI
  24. [24]Di Wu, Xiujun Chen, Xun Yang, Hao Wang, Qing Tan, Xiaoxun Zhang, Jian Xu, and Kun Gai. 2018. Budget Constrained Bidding by Model-Free Reinforcement Learning in Display Advertising. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM). ACM, New York, NY, USA, 1443–1451.
  25. [25]Jian Xu, Kuang-chih Lee, Wentong Li, Hang Qi, and Quan Lu. 2015. Smart Pacing for Effective Online Campaign Optimization. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, NY, USA, 2217–2226.DOI
  26. [26]Xiao Yang, Mehdi Ben Ayed, Longyu Zhao, Fan Zhou, Yuchen Shen, Abe Engle, Jinfeng Zhuang, Ling Leng, Jiaxing Xu, Charles Rosenberg, and Prathibha Deshikachar. 2025. Deep Reinforcement Learning for Ranking Utility Tuning in the Ad Recommendation System at Pinterest. In Proceedings of the Nineteenth ACM Conference on Recommender Systems (RecSys). ACM, New York, NY, USA.arxiv.org/abs/2509.05292
  27. [27]Xun Yang, Yasong Li, Hao Wang, Di Wu, Qing Tan, Jian Xu, and Kun Gai. 2019. Bid Optimization by Multivariable Control in Display Advertising. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, NY, USA, 1966–1974.DOI
  28. [28]Weinan Zhang, Yifei Rong, Jun Wang, Tianchi Zhu, and Xiaofan Wang. 2016. Feedback Control of Real-Time Display Advertising. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining (WSDM). ACM, New York, NY, USA, 407–416.DOI

Paper details

vlnc:W_7jru27b6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents