WorksMathematics
DynamAuction: A Reinforcement Learning Environment for Repeated Auctions with Dynamic Value
Abstract
A buyer who repeatedly participates in auctions for related items often faces a dynamic value problem: what they win today changes what tomorrow's items are worth. Despite the empirical importance of such effects, especially in display advertising, the algorithmic literature on repeated auctions has largely focused on the learning problem, by treating each opportunity as a contextual bandit, while sidestepping the planning problem induced by the dependence of current value on past wins. We introduce DynamAuction, a synthetic environment for single-buyer sequential bidding. The reward from each win depends on the buyer's history of previous wins, summarized through a carry-over state. The environment supports first- and second-price auctions, non-homogeneous Poisson arrivals, delayed feedback, saturation, and recency effects. We also introduce a new class of Model Predictive Control (MPC) algorithms. On our benchmark scenario, these MPC policies outperform off-the-shelf deep RL baselines (PPO, SAC, TD3) as well as a tuned shading baseline. We release DynamAuction to support RL research on repeated auctions.
DynamAuction
Introduction
Motivation. In many real-world bidding settings, a buyer participates in a long sequence of auctions for related items, and the marginal value of winning the next one depends on what they have already won. The simplest instance is a buyer with single-unit demand: a firm that needs to acquire exactly one piece of equipment over a quarter through a series of auctions derives no additional value from a second win, so the value of the next opportunity drops to zero as soon as one auction has been won. More complex dynamics arise in display advertising. For example, showing the same banner to a user multiple times within a short period typically leads to diminishing returns, so the value of the next impression depends on the timing and outcomes of previous auctions.
Learning vs. planning in repeated auctions. Most existing work on repeated auctions focuses on learning how much to bid from contextual information and an unknown competition distribution [42, 1, 20]. The only planning component typically included is a global budget or ROI constraint, which couples bids over time through a Lagrangian or dual variable [20, 41, 7, 5]. Beyond this constraint, these approaches generally assume that the value of each opportunity depends only on its current context, without accounting for how past wins affect future values. In many applications, improving planning by accounting for these dynamics may be more beneficial than further improving learning under a static value model. Our work aims to study this question by introducing a reinforcement learning environment in which these dependencies can be explicitly modeled and evaluated.
Motivations for a synthetic environment. Evaluating bidding algorithms on real-world data presents several challenges. Logged auction data only reflect the decisions made by the bidding policy used to collect them, making it difficult to evaluate alternative strategies without additional assumptions. Online evaluation through A/B tests is the textbook alternative, but it is slow, costly, constrained by business and ethical risk, and does not let the researcher independently vary delay, saturation, recency, or interference. A simulator avoids these limitations by allowing each factor to be varied under controlled conditions. Although DYNAMAUCTION can be used with general reinforcement learning algorithms, our experiments focus on planning and on the value of accounting for past wins.
Contributions.
We formalize a model of single-buyer sequential bidding in which the per-display reward is generated by a kernel conditioned on a carry-over context obtained by aggregating past wins through a commutative monoid. This framework accommodates delayed feedback, saturation and recency effects, exogenous interference, and user heterogeneity.
We release DYNAMAUCTION, an open-source, Gymnasium-compatible implementation of this model, together with a default benchmark task and guidelines for users to define new tasks.
We implement and evaluate a panel of baselines (constant bid, prediction-based bidding, greedy myopic bidding with shading, PPO, SAC, TD3) and introduce a class of Model Predictive Control algorithms. We show empirically that, on the benchmark scenario, this myopic-optimal MPC outperforms the deep-RL baselines and a tuned shading baseline.
Outline. Section 2 reviews the related literature and motivates the design of the environment. Section 3 presents the model, and Section 4 introduces DYNAMAUCTION. Section 5 describes our MPC approach. Section 6 presents the experiments, and Section 7 concludes. We use display advertising as the main running example throughout the paper. Accordingly, we refer to a won auction as a display and sometimes to its reward as a conversion.
Related work
Repeated auctions, attribution, and budget/ROI constraints
Much of the literature on repeated auctions studies a single bidder facing an unknown competition distribution, using bandit [32] or online learning methods. For example, [42, 1] consider repeated second-price auctions where the bidder learns the value of an item only after winning it. They develop bidding strategies with theoretical guarantees in stochastic and adversarial settings. Other works extend this framework to include global budget or ROI constraints, often using knapsack formulations [7, 5, 14, 13, 20, 19, 26]. This perspective has received considerable attention in the theoretical literature and has led to several important modeling contributions [4, 6, 17]. A recent survey is provided in [2].
A second line of work uses reinforcement learning (RL) to model sequential bidding as a Markov Decision Process (MDP). [11] formulate real-time bidding as an MDP whose state encodes the campaign’s remaining budget and lifetime together with the current bid request, and learn a value-function approximation for the resulting bidding policy. [43] propose a model-free RL approach for budget-constrained bidding, while [44] treat bid optimization as a multivariable control problem for a set of campaigns. A more general RL framework with budget and KPI constraints is introduced in [25]. More recent work has questioned whether the current state contains enough information for effective bidding decisions. This has motivated generative approaches that model entire bidding trajectories rather than relying only on the current state [22, 35]. However, these approaches mainly capture dependencies between bidding decisions through campaign-level constraints, such as budgets or ROI targets. The value of an impression is generally determined by its current context, without explicitly modeling how previous wins affect the value of future opportunities.
A third line of work focuses on attributing observed conversions back to the displays that caused them, and on bidding directly on incremental rather than observed outcomes. The general framing of computational advertising as a counterfactual learning problem goes back to [10]. On the bidding side, [18] show empirically that attribution-aware bidding outperforms last-touch baselines, and [9] and [8] develop causal models for repeated user interactions, the latter introducing the myopic-optimal valuation that we will reuse. [33] formalize the link between incrementality and attribution. [27] introduce the concept of repeated auctions with dynamic value and consider a setting simple enough to be solved with a differential equation, [28] leverage the analysis to build a no-regret learning algorithm based on plugin estimators, and [3] study incrementality bidding via reinforcement learning under mixed and delayed rewards. These works highlight the importance of accounting for past interactions, diminishing returns, and delayed outcomes. Our environment allows these effects to be modeled explicitly and studied under controlled conditions.
RL benchmarks, simulators, and datasets for auctions
A growing collection of open resources targets the bidding problem in online advertising. We list below those most relevant to our work.
Datasets. These contributions mostly cast the auction as a supervised learning problem. [34, 45] release the iPinYou dataset, the first publicly available large-scale RTB log, consisting of GB of bid, impression, click, and conversion logs from a 2013 DSP competition. No environment for off-policy evaluation is provided. [18] release the Criteo Attribution Modeling for Bidding dataset, which contains 30 days of real advertising data with attribution labels. The dataset can be used to study attribution and incrementality, but does not provide an environment for simulating auction outcomes under different bidding strategies.
Simulators. [30] release AUCTIONGYM: a Python simulator of first- and second-price advertising auctions designed for the off-policy evaluation of bandit-style learning-to-bid algorithms. It provides configurable competition; however, the per-impression value is a function of the current context only.
Datasets paired with simulators. More recently, [39] release the simulated environment of the Alibaba/NeurIPS 2024 auto-bidding competition: it consists of a large dataset and an RL-ready state/action/reward interface designed for cost-per-action-constrained bidding. [31] release BAT, a benchmark built on the Avito platform, comprising 10 million first-price and 1 million Vickrey–Clarke–Groves (VCG) auction records collected over a month, together with a simulator and budget-pacing and cost-per-click (CPC) constrained baselines.
None of these resources is designed to exhibit the dependence of a single bidder’s per-opportunity value on the history of past wins. DYNAMAUCTION is therefore complementary: AUCTIONGYM, BAT, and the AuctionNet environment are natural sandboxes for studying learning-to-bid and budget/ROI/Cost Per Auction (CPA)-constrained pacing; the iPinYou and Criteo datasets are appropriate for Click-Through Rate (CTR)/Conversion Rate (CVR) and attribution modeling on real logs; and DYNAMAUCTION is intended as a sandbox for studying planning under the dynamics induced by past wins on the value of future opportunities.
Model Predictive Control and model-based RL
Model Predictive Control (MPC) [12] uses a predictive model to optimize a sequence of actions over a finite horizon, executes only the first action, and replans at the next decision step. By evaluating candidate actions through model predictions rather than direct interactions with the environment, MPC is particularly appealing in settings where real-world experimentation is costly, provided that a sufficiently accurate model is available. MPC is an instance of a broader class of model-based approaches that use learned or known dynamics to guide decision-making. In model-based reinforcement learning, these approaches are commonly distinguished by how and when the model is used. In decision-time planning, the model is used at each decision step to evaluate candidate action sequences and select an action. PETS [16] is a representative example, as are the MPC algorithms proposed in Section 5. In background planning, the model is used during training to simulate trajectories and improve a policy or value function. Unlike decision-time planning, these methods learn a policy using model-generated trajectories, rather than using the model to optimize actions at each decision step. MBPO [29] and Dreamer [24] follow this approach. MuZero [37] combines elements of both: it learns a compact latent dynamics model used for decision-time planning through Monte Carlo tree search, while training the model to support value and reward predictions rather than reconstructing the full environment. Model-based approaches have also been explored in real-time bidding, primarily to address budget allocation, pacing, and ROI or CPA constraints [11, 15]. In contrast, our setting focuses on a different source of temporal dependence: rather than campaign-level constraints, past auction wins influence the future marginal value of targeting the same user. The closest related work is repeated bidding with dynamic valuations [27]. Building on this perspective, DYNAMAUCTION provides a configurable benchmark for studying such sequential bidding decisions.
Benchmark Design Requirements
Based on the literature reviewed above, we identify three key properties for benchmarking sequential bidding with dynamic valuations. Delayed feedback. The reward from winning an auction may be observed only after a stochastic delay, which can be long. Diminishing returns. The marginal value of winning an auction may decrease as the bidder accumulates wins for the same user, reflecting saturation effects. Recency effects. The marginal value of a win may also depend on the time elapsed since previous wins, rather than only on their number. Beyond these three properties, the environment should allow for heterogeneous users, exogenous events affecting user behavior, different opponent bid distributions and auction formats, and configurable auction arrival processes. We do not consider campaign-level budget or ROI constraints, which have been extensively studied and are separate from the user-level dynamics considered here. They could nevertheless be incorporated in future extensions.
Model: sequential bidding with carry-over value
Buyer, opportunities, and arrival process
We consider a single buyer interacting over a finite, possibly random, time horizon where , with a user drawn from a set according to a probability measure . A bidding opportunity for is an event at which the buyer is offered the chance to bid in an auction for an item3 described by a feature vector , where is a measurable feature space. Bidding opportunities for a fixed user arrive at random times generated by a non-homogeneous Poisson process on with intensity , and item feature is drawn from a conditional distribution . The buyer observes , places a bid , and competes against an opponent’s bid drawn independently from a competition distribution .4 The auction is resolved by an allocation rule (winning indicator) and a payment rule . For instance, the first-price rule corresponds to , and the second-price rule corresponds to , . We let denote whether opportunity is won. We write for the corresponding win times, that is, the opportunity times such that . Finally, denotes the feature of the item won at time . User heterogeneity is represented through the dependence of the model primitives on : the arrival intensity , the item-feature distribution , the law of the exchange , and the reward kernel may all depend on (see below for the definition of these terms). The population measure on then determines how this heterogeneity aggregates into the buyer’s objective (4). For , let , and we write for the wins strictly before .
Carry-over context as a commutative monoid
To capture the dependence of the buyer’s per-opportunity value on past wins, we summarize the user’s display history into a single state. We fix a (measurable) space5 together with a commutative monoid6 structure . The binary operation is associative and commutative, and is its identity element. We further fix a decay map , , intended to be evaluated only at , and which encodes the residual contribution at time of a display with feature that was shown at the earlier time . For a fixed user with realized win sequence at time , we define the carry-over context at time as
with the empty sum convention when . Since is associative and commutative, the value of the sum does not depend on the order in which past wins are aggregated. By construction, is càdlàg in : it jumps at each win time and evolves smoothly between wins, through the dependence of on its third argument. Familiar instances of that we use in the experiments include for additive counters, for multiplicatively decreasing marginal effects, and for the time since the most recent qualifying win. For the recency context, a qualifying display contributes , while a nonqualifying display contributes . Taking the minimum gives the time since the most recent qualifying win, or if there has been none. A different choice is to retain the full display history. We use , where is the collection of finite sets of display–time pairs . Each display contributes , and these contributions are combined by set union. The timestamps allow us to recover the chronological order of the displays.
Reward kernel
We define a reward kernel that describes how the expected reward of a display is distributed over time. It is modeled as a positive measure over conversion delays, parametrized by the user, the displayed item, the carry-over context at the time of display, and an exogenous information process (introduced in Section 3.4). Formally, we define a measurable map
where is the value space of the exogenous information process and denotes the set of positive Borel measures on . For a display occurring at time , the kernel specifies the expected reward associated with different delays after the display. For any Borel measurable set of such delays, we write
where the Dirac component models the instantaneous part of the reward, while the absolutely continuous component models the delayed part as a density over the delay since the display. For a single won display at time on user with feature , the expected reward attributable to this display and generated over a time window for is
where denotes the left-limit of the carry-over context at , i.e., the carry-over context evaluated just before the -th win is incorporated into it.7 The buyer’s expected cumulative reward over the horizon is then obtained by integrating per-display contributions over all wins occurring before against the population measure ,
where the expectation is over the arrival process, the item-feature distribution, the competition, the exogenous process, and any randomness in the bidding policy, all conditional on the user . The expected cumulative payment is , and the buyer’s net cumulative profit is what they seek to maximize.
Exogenous information and information available to the buyer
Exogenous information process. For each user , we allow exogenous events that affect the value of future displays but are independent of the buyer’s actions. These events occur at times and have marks . We treat the marked point process as a primitive of the model and assume it is independent of the buyer’s bidding policy. We then summarize the past of this process into a càdlàg exogenous information process
The resulting state affects the reward generated by a display at time but evolves independently of the buyer’s actions.
Information available to the buyer. Let be a measurable observation map, where is the space of reported carry-over summaries. Full observation corresponds to and . At an opportunity time on user , we assume the buyer observes the candidate item feature , the user identity , the carry-over summary , the exogenous process value , and the time itself. We collect this information in the observable state.
After each auction, the buyer observes the win indicator and payment. The scenario specifies whether the competing bid is also disclosed and how delayed rewards are reported. Policies may use current and past observations, past bids, feedback already received, and private randomization. The observation need not be Markov. Writing restricts attention to policies using only the current observation. Figure 1 summarizes the interaction process.

Figure 1. Overview of DYNAMAUCTION, illustrating the interaction between the bidding agent and the environment, the dependence of future values on past wins, and the timing of rewards.
DynamAuction
DYNAMAUCTION implements the model introduced in the previous section as a Gymnasium environment. It allows researchers to evaluate bidding strategies under different auction dynamics through configurable parameters and provides a set of baseline policies for comparison. Additional information is provided in Appendix A.
Scenario configuration. In DYNAMAUCTION, a scenario defines the model parameters, including auction mechanism, reward kernel, carry-over context, competition distribution, campaign horizon, and arrival process. Scenarios can be saved and reused across experiments, making it easy to compare bidding strategies under the same conditions. Table 2 presents the predefined scenarios available in DYNAMAUCTION. Gymnasium interface. DYNAMAUCTION follows the Gymnasium API [40] and can be used with RL libraries such as Stable-Baselines3 [36]. We provide three wrappers to adapt the environment for RL training (Table 3). The recommended order is: (i) oracle masking, which removes expected-reward information that would not be available to a real bidder; (ii) observation flattening, which converts the observations into a single vector for standard neural network policies; and (iii) bid clipping, which limits bids to during training for stable exploration.
| Preset ID | Kernel | Carry-over context |
| survival-second-price-exponential | Survival | multiplicative fatigue |
| survival-recovering-second-price-exponential-burst-arrivals | Survival | recovering multiplicative fatigue |
| survival-second-price-exponential-immediate | Immediate survival | multiplicative fatigue |
| saturation-second-price-lognormal | Saturation | number of wins |
| recency-second-price-lognormal | Recency | time of the last win |
| effective-frequency-second-price-lognormal | Effective frequency | number of wins |
| reminder-second-price-lognormal | Reminder | time of the last win |
Table 2. Built-in scenarios available in DYNAMAUCTION, with their reward kernels and carry-over contexts. Unless otherwise specified, all scenarios use second-price auctions, an exponentially distributed campaign duration with mean , and competing bids drawn from LogNormal(). See the GitHub repository for the full scenario configurations
| Class | Type | Effect |
| MaskExpectedRewardWrapper | Observation | Removes expected_reward from the observation dict. Prevents RL agents from accessing the oracle conversion value that FiPla bids directly, ensuring a fair comparison. |
| FlatAuctionObsWrapper | Observation | Concatenates all remaining Dict observation fields into a single 1-D float32 vector, compatible with standard MLP policies. |
| BidScaleWrapper | Action | Clips the action space from to (default ), restricting exploration to a practically relevant bid range. |
Table 3. Observation and action wrappers provided by DYNAMAUCTION. The recommended stacking order for RL training is top to bottom.
Model Predictive Control with Myopic Valuations
Myopic-optimal valuation
The planning problem introduced in § 3 can, in principle, be solved using dynamic programming. The optimal bid at time must account not only for the reward obtained by winning the current auction, but also for its impact on future rewards. As shown in [9 Theorem 3.1], this trade-off can be characterized using the Bellman value function , which represents the maximum expected cumulative reward achievable from a given state . However, estimating is challenging. The state space is high-dimensional, online exploration is costly, offline counterfactual estimation suffers from high variance, and the environment is non-stationary. Instead, we build on the notion of myopic-optimal valuation introduced in [8]. The myopic-optimal value is defined as the optimal bid for the current display under two assumptions: (i) the bidder cannot participate in any future auction on the same user, and (ii) the current auction is second-price. In our model, corresponds to the expected reward generated by the current display, obtained by integrating Equation (3) over the full delay support. It can be estimated using a fixed continuation policy that places no further bids on the same user, thereby isolating the reward of the current display from future bidding decisions. Additional algorithmic details are provided in Appendix § 11.
Planning with Myopic-optimal Valuations
At each decision step, our Model Predictive Control (MPC) approach compares candidate bids through Monte Carlo rollouts over a finite planning horizon. It submits the bid with the highest estimated net reward and replans at the next auction. An estimator of the myopic-optimal value provides the scale for both the candidate bids and a fixed continuation policy. Rollout scores are computed from simulated conversions and payments. Thus, the optimization concerns a single multiplier for the current bid, while accounting for its consequences for future opportunities.
Candidate bid. We express each candidate bid as a multiple of the estimated myopic-optimal value:
where is selected from a finite set of candidate multipliers. In a second-price auction, bids the estimated myopic value, while allows the planner to reduce the bid when winning the current auction has an adverse effect on future opportunities.
Simulating future auctions. At auction , at time , we generate rollouts from copies of the current simulator state, retaining past wins and pending conversions. The current observation is held fixed, while the remaining campaign duration and future events are resampled according to the scenario model. Let denote the last auction time in the complete sampled campaign, including the current auction. The planning window contains the auction indices with , where is measured in time units. The current auction uses , and auctions in this window use , with fixed across candidates. The continuation rule is fixed, but its bids depend on the observations reached in each rollout. All candidates are evaluated on the same sampled campaigns, using common random numbers for competing bids, display features, and conversions, with draws aligned by auction to reduce the variance of score differences.
Evaluating simulated trajectories. For each simulated win, the display’s conversions are sampled using its pre-win carry-over context. Let denote the total value of these conversions occurring no later than . The contribution of auction to the rollout score is
where , , and for . The allocation and payment rules and are those defined in § 3, with zero payment after a loss. For a lost auction, we set .
The planning horizon limits the auctions for which decisions are simulated, not the delays of their conversions. A display won within the window may therefore contribute conversions after , provided they occur before , consistently with the environment’s reward reporting. Contributions from displays won before auction are omitted from the score because they do not depend on the candidate. The effects of simulated wins on subsequent opportunities are captured through the evolution of the carry-over context.
For each candidate, we average these returns over the rollouts:
Selecting the bid. We select the multiplier with the highest estimated net reward, breaking ties in favor of the smallest multiplier:
After submitting , the bidder receives the auction feedback and repeats the optimization at the next opportunity. Pseudocode and implementation details are provided in Appendix § 11.
Experiments
Algorithms. To illustrate the use of DynamAuction as a benchmark for bidding with dynamic valuations, we evaluate eight bidding policies, ranging from simple baselines to model-free reinforcement learning as well as our MPC approach. The policies are the following:
Constant: Bids a fixed amount selected by grid search.
FiPla, introduced by [8] that bids the true myopic-optimal value .
Greedy Myopic, that estimates the marginal return of winning the current auction by comparing Monte Carlo rollouts that force a win or a loss, followed by a zero-bid continuation policy. The estimated difference is then used to determine the bid.
Shading uses the same estimator as Greedy Myopic but scales the resulting bid by a constant multiplier optimized through grid search.
MPC, introduced in Section 5.
Deep RL: We consider three model-free algorithms implemented with Stable-Baselines3 [36]: PPO [38], SAC [23], and TD3 [21].
All policies are evaluated on the same 1,000 test campaigns to ensure consistent comparisons. For each RL algorithm, we select hyperparameters using 200 validation campaigns and train three models with different random seeds. We report the average performance across these three runs. Further experimental details are provided in Appendix § 12.
Scenario. We choose a scenario in which current bidding decisions have lasting effects on future rewards, making planning a central aspect of the task. This can be modelled by advertising fatigue: a user repeatedly exposed to the same advertisement over a short period may become less receptive to subsequent impressions, with an effect that persists throughout the campaign. The bidder must therefore balance the expected reward of winning the current auction against its effect on the value of subsequent opportunities, making the timing and spacing of wins relevant to campaign performance. This is a deliberate modeling choice for our benchmark. Users of DynamAuction are free to specify alternative carry-over dynamics, reward kernels, and scenario parameters to study different mechanisms or applications.
Formally, all main experiments use the survival-second-price-exponential scenario, a second-price (Vickrey) auction environment with a random campaign duration , where is the mean duration. We consider , the corresponding test campaigns contain 47.7 auctions on average. Auction opportunities arrive according to a Poisson process of rate 1, and the competing bid is . A user feature vector is drawn once per campaign, and a display feature vector is drawn at each auction. Performance is reported as the mean episodic return over 1 000 test campaigns. The remaining parameters are listed in Table 5 and Table 6.
Table 5: Configurable components of a DYNAMAUCTION scenario: auction, horizon, competition, arrival, user features, and display features. A dash (—) indicates inactive components.
(a) Auction, horizon, competition, arrival
| Component | Parameter | Options / Type | Preset |
| Auction | mechanism | second-price, first-price | second-price |
| Horizon | type | exponential, fixed | exponential |
| Horizon | (exponential) | , mean campaign length | 50 |
| Horizon | (fixed) | , campaign length | — |
| Competition | distribution | lognormal, uniform, exponential, constant | lognormal |
| Competition | (lognormal) | −1.0, 1.0 | |
| Competition | low, high (uniform) | — | |
| Competition | scale (exponential) | — | |
| Competition | value (constant) | , the same competing bid at every auction | — |
| Arrival | process | cstpoisson, pwpoisson, decaypoisson | cstpoisson |
| Arrival | rate (cstpoisson) | 1.0 | |
| Arrival | breakpoints, intensities (pwpoisson) | piecewise-constant rate: lists in | — |
| Arrival | rate, period, factor (decaypoisson) | rate factor, in | — |
(b) User and display features
| Component | Parameter | Options / Type | Preset |
| User features | sampler type | uniform, normal, fixed | normal |
| User features | full dim | 3 | |
| User features | observable dim | 3 | |
| User features | mean | 0.0 | |
| User features | std | 1.0 | |
| User features | feature vector | — | |
| Display features | sampler type | iid-uniform, catalogue, fixed, normal | normal |
| Display features | IIDUniformSampler | — | |
| Display features | CatalogueSampler | — | |
| Display features | FixedSampler | — | |
| Display features | NormalSampler |
Table 5.
| Component | Parameter | Options / Type | Preset |
| Value kernel | type | survival, immediate-survival, saturation, recency, competitor-extinction, effective-frequency, reminder, mixed | survival |
| , (survival, immediate-survival; optional relevance for effective-frequency, reminder) | 1, 1 | ||
| delay rate (all but mixed) | 1.0 | ||
| base rate, saturation scale (saturation) | — | ||
| recency rate (recency) | — | ||
| base rate (competitor-extinction) | — | ||
| base rate, wear-in midpoint and slope, saturation scale (effective-frequency) | — | ||
| , , peak time (reminder) | , , | — | |
| base kernel, (mixed) | kernel type, | — | |
| Carry-over and decay map | type | multiplicative, recovering-multiplicative, additive, last-won-time, time-since-last-win, constant | multiplicative |
| decay (multiplicative, recovering-multiplicative) | 0.9 | ||
| recovery rate (multiplicative, recovering-multiplicative) | 1.0 | ||
| initial value (multiplicative, additive, last-won-time, time-since-last-win) | , context before the first win | 1.0 | |
| value (constant) | — | ||
| Exogenous process | type | trivial (), competitor-click | trivial |
| click rate (competitor-click) | , Poisson rate of competitor clicks; is the time of the last click | — |
Table 6. Configurable components of a DYNAMAUCTION scenario: value model, context, exogenous process, and conversion generation.
Carry-over context and observations. To instantiate the commutative-monoid framework introduced in § 3.2, we take the carry-over context to be the list of past wins:
At time , the buyer only receives a scalar fatigue factor computed from this history. The buyer therefore does not have access to the full carry-over context, making the scenario partially observable. For past wins at times , define
We use the empty-product convention and set before the first win. Each completed interval between consecutive wins contributes a fixed multiplicative penalty. Closely spaced wins cause a persistent reduction in subsequent conversion propensity within the campaign. If the current auction at time is won, its timestamp is added to the history after evaluating the current display’s reward kernel, so it affects later displays through their carry-over context. The observation fields are described in Table 4.
| Quantity | Dim. | Description |
| 1 | Current auction time | |
| 1 | Carry-over summary just before the auction (§ 3): the context itself when it is fully observed, or a scalar computed from it. | |
| 1 | Auction index within episode | |
| Observable user features | ||
| Display feature vector | ||
| 1 | Exogenous process value | |
| 1 | Oracle expected reward | |
| Times of the delayed conversions matured since : the most recent ones, zero-padded if fewer (, 1 in all presets) | ||
| 1 | Cumulative reward generated by the delayed conversions | |
| 1 | Highest competing bid | |
| 1 | Win indicator | |
| 1 | Payment |
Table 4. Observation variables of the DYNAMAUCTION environment. Fields marked with † describe the previous auction and are excluded from policy inputs. The oracle value is used by FIPLA but masked from RL observations.
Delayed reward. There is no exogenous signal () and no instantaneous reward component (). For delay , the reward kernel is
Conditional on the realized campaign end , the expected conversion reward attributable to a display won at is
We call this quantity the oracle value because it depends on the realized end time , which is unknown to the bidder. The oracle value is available to FiPla but hidden from the other agents (Table 3).
Results. Table 1 and Figure 2 report the performance of the different policies. MPC outperforms the tuned Shading baseline when the planning horizon is sufficiently long (). The best configuration, MPC(, , ), achieves a mean episodic return of 4.33, compared with 3.68 for Shading and 2.82 for FiPla. The gain over Shading is . With a shorter planning horizon (), the improvement is smaller but remains positive (). Using a shaded continuation policy () also improves MPC performance, although the gains are smaller. The deep RL algorithms perform worse than Shading on average. Their performance is also sensitive to training: PPO shows substantial variability across seeds, one of the three TD3 runs converges to a policy that never bids, and SAC wins considerably more auctions than FiPla but achieves a lower return. Additional experiments are reported in Appendix D.
Figure 2. Mean episode reward per policy at over the 1 000 test campaigns. Error bars are 95% bootstrap confidence intervals. RL agents: mean of their three trainings per campaign.
Mean episode reward ( CI) over the 1 000 test campaigns, and paired difference with Shading(, ): its CI is computed from the per-campaign differences, which removes the variability between campaigns. RL agents: mean of their three trainings per campaign. Settings in Appendix C.
| Policy | Reward | vs Shading |
| MPC(, , ) | ||
| MPC(, , ) | ||
| MPC(, ) | ||
| MPC(, ) | ||
| MPC(, ) | ||
| MPC(, ) | ||
| Shading(, ) | — | |
| PPO (mean of 3 trainings) | ||
| GreedyMyopic | ||
| FiPla | ||
| TD3 (mean of 3 trainings) | ||
| Constant() | ||
| SAC (mean of 3 trainings) |
Table 1.
Conclusion
We introduced a generic model for describing dynamic value in repeated auctions, along with DynamAuction, an RL environment for studying sequential bidding. The environment is challenging for deep reinforcement learning methods while accommodating settings that can be studied analytically, as in [27]. On our benchmark, which illustrates the motivation for planning in this setting, simple shading and model-predictive control policies outperform the tested deep RL methods. Our design focuses on isolating the effects of dynamic value. Accordingly, we use synthetic data and leave budget and ROI constraints outside the scope of this work.
Acknowledgments and Disclosure of Funding
This work was supported in part by the French National Research Agency (ANR) through grants ANR-20-CE23-0007 and ANR-23-CE23-0002, and through the PEPR IA FOUNDRY project (ANR-23-PEIA-0003). It was also supported by Hi! PARIS and the ANR/France 2030 program (ANR-23-IACL-0005).
References
Environment implementation and configuration
DynamAuction provides seven predefined scenarios (Table 2), each of which can be customized using the parameters listed in Table 5 and Table 6. Our main experiments use the survival-second-price-exponential scenario. Additional experiments consider immediate conversions and the saturation and recency scenarios (Appendix § 13).
Gymnasium Interface and Wrappers
DYNAMAUCTION follows the Gymnasium API and can be used with Stable-Baselines3. For RL training, we apply three wrappers in order: remove the oracle expected reward from the observation, flatten the remaining observation fields into a vector, and clip bids to (Table 3).
Observation and Action Spaces
Table 4 describes the observation fields provided by the environment. The user features in (§ 3.4) consist of an observed user vector , with , while the reward kernel depends on the full vector . In all our experiments, the user features are fully observed ().
Episode initialization and reward reporting
At the start of each episode, the environment samples the campaign horizon and the auction times over . Each step corresponds to an auction and advances the simulation to the next auction time. When an auction is won, the conversion times of the corresponding display are sampled. Delayed conversions are reported at the next auction, without identifying the display that generated them. Let denote the number of immediate conversions from winning auction , and the number of delayed conversions occurring in , with . The reward reported at step is
where is the value of a conversion, indicates whether auction is won, and is the payment, which is zero if the auction is lost. Unlike the expected reward associated with a single display in Equation (3), the step reward may include conversions from several earlier wins. The episode ends after the last auction before , and conversions occurring after this auction are not credited, even if they occur before .
Bidding Algorithms
We provide pseudocode and implementation details for MPC, followed by the GreedyMyopic and Shading baselines.
MPC
At auction , which takes place at time , the bidder receives the observation . The learned estimator predicts the conversion value of the current display, before payment. MPC uses this estimate to construct candidate bids, then compares them by simulating their consequences.
Candidate bids and future decisions. Let be a finite set of nonnegative multipliers. For each , the candidate bid is . We evaluate this bid with simulated trajectories, called rollouts. Each rollout processes the current auction and the following auctions up to time , where the planning horizon is measured in time units.
After the current auction, bids follow a fixed continuation policy. At a later simulated auction , this policy bids , where is the observation reached in that rollout and is a fixed nonnegative multiplier. Thus, only the first multiplier is optimized; subsequent bids depend on the simulated observations through .
Simulated campaigns and shared random draws. Each rollout starts from a copy of the current simulator state, including past wins and pending conversions. The current observation is kept fixed, while future events are sampled anew. For the exponential campaign durations used here, rollout draws a remaining duration , where is the mean campaign length. It then samples the future auction times and exogenous events up to . We write for the auction times in this sampled campaign, with .
The same sampled campaign is used for every candidate . Competing bids and future display features are drawn from separate random streams, and conversion draws are indexed by auction. This keeps the random inputs at auction aligned across candidates, even when earlier auction outcomes differ. The conversion outcomes can still differ, because their probabilities depend on the history of wins. These common random numbers are used to reduce the variance of differences between candidate returns.
Scoring a rollout. For each candidate , rollout simulates the current auction and subsequent auctions up to the planning horizon . Its return is the net reward of the displays won during this window:
where and for . Here, is the last auction within the planning horizon, and is the value of the conversions generated by the display won at auction that occur before the last auction of the simulated campaign, at time .
Importantly, the planning horizon limits which auctions are simulated, not which conversions are counted. A display won within the planning horizon may generate conversions after , and these are included in its return as long as they occur before , consistently with the environment’s reward reporting. Conversions from displays won before auction are omitted because they do not depend on the candidate .
The estimated return of each candidate is then
MPC selects the candidate with the largest mean score, breaking ties in favor of the smallest multiplier. It submits this bid and repeats the procedure at the next real auction. Algorithm 1 gives the steps.
Algorithm 1 MPC: selecting the bid at auction
Require: Current simulator state, observation , auction time Require: Estimator , multipliers , continuation multiplier , horizon , number of rollouts
for do
Draw a remaining duration
Sample future auctions and exogenous events up to
last auction time in the sampled campaign, including
last auction index with
Store the shared random streams for competing bids, display features and conversions
end for
for each do
for do
Copy the current simulator state, keeping past wins and pending conversions
Load rollout ’s auction schedule and exogenous events; reset its random streams
for do
Advance the copy to auction and read its observation
if then
else
end if
Determine the outcome and payment using and the competing bid of rollout at auction
if the auction is won then
Sample and schedule the display’s conversions using its pre-win context and the draws for
total value of these conversions occurring no later than
end if
Update the simulator copy with the auction outcome
end for
end for
end for
return
Special cases. With , MPC returns the bid ; using the oracle gives the FIPLA bid. With , each rollout evaluates only the current auction, including its delayed conversions credited before . In a second-price auction, is optimal in expectation if the valuation is exact for these credited conversions and . A learned valuation and a finite number of rollouts can lead to a different choice.
Computational cost. Each decision evaluates rollouts. The number of auctions processed in each rollout depends on , the arrival process and the sampled campaign duration. The rollouts use the simulator’s dynamics directly; is the learned component used to set bids.
GreedyMyopic
GreedyMyopic estimates the conversion value of winning the current auction, assuming that no further auctions will be won. Let denote the continuation policy that bids zero at every subsequent auction.
At auction , we run pairs of simulations from the current simulator state. In each pair, one simulation forces a win at auction and the other forces a loss. Both then follow until the last auction of the simulated campaign. They share the same remaining duration , future auction times and exogenous events, and start with the same noise seed .
Let and denote the total conversion value credited in these two simulations. Payments are excluded, since the quantity being estimated is the display’s value before payment. As in the environment, conversions occurring after the last auction are not credited. The bid is
The paired simulations use common random numbers to cancel the contributions of previously won displays, as detailed in Remark 1.
Algorithm 2 GreedyMyopic: selecting the bid at auction
Require: Current simulator state at auction (observation ), zero-bid continuation policy , number of pairs
Ensure: Bid
;
for do
Draw a scenario seed and a noise seed
Use the scenario seed to sample , future auction times and exogenous events
Forced win:
Copy the current simulator state, keeping past wins and pending conversions
Load the sampled campaign and initialize the noise generator with
Force auction to be won
Follow through the last auction of the sampled campaign
total conversion value credited in this simulation
Forced loss:
Make a fresh copy of the same current simulator state
Load the same sampled campaign and reset the noise generator to
Force auction to be lost
Follow through the last auction of the sampled campaign
total conversion value credited in this simulation
end for
return
Remark 1 (Common Random Numbers). The scenario seed fixes the remaining campaign duration, auction times and exogenous events. The noise seed initializes the single generator used for competing bids, display features and conversions. The forced-win simulation draws conversions for the current display, which shifts the subsequent draws of this generator. These later draws do not affect conversion rewards because wins no further auctions. Conversions already pending at auction are copied unchanged into both simulations and are credited at the same auction times. Their contributions therefore cancel in , leaving the conversion value credited to the current display. This value remains random, and averages it over the pairs.
Shading
Shading uses the same value estimate as GreedyMyopic (Algorithm 2), but multiplies it by a fixed factor . This reduces the bid to account for the effect of winning the current auction on the value of future opportunities. When , Shading is equivalent to GreedyMyopic.
Algorithm 3 Shading() — bid at auction
Require: current state , shading factor , GreedyMyopic estimator (Algorithm 2) Ensure: bid
return
Experimental details
Scenario configuration
Tables 5 and 6 list the configurable components of DYNAMAUCTION and their preset values. The main experiments use this preset with .
| Component | Parameter | Options / Type | Preset |
| Conversion probe | type | poisson, bernoulli, null | bernoulli |
| Poisson probe | sampling method | Ogata thinning process | — |
| event generation | point process over horizon | — | |
| Bernoulli probe | conversion gate | Bernoulli + exponential delay | ✓ |
| max conversions | 0 or 1 per display | — | |
| Null probe | rule | no delayed conversion (immediate kernels) | — |
| Delay sampler | type | single, multi | single |
| Single sampler | rule | at most one conversion possible per display | ✓ |
| Multi sampler | rule | multiple conversions allowed | — |
| Display history | max_history | 1 |
Table 6d. (d) Conversion generation and delay modeling
Evaluation protocol
All policies are evaluated on the same 1 000 test campaigns, generated with environment seeds 0–999. This allows us to compare policies on identical campaigns, rather than on independently sampled ones. For stochastic policies, campaign uses the policy seed .
Training and model selection use different campaigns. The MPC valuation head is trained on 1 000 FiPLA campaigns (seeds 10 000–10 999), while the RL agents generate their training campaigns from their own random streams. Intermediate evaluations during RL training use seeds . RL hyperparameters and checkpoints are selected using 200 validation campaigns (seeds 20 000–20 199). Each selected RL configuration is then trained with three different random seeds.
We report mean rewards with 95% confidence intervals, written as the mean , where is the sample standard deviation over the test campaigns. For comparisons between policies and , we compute the reward difference on each campaign and use the standard deviation of these paired differences to construct the confidence interval. For RL policies, is the average reward of the three trained models on campaign . Since both policies face the same campaigns, pairing reduces the variability due to differences in campaign length and user characteristics, yielding narrower confidence intervals for their performance differences.
Hyperparameters and training
We report the configurations used in the experiments and how their parameters were selected.
MPC. MPC evaluates candidate bids using the multiplier grid , with ties resolved in favor of the smallest multiplier. We consider planning horizons , rollouts per candidate, and continuation multipliers , where 0.4 is the multiplier selected for Shading. Unless otherwise specified, in all tables and figures.
As described in Algorithm 1, candidate bids are compared using common random numbers for competing bids, display features, and conversions. Each rollout is scored by the net reward of the displays won within the planning horizon, including their conversions occurring before the last auction of the simulated campaign.
MPC uses a learned valuation head to estimate the oracle value from observations that exclude itself. We train this estimator on 1 000 FiPLA campaigns, corresponding to 49 358 auctions at , with 15% of the data held out for validation. Training uses Adam with a learning rate of , a batch size of 256, and 500 epochs.
We compare MLPs with two hidden layers of 32, 64, or 128 units, using ReLU or Leaky ReLU activations (see Figure 3 and Table 7). Although the larger networks achieve lower training MSE (0.00097 and 0.00068, compared with 0.00147 for 32-32 ReLU), their test errors are higher (0.00208 and 0.00209, compared with 0.00184). Leaky ReLU performs similarly to ReLU (test MSE 0.00181 for 32-32 Leaky ReLU). We therefore use the 32-32 ReLU network, which has the lowest test error among the ReLU architectures and requires only 1 505 parameters, compared with 5 057 and 18 305 for the larger networks. Its smaller size also reduces the cost of the repeated predictions made during MPC rollouts.
Figure 3. Training and validation mean squared error (MSE) of MLPs used to estimate the oracle value . We compare three network sizes (32-32, 64-64, and 128-128 hidden units) with ReLU and Leaky ReLU activations. All models are trained and evaluated on the same data split from the main scenario. Curves show the mean over three random initializations, with shaded areas indicating the minimum and maximum values. MPC uses the 32-32 ReLU network.
Table 7: MLP valuation heads of different sizes and activations (mean over 3 initialization seeds). Test: 200 new FIPLA campaigns.
| hidden layers | activation | parameters | train MSE | validation MSE | test MSE | test corr. | minutes |
| 32-32 | relu | 1505 | 0.00147 | 0.00190 | 0.00184 | 0.988 | 5.7 |
| 32-32 | leaky relu | 1505 | 0.00145 | 0.00186 | 0.00181 | 0.988 | 5.7 |
| 64-64 | relu | 5057 | 0.00097 | 0.00216 | 0.00208 | 0.986 | 6.0 |
| 64-64 | leaky relu | 5057 | 0.00101 | 0.00216 | 0.00227 | 0.986 | 6.0 |
| 128-128 | relu | 18305 | 0.00068 | 0.00211 | 0.00209 | 0.987 | 6.7 |
| 128-128 | leaky relu | 18305 | 0.00057 | 0.00207 | 0.00208 | 0.987 | 6.5 |
Table 7.
GreedyMyopic and Shading. Both policies estimate the value of the current display as the mean difference between Monte Carlo rollouts that force a win and that force a loss, continued with the bid 0; each pair of rollouts shares its random numbers, so that environment noise cancels in the difference. The shading multiplier is selected by a grid search over with rollouts, on the 1 000 test campaigns: the best value is (Figure 4), and GreedyMyopic is Shading with and . Unlike the RL hyperparameters, which are selected on validation campaigns, the Shading multiplier (and hence in MPC) and the constant bid are selected on the test campaigns, which slightly favors these policies.
Figure 4. Average episode reward of Shading() with rollouts as a function of the shading multiplier , evaluated over the 1 000 test campaigns at . The shaded band shows standard deviation across episodes.
Constant bidder. The constant bid is selected by a two-stage grid search on the 1 000 test campaigns, first on a coarse grid of step 0.1 over , then on a grid of step 0.01 within of the best coarse value: (Figure 5).
Figure 5. (a) Coarse grid, , reward = 0.951. (b) Refined grid, , reward = 1.026. Average episode reward as a function of the constant bid at , using two grid resolutions: coarse ( on a 0.1 grid) and refined (0.01 grid within of the best coarse value). Results are evaluated over the 1 000 test campaigns. The shaded band shows standard deviation across episodes.
Deep RL training details
Observations and actions. PPO, SAC, and TD3 receive flattened observations from which the oracle value used by FiPLa has been removed. This value is unavailable during both training and evaluation. Observations and rewards are normalized using VecNormalize, and bids are clipped to . We use the default Stable-Baselines3 network architectures: two hidden layers of 64 units with tanh activations for PPO, and two hidden layers of 256 units with ReLU activations for SAC and TD3.
Hyperparameter search. We select the hyperparameters of each algorithm through a grid search at , using the Inria Margaret cluster. The search covers 144 configurations for PPO, 48 for SAC, and 96 for TD3. The tested values and fixed parameters are listed in Table 8.
| Parameter | Search grid / Fixed | PPO best | SAC best | TD3 best |
| Shared | ||||
| Total timesteps | search ; final (3 trainings) | — | — | — |
| Parallel environments | 4 | — | — | — |
| Normalization | VecNormalize (observations, rewards) | — | — | — |
| Eval. during training | 20 campaigns every steps | — | — | — |
| Selection | 200 validation campaigns, best or final checkpoint | — | — | — |
| PPO | ||||
| Learning rate | — | — | ||
| Rollout length (steps) | 256 | — | — | |
| Optimization epochs | 10 | — | — | |
| Entropy coefficient | 0 | — | — | |
| Discount factor | 0.99 | — | — | |
| Mini-batch size | 128 (fixed) | — | — | — |
| GAE parameter | 0.90 (fixed) | — | — | — |
| Clip range | 0.20 (fixed) | — | — | — |
| SAC | ||||
| Learning rate | — | — | ||
| Batch size | — | 512 | — | |
| Learning starts (warmup) | — | 500 | — | |
| Discount factor | — | 0.99 | — | |
| Entropy coefficient | auto-tuned | — | — | — |
| TD3 | ||||
| Learning rate | — | — | ||
| Batch size | — | — | 512 | |
| Exploration noise | — | — | 0.2 | |
| Learning starts (warmup) | — | — | 500 | |
| Discount factor | — | — | 0.99 | |
| Policy update delay | 2 (fixed) | — | — | — |
| Target policy noise | 0.2 (fixed) | — | — | — |
| Total grid configurations | 144 | 48 | 96 | |
| Validation reward of the selected configuration | 4.174 | 0.863 | 3.769 | |
| Test reward of the 3 trainings | 3.52 / 3.53 / 1.84 | 0.95 / 0.74 / 0.43 | 2.89 / 3.13 / 0.00 |
Table 8. Hyperparameter search for PPO, SAC, and TD3 at . The second column lists the values tested for each parameter or its fixed value. Bold entries indicate the hyperparameters selected using the 200 validation campaigns. The last rows report the validation reward of the selected configuration during the search and the test rewards of its three final trainings.
Each configuration is trained for timesteps with seed 42 and four parallel environments. During training, we evaluate the policy on 20 campaigns every timesteps and save the checkpoint with the highest evaluation reward. At the end of training, we compare this checkpoint with the final checkpoint on a separate set of 200 validation campaigns, using the normalization statistics saved with each checkpoint. For each algorithm, we select the configuration and checkpoint with the highest mean validation reward.
Final training. Once the hyperparameters have been selected, we train the chosen configuration again for timesteps with three different seeds (42, 43, and 44). For each training, we compare the best intermediate checkpoint and the final checkpoint on the 200 validation campaigns, then evaluate the better one on the 1 000 test campaigns. The results reported for each RL algorithm are averaged over these three independently trained models.
Table 8 reports the selected hyperparameters, their validation rewards during the search, and the test reward obtained by each of the three final trainings.
Results of the hyperparameter search. Figure 6 shows the validation reward obtained for each tested configuration. Most TD3 configurations converge to policies that never bid (82 out of 96), as do 5 of the 48 SAC configurations. PPO is less affected by this failure mode, although its final performance still depends on the training seed, as discussed below.
Figure 6. Mean reward on 200 validation campaigns for each hyperparameter configuration tested at . For each configuration, the better of the best intermediate and final checkpoints is shown. Stars indicate the configurations selected for the final trainings. Among the tested configurations, 82 of 96 for TD3 and 5 of 48 for SAC converge to policies that never bid (reward approximately zero).
Training behavior. Figure 7 shows the learning curves of the three final trainings for each algorithm. The runs reveal substantial differences across algorithms and training seeds.
Figure 7. Evaluation reward during the three final trainings of PPO, SAC, and TD3 at . Policies are evaluated on 20 campaigns every steps; curves show a rolling mean over 10 evaluations. The legend reports the test reward of each training on 1 000 campaigns. Horizontal lines indicate the test rewards of the best MPC configuration, Shading(0.4, ), and FiPla. The earliest SAC evaluations, with rewards as low as , are omitted from the vertical scale.
TD3 is particularly unstable: one of its three final trainings converges to a policy that never bids, as observed in most configurations during the search. Once its deterministic bid falls to zero, the agent stops winning auctions and does not recover in these runs.
PPO also shows substantial variability across seeds, with test rewards of 3.52, 3.53, and 1.84 despite using the same hyperparameters. SAC, by contrast, reaches its final performance level within approximately steps, but learns a policy that wins many cheap auctions and keeps users fatigued.
Longer PPO training. To check whether PPO benefits from a larger training budget, we extend the selected configuration to timesteps, using the same three training seeds. Figure 8 shows that test performance decreases over the extended runs. For example, seed 42 achieves a test reward of 3.59 after steps, 3.31 after steps, and 2.61 after steps. Increasing the training budget therefore does not improve PPO’s performance in this experiment.
Figure 8. Test reward of the selected PPO configuration trained for steps at , using three training seeds. Each point corresponds to a checkpoint saved every steps and evaluated on the same 1 000 test campaigns. Error bars show 95% confidence intervals.
Additional experiments
Bidding under varying auction frequency and recovering user fatigue
We consider a variant of the main scenario to test whether MPC adjusts its bids when the cost of winning changes over the course of a campaign. Two components are modified:
Recovering fatigue. Each past win at time reduces the value of future displays through a factor that gradually returns to 1:
This corresponds to (1) with the monoid and decay map . Each factor starts at 0.1 immediately after a win and approaches 1 independently of subsequent wins. After five time units, a win still reduces conversion propensity by about 20%. The reward kernel uses instead of , giving a conversion propensity . The bidder observes , but not the past win times that determine its future evolution.
Higher auction frequency at the start of the campaign. Auctions arrive according to a non-homogeneous Poisson process with rate 10 on and rate 1 thereafter, instead of rate 1 throughout the campaign. The campaign duration remains exponential with mean . This produces 131 auctions per campaign on average, including 92 during the initial high-frequency period.
During the first ten time units, auctions arrive roughly 0.1 time units apart, so fatigue from successive wins accumulates before it can dissipate. Later, auctions are less frequent and users have more time to recover. Winning an auction can therefore reduce the value of many more future opportunities early in the campaign than later on, making a fixed shading multiplier potentially suboptimal.
Experimental protocol. All policies are evaluated on the same 200 test campaigns (environment seeds 0–199, policy seed for campaign ). Comparisons are paired, as in the main benchmark, but use one fifth as many campaigns, giving confidence intervals approximately 2.2 times wider.
Shading uses rollouts, with its multiplier selected from on these test campaigns (). GreedyMyopic corresponds to , while the two-stage search for the constant bidder gives .
MPC uses rollouts per candidate, with , and a 32-32 ReLU valuation head trained on 1 000 FIPLA campaigns of this scenario, following the main benchmark procedure. We compare three continuation policies:
Fixed: , the selected Shading multiplier, with .
Shaded: the candidate multiplier is also used for subsequent bids in each rollout, with .
Two-phase: before and 0.7 thereafter, with .
The two-phase coefficients are selected by searching pairs on a grid with step 0.1, using an oracle policy that bids before and afterwards. Selection uses 200 validation campaigns (seeds 20 000–20 199).
For PPO, we conduct a smaller hyperparameter search centered on the configuration selected in the main benchmark. The initial grid combines learning rates , rollout lengths and optimization epochs (12 configurations). Since its best configuration uses 512 steps and 10 epochs, at the edge of this grid, we extend the search to learning rates , rollout lengths and epochs . This adds 23 new configurations, for 35 in total. The entropy coefficient remains 0 and ; all other settings follow Table 8.
Each configuration is trained for steps and evaluated on 200 validation campaigns using its best intermediate and final checkpoints. The selected configuration (learning rate , 1 024 steps per update, 5 epochs, final checkpoint) is retrained with three seeds for steps, and each training is evaluated on the 200 test campaigns. Both training budgets are shorter than in the main benchmark ( and steps, respectively). SAC and TD3 are not evaluated here, as both performed below PPO in the main benchmark.
| Policy | reward | vs Shading() | vs FIPLA |
| MPC(, , ) | |||
| MPC(, , ) | |||
| MPC(, , ) | |||
| MPC(, , shared ) | |||
| MPC(, , shared ) | |||
| MPC(, , shared ) | |||
| MPC(, , shared ) | |||
| MPC(, , two-phase ) | |||
| MPC(, , two-phase ) | |||
| MPC(, , two-phase ) | |||
| PPO (5.80 / 3.31 / 5.72) | |||
| Shading(, ) | — | ||
| FIPLA | — | ||
| GreedyMyopic | |||
| Constant() |
Table 9. Burst scenario: mean episode reward ( 95% CI) over the first 200 test campaigns, and paired differences (RL: mean of the 3 trainings; the test reward of each training in parentheses). MPC continuation (multiplier of the bids after the first one in each rollout): (the best Shading multiplier), shared (each rollout bids its candidate at every auction), or two-phase ( for , then ).
Results. Shading improves on FIPLA by (+13%). MPC performs better than Shading across the three continuation policies. Its best variant (, , ) gains (+7%) over Shading and (+21%) over FIPLA. The 95% confidence interval for the gain over Shading excludes zero for 8 of the 10 MPC variants.
MPC bids more selectively during the high-frequency period, winning 5.7% of auctions, compared with 6.6% for Shading and 7.7% for FIPLA. After , its win rate rises to 17.9%, close to Shading’s 17.6%. With the shared continuation, MPC selects an average multiplier of 0.45 before and 0.74 afterwards, close to the oracle two-phase coefficients (0.3, 0.7).
However, allowing the continuation policy to vary with time brings no clear improvement over the fixed multiplier . Most variants perform similarly, and the shared continuation at is the only one significantly below the best fixed-continuation variant (). Replanning at every auction appears sufficient to adapt the current bid to the campaign phase, without requiring a time-dependent continuation policy or a longer planning horizon.
PPO remains below Shading on average () and below the best MPC (). Two of its three trainings outperform FIPLA ( and ) but remain below Shading ( and ). The third learns to bid too little, winning only 5.2% of auctions, compared with 9.3–12.2% for the other policies except Constant.
The selected PPO run reaches a validation reward of 6.62 after steps, but its three retrainings achieve only 6.10, 2.46 and 6.29 on the same validation campaigns after steps. The high reward of the selected run thus appears optimistic, reflecting the selection of the best among 35 configurations on only 200 validation campaigns.
Computational cost and performance of bidding policies
Table 10 reports the time to compute one bid, measured on the same 80 decision states of the main scenario (), on one CPU core of a laptop (Intel Core Ultra 7 265H) and of a cluster node (Intel Xeon E5-2695 v3); the time per campaign is the time per bid times 47.7 auctions. MPC’s cost is linear in the number of rollouts per and in the horizon (each rollout simulates the next time units), whereas its reward saturates around and –80; the best setting, MPC(, , ), costs about 2.3 s per bid on the laptop. Changing does not change the cost. One prediction of the valuation head takes 0.13 ms, and the RL policies need less than 1 ms per bid.
| Policy | ms / bid (laptop) | ms / bid (cluster node) | s / campaign (laptop) | reward () |
| FiPla | < 0.01 | < 0.01 | < 0.01 | 2.821 ± 0.160 |
| Shading(, ) | 66.1 | 147 | 3.15 | 3.680 ± 0.234 |
| Shading(, ) | 133 | 297 | 6.33 | 3.680 ± 0.234 |
| Shading(, ) | 202 | 446 | 9.64 | 3.654 ± 0.232 |
| MPC(, ) | 155 | 308 | 7.40 | 3.458 ± 0.207 |
| MPC(, ) | 311 | 611 | 14.8 | 3.360 ± 0.193 |
| MPC(, ) | 769 | 1 538 | 36.7 | 3.309 ± 0.190 |
| MPC(, ) | 1 570 | 3 067 | 74.9 | 3.307 ± 0.190 |
| MPC(, ) | 373 | 878 | 17.8 | 3.911 ± 0.236 |
| MPC(, , ) | same cost as | 3.898 ± 0.239 | ||
| MPC(, ) | 734 | 1 745 | 35.0 | 3.862 ± 0.226 |
| MPC(, , ) | same cost as | 3.829 ± 0.228 | ||
| MPC(, ) | 1 120 | 2 615 | 53.4 | 3.938 ± 0.238 |
| MPC(, ) | 639 | 1 587 | 30.5 | 4.040 ± 0.256 |
| MPC(, , ) | same cost as | 4.120 ± 0.264 | ||
| MPC(, ) | 1 294 | 3 156 | 61.7 | 4.153 ± 0.262 |
| MPC(, , ) | same cost as | 4.228 ± 0.263 | ||
| MPC(, ) | 1 932 | 4 742 | 92.2 | 4.112 ± 0.257 |
| MPC(, , ) | same cost as | 4.285 ± 0.268 | ||
| MPC(, ) | 2 523 | 6 320 | 120 | 4.282 ± 0.268 |
| MPC(, ) | 781 | 1 950 | 37.2 | 4.030 ± 0.260 |
| MPC(, ) | 1 539 | 3 885 | 73.4 | 4.225 ± 0.273 |
| MPC(, ) | 2 278 | 5 859 | 109 | 4.206 ± 0.269 |
| MPC(, , ) | same cost as | 4.328 ± 0.275 | ||
| PPO (inference) | 0.574 | 0.739 | 0.0274 | 2.965 ± 0.182 |
| SAC (inference) | 0.441 | 0.697 | 0.0211 | 0.707 ± 0.119 |
| TD3 (inference) | 0.302 | 0.538 | 0.0144 | 2.006 ± 0.121 |
Table 10. Decision time per bid (mean over 80 decision states, one CPU core) and mean episode reward at .
Robustness of MPC to valuation errors and fewer rollouts
We study how errors in the learned valuation head and the number of planning rollouts affect MPC’s performance. Our reference is MPC(, , ), using the 32-32 ReLU head trained on 1 000 FiPla campaigns. We consider four types of changes (Table 11): (i) multiplicative noise, scaling each prediction by , with and ; (ii) heads trained on only 10, 30, 100, or 300 campaigns; (iii) misspecified heads that ignore fatigue (), are trained with higher competition (LogNormal(0, 1)), or have their predictions multiplied by 0.5 or 2; and (iv) fewer planning rollouts, keeping the reference head unchanged.
| family | variant | corr. | reward | vs reference | vs Shading | |
| reference | exact head (1 000 campaigns), | 0.97 | 0.050 | — | ||
| noisy | 0.95 | 0.069 | ||||
| 0.88 | 0.125 | |||||
| 0.69 | 0.225 | |||||
| 0.34 | 0.455 | |||||
| fewer training campaigns | 10 campaigns | 0.16 | 0.470 | |||
| 30 campaigns | 0.62 | 0.265 | ||||
| 100 campaigns | 0.75 | 0.212 | ||||
| 300 campaigns | 0.94 | 0.082 | ||||
| misspecified head | ignores fatigue () | 0.76 | 0.167 | |||
| trained on higher competition | 0.98 | 0.039 | ||||
| scaled | 0.96 | 0.314 | ||||
| 0.97 | 0.492 | |||||
| planning rollouts | — | — | ||||
| — | — | |||||
| — | — | |||||
| — | — | |||||
| baseline | Shading(, ) | — | — | — |
Table 11. Performance of MPC under valuation errors and reduced rollout counts. The reference uses , and the 32-32 ReLU valuation head trained on 1 000 campaigns. Correlation and mean absolute error are measured against the oracle .
All variants are evaluated on the same 1 000 test campaigns, with paired differences and 95% confidence intervals. The valuation-head experiments also use the same rollout random numbers as the reference. Prediction errors are measured against the oracle on auctions with .
MPC tolerates moderate valuation errors (Table 11, Figure 9). Heads trained on 100 or 300 campaigns perform close to the reference ( and ). Multiplicative noise reduces the reward by about 0.17 for and 0.35 for , but MPC still outperforms Shading. With , it falls to Shading’s level; with only 10 training campaigns, it performs substantially worse. Ignoring fatigue reduces the reward by , bringing MPC close to Shading, whereas a head trained under higher competition performs similarly to the reference ().
Figure 9. Mean episode reward of MPC(, ) as a function of the mean absolute error of its valuation head , for the different perturbations.
Errors in the scale of have asymmetric effects. Halving its predictions does not reduce performance (), as it gives a finer grid of candidate bids in the relevant range. Doubling them leads to excessive bids, including during rollout continuation, and reduces the reward by . Reducing the number of rollouts has a smaller effect than large valuation errors: using – instead of 80 lowers the reward by 0.13–0.24, while even remains above Shading ().
Evaluating bidding policies without delayed rewards
We evaluate a variant of the main scenario in which conversions occur immediately after a winning auction, with the same probability as in the original scenario (Table 12). This removes delayed credit assignment while keeping the fatigue dynamics, competition, and campaign lengths unchanged. The variant, survival-second-price-exponential-immediate, also ensures that no conversion is lost after the last auction. MPC uses a valuation head trained on this scenario, while the RL agents follow the same hyperparameter search and training protocol as in the main benchmark.
| Main scenario | Immediate rewards | |
| Carry-over context | , the past wins | Same |
| Conversion probability of a won display | Same | |
| Conversions per won display | At most 1 | At most 1 |
| Time of conversion | , delay | (no delay) |
| Conversions after the last auction | Never credited (including after ) | Cannot occur |
| Reward received by the agent | Payment at the winning auction; conversion value at the first auction after | Both at the winning auction |
| Oracle value |
Table 12. Comparison of the main scenario and its immediate-conversion variant. Only the timing and reporting of conversions change; the fatigue dynamics and conversion probabilities remain unchanged.
| Policy | reward | vs MPC | vs Shading |
| MPC(, ) | — | ||
| PPO (4.11 / 4.23 / 3.86) | |||
| Shading(best , ) | — | ||
| FiPla | |||
| GreedyMyopic | |||
| TD3 (3.93 / 0.00 / 0.00) | |||
| Constant(best ) |
Table 13. Performance of the bidding policies with immediate conversions. Results are mean episode rewards ( 95% CI) over 1 000 test campaigns, with paired differences relative to MPC and Shading. For RL, the mean is taken over three trainings, whose individual test rewards are shown in parentheses.
With immediate conversions, PPO outperforms all myopic baselines, including Shading (), but remains below MPC (). Thus, removing delayed feedback does not eliminate the advantage of MPC, which can still plan around the effects of fatigue on future auctions. TD3 remains unstable: one training reaches 3.93, above Shading, while the other two converge to policies that never bid, as did 44 of its 96 hyperparameter configurations during the search.
Immediate conversions also prevent rewards from being lost after the last auction, unlike in the main scenario. Since the two scenarios do not share the same random draws, performance comparisons are made within each scenario.
Evaluating bidding policies against the optimal policy under saturation and recency
We consider two simpler scenarios in which the optimal bidding policy can be computed by dynamic programming (DP). Neither scenario has user or display features, and the carry-over state is one-dimensional. They differ in how past wins affect conversion probability:
Saturation: , where is the number of past wins.
Recency: , where is the time since the last win.
Both scenarios use exponentially distributed conversion delays with rate 0.5 and competing bids drawn from .
We compare the bidding policies with the DP optimum on the same 1 000 test campaigns. Unless otherwise stated, MPC uses rollouts and , while the RL agents follow the training protocol of the main benchmark. We also evaluate PPO with the time since the last win added to its observations, and a diagnostic Cooldown policy that bids the oracle except during a fixed period after each win. The tables report the fraction of the improvement over FIPLA achieved by each policy, defined as , where denotes the mean episode reward.
Computing the optimal policy. In both scenarios, the campaign duration is exponential with mean , giving a termination rate . Auction arrivals have rate 1. Because the remaining campaign duration is memoryless, the optimal policy depends only on the current state : the number of past wins for saturation, or the time since the last win for recency. Before the first win, and .
A display won in state has expected conversion value
where is the conversion-delay rate. The factor accounts for the probability that the conversion occurs before the campaign ends.
The probability of reaching another auction before the campaign ends is . Conditional on reaching it, the elapsed time is . A win changes the state to in saturation or in recency; after a loss, the state is or , respectively.
In a second-price auction, the optimal bid is the marginal value of winning:
where is the optimal expected future reward and satisfies
The last expectation has a closed form. For saturation, we solve the Bellman equation by backward induction from , where conversion probability is negligible, using bisection to solve for each . For recency, we use value iteration on a grid with step 0.02, stopping when the maximum change falls below .
The resulting policy bids at every auction. Since it uses the known model, it provides an optimal reference in expectation, subject to numerical approximation and test-sample variability. Table 16 reports its effective shading multiplier .
| saturation, | 0 | 2 | 5 | 10 | 15 | 20 | |
| 0.39 | 0.42 | 0.46 | 0.55 | 0.66 | 0.76 | ||
| recency, | 0.25 | 1 | 2 | 3 | 5 | 10 | |
| 0.64 | 0.60 | 0.57 | 0.55 | 0.54 | 0.53 |
Table 16. Optimal effective shading of the DP by state (: past wins; : time since the last win).
Why constant shading is close to optimal. In both scenarios, the optimal bid is approximately a fixed fraction of the myopic value (Table 16). The effective multiplier ranges from 0.39 to 0.66 for the first 15 wins in saturation, and from 0.53 to 0.64 for the recency states considered. A single shading multiplier therefore approximates the optimal policy well. Indeed, Shading is statistically indistinguishable from the DP optimum in both scenarios: in saturation and in recency.
With rollouts, MPC remains below Shading in both scenarios. Increasing to 1 000 substantially reduces this gap, particularly in recency, where MPC matches Shading (). This suggests that the accuracy of MPC’s rollout estimates matters when the optimal multiplier varies little across states.
| Policy | reward | vs DP optimum | vs Shading | share of DP gain |
| Shading(best , ) | 4.351 0.158 | +0.03 0.08 | — | 105% |
| DP optimum | 4.323 0.162 | — | −0.03 0.08 | 100% |
| MPC(, ) | 4.237 0.157 | −0.09 0.09 | −0.11 0.10 | 86% |
| PPO (4.20 / 4.08 / 4.24) | 4.173 0.143 | −0.15 0.07 | −0.18 0.08 | 75% |
| MPC(, ) | 4.138 0.150 | −0.18 0.10 | −0.21 0.10 | 69% |
| MPC(, ) | 4.097 0.154 | −0.23 0.11 | −0.25 0.10 | 62% |
| MPC(, , ) | 4.097 0.152 | −0.23 0.10 | −0.25 0.11 | 62% |
| MPC(, ) | 4.056 0.154 | −0.27 0.11 | −0.30 0.10 | 56% |
| MPC(, ) | 4.051 0.156 | −0.27 0.10 | −0.30 0.10 | 55% |
| MPC(, , ) | 4.040 0.148 | −0.28 0.10 | −0.31 0.10 | 53% |
| MPC(, , ) | 4.039 0.149 | −0.28 0.10 | −0.31 0.10 | 53% |
| MPC(, ) | 4.029 0.154 | −0.29 0.11 | −0.32 0.11 | 51% |
| Constant(best ) | 3.856 0.169 | −0.47 0.12 | −0.50 0.12 | 22% |
| FiPla | 3.721 0.142 | −0.60 0.12 | −0.63 0.12 | 0% |
| GreedyMyopic | 3.605 0.151 | −0.72 0.13 | −0.75 0.12 | -19% |
Table 14. Saturation kernel (RL: mean of the 3 trainings, the test reward of each in parentheses). MPC: unless otherwise stated.
| Policy | reward | vs DP optimum | vs Shading | share of DP gain |
| DP optimum | 7.875 0.529 | — | +0.09 0.12 | 100% |
| MPC(, ) | 7.786 0.521 | −0.09 0.12 | +0.00 0.12 | 89% |
| Shading(best , ) | 7.784 0.520 | −0.09 0.12 | — | 88% |
| MPC(, ) | 7.670 0.516 | −0.20 0.13 | −0.11 0.13 | 74% |
| PPO + time since last win (7.66 / 7.63 / 7.63) | 7.643 0.510 | −0.23 0.10 | −0.14 0.10 | 70% |
| MPC(, ) | 7.542 0.510 | −0.33 0.13 | −0.24 0.14 | 57% |
| MPC(, , ) | 7.498 0.505 | −0.38 0.13 | −0.29 0.13 | 52% |
| MPC(, , ) | 7.488 0.504 | −0.39 0.13 | −0.30 0.13 | 50% |
| MPC(, , ) | 7.488 0.504 | −0.39 0.13 | −0.30 0.13 | 50% |
| MPC(, ) | 7.483 0.505 | −0.39 0.14 | −0.30 0.13 | 50% |
| MPC(, ) | 7.483 0.505 | −0.39 0.14 | −0.30 0.13 | 50% |
| MPC(, ) | 7.476 0.505 | −0.40 0.14 | −0.31 0.13 | 49% |
| Constant(best ) | 7.099 0.489 | −0.78 0.15 | −0.68 0.15 | 1% |
| FiPla | 7.095 0.490 | −0.78 0.18 | −0.69 0.17 | 0% |
| PPO (7.09 / 7.02 / 7.03) | 7.048 0.464 | −0.83 0.14 | −0.74 0.13 | -6% |
| Cooldown(oracle , best ) | 6.998 0.473 | −0.88 0.17 | −0.79 0.15 | -12% |
| GreedyMyopic | 6.856 0.478 | −1.02 0.18 | −0.93 0.16 | -31% |
Table 15. Recency kernel (RL: mean of the 3 trainings, the test reward of each in parentheses). MPC: unless otherwise stated.
Effect of state representation on PPO. PPO performs close to Shading in saturation, where the number of past wins is directly available in its observation. In recency, however, PPO receives the current time and the time of the last win as separate inputs (with indicating no previous win). Its bids barely depend on their difference, the time elapsed since the last win, and its performance is close to that of a constant bidder. Adding this elapsed time explicitly to the observation, while keeping the same hyperparameters and training seeds, improves PPO’s reward by about 0.6 and brings it close to Shading (). The gap in this scenario is therefore largely explained by the state representation.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: it does
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: it does
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes] Justification: it does
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: it does
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: it does
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: it does
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: it does
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: it does
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: we reviewed it
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: help practitioners and theoreticians to better train RL model on repeated auctions with dynamic value.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: no release of pre trained LLM or data.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: we are the creator of this simulator
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: it is the case
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: only used for writing and code cleaning purpose.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
References
- [1]Achddou, J., Cappé, O., & Garivier, A. (2021). Efficient algorithms for stochastic repeated second-price auctions. Algorithmic Learning Theory, 99–150.
- [2]G. Aggarwal, A. Badanidiyuru, S. R. Balseiro, K. Bhawalkar, Y. Deng, Z. Feng, G. Goel, C. Liaw, H. Lu, M. Mahdian, J. Mao, A. Mehta, V. Mirrokni, R. P. Leme, A. Perlroth, G. Piliouras, J. Schneider, A. Schvartzman, B. Sivan, K. Spendlove, Y. Teng, D. Wang, H. Zhang, M. Zhao, W. Zhu, and S. Zuo. Auto-bidding and auctions in online advertising: A survey. SIGecom Exch., 22(1):159–183, Oct. 2024.arxiv.org/abs/2408.07685
- [3]Badanidiyuru Varadaraja, A., Feng, Z., Li, T., & Xu, H. (2022). Incrementality bidding via reinforcement learning under mixed and delayed rewards. Advances in Neural Information Processing Systems, 35, 2142–2153.
- [4]Balseiro, S., Kim, A., Mahdian, M., & Mirrokni, V. (2017). Budget management strategies in repeated auctions. Proceedings of the 26th International Conference on World Wide Web, 15–23.
- [5]S. Balseiro, H. Lu, and V. Mirrokni. Dual mirror descent for online allocation problems. In International Conference on Machine Learning, pages 613–628. PMLR, 2020.arxiv.org/abs/2002.10421
- [6]S. R. Balseiro, O. Besbes, and G. Y. Weintraub. Repeated auctions with budgets in ad exchanges: Approximations and design. Management Science, 61(4):864–884, 2015.DOI
- [7]S. R. Balseiro and Y. Gur. Learning in repeated auctions with budgets: Regret minimization and equilibrium. In Proceedings of the 2017 ACM Conference on Economics and Computation, EC ’17, page 609, New York, NY, USA, 2017. Association for Computing Machinery.DOI
- [8]M. Bompaire, A. Désir, and B. Heymann. Fixed point label attribution for real-time bidding. Manufacturing & Service Operations Management, 26(3):1043–1061, 2024.arxiv.org/abs/2012.01767
- [9]M. Bompaire, A. Gilotte, and B. Heymann. Causal models for real time bidding with repeated user interactions. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, page 75–85, New York, NY, USA, 2021. Association for Computing Machinery.DOI
- [10]L. Bottou, J. Peters, J. Quiñonero-Candela, D. X. Charles, D. M. Chickering, E. Portugaly, D. Ray, P. Simard, and E. Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(11), 2013.
- [11]H. Cai, K. Ren, W. Zhang, K. Malialis, J. Wang, Y. Yu, and D. Guo. Real-time bidding by reinforcement learning in display advertising. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM ’17, page 661–670, New York, NY, USA, 2017. Association for Computing Machinery.DOI
- [12]Camacho, E. F., & Bordons, C. (2013). Model Predictive Control. In Advanced Textbooks in Control and Signal Processing.
- [13]M. Castiglioni, A. Celli, and C. Kroer. Online learning under budget and roi constraints via weak adaptivity. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2302.01203
- [14]M. Castiglioni, A. Celli, A. Marchesi, G. Romano, and N. Gatti. A unifying framework for online optimization with long-term constraints. Advances in Neural Information Processing Systems, 35:33589–33602, 2022.arxiv.org/abs/2209.07454
- [15]S. Chen, Q. Xu, L. Zhang, Y. Jin, W. Li, and L. Mo. Model-based reinforcement learning for auto-bidding in display advertising. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’23, page 1560–1568, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems.DOI
- [16]K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, volume 31, 2018.arxiv.org/abs/1805.12114
- [17]V. Conitzer, C. Kroer, E. Sodomka, and N. E. Stier-Moses. Multiplicative pacing equilibria in auction markets. Operations Research, 70(2):963–989, 2022.arxiv.org/abs/1706.07151
- [18]Diemert Eustache, Meynet Julien, P. Galland, and D. Lefortier. Attribution modeling increases efficiency of bidding in display advertising. In Proceedings of the AdKDD and TargetAd Workshop, KDD, Halifax, NS, Canada, August, 14, 2017, page To appear. ACM, 2017.arxiv.org/abs/1707.06409
- [19]Z. Feng, S. Padmanabhan, and D. Wang. Online bidding algorithms for return-on-spend constrained advertisers. In Proceedings of the ACM Web Conference 2023, WWW ’23, page 3550–3560, New York, NY, USA, 2023. Association for Computing Machinery.arxiv.org/abs/2208.13713
- [20]A. Flajolet and P. Jaillet. Real-time bidding with side information. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
- [21]Fujimoto, S., van Hoof, H., & Meger, D. (2018). Addressing function approximation error in actor-critic methods.
- [22]J. Guo, Y. Huo, Z. Zhang, T. Wang, C. Yu, J. Xu, B. Zheng, and Y. Zhang. Generative auto-bidding via conditional diffusion modeling. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5038–5049, 2024.DOI
- [23]T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018.arxiv.org/abs/1801.01290
- [24]D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.arxiv.org/abs/1912.01603
- [25]Y. He, X. Chen, D. Wu, J. Pan, Q. Tan, C. Yu, J. Xu, and X. Zhu. A unified solution to constrained bidding in online display advertising. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’21, page 2993–3001, New York, NY, USA, 2021. Association for Computing Machinery.DOI
- [26]B. Heymann. Cost per action constrained auctions. In Proceedings of the 14th Workshop on the Economics of Networks, Systems and Computation, pages 1–8, 2019.arxiv.org/abs/1809.08837
- [27]B. Heymann, A. Gilotte, and R. Chan-Renous. Repeated bidding with dynamic value. In International Conference on Web and Internet Economics, pages 548–563. Springer, 2024.arxiv.org/abs/2308.01755
- [28]B. Heymann and O. Sakhi. Learning to bid in repeated second-price auctions with dynamic values and aggregated feedback. to be published at Neurips 2026, 2026.arxiv.org/abs/2605.28133
- [29]M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1906.08253
- [30]O. Jeunen, S. Murphy, and B. Allison. Off-policy learning-to-bid with auctiongym. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4219–4228, 2023.DOI
- [31]A. Khirianova, E. Solodneva, A. Pudovikov, S. Osokin, E. Samosvat, Y. Dorn, A. Ledovsky, and Y. Zenkova. Bat: Benchmark for auto-bidding task. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 2657–2667, New York, NY, USA, 2025. Association for Computing Machinery.DOI
- [32]T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.DOI
- [33]R. Lewis and J. Wong. Incrementality bidding and attribution. arXiv preprint arXiv:2208.12809, 2022.arXiv
- [34]H. Liao, L. Peng, Z. Liu, and X. Shen. ipinyou global rtb bidding algorithm competition dataset. In *Proceedings of the Eighth International Workshop on Data Mining for Online Advertising*, ADKDD’14, page 1–6, New York, NY, USA, 2014. Association for Computing Machinery.DOI
- [35]Y. Peng, W. Shu, J. Sun, Y. Zeng, J. Pang, W. Bai, Y. Bai, X. Liu, and P. Jiang. Expert-guided diffusion planner for auto-bidding. In *Proceedings of the 34th ACM International Conference on Information and Knowledge Management*, CIKM ’25, page 5963–5970, New York, NY, USA, 2025. Association for Computing Machinery.DOI
- [36]A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. *Journal of Machine Learning Research*, 22(268):1–8, 2021.
- [37]Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., & Silver, D. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839), 604–609.
- [38]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.arxiv.org/abs/1707.06347
- [39]K. Su, Y. Huo, Z. Zhang, S. Dou, C. Yu, J. Xu, Z. Lu, and B. Zheng. Auctionnet: A novel benchmark for decision-making in large-scale games. *Advances in Neural Information Processing Systems*, 37:94428–94452, 2024.arxiv.org/abs/2412.10798
- [40]Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., & al., . et . (2024). Gymnasium: A standard interface for reinforcement learning environments.arXiv
- [41]Q. Wang, Z. Yang, X. Deng, and Y. Kong. Learning to bid in repeated first-price auctions with budgets. In *Proceedings of the 40th International Conference on Machine Learning*, ICML’23. JMLR.org, 2023.arxiv.org/abs/2304.13477
- [42]J. Weed, V. Perchet, and P. Rigollet. Online learning in repeated auctions. In *Conference on Learning Theory*, pages 1562–1583. PMLR, 2016.arxiv.org/abs/1511.05720
- [43]D. Wu, X. Chen, X. Yang, H. Wang, Q. Tan, X. Zhang, J. Xu, and K. Gai. Budget constrained bidding by model-free reinforcement learning in display advertising. In *Proceedings of the 27th ACM International Conference on Information and Knowledge Management*, CIKM ’18, page 1443–1451, New York, NY, USA, 2018. Association for Computing Machinery.DOI
- [44]X. Yang, Y. Li, H. Wang, D. Wu, Q. Tan, J. Xu, and K. Gai. Bid optimization by multivariable control in display advertising. In *Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, KDD ’19, page 1966–1974. ACM, 2019.DOI
- [45]W. Zhang, S. Yuan, J. Wang, and X. Shen. Real-time bidding benchmarking with ipinyou dataset. *arXiv preprint arXiv:1407.7073*, 2014.arxiv.org/abs/1407.7073







