WorksEconomics, Econometrics and Finance
Media Trading with Censored Auction Prices: A Stochastic Control and Reinforcement Learning Approach
Abstract
In real-time bidding, marketing agencies must balance campaign performance, transaction margins and the use of a finite advertising budget. A key difficulty is that market-clearing prices are typically observed only when bids are successful, so the information available for subsequent decisions depends on the bidding policy. We formulate this problem using a continuous-time stochastic control model with a latent price process. To construct a price signal, we consider parallel exploratory subcampaigns in which bids are randomized subject to an upper bound. Under a stylized cross-sectional sampling model, the resulting price estimator is consistent and satisfies a mean-square error bound of order , where is the number of exploratory observations. We then describe a model-based approach combining forward-equation calibration with Hamilton-Jacobi-Bellman policy iteration and a physics-informed neural network, alongside an entropy-regularized actor-critic approach based on observable information in a model-free setting. Synthetic experiments illustrate the estimator and the computational procedures. The framework clarifies the decision and information trade-offs faced by the agency, while distinguishing full-information benchmarks and estimated-price policies from the genuinely partially observed control problem.
MSC Classification: 90B60; 93E20; 93E35.
Introduction
Real-time bidding (RTB) has made advertising inventory a resource that must be purchased through frequent, automated auctions. For a marketing agency, the decision is not simply how much to bid on an individual impression: the bids determine campaign performance, the revenue associated with media trading, and the depletion of the client’s budget over the campaign horizon. Market prices and campaign outcomes are uncertain, and the agency must adjust its decisions as information arrives. This makes media trading a dynamic resource-allocation problem with an important operational constraint: the agency may not observe the clearing prices of auctions that it loses.
The observation constraint changes the nature of the optimization. A high bid is more likely to generate an impression and reveal the clearing price, but may also consume more of the available budget. A low bid preserves expenditure yet produces less information about the market. Thus, data collection cannot be treated independently of the bidding policy. Figure 1 summarizes the information and decision flow.

Figure 1. Main participants in a programmatic advertising auction.
The problem has two complementary aspects. From a modeling perspective, one needs to represent the market price, campaign performance and remaining budget, together with an observation mechanism that depends on the bid. From a computational perspective, the agency needs implementable policies despite imperfect information and uncertain dynamics. Stochastic control provides a natural description of the intertemporal objectives and constraints; reinforcement learning offers an alternative route to policy optimization when the dynamics are not fully calibrated.
Related work. Earlier studies of advertising effectiveness include the analysis of exposure and memory [20, 21], aggregate advertising models [8], and media-mix measurement [18]. Uncertainty in attributing advertising outcomes remains a major practical difficulty [7, 11].
Auction-based advertising has been studied using constrained optimization [17], online learning with side information [3], and deep reinforcement learning [19]. Continuous-time stochastic control approaches to bidding have been proposed by [2] and [10]; mean-field models also describe aggregate advertising effects [14]. The statistical problem of estimating auction-clearing prices from censored observations is related to [16].
Of particular relevance to the operational setting is the recent contribution in [9]. They study keyword-level bid optimization in sponsored-search advertising using limited aggregate feedback, a Bayesian model of campaign metrics, and a bandit-based online optimization method. Their study includes simulations using real advertising data. The present work addresses a different decision problem: continuous-time bidding for a media-trading agency, explicit censoring of clearing prices by auction outcomes, and a balance between performance, transaction margins and remaining budget. Accordingly, the informational assumptions, decision frequency, and objectives differ; we do not claim a direct empirical comparison with their method.
Modeling and methodological contributions. We first set out an agency-level stochastic decision model in which the market price is latent and bidding affects the information revealed by transactions. To construct a usable price signal, we consider randomized bids in parallel exploratory subcampaigns. In an explicit cross-sectional observation model, the winning frequency leads to a consistent estimator with a finite-sample mean-square error bound.
We next connect this price signal to two computational approaches. The first combines forward-equation calibration with a full-information Hamilton–Jacobi–Bellman (HJB) benchmark, approximated through policy iteration and physics-informed neural networks (PINNs) [13]. The second describes an entropy-regularized actor–critic method, defined on observable features and inspired by continuous-time reinforcement learning [5, 6, 15]. These are established numerical ingredients; the interest of combining them here is to clarify their use and limitations for the media-trading decision problem. In particular, the estimated-price feedback is a plug-in strategy, not the solution of the underlying partially observed control problem.
The paper is organized as follows. Section 2 describes the RTB decision problem and a simple benchmark. Section 3 introduces the stochastic model and the observation structure. Estimation and calibration are considered in Section 4. Sections 5 and 6 present the HJB-based and reinforcement-learning approaches. Section 7 discusses the numerical experiments and their limitations, before the concluding section.
The media trading decision problem
We begin with the agency’s decisions, observations and budget accounting, then use an elementary linear–quadratic example to illustrate how the auction-winning constraint enters the optimization. The example is a full-information benchmark; the incomplete-information model follows in the next section.
Auction mechanism and information
Consider an advertising campaign over a finite horizon . The agency receives an initial budget from its client and submits bids to a sequence of RTB auctions through a demand-side platform.
At time , the agency chooses a bid . An auction is won if , where denotes the market-clearing price. A successful bid contributes to campaign performance and consumes part of the budget.
The agency observes the clearing price only when it wins. Its observation history thus depends on the previous bids, and the decision problem combines partial observation with control-dependent information acquisition.
Following the continuous-time approaches of [2] and [10], we use a diffusion approximation to represent the frequent auctions and the resulting campaign dynamics.
We use the following variables:
, the latent market-clearing price;
, the campaign performance (for instance the cumulative number of conversions or sales);
, the remaining campaign budget;
, the bid, which is the control rather than a state variable.
The trade-offs have a direct operational interpretation. Raising the bid increases the chance of obtaining an impression, while successful transactions affect both campaign value and the agency’s margin. The remaining budget couples today’s bid with tomorrow’s opportunities. Moreover, a lost auction reveals less price information than a won auction. We first examine these mechanisms in a simple complete-information model; a joint optimization of exploration expenditure is beyond the scope of the present framework.
A linear–quadratic benchmark
As an elementary illustration, we first consider a first-price stylization in which a winning bid is paid at its submitted value. Unlike the market-clearing-price settlement in the general model below, this convention leads to a particularly transparent quadratic maximization. Suppose that the market price is fully observed and follows the Ornstein–Uhlenbeck dynamics
The agency controls its bidding price . An auction is won whenever
Campaign performance evolves according to
where measures the average increase in performance generated by a successful auction. The remaining budget satisfies
where denotes the agency margin.
The agency seeks to maximize the performance criterion
The running reward combines campaign performance, the margin obtained from successful bids, and a quadratic penalty on the bidding cost. The terminal term penalizes any deviation of the final budget from zero, including an overdraft. This is a soft penalty, not a hard budget constraint.
Let and be generic variables. The (maximized) Hamiltonian of this benchmark is defined by
Here is the space of symmetric matrices. The state variable does not appear explicitly in the present coefficients, but is retained as an argument of the Hamiltonian. For a smooth value function , the arguments in (2.1) are replaced by , where . The HJB equation therefore reads
The optimization over the bid can be performed explicitly, taking the winning constraint into account. For generic , set
If , the best bid conditional on winning is
Writing and , we obtain
For , the optimal bid wins the auction only if ; otherwise any bid below avoids the transaction. If , no admissible bid wins the auction. In these formulas the generic vector is evaluated at when defining the optimal feedback; the matrix is evaluated at in the HJB equation.

Figure 2. Bidding policy from the benchmark implementation

Figure 3. Benchmark trajectories
The preceding calculations rely on observing continuously. We now turn to the actual information structure, in which the market price is hidden except for the transactions won by the agency.
Stochastic dynamics and information structure
We now introduce the general stochastic control framework studied in the remainder of the paper. Contrary to the benchmark model of § 2, we allow for general nonlinear dynamics of the market-clearing price and of the campaign performance. The presentation proceeds in three steps: we first formulate the problem under full information, that is, as if the agency had access to the whole filtration (§ 3.1); we then explain how the agency can enrich this information by running auxiliary sub-campaigns, and we state the problem addressed in this paper: constructing an approximate solution of the control problem from these limited observations (§ 3.2).
The full-information model
Let be a filtered probability space satisfying the usual conditions and supporting two independent one-dimensional -Brownian motions and . The filtration represents the full information of the model: it contains every source of randomness affecting the market and the campaign, including quantities that the agency does not observe. It serves as a reference framework, in which all processes below are defined; the information actually available to the agency is described in § 4.1.
The latent market-clearing price is the solution of
where is an -measurable, square-integrable random variable. The functions and describe the local trend and the volatility of the market-clearing price. The price is exogenous: the agency’s bids are assumed to be too small, relative to the size of the market, to affect .
The agency controls its bidding strategy through a process taking values in the compact set
In the full-information framework, the set of bidding strategies is
Since is bounded, every satisfies , so that no additional integrability condition is required. All processes introduced below depend on the chosen strategy , which we indicate by a superscript.
At time , the auction is won if and only if the submitted bid exceeds the market-clearing price, that is, on the event . The cumulative number of auctions won is thus given by
The performance process of the campaign represents, depending on the application, the cumulative number of clicks, conversions or sales generated by the campaign. It evolves according to
The drift and diffusion coefficients may depend simultaneously on the current market conditions and on the bid selected by the agency.
The agency starts the campaign with an initial budget , and denotes the remaining budget. Whenever an auction is won, the agency pays the market-clearing price together with its trading margin. For a fixed margin parameter , the budget dynamics are
Throughout the following, we assume that:
(H1) are Borel measurable, Lipschitz continuous in uniformly in , and of linear growth in ;
(H2) are Borel measurable, Lipschitz continuous in uniformly in , and satisfy
for some constant ;
(H3) and are Borel measurable and of quadratic growth:
Under (H1)–(H2), equation (2) admits a unique strong solution and, for every , equation (3) (whose coefficients are random through and ) admits a unique strong solution , with ; see, e.g., [12]. The processes and are then well defined, continuous and -adapted, and .
The agency aims at jointly maximizing campaign performance and trading revenue, while ensuring that the available budget is efficiently spent over the campaign horizon. For , we define the performance criterion
which is well defined under (H1)–(H3). The function measures the instantaneous performance of the campaign: for instance, corresponds to maximizing the cumulative number of sales, while other choices model cost-per-click or cost-per-acquisition objectives. The terminal function penalizes unspent budget, e.g. , and reflects the practical requirement of spending the campaign budget before the end of the planning horizon. The full-information problem is
Problem (5) is a standard Markovian stochastic control problem in the state variables . It is, however, not the problem faced by the agency: an optimal strategy for (5) generally requires the knowledge of the current price , which is not available in practice. The value will therefore only be used as a benchmark.
Sub-campaigns and approximate problem
To overcome this lack of information, the agency may allocate part of its budget to auxiliary sub-campaigns. This possibility is at the core of the randomization procedure developed in this paper. The purpose of the sub-campaigns is not to optimize their own results, but to generate observations of the market price that will be used to control the main campaign. They are described in more detail in Section 4.
Each sub-campaign faces its own market-clearing price , a continuous, -progressively measurable process obtained as a perturbed version of ; its construction is detailed in § 4.1. On sub-campaign , the agency uses a prescribed -valued randomized bidding process , which is not subject to optimization. In the exploration scheme of § 4.1, the bids are generated independently of the latent price process and of the subcampaign price noises. The reference filtration is understood to include this exogenous randomization. The cumulative winning-time process on sub-campaign is
Our objective is then to construct an approximate solution of (5) from the limited information available, that are and . To this end, we use the sub-campaign observations to build a proxy of the latent price (§ 4.1). Substituting for in the solution of the full-information problem (5) yields a strategy in , whose performance can be compared with the benchmark value .
Price estimation and model calibration
The operational model raises two distinct estimation questions. The current market price must be inferred from censored auction outcomes, while the dynamics of prices and campaign performance have to be fitted to historical data. We address these questions separately, starting with the information obtained from randomized exploratory bids. This separation makes the resulting method tractable but does not optimize exploration and exploitation jointly.
Exploration through randomized bids
Consider exploratory subcampaigns with a common latent market factor . At each observation time, the price of subcampaign is
where is known and the noises are conditionally independent across subcampaigns. This is a stylised cross-sectional sampling model, not an assertion that auction prices are exactly Gaussian in practice.
The main-campaign bids belong to . To respect the same bidding cap during exploration, we draw exploratory bids independently from an exponential distribution conditioned on :
The exploratory bids are independent of and of the price noises. The agency observes the winning indicators, hence the empirical winning fraction
Writing for the standard Gaussian distribution function, the conditional winning probability is
In particular, is a continuous strictly decreasing function from onto , since
where is the standard Gaussian density. The empirical win rate can therefore be inverted to estimate the hidden price. The assumption of an exogenous independent exploration channel is important: it allows elementary statistical estimation, but does not solve the joint exploration–exploitation problem for the full campaign.
Remark 4.1. The mean-square error bound in Lemma 4.1 assumes that the latent price lies almost surely in a fixed compact interval. For the Ornstein–Uhlenbeck model considered in § 7, this assumption does not hold, and the projection in (4.4) introduces a truncation error that is not covered by the bound.
Estimation error and stability
Proposition 4.1. For each fixed with , suppose that the pairs are conditionally i.i.d. given , with the distributions specified above. Then
and in and probability. If the noise scales are measurable and strictly positive, then
In particular, these assertions concern fixed-time and integrated mean-square convergence.
Let be a compact interval containing the latent price. We define the inverse estimator
The projection in (4.4) makes the estimator well-defined even when the empirical winning fraction is zero or one. Since is strictly decreasing for , the inverse is unique; compactness provides a uniform stability bound.
Lemma 4.1. Suppose almost surely, and
Then
If , then .
Proof. Conditional on , is binomial with success probability , and therefore its conditional variance is at most . Projection onto an interval does not increase distance to a point in that interval. On the inverse of is -Lipschitz. The result follows by conditioning on . □
Proposition 4.2. Fix a strategy that is admissible for the original observation problem. Let and , and define
Let denote the payoff in (4), and let be the corresponding auxiliary payoff with , and replaced by , and (the performance process is held fixed in this comparison). Suppose the estimates in Lemma 4.1 hold uniformly in time, , take values in a compact interval , and and are Lipschitz on the relevant ranges. Assume, moreover, that for all small enough
with independent of . Then
where is independent of .
Proof. For each ,
Markov’s inequality, (10) and the mean-square estimator bound imply
Take . Since both price variables remain bounded,
Integrating this inequality and applying the Lipschitz assumptions to and yields the asserted bound. □
Implications for optimal control. Proposition 4.2 concerns a fixed policy satisfying the anti-concentration assumption. It does not provide a uniform estimate over arbitrary observation-adapted policies. Indeed, the choice wins by construction in the proxy model, while its outcome remains uncertain in the true model. Therefore convergence of the price estimator alone does not establish near-optimality of an optimizer of the proxy problem. Such a conclusion requires additional stability conditions on the admissible policy class. In what follows, is used as an observable price signal in a plug-in control approximation. This is not an exact separation of the partially observed problem.
Example 4.1 (A synthetic check of the price estimator). We illustrate the estimator of (9) in a cross-sectional experiment. Fix , , , , and . For each value of , we generated 3000 independent samples of exploratory bids and Gaussian price noises, using the capped exponential law (4.1). Figure 4 displays the empirical mean-square estimation error. The observed decrease is close to , as expected from Lemma 4.1. This numerical check concerns price estimation only, not the performance of a bidding strategy.

Figure 4. Mean-square error of the capped-bid estimator versus sample size, with 3000 independent replications and ; dashed reference line of slope
Forward-equation calibration
We now turn to the coefficients that determine the evolution of the price and campaign-performance processes. Even when contemporaneous prices are censored, historical campaign batches may provide data from which to estimate
As bids affect campaign performance, calibration must take the joint distribution of states and actions into account. The observation or experimental policy must be recorded; otherwise parameters of the performance equation may not be identifiable.
Let
denote the joint density of , provided it exists. For deterministic feedback controls, the conditional law of the action may be singular; the corresponding expressions should then be understood in terms of state–action occupation measures. For any test function , Itô’s formula gives
Writing for the state marginal, the weak forward equation takes the form
Suppose that historical data provide observations
where is reconstructed from the collected auction data. We estimate the marginal densities of and using kernel estimators:
and
Here is a symmetric kernel satisfying
and the bandwidths and may be selected by cross-validation.
We consider parametric families
where is an unknown parameter vector. Calibration amounts to controlling the residual of (4.6), together with the discrepancy between observed and model-implied data. We use physics-informed neural networks to approximate the forward densities and their derivatives.
Let
be collocation points sampled in the state-time domain. The price-density residual is
Similarly, the -marginal density satisfies, assuming vanishing boundary fluxes,
A residual associated with the -marginal equation must integrate over both price and action:
Here estimates the joint state–action occupation density, or an appropriate discretisation of the occupation measure. The time derivatives in the residual require a sufficiently regular interpolation of the empirical densities.
The calibration loss is then
Here measures agreement with observed campaign data.
A small residual of the forward equation does not, by itself, identify the unknown coefficients: different models may yield the same observed marginal distributions. For completeness, the following elementary local stability bound isolates the additional identifiability assumption; its proof is deferred to Appendix § 10.
Proposition 4.3. Let be finite-dimensional, and let denote the model-implied state density as an element of . Suppose is continuously Fréchet differentiable near the true parameter and
for some . Suppose also that the forward equations admit a uniformly bounded evolution family on . For an approximate density with initial data , define the forward residual , where is the forward (Fokker–Planck) operator. Then, locally in ,
The proposition separates two sources of error: approximation of the forward equation and identification of the parameters from the induced densities. The latter depends on the observation design and cannot be inferred from the PDE alone. We now consider the associated optimal control problem.
Example 4.2 (Calibration on synthetic data). Figure 5 illustrates calibration on the seasonal model of Section 7 with assumed historical observation of prices. The plot shows the learning dynamics and the reconstructed coefficients in the experiment.

Figure 5. Parameter trajectories and reconstructed dynamics from the PINN calibration experiment
Model-based control: HJB and policy iteration
We next consider a benchmark problem in which the current market price and the model coefficients are known. This complete-information formulation leads to an HJB equation and provides a reference policy against which an estimated-price strategy may be compared. Under incomplete information, the same numerical method can be used as a plug-in approximation, but it does not give the HJB equation of the original partially observed problem.
Full-information dynamic programming
Let denote the class of admissible policies adapted to the filtration generated by the full state . For this benchmark, define the value function
where denotes expectation conditional on under the policy . This value represents the maximal expected reward over for a given initial state. Under the usual well-posedness and dynamic programming assumptions, the value function satisfies the following principle.
Dynamic programming principle. For any stopping time valued in with respect to the full-information filtration,
where
This is the standard dynamic programming identity for controlled diffusions; see [4] and [12].
The Hamilton–Jacobi–Bellman equation
As in the linear–quadratic benchmark, we define the Hamiltonian using generic arguments and :
Since the Brownian motions driving and are independent, no mixed second-order derivatives appear in (12). Under the usual dynamic programming assumptions, the corresponding HJB equation is
The maximization in (5.1) accounts for the campaign reward, the margin earned on successful transactions, and the effect of expenditure on future value. The gradient and Hessian in (5.2) are evaluated at .
If is smooth, a verification theorem applies, and the pointwise maximum is attained, an optimal feedback control may be chosen from
The terms involving and belong to the full Hamiltonian but do not affect the pointwise maximization because the price dynamics are uncontrolled.
Under incomplete information, however, the relevant state includes the conditional distribution of the hidden price. Because the observation mechanism depends on the policy, this conditional distribution has controlled filtering dynamics. The separated problem is consequently measure-valued. Replacing by in the benchmark equation is an approximation rather than an exact separation principle.
The value function need not be smooth, and the viscosity-solution framework is natural for an HJB equation of this type; see [1]. The discontinuity of the winning indicator, together with possible loss of attainment at a threshold, requires care in establishing a rigorous comparison or verification principle. We do not address this technical question here, and use (13) as the basis for numerical approximation.
Policy iteration with physics-informed neural networks
To translate the control formulation into a bidding rule, one needs to approximate the value function and perform the pointwise maximization in the HJB equation. Grid-based schemes may become expensive for several state variables. We therefore consider a physics-informed neural network (PINN) approximation of the value function.
Let
be a neural-network approximation of , with trainable parameters .
The terminal condition is enforced through the parameterization
where is an unconstrained neural network. A strict budget constraint , if imposed, would also require an appropriate boundary condition at .
We use a generalized policy-iteration procedure, alternating between
policy evaluation, during which the value function is updated for a fixed policy;
policy improvement, during which the policy is updated using the current approximation of the value function.
At each policy-evaluation step, automatic differentiation provides the first- and second-order derivatives entering the linear equation. The policy-improvement step maximizes the Hamiltonian computed from the current network.
Algorithm 1 summarizes the computational steps.
Algorithm 1 Policy iteration with a PINN value-function approximation
Initialize value network
repeat
Policy evaluation
Fix the current policy and solve the corresponding linear policy-evaluation equation approximately with
Sample collocation points
Evaluate the HJB residual using automatic differentiation
Update by stochastic gradient descent
Policy improvement: maximize the Hamiltonian pointwise using the current derivatives of
until convergence
The procedure follows the policy-iteration principle, but viscosity-solution theory alone does not establish convergence of the neural approximation. In particular, approximation, optimization, and policy-improvement errors must be analysed separately. We assess the numerical behavior of the method below.
Policy learning through reinforcement learning
The HJB approach relies on a specified and calibrated dynamics model. A practical alternative is to update bidding policies from observed actions and rewards, without first estimating every model coefficient. We describe this second approach using randomized controls and an entropy-regularized actor–critic approximation. Both methods must respect the same information constraint if their outcomes are to be compared.
Randomized policies and relaxed controls
Instead of selecting a single bid, the controller chooses at time a probability measure
and the bid is sampled according to
The process is progressively measurable and takes values in . The continuous-time model below uses a relaxed (measure-valued) control; averaging the infinitesimal drift and variance should not be confused with replacing the diffusion coefficient by its ordinary average.
For a probability measure , define
and
The relaxed state dynamics are then written as
The expected budget expenditure under the relaxed policy is
Similarly, the average trading revenue is
This formulation is related to entropy-regularized stochastic control and continuous-time reinforcement learning; see, for example, [15], [5], and [6].
Entropy-regularized objective
To favor exploration, consider the entropy penalty associated with a policy density on :
For an entropy temperature (distinct from the exploration rate ), the criterion becomes
where
The parameter determines the relative weight assigned to entropy. Larger values favor more diffuse bidding distributions. Because entropy is differential entropy, its numerical value also depends on the units of the bid; the action range is fixed here.
Actor–critic approximation
For numerical implementation, we use an actor–critic scheme on a time grid . The information fed to the networks must be observable. Let be a feature vector that may contain the estimated price , observed performance and budget, or a finite-dimensional approximation of the conditional price distribution. The actor represents a policy density on (for example, a truncated Gaussian), while the critic approximates its value.
For an action chosen at time , let be the observed reward earned over , including the entropy regularisation. The terminal condition is , with no additional terminal term in . Define the temporal-difference innovation
For an exact continuation value and a sufficient observation state, the temporal-difference increment has zero conditional mean. In a finite-dimensional feature approximation one instead seeks approximately
One possible critic loss is therefore the squared conditional expectation
With an advantage estimator , the actor update takes the form
This condition does not require the realised temporal-difference error to vanish pathwise. The above discretisation defines an implementable scheme, but its convergence to a continuous-time optimal policy is not established here.
Unless is a sufficient statistic for the observation history, this reduction also introduces a state-approximation error. Algorithm 2 summarizes the implementation.
Algorithm 2 Discrete actor–critic approximation of a continuous-time control problem
Initialize actor parameters and critic parameters
repeat
Simulate trajectories under the randomized policy
Compute rewards and budget updates
Update the critic by minimizing the temporal-difference residual
Update the actor using the entropy-regularized policy gradient
until convergence
return Learned randomized policy
Unlike the HJB-based strategy, this approach does not require an explicit parameterization of the drift and volatility coefficients. It does, however, require sufficiently informative observations and rewards. We compare its numerical behavior with that of the model-based approximation in § 7.
Numerical experiments and operational interpretation
Experimental design
We consider the following seasonal diffusion model, together with a performance criterion consistent with the payment rule of § 3:
The quadratic bidding penalty is included in the running reward. Since the Ornstein–Uhlenbeck price is not almost surely positive, a positivity-preserving specification or a documented truncation would be needed for a literal auction-price interpretation. The parameters used in the experiments are collected in Table 1.
| Parameter | Value |
| 2.0 | |
| 0.5 | |
| 0.3 | |
| 0.3 | |
| 1.0 | |
| 0.4 | |
| 0.1 | |
| (seasonal frequency) | 1.0 |
| 0.5 | |
| 0.2 | |
| 1.0 | |
| 3.0 | |
| 0.1 |
Table 1. Parameters used in the seasonal-price simulations. is the share of budget allocated to price estimation.
Model-based control with observed and estimated prices
Figure 6 displays the evolution of the PDE residual during the policy-iteration procedure. The full-information benchmark yields the trajectories in Figure 7, illustrating the interaction between the market price, bids, performance, and remaining budget.

Figure 6. PDE residual during the PINN policy-iteration computation

Figure 7. State and bidding trajectories from the full-information PINN implementation
Figure 8 compares the trajectories with and without price estimation, using an exploratory allocation corresponding to 10% of the initial budget.

Figure 8. Estimated-price trajectories: prices, bidding, win frequency, performance, trading margin, and budget
Randomized actor–critic policy
Figure 9 illustrates the evolution of the randomized-policy parameters, together with performance and expenditure under reinforcement learning.

Figure 9. Performance, budget expenditure, and randomized-bid parameters from the actor–critic experiment
A budget-pacing benchmark
To compare the model-based and learning-based policies with a rule closer to operational practice, we consider a budget-pacing heuristic based on a proportional–integral–derivative (PID) controller. The rule does not attempt to solve the optimization problem (5). Its only objective is to keep the expenditure of the campaign close to a prescribed spending schedule.
Let be a deterministic target for the remaining budget, decreasing from at time 0. In the experiments we use uniform pacing,
although a seasonal schedule following the expected price level could be used instead. The tracking error is
A positive error indicates underspending relative to the schedule, and a negative error indicates overspending. Given a baseline bid and nonnegative gains , , , the PID bid is
Underspending therefore raises the bid, which increases the probability of winning auctions, and overspending lowers it. The integral term corrects persistent deviations from the schedule. The derivative term damps the response to rapid changes in expenditure.
Since is absolutely continuous, exists for almost every , but it inherits the discontinuities of the winning indicator. In practice, the rule is implemented on the time grid of § 6.3:
where and is the grid step. When the bid saturates at 0 or , the accumulated error may grow without improving the tracking. A standard anti-windup correction, freezing the integral term during saturation, prevents this effect.
The discrete-time PID rule (14) depends only on the current and past values of the observed remaining budget. It can therefore be implemented without access to the latent market price and requires neither the price estimator nor expenditure on exploratory subcampaigns.
Budget pacing reflects a common practice in media trading: ensuring that the client’s budget is spent regularly over the campaign horizon. The rule is transparent, easy to tune, and robust to misspecification of the price dynamics. Its limitations follow from the same simplicity. It ignores the market price, campaign performance and the agency margin, and it does not account for the quadratic bidding cost in the criterion. In particular, it may win many auctions at unfavorable prices, which increases performance but not necessarily the value of (4). The rule is therefore a useful operational benchmark rather than a competing solution of the control problem. The gains and the baseline bid used in the experiments are given in Table 2.
| Parameter | Value |
| 0.5 | |
| 2.0 | |
| 4.0 | |
| 0.05 | |
| Time step | |
| Bounds |
Table 2. Parameters of the PID budget-pacing rule
Outcomes and comparison
Figure 10 gathers the outcomes of five bidding rules: a finite-difference approximation of the HJB benchmark and the GPI–PINN policy, both with observed prices; the GPI–PINN feedback evaluated at the estimated price ; the actor–critic policy of § 6, fed with the same estimated price; and a PID budget-pacing rule. The PID rule adjusts the bid so that cumulative expenditure tracks a prescribed spending schedule. It uses neither the value function nor the price estimator, and serves as an operational heuristic rather than as a solution of (5).

Figure 10. Performance comparison between the different methods on 1000 common trajectories
The three panels display the mean remaining budget, the mean cumulative performance, and the empirical distribution of the realized criterion. Several features are visible. First, the policies based on the estimated price start from a budget of 0.9, since 10% of the initial budget is allocated to the exploratory subcampaigns. Second, the observed-price policies spend their budget at a comparable rate and end the horizon with approximately one third of the initial budget. The estimated-price GPI–PINN policy spends more slowly and ends with roughly half of it. Third, the PID rule produces the largest cumulative performance, whereas the estimated-price GPI–PINN policy produces the smallest.
The distribution of the realized criterion reverses this ordering. The GPI–PINN value with observed prices exceeds the other values, whereas the PID rule yields more sales together with a markedly lower and more dispersed criterion. This is consistent with the structure of the objective: the PID rule tracks expenditure without accounting for the quadratic bidding cost or for the margin earned on each transaction. The policies based on the price signal lie between these two cases.
The gap between the observed-price and estimated-price GPI–PINN policies has two distinct sources. The first is the exploration cost, which reduces the budget available to the main campaign. The second is the threshold effect discussed after Proposition 4.2. A feedback calibrated to bid close to the estimated price wins with high probability in the proxy model, but only with probability close to one half in the true model when is approximately centred. Figure 8 shows this reduction in the win rate. It explains both the slower expenditure and the lower performance of the estimated-price policy. Consequently, the observed gap measures the cost of using a plug-in policy, not only the cost of estimating the price.
Discussion and conclusion
We have formulated media trading as a dynamic bidding and budget-allocation problem in which the agency sees market-clearing prices only after successful auctions. The model makes the three components of the agency’s decision explicit: campaign performance, trading margin and the use of the client’s remaining budget. An additional feature is the endogenous information structure, as the bids determine which clearing prices are observed.
Randomized exploratory subcampaigns provide a simple statistical instrument for recovering a price signal. Under the specified cross-sectional model, the estimation error admits a mean-square bound of order . We then consider two ways to turn price and performance information into bidding policies: a calibrated stochastic model with an HJB/PINN approximation, and an entropy-regularized actor–critic approach. These methods offer complementary computational routes rather than new results in stochastic control theory.
Operational implications. The formulation highlights why good bidding decisions cannot be judged by conversions alone. A policy generating more sales may also erode the agency’s margin or exhaust the client’s budget too early. Likewise, exploration has a potential value through improved price information but entails a cost in advertising expenditure. These trade-offs motivate the use of a common, transparent objective when evaluating competing policies.
Limitations and outlook. Our estimator result relies on an independent exploratory sampling scheme and does not establish optimality for control with censored data. The HJB equation is a full-information benchmark, and substituting an estimated price leads to a plug-in policy. The exploratory budget is not jointly optimized with the main campaign.
Expression of as a diffusion process
Let and suppose . Let be a possibly random family of probability measures with on . Suppose is in and absolutely continuous with finite variation in time,
with sufficient integrability to apply the Itô formula for a random finite-variation field. Define the randomized-policy winning probability
Since and , Itô’s formula yields
The second-order term therefore has a negative sign. Because is strictly positive on the price interval, is strictly decreasing on that interval. For let denote its inverse. Then and (A.1) can be rewritten entirely in terms of . Atomlessness alone is not sufficient for invertibility (a non-atomic probability density may vanish on an interval). Outside the support of , equals zero or one, so the latent price cannot be recovered by inversion. The finite-variation condition is important here: if the policy distribution had a martingale component, the formula would include additional Itô–Wentzell terms.
Proof of the Calibration Consistency Bound
We prove Proposition 4.3. Denote by the parameter-to-density map, with values in . By Fréchet differentiability,
The assumed injective derivative with a quantitative lower bound implies, for in a sufficiently small neighborhood,
The estimate is local and relies on the quantitative injectivity assumption.
Let denote the forward evolution family of , assumed uniformly bounded by on . Duhamel’s formula for the approximate density gives
By Cauchy–Schwarz and the uniform bound on ,
Finally,
and the claimed estimate follows. For controlled dynamics, the forward operator must incorporate the experimental policy or the corresponding state–action occupation measure. If the parameter-to-density map is not locally identifiable, the same argument does not yield parameter consistency.
Convergence of the Randomized Observation Process
Fix and condition on . By the conditional independence of exploratory bids and price noises, the variables are independent Bernoulli variables with success probability given by (7). Consequently,
Taking expectations gives the fixed-time bound. Integration and Tonelli’s theorem yield the integrated result without any temporal independence assumption.
To establish strict monotonicity, differentiate the Gaussian distribution function under the bounded-action integral in (7). This gives (8). On a compact price interval , the derivative is bounded away from zero, and the clipped inverse in (9) is Lipschitz. The price-estimation bound then follows as in Lemma 4.1.
We do not claim convergence in a process topology. A continuous-time interpretation of the threshold-based winning indicators requires an explicit temporal sampling model.
References
- [1]Michael G Crandall, Hitoshi Ishii, and Pierre-Louis Lions. User’s guide to viscosity solutions of second order partial differential equations. Bulletin of the American mathematical society, 27(1):1–67, 1992.arxiv.org/abs/math/9207212
- [2]Joaquin Fernandez-Tapia, Olivier Guéant, and Jean-Michel Lasry. Optimal real-time bidding strategies. Applied Mathematics Research Express, 2017(1):142–183, 2017.
- [3]Arthur Flajolet and Patrick Jaillet. Real-time bidding with side information. http://web.mit.edu/jaillet/www/general/bidding-nips-17.pdf, 2017.
- [4]Wendell H. Fleming and H. Mete Soner. Controlled Markov Processes and Viscosity Solutions, volume 25 of Stochastic Modelling and Applied Probability. Springer, second edition, 2006.DOI
- [5]Yanwei Jia and Xun Yu Zhou. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach. Journal of Machine Learning Research, 23(154):1–55, 2022.arxiv.org/abs/2108.06655
- [6]Yanwei Jia and Xunyu Zhou. q-Learning in Continuous Time. Journal of Machine Learning Research, 24(161):1–61, 2023.arxiv.org/abs/2207.00713
- [7]Randall Lewis and Justin Rao. The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4):1941–1973, 2015.DOI
- [8]John D. C. Little. Aggregate advertising models: the state of the art. Operations Research, 27(4):629–667, 1979.
- [9]Kaito Majima, Kosuke Kawakami, Kota Ishizuka, and Kazuhide Nakata. Keyword-level bayesian online bid optimization for sponsored search advertising. Operations Research Forum, 5(45), 2024.
- [10]Médéric Motte and Huyền Pham. Optimal bidding strategies for digital advertising, 2021.arxiv.org/abs/2111.08311
- [11]Thomas Mulc, Mike Anderson, Paul Cubre, Huikun Zhang, Ivy Liu, and Saket Kumar. Nnn: Next-generation neural networks for marketing measurement, 2025.arxiv.org/abs/2504.06212
- [12]Huyền Pham. Continuous-time stochastic control and optimization with financial applications, volume 61 of Stochastic Modelling and Applied Probability. Springer-Verlag Berlin Heidelberg, 2009.DOI
- [13]Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.DOI
- [14]Michele Ricciardi and Mauro Rosestolato. Mean field games incorporating carryover effects: Optimizing advertising models. Decisions in Economics and Finance, 49:475–495, 2026.
- [15]Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 21(198):1–34, 2020.
- [16]Wush Chi-Hsuan Wu, Mi-Yen Yeh, and Ming-Syan Chen. Predicting winning price in real time bidding with censored data. http://www0.cs.ucl.ac.uk/staff/w.zhang/rtb-papers/win-price-pred.pdf, 2015.DOI
- [17]Weinan Zhang, Shuai Yuan, and Jun Wang. Optimal real-time bidding for display advertising. In KDD ’14: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1077–1086, 2014.DOI
- [18]Yingxiang Zhang, Mike Wurm, Alexander Wakim, Eddie Li, and Ying Liu. Bayesian hierarchical media mix model incorporating reach and frequency data. research.google, 2023.
- [19]Jun Zhao et al. Deep reinforcement learning for sponsored search real-time bidding. https://arxiv.org/abs/1803.00259v1, 2018. Accessed: 1 Mar. 2018.
- [20]Hubert A. Zielske. The remembering and forgetting of advertising. Journal of marketing, 1959.DOI
- [21]Hubert A. Zielske and Walter A. Henry. Remembering and forgetting television ads. Journal of Advertising Research, 1980.DOI