betaFeedback

WorksEconomics, Econometrics and Finance

Media Trading with Censored Auction Prices: A Stochastic Control and Reinforcement Learning Approach

Abstract

In real-time bidding, marketing agencies must balance campaign performance, transaction margins and the use of a finite advertising budget. A key difficulty is that market-clearing prices are typically observed only when bids are successful, so the information available for subsequent decisions depends on the bidding policy. We formulate this problem using a continuous-time stochastic control model with a latent price process. To construct a price signal, we consider parallel exploratory subcampaigns in which bids are randomized subject to an upper bound. Under a stylized cross-sectional sampling model, the resulting price estimator is consistent and satisfies a mean-square error bound of order N−1N^{-1}, where NN is the number of exploratory observations. We then describe a model-based approach combining forward-equation calibration with Hamilton-Jacobi-Bellman policy iteration and a physics-informed neural network, alongside an entropy-regularized actor-critic approach based on observable information in a model-free setting. Synthetic experiments illustrate the estimator and the computational procedures. The framework clarifies the decision and information trade-offs faced by the agency, while distinguishing full-information benchmarks and estimated-price policies from the genuinely partially observed control problem.

MSC Classification: 90B60; 93E20; 93E35.

Introduction

Real-time bidding (RTB) has made advertising inventory a resource that must be purchased through frequent, automated auctions. For a marketing agency, the decision is not simply how much to bid on an individual impression: the bids determine campaign performance, the revenue associated with media trading, and the depletion of the client’s budget over the campaign horizon. Market prices and campaign outcomes are uncertain, and the agency must adjust its decisions as information arrives. This makes media trading a dynamic resource-allocation problem with an important operational constraint: the agency may not observe the clearing prices of auctions that it loses.

The observation constraint changes the nature of the optimization. A high bid is more likely to generate an impression and reveal the clearing price, but may also consume more of the available budget. A low bid preserves expenditure yet produces less information about the market. Thus, data collection cannot be treated independently of the bidding policy. Figure 1 summarizes the information and decision flow.

Main participants in a programmatic advertising auction

Figure 1. Main participants in a programmatic advertising auction.

The problem has two complementary aspects. From a modeling perspective, one needs to represent the market price, campaign performance and remaining budget, together with an observation mechanism that depends on the bid. From a computational perspective, the agency needs implementable policies despite imperfect information and uncertain dynamics. Stochastic control provides a natural description of the intertemporal objectives and constraints; reinforcement learning offers an alternative route to policy optimization when the dynamics are not fully calibrated.

Related work. Earlier studies of advertising effectiveness include the analysis of exposure and memory [20, 21], aggregate advertising models [8], and media-mix measurement [18]. Uncertainty in attributing advertising outcomes remains a major practical difficulty [7, 11].

Auction-based advertising has been studied using constrained optimization [17], online learning with side information [3], and deep reinforcement learning [19]. Continuous-time stochastic control approaches to bidding have been proposed by [2] and [10]; mean-field models also describe aggregate advertising effects [14]. The statistical problem of estimating auction-clearing prices from censored observations is related to [16].

Of particular relevance to the operational setting is the recent contribution in [9]. They study keyword-level bid optimization in sponsored-search advertising using limited aggregate feedback, a Bayesian model of campaign metrics, and a bandit-based online optimization method. Their study includes simulations using real advertising data. The present work addresses a different decision problem: continuous-time bidding for a media-trading agency, explicit censoring of clearing prices by auction outcomes, and a balance between performance, transaction margins and remaining budget. Accordingly, the informational assumptions, decision frequency, and objectives differ; we do not claim a direct empirical comparison with their method.

Modeling and methodological contributions. We first set out an agency-level stochastic decision model in which the market price is latent and bidding affects the information revealed by transactions. To construct a usable price signal, we consider randomized bids in parallel exploratory subcampaigns. In an explicit cross-sectional observation model, the winning frequency leads to a consistent estimator with a finite-sample mean-square error bound.

We next connect this price signal to two computational approaches. The first combines forward-equation calibration with a full-information Hamilton–Jacobi–Bellman (HJB) benchmark, approximated through policy iteration and physics-informed neural networks (PINNs) [13]. The second describes an entropy-regularized actor–critic method, defined on observable features and inspired by continuous-time reinforcement learning [5, 6, 15]. These are established numerical ingredients; the interest of combining them here is to clarify their use and limitations for the media-trading decision problem. In particular, the estimated-price feedback is a plug-in strategy, not the solution of the underlying partially observed control problem.

The paper is organized as follows. Section 2 describes the RTB decision problem and a simple benchmark. Section 3 introduces the stochastic model and the observation structure. Estimation and calibration are considered in Section 4. Sections 5 and 6 present the HJB-based and reinforcement-learning approaches. Section 7 discusses the numerical experiments and their limitations, before the concluding section.

The media trading decision problem

We begin with the agency’s decisions, observations and budget accounting, then use an elementary linear–quadratic example to illustrate how the auction-winning constraint enters the optimization. The example is a full-information benchmark; the incomplete-information model follows in the next section.

Auction mechanism and information

Consider an advertising campaign over a finite horizon [0,T][0,T]. The agency receives an initial budget from its client and submits bids to a sequence of RTB auctions through a demand-side platform.

At time tt, the agency chooses a bid αt\alpha_t. An auction is won if αt≥St\alpha_t \ge S_t, where StS_t denotes the market-clearing price. A successful bid contributes to campaign performance and consumes part of the budget.

The agency observes the clearing price only when it wins. Its observation history thus depends on the previous bids, and the decision problem combines partial observation with control-dependent information acquisition.

Following the continuous-time approaches of [2] and [10], we use a diffusion approximation to represent the frequent auctions and the resulting campaign dynamics.

We use the following variables:

  • StS_t, the latent market-clearing price;

  • XtX_t, the campaign performance (for instance the cumulative number of conversions or sales);

  • YtY_t, the remaining campaign budget;

  • αt\alpha_t, the bid, which is the control rather than a state variable.

The trade-offs have a direct operational interpretation. Raising the bid increases the chance of obtaining an impression, while successful transactions affect both campaign value and the agency’s margin. The remaining budget couples today’s bid with tomorrow’s opportunities. Moreover, a lost auction reveals less price information than a won auction. We first examine these mechanisms in a simple complete-information model; a joint optimization of exploration expenditure is beyond the scope of the present framework.

A linear–quadratic benchmark

As an elementary illustration, we first consider a first-price stylization in which a winning bid is paid at its submitted value. Unlike the market-clearing-price settlement in the general model below, this convention leads to a particularly transparent quadratic maximization. Suppose that the market price is fully observed and follows the Ornstein–Uhlenbeck dynamics

dSt=μ(Sˉ−St) dt+σ dWt1dS_t = \mu(\bar{S} - S_t)\,dt + \sigma\,dW_t^1

The agency controls its bidding price αt∈A=[0,Amax⁡]\alpha_t \in A = [0,A_{\max}]. An auction is won whenever

αt≥St.\alpha_t \ge S_t.

Campaign performance evolves according to

{dXt=γ 1{αt≥St} dt+θ dWt2,X0=0,\begin{cases} dX_t = \gamma\,\mathbf{1}_{\{\alpha_t\ge S_t\}}\,dt + \theta\,dW_t^2,\\ X_0 = 0, \end{cases}

where γ\gamma measures the average increase in performance generated by a successful auction. The remaining budget satisfies

{dYt=−(1+β)αt1{αt≥St} dt,Y0=B,\begin{cases} dY_t = -(1+\beta)\alpha_t\mathbf{1}_{\{\alpha_t\ge S_t\}}\,dt,\\ Y_0 = B, \end{cases}

where β>0\beta> 0 denotes the agency margin.

The agency seeks to maximize the performance criterion

J(α)=E[∫0T(Xt+βαt1{αt≥St}−κ(1+β)2αt21{αt≥St}) dt−YT2].J(\alpha) = \mathbb{E}\left[\int_0^T \left(X_t+\beta\alpha_t\mathbf{1}_{\{\alpha_t\ge S_t\}}-\kappa(1+\beta)^2\alpha_t^2\mathbf{1}_{\{\alpha_t\ge S_t\}}\right)\,dt-Y_T^2\right].

The running reward combines campaign performance, the margin obtained from successful bids, and a quadratic penalty on the bidding cost. The terminal term penalizes any deviation of the final budget from zero, including an overdraft. This is a soft penalty, not a hard budget constraint.

Let p=(ps,px,py)∈R3p=(p_s,p_x,p_y)\in\mathbb{R}^3 and M=(Mij)i,j∈{s,x,y}∈S3M=(M_{ij})_{i,j\in\{s,x,y\}}\in\mathbb{S}^3 be generic variables. The (maximized) Hamiltonian of this benchmark is defined by

HLQ(s,x,y,p,M)=μ(Sˉ−s)ps+σ22Mss+θ22Mxx+x+sup⁡a∈[0,Amax⁡]1{a≥s}[γpx+(β−(1+β)py)a−κ(1+β)2a2].(1)\begin{aligned} H_{\mathrm{LQ}}(s,x,y,p,M)={}&\mu(\bar{S}-s)p_s+\frac{\sigma^2}{2}M_{ss}+\frac{\theta^2}{2}M_{xx}+x\\ &+\sup_{a\in[0,A_{\max}]}\mathbf{1}_{\{a\ge s\}}\left[\gamma p_x+\left(\beta-(1+\beta)p_y\right)a-\kappa(1+\beta)^2a^2\right]. \tag*{(1)} \end{aligned}

Here S3\mathbb{S}^3 is the space of symmetric 3×33\times3 matrices. The state variable yy does not appear explicitly in the present coefficients, but is retained as an argument of the Hamiltonian. For a smooth value function VV, the arguments (p,M)(p,M) in (2.1) are replaced by (DV,D2V)(DV,D^2V), where DV=(Vs,Vx,Vy)DV=(V_s,V_x,V_y). The HJB equation therefore reads

{Vt+HLQ(s,x,y,DV,D2V)=0,V(T,s,x,y)=−y2.\begin{cases} V_t+H_{\mathrm{LQ}}(s,x,y,DV,D^2V)=0,\\ V(T,s,x,y)=-y^2. \end{cases}

The optimization over the bid can be performed explicitly, taking the winning constraint into account. For generic p∈R3p\in\mathbb{R}^3, set

Qp(a)=γpx+(β−(1+β)py)a−κ(1+β)2a2,a0(p)=β−(1+β)py2κ(1+β)2.\begin{aligned} Q_p(a)&=\gamma p_x+\left(\beta-(1+\beta)p_y\right)a-\kappa(1+\beta)^2a^2,\\ a_0(p)&=\frac{\beta-(1+\beta)p_y}{2\kappa(1+\beta)^2}. \end{aligned}

If s≤Amax⁡s\le A_{\max}, the best bid conditional on winning is

awin(s,p)=proj⁡[max⁡{0,s},Amax⁡]a0(p).a_{\mathrm{win}}(s,p)=\operatorname{proj}_{[\max\{0,s\},A_{\max}]}a_0(p).

Writing B(s,x,p,M)=μ(Sˉ−s)ps+σ22Mss+θ22Mxx+xB(s,x,p,M)=\mu(\bar{S}-s)p_s+\frac{\sigma^2}{2}M_{ss}+\frac{\theta^2}{2}M_{xx}+x and awin=awin(s,p)a_{\mathrm{win}}=a_{\mathrm{win}}(s,p), we obtain

HLQ(s,x,y,p,M)=B(s,x,p,M)+{0,s>Amax⁡,Qp(awin),s≤0,max⁡{0,Qp(awin)},0<s≤Amax⁡.H_{\mathrm{LQ}}(s,x,y,p,M)=B(s,x,p,M)+ \begin{cases} 0, & s>A_{\max},\\ Q_p(a_{\mathrm{win}}), & s\le0,\\ \max\{0,Q_p(a_{\mathrm{win}})\}, & 0<s\le A_{\max}. \end{cases}

For 0<s≤Amax⁡0<s\le A_{\max}, the optimal bid wins the auction only if Qp(awin)≥0Q_p(a_{\mathrm{win}})\ge0; otherwise any bid below ss avoids the transaction. If s>Amax⁡s>A_{\max}, no admissible bid wins the auction. In these formulas the generic vector pp is evaluated at DV(t,s,x,y)D V(t,s,x,y) when defining the optimal feedback; the matrix MM is evaluated at D2VD^2V in the HJB equation.

Bidding policy from the benchmark implementation

Figure 2. Bidding policy from the benchmark implementation

Benchmark trajectories

Figure 3. Benchmark trajectories

The preceding calculations rely on observing (St)(S_t) continuously. We now turn to the actual information structure, in which the market price is hidden except for the transactions won by the agency.

Stochastic dynamics and information structure

We now introduce the general stochastic control framework studied in the remainder of the paper. Contrary to the benchmark model of § 2, we allow for general nonlinear dynamics of the market-clearing price and of the campaign performance. The presentation proceeds in three steps: we first formulate the problem under full information, that is, as if the agency had access to the whole filtration F\mathcal{F} (§ 3.1); we then explain how the agency can enrich this information by running auxiliary sub-campaigns, and we state the problem addressed in this paper: constructing an approximate solution of the control problem from these limited observations (§ 3.2).

The full-information model

Let (Ω,F,F=(Ft)t∈[0,T],P)(\Omega,\mathcal{F},\mathbb{F}=(\mathcal{F}_t)_{t\in[0,T]},\mathbb{P}) be a filtered probability space satisfying the usual conditions and supporting two independent one-dimensional F\mathcal{F}-Brownian motions W1W^1 and W2W^2. The filtration F\mathcal{F} represents the full information of the model: it contains every source of randomness affecting the market and the campaign, including quantities that the agency does not observe. It serves as a reference framework, in which all processes below are defined; the information actually available to the agency is described in § 4.1.

The latent market-clearing price S=(St)t∈[0,T]S=(S_t)_{t\in[0,T]} is the solution of

dSt=μ(t,St) dt+σ(t,St) dWt1,S0=s0,(2)dS_t=\mu(t,S_t)\,dt+\sigma(t,S_t)\,dW_t^1,\qquad S_0=s_0, \tag*{(2)}

where s0s_0 is an F0\mathcal{F}_0-measurable, square-integrable random variable. The functions μ\mu and σ\sigma describe the local trend and the volatility of the market-clearing price. The price is exogenous: the agency’s bids are assumed to be too small, relative to the size of the market, to affect SS.

The agency controls its bidding strategy through a process α=(αt)t∈[0,T]\alpha=(\alpha_t)_{t\in[0,T]} taking values in the compact set

A=[0,Amax⁡],Amax⁡>0.A=[0,A_{\max}],\qquad A_{\max}>0.

In the full-information framework, the set of bidding strategies is

AF={α=(αt)t∈[0,T]: α is F-progressively measurable and A-valued}.\mathcal{A}^{\mathcal{F}}=\left\{\alpha=(\alpha_t)_{t\in[0,T]}:\ \alpha\text{ is }\mathbb{F}\text{-progressively measurable and }A\text{-valued}\right\}.

Since AA is bounded, every α∈AF\alpha\in\mathcal{A}^{\mathcal{F}} satisfies E[∫0Tαt2 dt]<∞\mathbb{E}\left[\int_0^T\alpha_t^2\,dt\right]<\infty, so that no additional integrability condition is required. All processes introduced below depend on the chosen strategy α\alpha, which we indicate by a superscript.

At time tt, the auction is won if and only if the submitted bid exceeds the market-clearing price, that is, on the event {αt≥St}\{\alpha_t\ge S_t\}. The cumulative number of auctions won is thus given by

Ztα=∫0t1{αu≥Su} du,t∈[0,T].Z_t^\alpha=\int_0^t\mathbf{1}_{\{\alpha_u\ge S_u\}}\,du,\qquad t\in[0,T].

The performance process Xα=(Xtα)t∈[0,T]X^\alpha=(X_t^\alpha)_{t\in[0,T]} of the campaign represents, depending on the application, the cumulative number of clicks, conversions or sales generated by the campaign. It evolves according to

dXtα=γ(t,Xtα,St,αt) dt+θ(t,Xtα,St,αt) dWt2,X0α=x0∈R.(3)dX_t^\alpha=\gamma(t,X_t^\alpha,S_t,\alpha_t)\,dt+\theta(t,X_t^\alpha,S_t,\alpha_t)\,dW_t^2,\qquad X_0^\alpha=x_0\in\mathbb{R}. \tag*{(3)}

The drift and diffusion coefficients may depend simultaneously on the current market conditions and on the bid selected by the agency.

The agency starts the campaign with an initial budget B0>0B_0 > 0, and Yα=(Ytα)t∈[0,T]Y^\alpha= (Y^\alpha_t)_{t\in[0,T]} denotes the remaining budget. Whenever an auction is won, the agency pays the market-clearing price together with its trading margin. For a fixed margin parameter β>0\beta> 0, the budget dynamics are

Ytα=B0−(1+β)∫0tSu1{αu≥Su} du,t∈[0,T].Y^\alpha_t = B_0 - (1+\beta)\int_0^t S_u \mathbf{1}_{\{\alpha_u\ge S_u\}}\,du,\qquad t\in[0,T].

Throughout the following, we assume that:

  • (H1) μ,σ:[0,T]×R→R\mu,\sigma:[0,T]\times\mathbb{R}\to\mathbb{R} are Borel measurable, Lipschitz continuous in ss uniformly in tt, and of linear growth in ss;

  • (H2) γ,θ:[0,T]×R×R×A→R\gamma,\theta:[0,T]\times\mathbb{R}\times\mathbb{R}\times A\to\mathbb{R} are Borel measurable, Lipschitz continuous in xx uniformly in (t,s,a)(t,s,a), and satisfy

    ∣γ(t,x,s,a)∣+∣θ(t,x,s,a)∣≤C(1+∣x∣+∣s∣)|\gamma(t,x,s,a)|+|\theta(t,x,s,a)|\le C(1+|x|+|s|)

    for some constant C>0C>0;

  • (H3) f:A×R×R→Rf:A\times\mathbb{R}\times\mathbb{R}\to\mathbb{R} and g:R→Rg:\mathbb{R}\to\mathbb{R} are Borel measurable and of quadratic growth:

    ∣f(a,s,x)∣≤C(1+∣s∣2+∣x∣2)and∣g(y)∣≤C(1+∣y∣2).|f(a,s,x)|\le C(1+|s|^2+|x|^2)\quad\text{and}\quad|g(y)|\le C(1+|y|^2).

Under (H1)–(H2), equation (2) admits a unique strong solution and, for every α∈AF\alpha\in\mathcal{A}^{\mathrm{F}}, equation (3) (whose coefficients are random through SS and α\alpha) admits a unique strong solution XαX^\alpha, with E[sup⁡t≤T(∣St∣2+∣Xtα∣2)]<∞\mathbb{E}[\sup_{t\le T}(|S_t|^2+|X^\alpha_t|^2)]<\infty; see, e.g., [12]. The processes ZαZ^\alpha and YαY^\alpha are then well defined, continuous and F\mathbb{F}-adapted, and E[∣YTα∣2]<∞\mathbb{E}[|Y^\alpha_T|^2]<\infty.

The agency aims at jointly maximizing campaign performance and trading revenue, while ensuring that the available budget is efficiently spent over the campaign horizon. For α∈AF\alpha\in\mathcal{A}^{\mathrm{F}}, we define the performance criterion

J(α)=E[∫0T(βSt1{αt≥St}+f(αt,St,Xtα))dt−g(YTα)].(4)J(\alpha)=\mathbb{E}\left[\int_0^T\left(\beta S_t\mathbf{1}_{\{\alpha_t\ge S_t\}}+f(\alpha_t,S_t,X^\alpha_t)\right)dt-g(Y^\alpha_T)\right]. \tag*{(4)}

which is well defined under (H1)–(H3). The function ff measures the instantaneous performance of the campaign: for instance, f(a,s,x)=xf(a,s,x)=x corresponds to maximizing the cumulative number of sales, while other choices model cost-per-click or cost-per-acquisition objectives. The terminal function gg penalizes unspent budget, e.g. g(y)=y2g(y)=y^2, and reflects the practical requirement of spending the campaign budget before the end of the planning horizon. The full-information problem is

VF=sup⁡α∈AFJ(α).(5)V^{\mathrm{F}}=\sup_{\alpha\in\mathcal{A}^{\mathrm{F}}}J(\alpha). \tag*{(5)}

Problem (5) is a standard Markovian stochastic control problem in the state variables (S,X,Y)(S,X,Y). It is, however, not the problem faced by the agency: an optimal strategy for (5) generally requires the knowledge of the current price StS_t, which is not available in practice. The value VFV^{\mathrm{F}} will therefore only be used as a benchmark.

Sub-campaigns and approximate problem

To overcome this lack of information, the agency may allocate part of its budget to NN auxiliary sub-campaigns. This possibility is at the core of the randomization procedure developed in this paper. The purpose of the sub-campaigns is not to optimize their own results, but to generate observations of the market price that will be used to control the main campaign. They are described in more detail in Section 4.

Each sub-campaign i∈{1,…,N}i \in\{1,\ldots,N\} faces its own market-clearing price Si=(Sti)t∈[0,T]S^i=(S^i_t)_{t\in[0,T]}, a continuous, F\mathbb{F}-progressively measurable process obtained as a perturbed version of SS; its construction is detailed in § 4.1. On sub-campaign ii, the agency uses a prescribed AA-valued randomized bidding process αi=(αti)t∈[0,T]\alpha^i=(\alpha^i_t)_{t\in[0,T]}, which is not subject to optimization. In the exploration scheme of § 4.1, the bids are generated independently of the latent price process SS and of the subcampaign price noises. The reference filtration F\mathbb{F} is understood to include this exogenous randomization. The cumulative winning-time process on sub-campaign ii is

Zti=∫0t1{αui≥Sui} du,t∈[0,T].Z^i_t=\int_0^t \boldsymbol{1}_{\{\alpha^i_u\ge S^i_u\}}\,du,\qquad t\in[0,T].

Our objective is then to construct an approximate solution of (5) from the limited information available, that are XαX^\alpha and (Zi)i∈N(Z^i)_{i\in\mathbb{N}}. To this end, we use the sub-campaign observations to build a proxy S^\widehat{S} of the latent price SS (§ 4.1). Substituting S^\widehat{S} for SS in the solution of the full-information problem (5) yields a strategy in AA, whose performance can be compared with the benchmark value VFV^{\mathrm{F}}.

Price estimation and model calibration

The operational model raises two distinct estimation questions. The current market price must be inferred from censored auction outcomes, while the dynamics of prices and campaign performance have to be fitted to historical data. We address these questions separately, starting with the information obtained from randomized exploratory bids. This separation makes the resulting method tractable but does not optimize exploration and exploitation jointly.

Exploration through randomized bids

Consider NN exploratory subcampaigns with a common latent market factor SS. At each observation time, the price of subcampaign ii is

Sti=St+εti,εti∼N(0,vt2),S^i_t=S_t+\varepsilon^i_t,\qquad\varepsilon^i_t\sim N(0,v_t^2),

where vt>0v_t>0 is known and the noises are conditionally independent across subcampaigns. This is a stylised cross-sectional sampling model, not an assertion that auction prices are exactly Gaussian in practice.

The main-campaign bids belong to A=[0,Amax⁡]A=[0,A_{\max}]. To respect the same bidding cap during exploration, we draw exploratory bids independently from an exponential distribution conditioned on [0,Amax⁡][0,A_{\max}]:

qλ(a)=λe−λa1−e−λAmax⁡1[0,Amax⁡](a),λ>0.(6)q_\lambda(a)=\frac{\lambda e^{-\lambda a}}{1-e^{-\lambda A_{\max}}}\boldsymbol{1}_{[0,A_{\max}]}(a),\qquad\lambda>0. \tag*{(6)}

The exploratory bids are independent of SS and of the price noises. The agency observes the winning indicators, hence the empirical winning fraction

ztN=1N∑i=1N1{αti≥St+εti}.z^N_t=\frac{1}{N}\sum_{i=1}^{N}\boldsymbol{1}_{\{\alpha^i_t\ge S_t+\varepsilon^i_t\}}.

Writing Φ\Phi for the standard Gaussian distribution function, the conditional winning probability is

ht(s):=P(αti≥s+εti∣St=s)=∫0Amax⁡qλ(a)Φ(a−svt) da.(7)h_t(s):=\mathbb{P}(\alpha^i_t\ge s+\varepsilon^i_t\mid S_t=s)=\int_0^{A_{\max}}q_\lambda(a)\Phi\left(\frac{a-s}{v_t}\right)\,da. \tag*{(7)}

In particular, hth_t is a continuous strictly decreasing function from R\mathbb{R} onto (0,1)(0,1), since

ht′(s)=−1vt∫0Amax⁡qΛ(a)ϕ(a−svt) da<0,(8)h_t'(s)=-\frac{1}{v_t}\int_0^{A_{\max}}q_\Lambda(a)\phi\left(\frac{a-s}{v_t}\right)\,da<0, \tag*{(8)}

where ϕ\phi is the standard Gaussian density. The empirical win rate can therefore be inverted to estimate the hidden price. The assumption of an exogenous independent exploration channel is important: it allows elementary statistical estimation, but does not solve the joint exploration–exploitation problem for the full campaign.

Remark 4.1. The mean-square error bound in Lemma 4.1 assumes that the latent price lies almost surely in a fixed compact interval. For the Ornstein–Uhlenbeck model considered in § 7, this assumption does not hold, and the projection in (4.4) introduces a truncation error that is not covered by the bound.

Estimation error and stability

Proposition 4.1. For each fixed tt with vt>0v_t>0, suppose that the pairs (αi,ϵti)(\alpha^i,\epsilon^i_t) are conditionally i.i.d. given StS_t, with the distributions specified above. Then

E[ztN∣St]=ht(St),E[∣ztN−ht(St)∣2]≤14N,\mathbb{E}[z_t^N\mid S_t]=h_t(S_t),\qquad\mathbb{E}[|z_t^N-h_t(S_t)|^2]\leq\frac{1}{4N},

and ztN→ht(St)z_t^N\to h_t(S_t) in L2L^2 and probability. If the noise scales vtv_t are measurable and strictly positive, then

E∫0T∣ztN−ht(St)∣2 dt≤T4N.\mathbb{E}\int_0^T|z_t^N-h_t(S_t)|^2\,dt\leq\frac{T}{4N}.

In particular, these assertions concern fixed-time and integrated mean-square convergence.

Let K=[s−,s+]K=[s_-,s_+] be a compact interval containing the latent price. We define the inverse estimator

S^tN=(ht∣K)−1(proj⁡[ht(s+),ht(s−)]ztN).(9)\widehat{S}_t^N=(h_t|_K)^{-1}\left(\operatorname{proj}_{[h_t(s_+),h_t(s_-)]}z_t^N\right). \tag*{(9)}

The projection in (4.4) makes the estimator well-defined even when the empirical winning fraction is zero or one. Since hth_t is strictly decreasing for vt>0v_t>0, the inverse is unique; compactness provides a uniform stability bound.

Lemma 4.1. Suppose St∈KS_t\in K almost surely, and

mt:=inf⁡s∈K∣ht′(s)∣>0.m_t:=\inf_{s\in K}|h_t'(s)|>0.

Then

E∣St−S^tN∣2≤14mt2N.\mathbb{E}|S_t-\widehat{S}_t^N|^2\leq\frac{1}{4m_t^2N}.

If inf⁡0≤t≤Tmt=m>0\inf_{0\leq t\leq T}m_t=m>0, then E∫0T∣St−S^tN∣2dt≤T/(4m2N)\mathbb{E}\int_0^T|S_t-\widehat{S}_t^N|^2dt\leq T/(4m^2N).

Proof. Conditional on StS_t, NztNNz_t^N is binomial with success probability ht(St)h_t(S_t), and therefore its conditional variance is at most N/4N/4. Projection onto an interval does not increase distance to a point in that interval. On KK the inverse of hth_t is mt−1m_t^{-1}-Lipschitz. The result follows by conditioning on StS_t. □

Proposition 4.2. Fix a strategy α\alpha that is admissible for the original observation problem. Let It=1{αt≥St}I_t=\mathbf{1}_{\{\alpha_t\geq S_t\}} and I^t=1{αt≥S^tN}\widehat{I}_t=\mathbf{1}_{\{\alpha_t\geq\widehat{S}_t^N\}}, and define

YT=B0−(1+β)∫0TStIt dt,Y^TN=B0−(1+β)∫0TS^tNI^t dt.Y_T=B_0-(1+\beta)\int_0^T S_tI_t\,dt,\qquad\widehat{Y}_T^N=B_0-(1+\beta)\int_0^T\widehat{S}_t^N\widehat{I}_t\,dt.

Let J(α)J(\alpha) denote the payoff in (4), and let JN(α)J^{N}(\alpha) be the corresponding auxiliary payoff with SS, II and YTY_T replaced by S^N\widehat{S}^{N}, I^\widehat{I} and Y^T\widehat{Y}_T (the performance process XX is held fixed in this comparison). Suppose the estimates in Lemma 4.1 hold uniformly in time, StS_t, S^tN\widehat{S}^{N}_t take values in a compact interval KK, and ff and gg are Lipschitz on the relevant ranges. Assume, moreover, that for all δ>0\delta> 0 small enough

P(∣St−αt∣≤δ)≤Mαδfor a.e. t∈[0,T],(10)\mathbb{P}(|S_t-\alpha_t|\leq\delta)\leq M_\alpha\delta\qquad\text{for a.e. }t\in[0,T], \tag*{(10)}

with MαM_\alpha independent of NN. Then

∣J(α)−JN(α)∣≤CαN−1/3,|J(\alpha)-J^{N}(\alpha)|\leq C_\alpha N^{-1/3},

where CαC_\alpha is independent of NN.

Proof. For each δ>0\delta>0,

∣It−I^t∣≤1{∣St−αt∣≤δ}+1{∣St−S^tN∣>δ}.|I_t-\widehat{I}_t|\leq\mathbf{1}_{\{|S_t-\alpha_t|\leq\delta\}}+\mathbf{1}_{\{|S_t-\widehat{S}^{N}_t|>\delta\}}.

Markov’s inequality, (10) and the mean-square estimator bound imply

E∣It−I^t∣≤Mαδ+CNδ2.\mathbb{E}|I_t-\widehat{I}_t|\leq M_\alpha\delta+\frac{C}{N\delta^2}.

Take δ=N−1/3\delta=N^{-1/3}. Since both price variables remain bounded,

E∣StIt−S^tNI^t∣≤E∣St−S^tN∣+sup⁡s∈K∣s∣E∣It−I^t∣.\mathbb{E}|S_tI_t-\widehat{S}^{N}_t\widehat{I}_t|\leq\mathbb{E}|S_t-\widehat{S}^{N}_t|+\sup_{s\in K}|s|\mathbb{E}|I_t-\widehat{I}_t|.

Integrating this inequality and applying the Lipschitz assumptions to ff and gg yields the asserted bound. □

Implications for optimal control. Proposition 4.2 concerns a fixed policy satisfying the anti-concentration assumption. It does not provide a uniform estimate over arbitrary observation-adapted policies. Indeed, the choice αt=S^tN\alpha_t=\widehat{S}^{N}_t wins by construction in the proxy model, while its outcome remains uncertain in the true model. Therefore convergence of the price estimator alone does not establish near-optimality of an optimizer of the proxy problem. Such a conclusion requires additional stability conditions on the admissible policy class. In what follows, S^N\widehat{S}^{N} is used as an observable price signal in a plug-in control approximation. This is not an exact separation of the partially observed problem.

Example 4.1 (A synthetic check of the price estimator). We illustrate the estimator of (9) in a cross-sectional experiment. Fix St=0.5S_t=0.5, Amax⁡=1A_{\max}=1, λ=3\lambda=3, vt=0.15v_t=0.15, and K=[0,1]K=[0,1]. For each value of NN, we generated 3000 independent samples of exploratory bids and Gaussian price noises, using the capped exponential law (4.1). Figure 4 displays the empirical mean-square estimation error. The observed decrease is close to N−1N^{-1}, as expected from Lemma 4.1. This numerical check concerns price estimation only, not the performance of a bidding strategy.

Mean-square error of the capped-bid estimator versus sample size

Figure 4. Mean-square error of the capped-bid estimator versus sample size, with 3000 independent replications and S=0.5S = 0.5; dashed reference line of slope −1-1

Forward-equation calibration

We now turn to the coefficients that determine the evolution of the price and campaign-performance processes. Even when contemporaneous prices are censored, historical campaign batches may provide data from which to estimate

μ,σ,γ,θ.\mu,\quad\sigma,\quad\gamma,\quad\theta.

As bids affect campaign performance, calibration must take the joint distribution of states and actions into account. The observation or experimental policy must be recorded; otherwise parameters of the performance equation may not be identifiable.

Let

p(t,s,x,a)p(t,s,x,a)

denote the joint density of (St,Xt,αt)(S_t,X_t,\alpha_t), provided it exists. For deterministic feedback controls, the conditional law of the action may be singular; the corresponding expressions should then be understood in terms of state–action occupation measures. For any test function Δ∈C1,2((0,T)×R2)\Delta\in C^{1,2}((0,T) \times\mathbb{R}^{2}), Itô’s formula gives

E[Δ(t,St,Xt)]=E[Δ(0,S0,X0)]+E[∫0t(∂rΔ+μ(r,Sr)∂sΔ+γ(r,Xr,Sr,αr)∂xΔ+12σ2(r,Sr)∂ssΔ+12θ2(r,Xr,Sr,αr)∂xxΔ) dr].\mathbb{E}[\Delta(t,S_t,X_t)] = \mathbb{E}[\Delta(0,S_0,X_0)] + \mathbb{E}\left[\int_0^t \left(\partial_r\Delta+ \mu(r,S_r)\partial_s\Delta+ \gamma(r,X_r,S_r,\alpha_r)\partial_x\Delta+ \frac{1}{2}\sigma^2(r,S_r)\partial_{ss}\Delta+ \frac{1}{2}\theta^2(r,X_r,S_r,\alpha_r)\partial_{xx}\Delta\right)\,dr\right].

Writing ρ(t,s,x)=∫Ap(t,s,x,a) da\rho(t,s,x) = \int_A p(t,s,x,a)\,da for the state marginal, the weak forward equation takes the form

0=∫A[∂tp+∂s(μp)+∂x(γp)−12∂ss(σ2p)−12∂xx(θ2p)](t,s,x,a) da.(11)0 = \int_A \left[\partial_t p + \partial_s(\mu p) + \partial_x(\gamma p) - \frac{1}{2}\partial_{ss}(\sigma^2p) - \frac{1}{2}\partial_{xx}(\theta^2p)\right](t,s,x,a)\,da. \tag*{(11)}

Suppose that historical data provide observations

D={(ti,Xtij,S^tij,αtij):1≤i≤n, 1≤j≤Mi},\mathcal{D} = \left\{(t_i,X_{t_i}^{j},\widehat{S}_{t_i}^{j},\alpha_{t_i}^{j}) : 1 \le i \le n,\ 1 \le j \le M_i\right\},

where S^\widehat{S} is reconstructed from the collected auction data. We estimate the marginal densities of StS_t and XtX_t using kernel estimators:

ϕ^i(s)=1MihiS∑j=1MiK(s−S^tijhiS),\widehat{\phi}_i(s) = \frac{1}{M_i h_i^S}\sum_{j=1}^{M_i} K\left(\frac{s-\widehat{S}_{t_i}^{j}}{h_i^S}\right),

and

ψ^i(x)=1MihiX∑j=1MiK(x−XtijhiX)\widehat{\psi}_i(x)=\frac{1}{M_i h_i^X}\sum_{j=1}^{M_i}K\left(\frac{x-X_{t_i}^j}{h_i^X}\right)

Here KK is a symmetric kernel satisfying

K≥0,K(u)=K(−u),∫RK(u) du=1,K\ge0,\qquad K(u)=K(-u),\qquad\int_{\mathbb{R}}K(u)\,\mathrm{d}u=1,

and the bandwidths hiSh_i^S and hiXh_i^X may be selected by cross-validation.

We consider parametric families

μη,ση,γη,θη,\mu^\eta,\qquad\sigma^\eta,\qquad\gamma^\eta,\qquad\theta^\eta,

where η\eta is an unknown parameter vector. Calibration amounts to controlling the residual of (4.6), together with the discrepancy between observed and model-implied data. We use physics-informed neural networks to approximate the forward densities and their derivatives.

Let

(tk,sk,xk)k=1P(t_k,s_k,x_k)_{k=1}^{P}

be collocation points sampled in the state-time domain. The price-density residual is

Lφ(η)=1P∑k=1P∣∂tφ^(tk,sk)+∂s(μη(tk,sk)φ^(tk,sk))−12∂ss((ση)2(tk,sk)φ^(tk,sk))∣2.\mathcal{L}_{\varphi}(\eta)=\frac{1}{P}\sum_{k=1}^{P}\left|\partial_t\widehat{\varphi}(t_k,s_k)+\partial_s\left(\mu^\eta(t_k,s_k)\widehat{\varphi}(t_k,s_k)\right)-\frac{1}{2}\partial_{ss}\left((\sigma^\eta)^2(t_k,s_k)\widehat{\varphi}(t_k,s_k)\right)\right|^2.

Similarly, the XX-marginal density ψ(t,x)=∫R∫Ap(t,s,x,a) da ds\psi(t,x)=\int_{\mathbb{R}}\int_A p(t,s,x,a)\,\mathrm{d}a\,\mathrm{d}s satisfies, assuming vanishing boundary fluxes,

0=∂tψ(t,x)+∂x∫R∫Aγ(t,x,s,a)p(t,s,x,a) da ds−12∂xx∫R∫Aθ2(t,x,s,a)p(t,s,x,a) da ds.0=\partial_t\psi(t,x)+\partial_x\int_{\mathbb{R}}\int_A\gamma(t,x,s,a)p(t,s,x,a)\,\mathrm{d}a\,\mathrm{d}s -\frac{1}{2}\partial_{xx}\int_{\mathbb{R}}\int_A\theta^2(t,x,s,a)p(t,s,x,a)\,\mathrm{d}a\,\mathrm{d}s.

A residual associated with the XX-marginal equation must integrate over both price and action:

Lψ(η)=1P∑k=1P∣∂tψ^(tk,xk)+∂x∫R∫Aγη(tk,xk,s,a)p^(tk,s,xk,a) da ds−12∂xx∫R∫A(θη)2(tk,xk,s,a)p^(tk,s,xk,a) da ds∣2.\mathcal{L}_{\psi}(\eta)=\frac{1}{P}\sum_{k=1}^{P}\left|\partial_t\widehat{\psi}(t_k,x_k) +\partial_x\int_{\mathbb{R}}\int_A\gamma^\eta(t_k,x_k,s,a)\widehat{p}(t_k,s,x_k,a)\,\mathrm{d}a\,\mathrm{d}s -\frac{1}{2}\partial_{xx}\int_{\mathbb{R}}\int_A(\theta^\eta)^2(t_k,x_k,s,a)\widehat{p}(t_k,s,x_k,a)\,\mathrm{d}a\,\mathrm{d}s\right|^2.

Here p^\widehat{p} estimates the joint state–action occupation density, or an appropriate discretisation of the occupation measure. The time derivatives in the residual require a sufficiently regular interpolation of the empirical densities.

The calibration loss is then

L(η)=Lφ(η)+Lψ(η)+Ldata(η),\mathcal{L}(\eta)=\mathcal{L}_{\varphi}(\eta)+\mathcal{L}_{\psi}(\eta)+\mathcal{L}_{\mathrm{data}}(\eta),

Here Ldata\mathcal{L}_{\mathrm{data}} measures agreement with observed campaign data.

A small residual of the forward equation does not, by itself, identify the unknown coefficients: different models may yield the same observed marginal distributions. For completeness, the following elementary local stability bound isolates the additional identifiability assumption; its proof is deferred to Appendix § 10.

Proposition 4.3. Let Θ⊂Rd\Theta\subset\mathbb{R}^{d} be finite-dimensional, and let F(η)=pηF(\eta)=p^{\eta} denote the model-implied state density as an element of H=L2((0,T)×Ω)H=L^{2}((0,T)\times\Omega). Suppose FF is continuously Fréchet differentiable near the true parameter η⋆\eta^{\star} and

∥DF(η⋆)h∥H≥c0∣h∣for all h∈Rd,\lVert DF(\eta^{\star})h\rVert_{H} \ge c_{0}\lvert h\rvert\qquad\text{for all } h\in\mathbb{R}^{d},

for some c0>0c_{0}>0. Suppose also that the forward equations admit a uniformly bounded evolution family on L2(Ω)L^{2}(\Omega). For an approximate density p^\widehat{p} with initial data p^0\widehat{p}_{0}, define the forward residual Rη=∂tp^−Lη∗p^R^{\eta}=\partial_{t}\widehat{p}-L_{\eta}^{*}\widehat{p}, where Lη∗L_{\eta}^{*} is the forward (Fokker–Planck) operator. Then, locally in η\eta,

∣η−η⋆∣2≤C(∥p^0−p0∥L2(Ω)2+∥Rη∥H2+∥p^−p⋆∥H2).\lvert\eta-\eta^{\star}\rvert^{2}\le C\left(\lVert\widehat{p}_{0}-p_{0}\rVert_{L^{2}(\Omega)}^{2}+\lVert R^{\eta}\rVert_{H}^{2}+\lVert\widehat{p}-p^{\star}\rVert_{H}^{2}\right).

The proposition separates two sources of error: approximation of the forward equation and identification of the parameters from the induced densities. The latter depends on the observation design and cannot be inferred from the PDE alone. We now consider the associated optimal control problem.

Example 4.2 (Calibration on synthetic data). Figure 5 illustrates calibration on the seasonal model of Section 7 with assumed historical observation of prices. The plot shows the learning dynamics and the reconstructed coefficients in the experiment.

Data-driven discovery of the parameters (forward-equation PINN calibration)

Figure 5. Parameter trajectories and reconstructed dynamics from the PINN calibration experiment

Model-based control: HJB and policy iteration

We next consider a benchmark problem in which the current market price and the model coefficients are known. This complete-information formulation leads to an HJB equation and provides a reference policy against which an estimated-price strategy may be compared. Under incomplete information, the same numerical method can be used as a plug-in approximation, but it does not give the HJB equation of the original partially observed problem.

Full-information dynamic programming

Let Afull\mathcal{A}^{\mathrm{full}} denote the class of admissible policies adapted to the filtration generated by the full state (S,X,Y)(S,X,Y). For this benchmark, define the value function

V(t,s,x,y)=sup⁡α∈Afull ⁣Et,s,x,yα[∫tT(βSτ1{ατ≥Sτ}+f(ατ,Sτ,Xτ)) dτ−g(YT)].V(t,s,x,y)=\sup_{\alpha\in\mathcal{A}^{\mathrm{full}}\!}\mathbb{E}^{\alpha}_{t,s,x,y}\left[\int_t^T\left(\beta S_\tau\mathbf{1}_{\{\alpha_\tau\ge S_\tau\}}+f(\alpha_\tau,S_\tau,X_\tau)\right)\,\mathrm{d}\tau-g(Y_T)\right].

where Et,s,x,yα\mathbb{E}^{\alpha}_{t,s,x,y} denotes expectation conditional on (St,Xt,Yt)=(s,x,y)(S_t,X_t,Y_t)=(s,x,y) under the policy α\alpha. This value represents the maximal expected reward over [t,T][t,T] for a given initial state. Under the usual well-posedness and dynamic programming assumptions, the value function satisfies the following principle.

Dynamic programming principle. For any stopping time τ\tau valued in [t,T][t,T] with respect to the full-information filtration,

V(t,s,x,y)=sup⁡α∈AfullE[∫tτr(u,Su,Xu,Yu,αu) du+V(τ,Sτ,Xτ,Yτ)].V(t,s,x,y)=\sup_{\alpha\in\mathcal{A}^{\mathrm{full}}}\mathbb{E}\left[\int_t^\tau r(u,S_u,X_u,Y_u,\alpha_u)\,\mathrm{d}u+V(\tau,S_\tau,X_\tau,Y_\tau)\right].

where

r(t,s,x,y,α)=βs1{α≥s}+f(α,s,x).r(t,s,x,y,\alpha)=\beta s\mathbf{1}_{\{\alpha\ge s\}}+f(\alpha,s,x).

This is the standard dynamic programming identity for controlled diffusions; see [4] and [12].

The Hamilton–Jacobi–Bellman equation

As in the linear–quadratic benchmark, we define the Hamiltonian using generic arguments p=(ps,px,py)∈R3p=(p_s,p_x,p_y)\in\mathbb{R}^3 and M∈S3M\in\mathcal{S}^3:

H(t,s,x,y,p,M)=μ(t,s)ps+12σ2(t,s)Mss+sup⁡a∈A{f(a,s,x)+βs1{a≥s}+γ(t,x,s,a)px+12θ2(t,x,s,a)Mxx−(1+β)s1{a≥s}py}.(12)\begin{aligned} H(t,s,x,y,p,M)={}&\mu(t,s)p_s+\frac{1}{2}\sigma^2(t,s)M_{ss}\\ &+\sup_{a\in A}\left\{f(a,s,x)+\beta s\mathbf{1}_{\{a\ge s\}}+\gamma(t,x,s,a)p_x\right.\\ &\left.\qquad+\frac{1}{2}\theta^2(t,x,s,a)M_{xx}-(1+\beta)s\mathbf{1}_{\{a\ge s\}}p_y\right\}. \tag*{(12)} \end{aligned}

Since the Brownian motions driving SS and XX are independent, no mixed second-order derivatives appear in (12). Under the usual dynamic programming assumptions, the corresponding HJB equation is

{Vt+H(t,s,x,y,DV,D2V)=0,V(T,s,x,y)=−g(y).(13)\begin{cases} V_t+H(t,s,x,y,DV,D^2V)=0,\\ V(T,s,x,y)=-g(y). \tag*{(13)} \end{cases}

The maximization in (5.1) accounts for the campaign reward, the margin earned on successful transactions, and the effect of expenditure on future value. The gradient and Hessian in (5.2) are evaluated at (s,x,y)(s,x,y).

If VV is smooth, a verification theorem applies, and the pointwise maximum is attained, an optimal feedback control may be chosen from

α∗(t,s,x,y)∈argmax⁡a∈A{f(a,s,x)+βs1{a≥s}+γ(t,x,s,a)Vx+12θ2(t,x,s,a)Vxx−(1+β)s1{a≥s}Vy}.\begin{aligned} \alpha^*(t,s,x,y)\in\operatorname*{argmax}_{a\in A}\Bigl\{&f(a,s,x)+\beta s\mathbf{1}_{\{a\ge s\}}+\gamma(t,x,s,a)V_x\\ &+\frac{1}{2}\theta^2(t,x,s,a)V_{xx}-(1+\beta)s\mathbf{1}_{\{a\ge s\}}V_y\Bigr\}. \end{aligned}

The terms involving VsV_s and VssV_{ss} belong to the full Hamiltonian but do not affect the pointwise maximization because the price dynamics are uncontrolled.

Under incomplete information, however, the relevant state includes the conditional distribution L(St∣Gtα)\mathcal{L}(S_t \mid\mathcal{G}_t^\alpha) of the hidden price. Because the observation mechanism depends on the policy, this conditional distribution has controlled filtering dynamics. The separated problem is consequently measure-valued. Replacing StS_t by S^tN\widehat{S}_t^N in the benchmark equation is an approximation rather than an exact separation principle.

The value function need not be smooth, and the viscosity-solution framework is natural for an HJB equation of this type; see [1]. The discontinuity of the winning indicator, together with possible loss of attainment at a threshold, requires care in establishing a rigorous comparison or verification principle. We do not address this technical question here, and use (13) as the basis for numerical approximation.

Policy iteration with physics-informed neural networks

To translate the control formulation into a bidding rule, one needs to approximate the value function and perform the pointwise maximization in the HJB equation. Grid-based schemes may become expensive for several state variables. We therefore consider a physics-informed neural network (PINN) approximation of the value function.

Let

Vθ(t,s,x,y)V_\theta(t,s,x,y)

be a neural-network approximation of VV, with trainable parameters θ\theta.

The terminal condition is enforced through the parameterization

Vθ(t,s,x,y)=(T−t)Nθ(t,s,x,y)−g(y),V_\theta(t,s,x,y) = (T-t)N_\theta(t,s,x,y) - g(y),

where NθN_\theta is an unconstrained neural network. A strict budget constraint Yt≥0Y_t \ge0, if imposed, would also require an appropriate boundary condition at y=0y=0.

We use a generalized policy-iteration procedure, alternating between

  1. policy evaluation, during which the value function is updated for a fixed policy;

  2. policy improvement, during which the policy is updated using the current approximation of the value function.

At each policy-evaluation step, automatic differentiation provides the first- and second-order derivatives entering the linear equation. The policy-improvement step maximizes the Hamiltonian computed from the current network.

Algorithm 1 summarizes the computational steps.

Algorithm 1 Policy iteration with a PINN value-function approximation

  1. Initialize value network VθV_\theta

  2. repeat

  3. Policy evaluation

  4. Fix the current policy and solve the corresponding linear policy-evaluation equation approximately with VθV_\theta

  5. Sample collocation points

  6. Evaluate the HJB residual using automatic differentiation

  7. Update θ\theta by stochastic gradient descent

  8. Policy improvement: maximize the Hamiltonian pointwise using the current derivatives of VθV_\theta

  9. until convergence

The procedure follows the policy-iteration principle, but viscosity-solution theory alone does not establish convergence of the neural approximation. In particular, approximation, optimization, and policy-improvement errors must be analysed separately. We assess the numerical behavior of the method below.

Policy learning through reinforcement learning

The HJB approach relies on a specified and calibrated dynamics model. A practical alternative is to update bidding policies from observed actions and rewards, without first estimating every model coefficient. We describe this second approach using randomized controls and an entropy-regularized actor–critic approximation. Both methods must respect the same information constraint if their outcomes are to be compared.

Randomized policies and relaxed controls

Instead of selecting a single bid, the controller chooses at time tt a probability measure

πt∈P(A),\pi_t \in\mathcal{P}(A),

and the bid is sampled according to

αt∼πt.\alpha_t \sim\pi_t.

The process π=(πt)t∈[0,T]\pi= (\pi_t)_{t\in[0,T]} is progressively measurable and takes values in P(A)\mathcal{P}(A). The continuous-time model below uses a relaxed (measure-valued) control; averaging the infinitesimal drift and variance should not be confused with replacing the diffusion coefficient by its ordinary average.

For a probability measure π∈P(A)\pi\in\mathcal{P}(A), define

γ~(t,x,s,π)=∫Aγ(t,x,s,a)π(da),\widetilde{\gamma}(t,x,s,\pi) = \int_A \gamma(t,x,s,a)\pi(da),

and

θ~(t,x,s,π)=(∫Aθ2(t,x,s,a)π(da))1/2.\widetilde{\theta}(t,x,s,\pi) = \left(\int_A \theta^2(t,x,s,a)\pi(da)\right)^{1/2}.

The relaxed state dynamics are then written as

dStπ=μ(t,Stπ)dt+σ(t,Stπ)dWt1,dXtπ=γ~(t,Xtπ,Stπ,πt)dt+θ~(t,Xtπ,Stπ,πt)dWt2.\begin{aligned} dS_t^\pi&= \mu(t,S_t^\pi)dt+\sigma(t,S_t^\pi)dW_t^1,\\ dX_t^\pi&= \widetilde{\gamma}(t,X_t^\pi,S_t^\pi,\pi_t)dt+\widetilde{\theta}(t,X_t^\pi,S_t^\pi,\pi_t)dW_t^2. \end{aligned}

The expected budget expenditure under the relaxed policy is

dYtπ=−(1+β)Stπ∫A1{a≥Stπ}πt(da) dt.dY_t^\pi= -(1+\beta)S_t^\pi\int_A \mathbf{1}_{\{a\ge S_t^\pi\}}\pi_t(da)\,dt.

Similarly, the average trading revenue is

R(t,Stπ,πt)=βStπ∫A1{a≥Stπ}πt(da).R(t,S_t^\pi,\pi_t)=\beta S_t^\pi\int_A \mathbf{1}_{\{a\ge S_t^\pi\}}\pi_t(da).

This formulation is related to entropy-regularized stochastic control and continuous-time reinforcement learning; see, for example, [15], [5], and [6].

Entropy-regularized objective

To favor exploration, consider the entropy penalty associated with a policy density πt(a)\pi_t(a) on AA:

E(πt)=∫Aπt(a)log⁡πt(a) da.\mathcal{E}(\pi_t) = \int_A \pi_t(a) \log\pi_t(a)\, da.

For an entropy temperature τent>0\tau_{\mathrm{ent}} > 0 (distinct from the exploration rate λ\lambda), the criterion becomes

sup⁡πE[∫0T(fπ(t,Xtπ,Stπ)+R(t,Stπ,πt)−τentE(πt)) dt−g(YTπ)].\sup_{\pi} \mathbb{E}\left[\int_0^T \left(f_{\pi}(t, X_t^{\pi}, S_t^{\pi}) + R(t, S_t^{\pi}, \pi_t) - \tau_{\mathrm{ent}}\mathcal{E}(\pi_t)\right)\,dt - g(Y_T^{\pi})\right].

where

fπ(t,x,s)=∫Af(a,s,x)πt(da).f_{\pi}(t,x,s) = \int_A f(a,s,x)\pi_t(da).

The parameter τent\tau_{\mathrm{ent}} determines the relative weight assigned to entropy. Larger values favor more diffuse bidding distributions. Because entropy is differential entropy, its numerical value also depends on the units of the bid; the action range is fixed here.

Actor–critic approximation

For numerical implementation, we use an actor–critic scheme on a time grid 0=t0<⋯<tK=T0 = t_0 < \cdots< t_K = T. The information fed to the networks must be observable. Let OkO_k be a feature vector that may contain the estimated price S^kN\widehat{S}_k^N, observed performance and budget, or a finite-dimensional approximation of the conditional price distribution. The actor πψ(⋅∣Ok)\pi_{\psi}(\cdot\mid O_k) represents a policy density on AA (for example, a truncated Gaussian), while the critic Jθ(tk,Ok)J_{\theta}(t_k, O_k) approximates its value.

For an action aka_k chosen at time tkt_k, let rkr_k be the observed reward earned over [tk,tk+1][t_k,t_{k+1}], including the entropy regularisation. The terminal condition is Jθ(T,OK)=−g(YT)J_{\theta}(T,O_K) = -g(Y_T), with no additional terminal term in rK−1r_{K-1}. Define the temporal-difference innovation

δk=rk+Jθ(tk+1,Ok+1)−Jθ(tk,Ok).\delta_k = r_k + J_{\theta}(t_{k+1}, O_{k+1}) - J_{\theta}(t_k, O_k).

For an exact continuation value and a sufficient observation state, the temporal-difference increment has zero conditional mean. In a finite-dimensional feature approximation one instead seeks approximately

E[δk∣Ok]≃0.\mathbb{E}[\delta_k \mid O_k] \simeq0.

One possible critic loss is therefore the squared conditional expectation

Lcritic(θ)=∑k=0K−1E[∣E[δk∣Ok]∣2].\mathcal{L}_{\mathrm{critic}}(\theta) = \sum_{k=0}^{K-1} \mathbb{E}\left[\left|\mathbb{E}[\delta_k \mid O_k]\right|^2\right].

With an advantage estimator A^k\widehat{A}_k, the actor update takes the form

ψ←ψ+ηψ∑k=0K−1A^k∇ψlog⁡πψ(ak∣Ok).\psi\leftarrow\psi+ \eta_{\psi}\sum_{k=0}^{K-1}\widehat{A}_k\nabla_{\psi}\log\pi_{\psi}(a_k \mid O_k).

This condition does not require the realised temporal-difference error to vanish pathwise. The above discretisation defines an implementable scheme, but its convergence to a continuous-time optimal policy is not established here.

Unless OkO_k is a sufficient statistic for the observation history, this reduction also introduces a state-approximation error. Algorithm 2 summarizes the implementation.

Algorithm 2 Discrete actor–critic approximation of a continuous-time control problem

  1. Initialize actor parameters ψ\psi and critic parameters θ\theta

  2. repeat

  3. Simulate trajectories under the randomized policy πψ\pi_{\psi}

  4. Compute rewards and budget updates

  5. Update the critic by minimizing the temporal-difference residual

  6. Update the actor using the entropy-regularized policy gradient

  7. until convergence

  8. return Learned randomized policy πψ\pi_{\psi}

Unlike the HJB-based strategy, this approach does not require an explicit parameterization of the drift and volatility coefficients. It does, however, require sufficiently informative observations and rewards. We compare its numerical behavior with that of the model-based approximation in § 7.

Numerical experiments and operational interpretation

Experimental design

We consider the following seasonal diffusion model, together with a performance criterion consistent with the payment rule of § 3:

dSt=μ(Sˉ(t)−St) dt+σS dWtS,Sˉ(t)=Sˉbase+AmpSsin⁡(2πft),dXt=γ(t)αt1{αt≥St} dt+θX dWtX,γ(t)=γbase(1+AmpXcos⁡(2πft)),dYt=−(1+β)St1{αt≥St} dt,J(α)=E[∫0T(Xt+βSt1{αt≥St}−κ(1+β)2αt21{αt≥St}) dt−YT2].\begin{aligned} dS_t &= \mu\left(\bar{S}(t)-S_t\right)\,dt+\sigma_S\,dW_t^S,\\ \bar{S}(t) &= \bar{S}_{\mathrm{base}}+\mathrm{Amp}_S\sin(2\pi f t),\\ dX_t &= \gamma(t)\alpha_t\mathbf{1}_{\{\alpha_t\geq S_t\}}\,dt+\theta_X\,dW_t^X,\\ \gamma(t) &= \gamma_{\mathrm{base}}\left(1+\mathrm{Amp}_X\cos(2\pi f t)\right),\\ dY_t &= -(1+\beta)S_t\mathbf{1}_{\{\alpha_t\geq S_t\}}\,dt,\\ J(\alpha) &= \mathbb{E}\left[\int_0^T\left(X_t+\beta S_t\mathbf{1}_{\{\alpha_t\geq S_t\}}-\kappa(1+\beta)^2\alpha_t^2\mathbf{1}_{\{\alpha_t\geq S_t\}}\right)\,dt-Y_T^2\right]. \end{aligned}

The quadratic bidding penalty is included in the running reward. Since the Ornstein–Uhlenbeck price is not almost surely positive, a positivity-preserving specification or a documented truncation would be needed for a literal auction-price interpretation. The parameters used in the experiments are collected in Table 1.

ParameterValue
μ\mu2.0
S‾base\overline{S}_{\mathrm{base}}0.5
AmpS\mathrm{Amp}_{S}0.3
σS\sigma_{S}0.3
γbase\gamma_{\mathrm{base}}1.0
AmpX\mathrm{Amp}_{X}0.4
θX\theta_{X}0.1
ff (seasonal frequency)1.0
κ\kappa0.5
β\beta0.2
BB1.0
λ\lambda3.0
dd0.1

Table 1. Parameters used in the seasonal-price simulations. dd is the share of budget allocated to price estimation.

Model-based control with observed and estimated prices

Figure 6 displays the evolution of the PDE residual during the policy-iteration procedure. The full-information benchmark yields the trajectories in Figure 7, illustrating the interaction between the market price, bids, performance, and remaining budget.

PDE residual during the PINN policy-iteration computation

Figure 6. PDE residual during the PINN policy-iteration computation

State and bidding trajectories from the full-information PINN implementation

Figure 7. State and bidding trajectories from the full-information PINN implementation

Figure 8 compares the trajectories with and without price estimation, using an exploratory allocation corresponding to 10% of the initial budget.

Estimated-price trajectories: prices, bidding, win frequency, performance, trading margin, and budget

Figure 8. Estimated-price trajectories: prices, bidding, win frequency, performance, trading margin, and budget

Randomized actor–critic policy

Figure 9 illustrates the evolution of the randomized-policy parameters, together with performance and expenditure under reinforcement learning.

Performance, budget expenditure, and randomized-bid parameters from the actor–critic experiment

Figure 9. Performance, budget expenditure, and randomized-bid parameters from the actor–critic experiment

A budget-pacing benchmark

To compare the model-based and learning-based policies with a rule closer to operational practice, we consider a budget-pacing heuristic based on a proportional–integral–derivative (PID) controller. The rule does not attempt to solve the optimization problem (5). Its only objective is to keep the expenditure of the campaign close to a prescribed spending schedule.

Let Yref=(Ytref)t∈[0,T]Y^{\mathrm{ref}} = (Y_t^{\mathrm{ref}})_{t\in[0,T]} be a deterministic target for the remaining budget, decreasing from B0B_0 at time 0. In the experiments we use uniform pacing,

Ytref=B0(1−tT),Y_t^{\mathrm{ref}} = B_0\left(1-\frac{t}{T}\right),

although a seasonal schedule following the expected price level could be used instead. The tracking error is

et=Yt−Ytref.e_t = Y_t - Y_t^{\mathrm{ref}}.

A positive error indicates underspending relative to the schedule, and a negative error indicates overspending. Given a baseline bid αˉ∈A\bar{\alpha} \in A and nonnegative gains KPK_P, KIK_I, KDK_D, the PID bid is

αtPID=proj⁡[0,Amax⁡](αˉ+KPet+KI∫0teu du+KDe˙t).\alpha_t^{\mathrm{PID}} = \operatorname{proj}_{[0,A_{\max}]}\left(\bar{\alpha} + K_P e_t + K_I \int_0^t e_u\,\mathrm{d}u + K_D \dot{e}_t\right).

Underspending therefore raises the bid, which increases the probability of winning auctions, and overspending lowers it. The integral term corrects persistent deviations from the schedule. The derivative term damps the response to rapid changes in expenditure.

Since YY is absolutely continuous, e˙t\dot{e}_t exists for almost every tt, but it inherits the discontinuities of the winning indicator. In practice, the rule is implemented on the time grid 0=t0<⋯<tK=T0 = t_0 < \cdots< t_K = T of § 6.3:

αtkPID=proj⁡[0,Amax⁡](αˉ+KPek+KI∑j≤kej Δt+KDek−ek−1Δt),(14)\alpha_{t_k}^{\mathrm{PID}} = \operatorname{proj}_{[0,A_{\max}]}\left(\bar{\alpha} + K_P e_k + K_I \sum_{j\leq k} e_j\,\Delta t + K_D \frac{e_k-e_{k-1}}{\Delta t}\right), \tag*{(14)}

where ek=etke_k = e_{t_k} and Δt\Delta t is the grid step. When the bid saturates at 0 or Amax⁡A_{\max}, the accumulated error may grow without improving the tracking. A standard anti-windup correction, freezing the integral term during saturation, prevents this effect.

The discrete-time PID rule (14) depends only on the current and past values of the observed remaining budget. It can therefore be implemented without access to the latent market price and requires neither the price estimator S^N\widehat{S}^{N} nor expenditure on exploratory subcampaigns.

Budget pacing reflects a common practice in media trading: ensuring that the client’s budget is spent regularly over the campaign horizon. The rule is transparent, easy to tune, and robust to misspecification of the price dynamics. Its limitations follow from the same simplicity. It ignores the market price, campaign performance and the agency margin, and it does not account for the quadratic bidding cost in the criterion. In particular, it may win many auctions at unfavorable prices, which increases performance but not necessarily the value of (4). The rule is therefore a useful operational benchmark rather than a competing solution of the control problem. The gains (KP,KI,KD)(K_P, K_I, K_D) and the baseline bid αˉ\bar{\alpha} used in the experiments are given in Table 2.

ParameterValue
αˉ\bar{\alpha}0.5
KpK_p2.0
KiK_i4.0
KdK_d0.05
Time stepΔt=T/250\Delta t = T/250
Bounds[0,Amax⁡]=[0,1.5][0, A_{\max}] = [0, 1.5]

Table 2. Parameters of the PID budget-pacing rule

Outcomes and comparison

Figure 10 gathers the outcomes of five bidding rules: a finite-difference approximation of the HJB benchmark and the GPI–PINN policy, both with observed prices; the GPI–PINN feedback evaluated at the estimated price S^N\widehat{S}^{N}; the actor–critic policy of § 6, fed with the same estimated price; and a PID budget-pacing rule. The PID rule adjusts the bid so that cumulative expenditure tracks a prescribed spending schedule. It uses neither the value function nor the price estimator, and serves as an operational heuristic rather than as a solution of (5).

Performance comparison between the different methods on 1000 common trajectories

Figure 10. Performance comparison between the different methods on 1000 common trajectories

The three panels display the mean remaining budget, the mean cumulative performance, and the empirical distribution of the realized criterion. Several features are visible. First, the policies based on the estimated price start from a budget of 0.9, since 10% of the initial budget is allocated to the exploratory subcampaigns. Second, the observed-price policies spend their budget at a comparable rate and end the horizon with approximately one third of the initial budget. The estimated-price GPI–PINN policy spends more slowly and ends with roughly half of it. Third, the PID rule produces the largest cumulative performance, whereas the estimated-price GPI–PINN policy produces the smallest.

The distribution of the realized criterion reverses this ordering. The GPI–PINN value with observed prices exceeds the other values, whereas the PID rule yields more sales together with a markedly lower and more dispersed criterion. This is consistent with the structure of the objective: the PID rule tracks expenditure without accounting for the quadratic bidding cost or for the margin earned on each transaction. The policies based on the price signal lie between these two cases.

The gap between the observed-price and estimated-price GPI–PINN policies has two distinct sources. The first is the exploration cost, which reduces the budget available to the main campaign. The second is the threshold effect discussed after Proposition 4.2. A feedback calibrated to bid close to the estimated price wins with high probability in the proxy model, but only with probability close to one half in the true model when S^tN−St\hat{S}^{N}_{t}-S_{t} is approximately centred. Figure 8 shows this reduction in the win rate. It explains both the slower expenditure and the lower performance of the estimated-price policy. Consequently, the observed gap measures the cost of using a plug-in policy, not only the cost of estimating the price.

Discussion and conclusion

We have formulated media trading as a dynamic bidding and budget-allocation problem in which the agency sees market-clearing prices only after successful auctions. The model makes the three components of the agency’s decision explicit: campaign performance, trading margin and the use of the client’s remaining budget. An additional feature is the endogenous information structure, as the bids determine which clearing prices are observed.

Randomized exploratory subcampaigns provide a simple statistical instrument for recovering a price signal. Under the specified cross-sectional model, the estimation error admits a mean-square bound of order N−1N^{-1}. We then consider two ways to turn price and performance information into bidding policies: a calibrated stochastic model with an HJB/PINN approximation, and an entropy-regularized actor–critic approach. These methods offer complementary computational routes rather than new results in stochastic control theory.

Operational implications. The formulation highlights why good bidding decisions cannot be judged by conversions alone. A policy generating more sales may also erode the agency’s margin or exhaust the client’s budget too early. Likewise, exploration has a potential value through improved price information but entails a cost in advertising expenditure. These trade-offs motivate the use of a common, transparent objective when evaluating competing policies.

Limitations and outlook. Our estimator result relies on an independent exploratory sampling scheme and does not establish optimality for control with censored data. The HJB equation is a full-information benchmark, and substituting an estimated price leads to a plug-in policy. The exploratory budget is not jointly optimized with the main campaign.

Expression of ZπZ^{\pi} as a diffusion process

Let A=[0,Amax⁡]A=[0,A_{\max}] and suppose St∈(0,Amax⁡)S_{t}\in(0,A_{\max}). Let Πt(da)=πt(a)da\Pi_{t}(da)=\pi_{t}(a)da be a possibly random family of probability measures with πt(a)>0\pi_{t}(a)>0 on (0,Amax⁡)(0,A_{\max}). Suppose πt(a)\pi_{t}(a) is C1C^{1} in aa and absolutely continuous with finite variation in time,

πt(a)=π0(a)+∫0tπ˙u(a) du,∫Aπ˙t(a) da=0,\pi_t(a) = \pi_0(a) + \int_0^t \dot{\pi}_u(a)\,du, \qquad\int_A \dot{\pi}_t(a)\,da = 0,

with sufficient integrability to apply the Itô formula for a random finite-variation field. Define the randomized-policy winning probability

Ztπ=∫StAmax⁡πt(a) da.Z_t^\pi= \int_{S_t}^{A_{\max}} \pi_t(a)\,da.

Since ∂sZ=−πt(s)\partial_s Z = -\pi_t(s) and ∂ssZ=−∂sπt(s)\partial_{ss} Z = -\partial_s\pi_t(s), Itô’s formula yields

dZtπ=[∫StAmax⁡π˙t(a) da−μ(t,St)πt(St)−12σ2(t,St)∂sπt(St)]dt−σ(t,St)πt(St) dWt1.(15)dZ_t^\pi= \left[\int_{S_t}^{A_{\max}} \dot{\pi}_t(a)\,da - \mu(t,S_t)\pi_t(S_t) - \frac{1}{2}\sigma^2(t,S_t)\partial_s\pi_t(S_t)\right]dt - \sigma(t,S_t)\pi_t(S_t)\,dW_t^1. \tag*{(15)}

The second-order term therefore has a negative sign. Because πt\pi_t is strictly positive on the price interval, s↦Πt([s,Amax⁡])s \mapsto\Pi_t([s,A_{\max}]) is strictly decreasing on that interval. For z∈(0,1)z \in(0,1) let s(Πt,z)s(\Pi_t,z) denote its inverse. Then St=s(Πt,Ztπ)S_t=s(\Pi_t,Z_t^\pi) and (A.1) can be rewritten entirely in terms of (Πt,Ztπ)(\Pi_t,Z_t^\pi). Atomlessness alone is not sufficient for invertibility (a non-atomic probability density may vanish on an interval). Outside the support of Πt\Pi_t, ZtπZ_t^\pi equals zero or one, so the latent price cannot be recovered by inversion. The finite-variation condition is important here: if the policy distribution had a martingale component, the formula would include additional Itô–Wentzell terms.

Proof of the Calibration Consistency Bound

We prove Proposition 4.3. Denote by F(η)=pηF(\eta)=p^\eta the parameter-to-density map, with values in H=L2((0,T)×Ω)H=L^2((0,T)\times\Omega). By Fréchet differentiability,

F(η)−F(η⋆)=DF(η⋆)(η−η⋆)+o(∣η−η⋆∣).F(\eta)-F(\eta^\star)=DF(\eta^\star)(\eta-\eta^\star)+o(|\eta-\eta^\star|).

The assumed injective derivative with a quantitative lower bound implies, for η\eta in a sufficiently small neighborhood,

∣η−η⋆∣≤2c0∥pη−pη⋆∥H.|\eta-\eta^\star| \le\frac{2}{c_0}\|p^\eta-p^{\eta^\star}\|_H.

The estimate is local and relies on the quantitative injectivity assumption.

Let Uη(t,s)U_\eta(t,s) denote the forward evolution family of ∂tp=Lη∗p\partial_t p=L_\eta^\ast p, assumed uniformly bounded by MM on 0≤s≤t≤T0\le s\le t\le T. Duhamel’s formula for the approximate density p^\widehat p gives

p^(t)−pη(t)=Uη(t,0)(p^0−p0)+∫0tUη(t,s)Rη(s) ds,Rη=∂tp^−Lη∗p^.\widehat p(t)-p^\eta(t)=U_\eta(t,0)(\widehat p_0-p_0)+\int_0^t U_\eta(t,s)R^\eta(s)\,ds,\qquad R^\eta=\partial_t\widehat p-L_\eta^\ast\widehat p.

By Cauchy–Schwarz and the uniform bound on UηU_\eta,

∥p^−pη∥H2≤CT(∥p^0−p0∥L2(Ω)2+∥Rη∥H2).\|\widehat p-p^\eta\|_H^2\le C_T\left(\|\widehat p_0-p_0\|_{L^2(\Omega)}^2+\|R^\eta\|_H^2\right).

Finally,

∥pη−pη⋆∥H≤∥pη−p^∥H+∥p^−pη⋆∥H,\|p^\eta-p^{\eta^\star}\|_H\le\|p^\eta-\widehat p\|_H+\|\widehat p-p^{\eta^\star}\|_H,

and the claimed estimate follows. For controlled dynamics, the forward operator must incorporate the experimental policy or the corresponding state–action occupation measure. If the parameter-to-density map is not locally identifiable, the same argument does not yield parameter consistency.

Convergence of the Randomized Observation Process

Fix t∈[0,T]t \in[0,T] and condition on St=sS_t=s. By the conditional independence of exploratory bids and price noises, the variables Iti=1{αi≥s+εi}I_t^i = 1_{\{\alpha_i \ge s+\varepsilon^i\}} are independent Bernoulli variables with success probability ht(s)h_t(s) given by (7). Consequently,

E[∣ztN−ht(St)∣2∣St]=ht(St)(1−ht(St))N≤14N.\mathbb{E}\left[\left|z_t^N-h_t(S_t)\right|^2 \mid S_t\right] = \frac{h_t(S_t)(1-h_t(S_t))}{N} \le\frac{1}{4N}.

Taking expectations gives the fixed-time bound. Integration and Tonelli’s theorem yield the integrated result without any temporal independence assumption.

To establish strict monotonicity, differentiate the Gaussian distribution function under the bounded-action integral in (7). This gives (8). On a compact price interval KK, the derivative is bounded away from zero, and the clipped inverse in (9) is Lipschitz. The price-estimation bound then follows as in Lemma 4.1.

We do not claim convergence in a process topology. A continuous-time interpretation of the threshold-based winning indicators requires an explicit temporal sampling model.

References

  1. [1]Michael G Crandall, Hitoshi Ishii, and Pierre-Louis Lions. User’s guide to viscosity solutions of second order partial differential equations. Bulletin of the American mathematical society, 27(1):1–67, 1992.arxiv.org/abs/math/9207212
  2. [2]Joaquin Fernandez-Tapia, Olivier Guéant, and Jean-Michel Lasry. Optimal real-time bidding strategies. Applied Mathematics Research Express, 2017(1):142–183, 2017.
  3. [3]Arthur Flajolet and Patrick Jaillet. Real-time bidding with side information. http://web.mit.edu/jaillet/www/general/bidding-nips-17.pdf, 2017.
  4. [4]Wendell H. Fleming and H. Mete Soner. Controlled Markov Processes and Viscosity Solutions, volume 25 of Stochastic Modelling and Applied Probability. Springer, second edition, 2006.DOI
  5. [5]Yanwei Jia and Xun Yu Zhou. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach. Journal of Machine Learning Research, 23(154):1–55, 2022.arxiv.org/abs/2108.06655
  6. [6]Yanwei Jia and Xunyu Zhou. q-Learning in Continuous Time. Journal of Machine Learning Research, 24(161):1–61, 2023.arxiv.org/abs/2207.00713
  7. [7]Randall Lewis and Justin Rao. The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4):1941–1973, 2015.DOI
  8. [8]John D. C. Little. Aggregate advertising models: the state of the art. Operations Research, 27(4):629–667, 1979.
  9. [9]Kaito Majima, Kosuke Kawakami, Kota Ishizuka, and Kazuhide Nakata. Keyword-level bayesian online bid optimization for sponsored search advertising. Operations Research Forum, 5(45), 2024.
  10. [10]Médéric Motte and Huyền Pham. Optimal bidding strategies for digital advertising, 2021.arxiv.org/abs/2111.08311
  11. [11]Thomas Mulc, Mike Anderson, Paul Cubre, Huikun Zhang, Ivy Liu, and Saket Kumar. Nnn: Next-generation neural networks for marketing measurement, 2025.arxiv.org/abs/2504.06212
  12. [12]Huyền Pham. Continuous-time stochastic control and optimization with financial applications, volume 61 of Stochastic Modelling and Applied Probability. Springer-Verlag Berlin Heidelberg, 2009.DOI
  13. [13]Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.DOI
  14. [14]Michele Ricciardi and Mauro Rosestolato. Mean field games incorporating carryover effects: Optimizing advertising models. Decisions in Economics and Finance, 49:475–495, 2026.
  15. [15]Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 21(198):1–34, 2020.
  16. [16]Wush Chi-Hsuan Wu, Mi-Yen Yeh, and Ming-Syan Chen. Predicting winning price in real time bidding with censored data. http://www0.cs.ucl.ac.uk/staff/w.zhang/rtb-papers/win-price-pred.pdf, 2015.DOI
  17. [17]Weinan Zhang, Shuai Yuan, and Jun Wang. Optimal real-time bidding for display advertising. In KDD ’14: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1077–1086, 2014.DOI
  18. [18]Yingxiang Zhang, Mike Wurm, Alexander Wakim, Eddie Li, and Ying Liu. Bayesian hierarchical media mix model incorporating reach and frequency data. research.google, 2023.
  19. [19]Jun Zhao et al. Deep reinforcement learning for sponsored search real-time bidding. https://arxiv.org/abs/1803.00259v1, 2018. Accessed: 1 Mar. 2018.
  20. [20]Hubert A. Zielske. The remembering and forgetting of advertising. Journal of marketing, 1959.DOI
  21. [21]Hubert A. Zielske and Walter A. Henry. Remembering and forgetting television ads. Journal of Advertising Research, 1980.DOI

Paper details

vlnc:W_35phncsm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Economics, Econometrics and Finance

Not recorded for this paper

View statement
Looking for related papers…
Contents