betaFeedback

WorksMathematics

When Predictive Fit Misleads: Selecting Marketing Mix Models Using Experimental RoAS Benchmarks

Abstract

Marketing Mix Models (MMMs) are used to estimate channel-level Return on Ad Spend (RoAS) and guide marketing investment, yet competing specifications are usually compared on predictive-fit metrics such as out-of-sample R-squared, MAPE, or RMSE. We show that predictive fit can diverge from recovery of the causal quantities that drive marketing decisions. In simulations with known RoAS, the best-fitting specification is often not the one that recovers RoAS most accurately; replicated studies show that this disagreement is systematic and persists under media-spend collinearity, noise, and a misspecified carryover shape. Selecting specifications against experimental RoAS benchmarks, even imprecise ones, improves RoAS recovery relative to fit-based selection, and a real-world application with experimental benchmarks shows the same pattern. We propose an applied workflow: calibrate MMM specifications against estimand-aligned experimental benchmarks when they exist, and otherwise report RoAS sensitivity across plausible specifications rather than relying on a single best-fitting model.

Highlights for Practitioners

  • Standard model accuracy metrics (R2R^2, MAPE, RMSE) measure how well a model predicts sales — not whether it correctly identifies which advertising channels are driving them. Using these metrics alone to select a marketing mix model (MMM) can lead to misguided budget decisions.

  • When causal experiments (such as geo-experiments) have been conducted, their RoAS estimates should be used to validate and select among competing model specifications — not just to benchmark the final chosen model.

  • When no experiments are available, practitioners should run multiple plausible model specifications and report the range of RoAS estimates across them, rather than presenting a single model’s output as definitive.

  • In 320 simulated datasets where competing models differed only in their saturation curve, the best-fitting model was the most accurate RoAS model less often than a random pick would be; even imprecise experiments (up to 50% noise) picked more accurate models than R2R^2 did.

  • Small differences in model fit can mask large differences in RoAS estimates: in our real-world application, two models with nearly identical R2R^2 values produced RoAS estimates that differed by 60% in accuracy against an experimental benchmark — a gap large enough to materially affect channel investment decisions.

Introduction

Marketing Mix Models (MMMs), also known as media mix models, have become a central framework for quantifying the effectiveness of marketing investments and informing budget allocation across channels. Organizations rely on MMMs to estimate channel-specific Return on Ad Spend (RoAS), so that media decisions can be tied to incremental business outcomes rather than to exposure or sales correlations alone. Given the scale of contemporary advertising investment, errors in RoAS estimation can translate directly into inefficient budget allocation and lost profitability.

Despite the clear advantages of causal inference, much of MMM practice remains anchored in fit-based diagnostics that are only indirectly related to the business objective of measuring incremental marketing effectiveness. Controlled experiments such as geo-experiments are widely regarded as credible benchmarks for advertising incrementality, but they are often costly and resource-intensive to implement at scale ([28, 20, 2]). As a result, many practitioners continue to evaluate MMM specifications primarily using goodness-of-fit metrics such as out-of-sample R2R^2, MAPE, and RMSE, even though these metrics reward historical fit rather than causal recovery. This tendency is reinforced by persistent practical and identification challenges in MMM, including multicollinearity, endogeneity, and specification sensitivity ([3, 10, 8]). Widely used MMM workflows also commonly include predictive or posterior predictive diagnostics as important components of model assessment ([22, 13]). These diagnostics are useful for assessing whether a model captures broad sales patterns, but they do not by themselves establish recovery of causal RoAS. For marketing investment decisions ([16]), a central decision-relevant quantity is incremental RoAS, with marginal RoAS needed for formal real-location decisions ([11]).

The applied problem is straightforward. In practice, analysts often face several defensible MMM specifications that produce similar sales fit but materially different channel-level RoAS estimates. The practical question is not simply which model predicts sales best, but which model produces RoAS estimates credible enough to inform investment decisions. The framework below addresses that question when experimental benchmarks exist and provides sensitivity reporting when they do not.

To study this problem, we compare specification choice based on predictive fit with specification choice based on RoAS recovery against causal benchmarks. Our evaluation combines simulation scenarios that vary in saturation form, model complexity, and adstock inclusion with a real-world calibration illustration in which experimental RoAS benchmark estimates are available. Across these settings, the criteria frequently disagree, and the models favored by predictive fit can yield materially worse RoAS estimates. Predictive diagnostics remain useful, but they are not sufficient evidence of causal validity when MMM outputs are used to inform marketing investment decisions.

The contribution is threefold. First, it documents, through controlled simulation scenarios and a Monte Carlo replication with stress tests on media-spend collinearity, noise, and adstock shape, that predictive-fit ranking and causal-RoAS ranking diverge systematically in MMM specification choice; when candidates differ only in saturation form, out-of-sample R2R^2 is essentially uninformative about which specification recovers RoAS best. Second, it proposes using experimental RoAS benchmarks, when available, as calibration targets for selecting among otherwise plausible MMM specifications, and shows that this remains beneficial when the experimental benchmarks are themselves noisy. Third, it recommends specification-sensitivity reporting when benchmarks are unavailable, using summaries such as the componentwise median RoAS vector as descriptive consensus diagnostics rather than causal substitutes.

The argument is not that predictive fit is irrelevant for MMM. Predictive diagnostics remain useful for assessing whether a model captures broad sales patterns. The claim is narrower: when the target quantity is channel-level causal RoAS, predictive fit alone is not a sufficient model-selection criterion. The benchmarked evaluation framework is therefore a calibration procedure for settings where experimental or otherwise credible causal RoAS estimates are available. It is not a universal solution to causal identification from observational data.

The remainder of this paper is organized as follows. § 3 reviews related literature on MMM methodology, causal inference in marketing, and open-source frameworks. § 4 presents a motivating example in which a misspecified model achieves superior R2R^2 yet produces substantially worse RoAS estimates. § 5 details the experimental-benchmark calibration framework. § 6 presents simulation evidence: 32 single-draw comparisons spanning four saturation function types, four complexity levels, and two adstock conditions, followed by replicated studies of selection accuracy under repeated sampling, media-spend collinearity, noise, and noisy experimental benchmarks. § 7 presents a real-world calibration illustration using observational marketing data and experimental RoAS benchmarks. § 8 summarizes managerial implications, and § 9 concludes with limitations and directions for future research.

Related Literature

Experiments, including geo-experiments, are widely regarded as credible tools for measuring the incremental impact of advertising [9, 28, 20], but they are costly and resource-intensive [2], so they are impractical for continuous measurement. They are nonetheless frequently used as external benchmarks for MMM-derived RoAS [26]. Our focus on causal parameter recovery also connects to the econometric literature on identification in marketing: instrumental variables for endogenous price and promotion effects [25], double/debiased machine learning, which formalizes the prediction–causal tension motivating our work [5], and synthetic control methods for regional counterfactuals [1].

MMM is a specialized area of marketing analytics [17, 7]. Building on foundational work from the early 1990s [14], Hanssens et al. [15] and Wang et al. [29] consolidated the use of MMM for decomposing sales into marketing contributions, typically with frequentist regression and predetermined adstock and saturation transforms [6]; more recent work includes hierarchical Bayesian models with sign constraints [4]. A persistent challenge is multicollinearity: Ebbes et al. [10] showed that highly correlated marketing variables can yield different parameter values with similar predictions, and Dew et al. [8] examined its negative implications for causal estimation. Practitioners have recently raised the same concern for carryover: Palu [24] reports that adstock kernels of different shapes fit the same data almost equally well yet imply channel multipliers that differ several-fold, and argues for richer, transfer-function representations of carryover. Carryover, saturation, and attribution have also been studied extensively [23, 21, 12], with industry perspectives highlighting the gap between academic methods and practice [3]. Comparative studies of MMM estimators have focused on prediction accuracy [31], while the importance of RoAS metrics has been emphasized in digital marketing [19] and more broadly in campaigns [11].

Open-source frameworks have democratized MMM. Meta’s Robyn [22] offers automated hyperparameter optimization, multi-objective model selection, and budget allocation, and Google’s Meridian [13] is fully Bayesian, building on earlier Bayesian hierarchical work and geo-experiment research [30, 18, 27]. These workflows improve access but also illustrate the need to distinguish predictive from causal diagnostics. Bayesian posterior predictive checks assess whether a model reproduces observed sales; used alone, they share the limitation of R2R^2-based selection, evaluating predictive adequacy rather than causal parameter recovery. Posterior distributions over RoAS do quantify estimation uncertainty, and a Bayesian analog of our criterion would compare posterior RoAS with experimental benchmarks. Existing work has expanded the available estimators, transformations, and software, but less attention has been given to the evaluation problem: how should competing MMM specifications be judged when the target is causal RoAS rather than sales prediction? That gap is the focus here.

Motivating Example: When Better Predictions Lead to Worse Decisions

We illustrate the importance of RoAS-focused model evaluation through a motivating example. Consider a MMM for a product sold across L=5L = 5 retail locations over T=208T = 208 weeks (4 years), with marketing activities in C=4C = 4 channels. The data-generating process follows:

log⁡(ylt)=β0+∑j=2LβL,j1{l=j}+∑i=1C(βi+∑j=2Lβi,j1{l=j})⋅mi,lt\log(y_{lt}) = \beta_0 + \sum_{j=2}^{L} \beta_{L,j}\mathbb{1}\{l=j\} + \sum_{i=1}^{C}\left(\beta_i + \sum_{j=2}^{L}\beta_{i,j}\mathbb{1}\{l=j\}\right)\cdot m_{i,lt}
+βplog⁡(plt)+βtt+βddlt+βuut+βcct{}+ \beta_p \log(p_{lt}) + \beta_t t + \beta_d d_{lt} + \beta_u u_t + \beta_c c_t
+∑k=13γkhk,t+βs(1)sin⁡(2πt52)+βs(2)cos⁡(2πt52)+ϵlt(1){}+ \sum_{k=1}^{3}\gamma_k h_{k,t} + \beta_s^{(1)}\sin\left(\frac{2\pi t}{52}\right) + \beta_s^{(2)}\cos\left(\frac{2\pi t}{52}\right) + \epsilon_{lt} \tag*{(1)}

where ylty_{lt} denotes sales units at location ll in week tt, the superscripts (1) and (2) index the sine and cosine components of the annual seasonality cycle, respectively, and the transformed media variable mi,ltm_{i,lt} is constructed through a two-stage process:

mi,lt=fsat(gadstock(xi,lt;αi);Ki,Oi)(2)m_{i,lt} = f_{\mathrm{sat}}\left(g_{\mathrm{adstock}}(x_{i,lt};\alpha_i);K_i,O_i\right) \tag*{(2)}
  • Adstock Transformation (Geometric Decay):

gadstock(xi,lt;αi)=xi,lt+αi⋅gadstock(xi,l(t−1);αi),0≤αi<1(3)g_{\mathrm{adstock}}(x_{i,lt};\alpha_i) = x_{i,lt} + \alpha_i \cdot g_{\mathrm{adstock}}(x_{i,l(t-1)};\alpha_i), \qquad0 \le\alpha_i < 1 \tag*{(3)}

with initialization gadstock(xi,l,0;αi)=0g_{\mathrm{adstock}}(x_{i,l,0};\alpha_i)=0 for each location.

  • Saturation Transformation (Hill Function):

fsat(zi,lt;Ki,Oi)=zi,ltOiKiOi+zi,ltOi,Ki>0, Oi>0(4)f_{\mathrm{sat}}(z_{i,lt};K_i,O_i) = \frac{z_{i,lt}^{O_i}}{K_i^{O_i}+z_{i,lt}^{O_i}}, \qquad K_i>0,\ O_i>0 \tag*{(4)}

where zi,lt=gadstock(xi,lt;αi)z_{i,lt}=g_{\mathrm{adstock}}(x_{i,lt};\alpha_i) represents the adstocked media exposure.

Because the outcome is modeled on the log-sales scale, RoAS must be computed from counterfactual predictions transformed back to the sales level. Let η^lt(i)\widehat{\eta}_{lt}^{(i)} denote the fitted linear predictor with all channels active, and let η^lt(−i)\widehat{\eta}_{lt}^{(-i)} denote the fitted linear predictor under the counterfactual in which channel ii is set to zero while all other covariates remain at their observed values. Under the Gaussian log-error assumption, the corresponding level predictions are

y^lt(i)=exp⁡(η^lt(i)+σ^22),y^lt(−i)=exp⁡(η^lt(−i)+σ^22).\widehat{y}_{lt}^{(i)}=\exp\left(\widehat{\eta}_{lt}^{(i)}+\frac{\widehat{\sigma}^{2}}{2}\right), \qquad\widehat{y}_{lt}^{(-i)}=\exp\left(\widehat{\eta}_{lt}^{(-i)}+\frac{\widehat{\sigma}^{2}}{2}\right).

We then define channel-level return on ad spend as

RoAS^i=∑l=1L∑t=1T(y^lt(i)−y^lt(−i))plt∑l=1L∑t=1Txi,lt(5)\widehat{\mathrm{RoAS}}_i = \frac{\sum_{l=1}^{L}\sum_{t=1}^{T}\left(\widehat{y}_{lt}^{(i)}-\widehat{y}_{lt}^{(-i)}\right)p_{lt}}{\sum_{l=1}^{L}\sum_{t=1}^{T}x_{i,lt}} \tag*{(5)}

where y^lt(i)\widehat{y}_{lt}^{(i)} denotes predicted sales with all channels active, and y^lt(−i)\widehat{y}_{lt}^{(-i)} denotes predicted sales with channel ii spending set to zero (while maintaining all other variables at observed levels). The RoAS estimand in (5) is an average incremental return over the observed spend path and evaluation window. Without redefining the counterfactual perturbation, it is not the marginal return to an additional dollar of spend. For budget allocation, average RoAS is informative, but marginal RoAS or finite-difference returns around the current budget are the more direct allocation quantities.

We partition the data into a training set (weeks 1–156, Ntrain=780N_{\mathrm{train}} = 780 observations) and a test set (weeks 157–208, Ntest=260N_{\mathrm{test}} = 260 observations). Two specifications are estimated on the training data: Model A (correct), with adstock (Equation (3)) and saturation (Equation (4)), and Model B (misspecified), a pure linear model with mi,lt=xi,ltm_{i,lt} = x_{i,lt}. On the test set, both are evaluated on sales prediction (out-of-sample R2R^2, MAPE, RMSE) and on RoAS recovery (MAE and MAPE against the known RoAS values implied by the data-generating parameters).

Table A.2 (Appendix § 10) lists the known parameter values and Table A.1 (Appendix § 10) lists all variables. The known simulation RoAS values are computed from these parameters via counterfactual predictions: for each channel, the difference in total revenue between all channels active and that channel’s spend set to zero, divided by its total spend. The question is whether a misspecified model with superior sales prediction can still provide reliable RoAS estimates, or whether response-specification credibility matters more than predictive fit.

Table 1 and Figure 1 (with Figure A.1 in Appendix § 10) reveal a paradox: the model with better sales prediction produces worse RoAS estimates. Model B (misspecified) has superior out-of-sample fit, with R2=0.93R^2 = 0.93 versus 0.90 for Model A, MAPE of 9.07% versus 10.38%, and RMSE of 138.55 versus 169.13, because its trend term acts as a flexible approximation to cumulative adstock effects. Yet Model A (correctly specified) recovers RoAS substantially better: RoAS MAE of 3.12 versus 4.40 and RoAS MAPE of 141.92% versus 209.59%. Model B overestimates RoAS in all four channels, with errors from +172% to +241%, and its estimates lie systematically above the 45-degree recovery line in panel (d) of Figure 1, whereas Model A’s estimates, though still biased upward, lie closer to it. By omitting adstock dynamics, Model B’s linear trend term absorbs long-term sales growth and credits it to concurrent spend. Thus Model B’s 3.3% advantage in R2R^2 coincides with a 41% disadvantage in RoAS recovery (MAE ratio 4.40/3.12 = 1.41): a modest gain in predictive fit can be accompanied by a materially larger deterioration in RoAS recovery, since a misspecified model can exploit trends, seasonality, and autocorrelation without capturing the adstock and saturation mechanisms that determine incremental RoAS.

RoAS Recovery Performance

Figure 1. RoAS Recovery Performance. (a-b) Model A achieves 29-32% lower RoAS estimation errors despite inferior predictive accuracy. (c) Channel-level comparison shows Model B systematically overestimates RoAS across all channels. (d) Scatter plot reveals Model A’s estimates cluster closer to the perfect recovery line (dashed diagonal), while Model B exhibits larger systematic deviations from known simulation RoAS values. Both models exhibit substantial absolute RoAS error; the figure illustrates relative RoAS recovery rather than adequate absolute recovery.

Performance MetricModel A (Correct)Model B (Misspecified)
Sales Prediction Metrics (Test Set)
R2R^20.900.93
MAPE (%)10.389.07
RMSE169.13138.55
RoAS Recovery Metrics (Test Set)
Mean Absolute Error3.124.40
MAPE (%)141.92209.59
Relative Performance1.0\times1.4\times worse
Channel-Level RoAS Estimates
Channel 1 (Known Sim. RoAS: 2.48)7.888.45
Channel 2 (Known Sim. RoAS: 2.44)5.346.64
Channel 3 (Known Sim. RoAS: 2.06)4.846.24
Channel 4 (Known Sim. RoAS: 1.46)2.874.73

Table 1. Comparison of Sales Prediction Accuracy and RoAS Recovery Accuracy

The absolute RoAS errors remain large even for the structurally correct model, which is informative: correct functional form does not guarantee precise RoAS recovery in finite samples when nonlinear transformation parameters, channel coefficients, location heterogeneity, and time-series controls must be estimated jointly. The example therefore concerns relative RoAS recovery, not satisfactory absolute recovery. The remainder of the paper formalizes this evaluation principle and examines it in simulation scenarios and a real-world calibration exercise.

Experimental-Benchmark Calibration Framework

Marketing Mix Models used for RoAS reporting should be evaluated against the causal quantity they are intended to estimate. The framework below is a calibration procedure for settings in which experimental or otherwise credible causal RoAS benchmark estimates are available; it is not a substitute for identification. We first specify the model structure and transformation functions, then describe the parameter estimation procedure, followed by RoAS calculation and benchmarked evaluation.

Model Specification

We consider a multiplicative sales response model with location-specific effects. For location l∈{1,…,L}l \in\{1,\ldots,L\} and time period t∈{1,…,T}t \in\{1,\ldots,T\}, the model takes the form:

log⁡(ylt)=β0+∑j=2LβL,j1{l=j}+∑i=1C(βi+∑j=2Lβi,j1{l=j})⋅mi,lt+Xlt′γ+ϵlt(6)\log(y_{lt}) = \beta_0 + \sum_{j=2}^{L}\beta_{L,j}\mathbb{1}\{l=j\} + \sum_{i=1}^{C}\left(\beta_i + \sum_{j=2}^{L}\beta_{i,j}\mathbb{1}\{l=j\}\right)\cdot m_{i,lt} + \mathbf{X}_{lt}^{\prime}\boldsymbol{\gamma} + \epsilon_{lt} \tag*{(6)}

where ylty_{lt} represents sales units, CC denotes the number of marketing channels, mi,ltm_{i,lt} is the transformed media variable for channel ii (described below), and Xlt\mathbf{X}_{lt} contains control variables including price (log), promotional discounts, macroeconomic indicators, holiday effects, and seasonality terms. The error term ϵlt∼N(0,σ2)\epsilon_{lt} \sim N(0,\sigma^2) captures unexplained variation. The transformed media variable mi,ltm_{i,lt} is constructed through a two-stage process that captures both carryover (adstock) and diminishing returns (saturation) effects:

mi,lt=fsat(gadstock(xi,lt;αi);Ki,Oi)(7)m_{i,lt} = f_{\mathrm{sat}}\left(g_{\mathrm{adstock}}(x_{i,lt};\alpha_i);K_i,O_i\right) \tag*{(7)}

where xi,ltx_{i,lt} is the raw media spend in channel ii at location ll and time tt.

Transformation Functions

Adstock Transformation

To capture the carryover effect of advertising, we employ a geometric decay adstock function. The adstocked media exposure at time tt is given by:

gadstock(xi,lt;αi)=xi,lt+αi⋅gadstock(xi,l(t−1);αi),0≤αi<1(8)g_{\mathrm{adstock}}(x_{i,lt};\alpha_i)=x_{i,lt}+\alpha_i\cdot g_{\mathrm{adstock}}(x_{i,l(t-1)};\alpha_i),\quad0\leq\alpha_i<1 \tag*{(8)}

with initialization gadstock(xi,l,0;αi)=0g_{\mathrm{adstock}}(x_{i,l,0};\alpha_i)=0 for each channel. The parameter αi\alpha_i controls the decay rate of advertising effectiveness, with higher values indicating longer-lasting effects.

Saturation Transformation

Each adstocked exposure zi,lt=gadstock(xi,lt;αi)z_{i,lt}=g_{\mathrm{adstock}}(x_{i,lt};\alpha_i) is passed through one of two saturation families,

fHill(z;K,O)=zOKO+zO,fWeibull(z;K,O)=1−exp⁡{−(zK)O},(9)f_{\mathrm{Hill}}(z;K,O)=\frac{z^O}{K^O+z^O},\qquad f_{\mathrm{Weibull}}(z;K,O)=1-\exp\left\{-\left(\frac{z}{K}\right)^O\right\}, \tag*{(9)}

with scale K>0K>0 and shape O>0O>0. For 0<O≤10<O\leq1 both curves are everywhere concave (C-shape); for O>1O>1 both are sigmoidal (S-shape), with a convex region at low spend followed by saturation. This yields four candidate specifications (Hill C, Hill S, Weibull C, Weibull S) that are all economically plausible but imply different average and marginal returns, particularly at low spend. Figure 2 illustrates the four forms; Appendix § 11 gives their inflection points and a comparison of the two families.

Four candidate saturation specifications at a fixed scale $K = 5000$ and varying shape $O$

Figure 2. The four candidate saturation specifications at a fixed scale K=5000K = 5000 and varying shape OO. Top: Hill S-shape (O>1O > 1, sigmoidal) and Hill C-shape (0<O≤10 < O \le1, concave). Bottom: Weibull S-shape and Weibull C-shape. S-shapes have a convex low-spend region followed by saturation; C-shapes give strong initial returns that diminish progressively.

Parameter Estimation with Sign Constraints

Writing ηlt(Θ)\eta_{lt}(\Theta) for the right-hand side of Equation (6) excluding the error, and assuming log⁡ylt=ηlt(Θ)+εlt\log y_{lt}=\eta_{lt}(\Theta)+\varepsilon_{lt} with εlt∼iidN(0,σ2)\varepsilon_{lt}\overset{\mathrm{iid}}{\sim}\mathcal{N}(0,\sigma^2), we estimate Θ\Theta by bound-constrained maximum likelihood subject to economically motivated sign and shape restrictions: nonnegative channel effects (βi≥0\beta_i\geq0 and βi+βi,j≥0\beta_i+\beta_{i,j}\geq0), 0≤αi<10\leq\alpha_i<1, Ki>0K_i>0, and Oi≤1O_i\leq1 for C-shape and Oi>1O_i>1 for S-shape specifications. Because the recursive adstock and nonlinear saturation make the problem non-convex, we use multi-start L-BFGS-B and retain the solution with the highest attained likelihood. The full likelihood and constraint set are given in Appendix § 11.

Evaluation Framework

We distinguish two applied evaluation settings depending on whether experimental RoAS benchmarks are available. The first supports benchmarked calibration; the second supports sensitivity reporting, not definitive causal model selection.

Scenario 1: Experimental RoAS Benchmark Available (Primary Framework)

When causal inference methods such as geo-experiments have been conducted, experimental RoAS benchmark estimates can serve as external calibration targets. The data are divided into training (the first TtrainT_{\mathrm{train}} weeks) and test (Ttest=T−TtrainT_{\mathrm{test}} = T - T_{\mathrm{train}} weeks) sets; for each candidate specification (Hill-S, Hill-C, Weibull-S, Weibull-C in the primary setting), the parameters Θ\Theta are estimated on the training data by solving the constrained problem in Equation B.10; the estimated RoAS for each channel ii is computed on the test set according to Equation (5); and the specification minimizing the RoAS recovery error is selected, where

RoAS-MAE=1C∑i=1C∣RoASitrue−RoASiest∣,(10)\mathrm{RoAS\text{-}MAE} = \frac{1}{C}\sum_{i=1}^{C}\left|\mathrm{RoAS}_{i}^{\mathrm{true}}-\mathrm{RoAS}_{i}^{\mathrm{est}}\right|, \tag*{(10)}

RoASitrue\mathrm{RoAS}_{i}^{\mathrm{true}} is the known simulation RoAS value or the experimental benchmark estimate in applications, and RoASiest\mathrm{RoAS}_{i}^{\mathrm{est}} is the estimate from the fitted model. Algorithm 1 summarizes the procedure.

Scenario 2: No Experimental RoAS Benchmark Available

In practice, causal inference is costly and may not always be feasible. When no experimental RoAS benchmark is available, the RoAS-best specification cannot be identified directly from observational fit metrics alone. In that setting, we do not propose a definitive selection rule. Instead, the analyst should treat the distribution of

Algorithm 1 RoAS-Focused Model Evaluation Framework

Require: Training data Dtrain\mathcal{D}_{\mathrm{train}}, test data Dtest\mathcal{D}_{\mathrm{test}}, benchmark values {RoASitrue}i=1C\{\mathrm{RoAS}^{\mathrm{true}}_i\}_{i=1}^{C} (known simulation values or experimental estimates), candidate model set M\mathcal{M}

Ensure: Selected model m⋆m^\star

  1. BestMAE ←∞\leftarrow\infty; m⋆←∅m^\star\leftarrow\emptyset

  2. for each model m∈Mm \in\mathcal{M} do

  3. Estimate Θ^m\widehat{\Theta}_m on Dtrain\mathcal{D}_{\mathrm{train}} by solving Equation B.10

  4. Compute RoAS^i,m\widehat{\mathrm{RoAS}}_{i,m} on Dtest\mathcal{D}_{\mathrm{test}} for i=1,…,Ci = 1,\ldots,C (Equation 5)

  5. RoAS-MAEm←1C∑i=1C∣RoASitrue−RoAS^i,m∣\mathrm{RoAS\text{-}MAE}_m \leftarrow\frac{1}{C}\sum_{i=1}^{C}\left|\mathrm{RoAS}^{\mathrm{true}}_i-\widehat{\mathrm{RoAS}}_{i,m}\right|

  6. if RoAS-MAEm<BestMAE\mathrm{RoAS\text{-}MAE}_m < \mathrm{BestMAE} then

  7. BestMAE ←RoAS-MAEm\leftarrow\mathrm{RoAS\text{-}MAE}_m; m⋆←mm^\star\leftarrow m

  8. return m⋆m^\star

RoAS estimates across plausible specifications as a sensitivity diagnostic. Section 5.5 illustrates one simple consensus diagnostic based on proximity to the componentwise median RoAS vector, while emphasizing that such summaries are descriptive and not causal substitutes.

Comparison to Traditional Evaluation

Traditional MMM evaluation focuses on sales prediction accuracy, using out-of-sample R2=1−∑l,t(ylt−y^lt)2∑l,t(ylt−yˉ)2R^2 = 1-\frac{\sum_{l,t}(y_{lt}-\widehat{y}_{lt})^2}{\sum_{l,t}(y_{lt}-\bar{y})^2}, MAPE =100%N∑l,t∣(ylt−y^lt)/ylt∣=\frac{100\%}{N}\sum_{l,t}\left|(y_{lt}-\widehat{y}_{lt})/y_{lt}\right|, and RMSE =1N∑l,t(ylt−y^lt)2=\sqrt{\frac{1}{N}\sum_{l,t}(y_{lt}-\widehat{y}_{lt})^2}, where NN is the number of test observations. As the motivating example (§ 4) shows, a model with better sales prediction (higher R2R^2, lower MAPE/RMSE) can produce severely biased RoAS estimates, because these metrics evaluate overall fit to observed data while RoAS recovery depends on correctly estimating the causal structure of marketing effects, namely the adstock and saturation transformations. In several scenarios in § 6, models with materially lower R2R^2 nonetheless achieved better RoAS recovery. The proposed framework therefore separates two tasks that are often conflated: predictive metrics diagnose whether a model captures sales patterns, whereas RoAS diagnostics assess whether the same model recovers the incremental quantities used for marketing evaluation. When experimental benchmarks exist, the latter can be evaluated directly; when they do not, RoAS dispersion across plausible specifications should be reported instead of being hidden behind a single predictive-fit-selected model.

Performance Evaluation

We compare predictive-fit ranking with RoAS-recovery ranking in two layers of simulation evidence. Sections 5.1–5.2 report a set of single-draw scenarios that span four saturation forms, four complexity levels, and two adstock conditions. Section 5.3 then replicates the design over hundreds of independent datasets and stresses it along three dimensions that are central to MMM identification: media-spend collinearity, noise, and the shape of carryover, and it also evaluates selection against noisy experimental benchmarks.

StepAnalyst actionOutput
1Fit substantively plausible MMM specificationsCandidate RoAS estimates and fit metrics
2Compare predictive fitR2R^2, MAPE, RMSE
3If an experimental benchmark exists, compare MMM-implied RoAS with the benchmarkRoAS-MAE or an uncertainty-weighted discrepancy
4If no benchmark exists, summarize RoAS dispersionMedian, interquartile range, and range across specifications
5Report decision riskStable versus specification-sensitive channel conclusions
6Use results for decision supportCalibrated or sensitivity-qualified RoAS recommendations

Table 2. Applied Workflow for MMM Specification Evaluation

Simulation Design

Each DGP follows Equation (1), combining geometric adstock with one of the four saturation forms (Hill C, Hill S, Weibull C, Weibull S). Four complexity scenarios vary the number of channels and locations: S1 (4 channels, 5 locations), S2 (6, 8), S3 (10, 12), and S4 (12, 15). All use 156 training weeks and 52 test weeks. Adstock decay rates are drawn from Uniform(0.5, 0.8); shape parameters from Uniform(0.3, 0.6) for C-shape and Uniform(2.0, 4.0) for S-shape DGPs; half-saturation points are linearly spaced from 8,000 to 4,000; media coefficients are linearly spaced from 1.5 to 0.3 with location-level deviations ∼N(0,0.12)\sim\mathcal{N}(0, 0.1^2) floored at 0.01; and the remaining coefficients follow Table A.2, with σ=0.02\sigma= 0.02. Weekly spend is drawn from a multivariate normal distribution with pairwise channel correlation ρ=0.3\rho= 0.3 (floored at $100).

For each DGP we fit the four candidate saturation specifications under two conditions: adstock-included candidates contain the carryover structure of the DGP (so exactly one candidate is fully correctly specified), while adstock-omitted candidates omit adstock entirely, so that every candidate is structurally wrong. In the simulations, the estimator adds a mild Gaussian shrinkage penalty that pulls location-specific media effects towards their channel mean. For each fitted model we compute out-of-sample sales metrics (R2R^2, MAPE, RMSE) and RoAS recovery metrics (RoAS-MAE, RoAS-MAPE, RoAS-RMSE) against the known simulation RoAS. The traditional criterion selects the highest-R2R^2 candidate; the benchmark criterion selects the lowest-RoAS-MAE candidate. With noiseless simulation RoAS, the benchmark criterion is an oracle, and its role is to measure how often, and by how much, predictive-fit ranking departs from RoAS-recovery ranking. The scenarios are deliberately difficult (many parameters relative to sample size, nonlinear and non-convex estimation), so absolute error magnitudes are not intended as benchmarks for typical MMM performance.

Single-Draw Scenario Results

Table 3 reports one simulated dataset per DGP and complexity scenario for adstock-included candidates. Even though the true saturation family is among the candidates, the highest-R2R^2 specification frequently differs from the specification with the lowest RoAS error. Averaged over scenarios, the RoAS-best specification has 1.33× (Weibull S) to 1.57× (Weibull C) lower RoAS-MAE than the highest-R2R^2 specification, and it typically does so with a markedly lower R2R^2: in Hill C S1, for example, the R2R^2-selected model attains R2=0.618R^2 = 0.618 and RoAS-MAE =0.554= 0.554, whereas the RoAS-best model attains R2=0.312R^2 = 0.312 and RoAS-MAE =0.446= 0.446. This is the predictive-fit paradox of the motivating example: higher sales fit does not reliably correspond to better recovery of incremental returns.

DGPScenarioTraditional (R2R^2 Max)Benchmark (RoAS-MAE Min)
R2R^2RoAS-MAERoAS-MAPER2R^2RoAS-MAERoAS-MAPE
Hill CS10.6180.55472.82%0.3120.44644.77%
S20.3761.94767.65%0.3761.94767.65%
S30.59825.11457.84%0.59820.50247.25%
S40.56433.42449.09%0.43920.11432.96%
Average0.53915.26061.85%0.43110.75248.16%
Improvement—1.42×
Hill SS10.8012.23568.99%0.7561.40736.57%
S20.6386.49742.64%0.6165.79040.51%
S30.78447.96834.71%0.67828.56822.35%
S40.70975.97937.84%0.35153.56429.01%
Average0.73333.17046.05%0.60122.33232.11%
Improvement—1.49×
Weibull CS10.6901.22878.81%0.5210.85348.77%
S20.6424.21333.24%0.3813.08924.94%
S30.53850.18737.27%0.28528.89418.97%
S40.61460.34728.89%0.22041.12323.70%
Average0.62128.99344.55%0.35218.49029.10%
Improvement—1.57×
Weibull SS10.6541.62434.19%0.6211.30827.35%
S20.67211.04142.85%0.6126.09219.86%
S30.56166.83133.78%0.52258.41924.38%
S40.62098.12930.84%0.58467.73321.22%
Average0.62744.40635.42%0.58533.38823.20%
Improvement—1.33×

Table 3. Notes: Traditional selection is based on highest R2R^2; benchmark selection is based on lowest RoAS-MAE against known simulation RoAS and therefore functions as an oracle benchmark for assessing disagreement with predictive-fit selection. Improvement factor = Traditional RoAS-MAE / Benchmark RoAS-MAE. Average rows show mean performance across the four scenarios for each DGP type.

When all candidates omit adstock (Appendix Table C.1), the disagreement persists but narrows, with average improvement factors between 1.27× and 1.35× (1.30× overall): when every candidate is a poor approximation of the DGP, less signal is available to either criterion. Across the 32 single-draw comparisons, the highest-R2R^2 specification had a higher RoAS-MAE than the RoAS-best specification in 30 cases.

Replicated Simulation Evidence

A single draw per scenario cannot separate systematic disagreement from the luck of one dataset. We therefore replicate the design and extend it in four directions; Appendix § 13 gives cell-level results. In every replication, the DGP parameters, media spend, controls, and noise are redrawn independently, and all four candidate specifications are re-estimated.

  1. Monte Carlo replication. The main design is replicated 40 times for each DGP, for S1 and S2, and for both adstock conditions (320 datasets per condition, 2,560 model fits).

  2. Media-spend collinearity. The pairwise spend correlation is varied over ρ∈{0,0.3,0.6,0.9}\rho\in\{0, 0.3, 0.6, 0.9\} (baseline 0.3), holding everything else fixed and reusing the same random seeds across ρ\rho (S2, adstock-included candidates, 60 datasets per value).

  3. Noise level. The error standard deviation is varied over σ∈{0.02,0.05,0.10,0.20}\sigma\in\{0.02, 0.05, 0.10, 0.20\} (baseline 0.02), again with common random seeds (S2, adstock-included, 60 datasets per value).

  4. Adstock form. Sales are generated with a delayed adstock whose effect peaks one or two weeks after spend (kernel wk∝α(k−θ)2w_k \propto\alpha^{(k-\theta)^2}, θ∈{1,2}\theta\in\{1, 2\}), while all candidates keep geometric adstock, so carryover is misspecified for every candidate (S2, 60 datasets).

In all four studies we also evaluate benchmark selection with noisy experiments. The analyst observes bi=RoAS⁡itrue(1+czi)b_i = \operatorname{RoAS}^{\mathrm{true}}_i(1 + c z_i) with zi∼N(0,1)z_i \sim\mathcal{N}(0,1) and a coefficient of variation c∈{0.10,0.25,0.50}c \in\{0.10, 0.25, 0.50\} per channel, selects the candidate closest to bb in RoAS-MAE, and the selected model is scored against the true RoAS (averaging over

500 benchmark draws per dataset). This removes the oracle element of the benchmark criterion and asks whether realistically imprecise experiments still improve on predictive-fit selection. Table 4 summarizes the results; Appendix Figure D.1 shows the Monte Carlo distribution of the RoAS-error penalty from R2R^2-based selection.

StudyDatasetsPr(agree) [95% CI]τˉ\bar{\tau}MAE ratioExp-10Exp-25Exp-50
Monte Carlo, S1, adstock-included1600.18 [0.13, 0.25]-0.051.570.730.770.83
Monte Carlo, S2, adstock-included1600.17 [0.12, 0.23]0.011.360.800.840.89
Monte Carlo, S1, adstock-omitted1600.54 [0.47, 0.62]0.621.210.890.931.09
Monte Carlo, S2, adstock-omitted1600.56 [0.49, 0.64]0.631.110.930.961.08
Collinearity, ρ=0\rho= 0600.33 [0.23, 0.46]-0.041.260.850.900.95
Collinearity, ρ=0.3\rho= 0.3600.30 [0.20, 0.43]0.051.340.820.860.92
Collinearity, ρ=0.6\rho= 0.6600.23 [0.14, 0.35]0.011.390.790.830.88
Collinearity, ρ=0.9\rho= 0.9600.20 [0.12, 0.32]-0.111.360.800.830.88
Noise, σ=0.02\sigma= 0.02600.23 [0.14, 0.35]0.041.270.830.870.92
Noise, σ=0.05\sigma= 0.05600.32 [0.21, 0.44]0.061.250.850.890.94
Noise, σ=0.1\sigma= 0.1600.30 [0.20, 0.43]0.101.270.850.880.93
Noise, σ=0.2\sigma= 0.2600.30 [0.20, 0.43]0.181.230.860.910.96
Adstock form, delayed-adstock DGP600.27 [0.17, 0.39]0.091.280.830.870.93

Table 4. Predictive-Fit Selection versus RoAS Recovery Across Replicated Simulation Studies

Notes: Each dataset is an independent draw of DGP parameters, media spend, controls and noise; four candidate saturation specifications are fitted to each. Pr(agree): share of datasets in which the highest out-of-sample R2R^2 candidate is also the lowest-RoAS-MAE candidate (Wilson 95% interval; random choice among four candidates gives 0.25). τˉ\bar{\tau}: mean Kendall rank correlation between the R2R^2 ranking and the RoAS-accuracy ranking. MAE ratio: mean RoAS-MAE of the R2R^2-selected model divided by that of the RoAS-best model. Exp-c: mean RoAS-MAE (against the true RoAS) of the model selected against a noisy experimental benchmark with c%c\% coefficient of variation per channel, divided by the RoAS-MAE of the R2R^2-selected model; values below one favour benchmark-based selection. Collinearity and noise studies use S2 with adstock-included candidates. The adstock-form study uses S2 with a delayed-adstock DGP and geometric-adstock candidates.

Finding 1: When candidates differ only in saturation form, R2R^2 carries essentially no information about RoAS accuracy. With adstock-included candidates, the highest-R2R^2 model coincides with the RoAS-best model in only 17.5% of the 320 datasets (95% CI 13.7–22.0%), below the 25% expected from choosing one of the four candidates at random. The mean Kendall correlation between the R2R^2 ranking and the RoAS-accuracy ranking is −0.02, and selecting on out-of-sample MAPE instead of R2R^2 gives the same picture (agreement 17.8%). The R2R^2-selected model has, on average, 1.47× the RoAS-MAE of the RoAS-best model (median 1.30×), equivalent to an average channel-level RoAS error of 41% of mean RoAS versus 29% for the RoAS-best model, while its R2R^2 is 0.20 higher. The pattern holds in every DGP-by-scenario cell (agreement between 7.5% and 37.5%; Appendix Table D.1). The single-draw results of § 6.2 are therefore not artifacts of particular random draws.

Finding 2: Under structural misspecification, predictive fit is more informative, and the gap narrows. When every candidate omits adstock, agreement rises to 55.3% (49.8–60.7%) and the rank correlation to 0.62; the mean RoAS-MAE penalty from R2R^2 selection falls to 1.16× (median 1.00×). Omitting carryover degrades sales fit and RoAS recovery jointly, so in that regime R2R^2 partly tracks causal accuracy. This replicates the narrowing documented in the single-draw results, and it qualifies the paper’s claim: predictive fit is least reliable precisely in the common applied situation in which several structurally sound specifications fit the data comparably well.

Finding 3: Noisy experimental benchmarks still outperform predictive-fit selection. Selecting against an imperfect experiment reduces the RoAS-MAE of the selected model relative to R2R^2 selection by 23%, 19%, and 14% for benchmark coefficients of variation of 10%, 25%, and 50% with adstock-included candidates, and with 25% benchmark noise the benchmark-selected model is at least as accurate as the R2R^2-selected model in 90% of datasets. With adstock-omitted candidates, precise benchmarks still help (9% and 6% reductions at c=0.10c = 0.10 and 0.25), but a benchmark with 50% noise is slightly worse than R2R^2 (+9%). Benchmark-based calibration is therefore valuable well beyond the oracle setting, with gains that shrink as experimental precision falls.

Finding 4: Collinearity and noise do not rescue predictive-fit selection. As the spend correlation rises from ρ=0\rho= 0 to 0.9, agreement falls from 33% to 20%, the mean RoAS-MAE penalty from R2R^2 selection rises from 1.26× to 1.36×, and the R2R^2 range across candidates narrows from 0.62 to 0.38: correlated channels constrain the total media response more tightly than its allocation across channels, so specifications that fit sales similarly can attribute returns very differently ([10]). The value of calibration grows accordingly (a 25%-noise benchmark reduces RoAS-MAE relative to R2R^2 selection by 10% at ρ=0\rho= 0 and 17% at ρ=0.9\rho= 0.9). A tenfold increase in noise, from σ=0.02\sigma= 0.02 to 0.20, changes little: agreement stays between 23% and 32%, near the random-choice rate, the penalty between 1.23× and 1.27×, and 25%-noise benchmarks reduce RoAS-MAE by 9–13% at every level; the rank correlation rises only to 0.18. The disagreement is thus not an artifact of a particularly clean or particularly collinear design (Appendix Figure D.2).

Finding 5: A misspecified carryover shape does not make R2R^2 informative. When sales are generated with a delayed adstock that peaks one or two weeks after spend while every candidate assumes geometric carryover, the highest-R2R^2 model is the RoAS-best model in 26.7% of datasets (95% CI 17.1–39.0%), indistinguishable from random choice; the rank correlation is 0.09, and the R2R^2-selected model has 1.28× the RoAS-MAE of the RoAS-best model (median 1.22×; Appendix Table D.4). Unlike omitting carryover altogether (Finding 2), getting its shape wrong degrades RoAS recovery without making sales fit a better guide, consistent with the practitioner concern raised by Palu [24]. Noisy benchmarks still help, reducing RoAS-MAE relative to R2R^2 selection by 17%, 13%, and 7% at c=0.10c = 0.10, 0.25, and 0.50.

Discussion

The two layers of evidence lead to the same conclusion: predictive-fit and RoAS-recovery rankings often disagree, the disagreement is systematic rather than an artifact of particular draws, and a modest gain in sales fit can coincide with a material loss in RoAS accuracy. Two qualifications matter for practice. RoAS-MAE is reported on the original RoAS scale, so the relevant comparisons are within-dataset rather than across scenarios. The value of benchmark-based selection also depends on the precision of the experiments, on media-spend collinearity, and on how far all candidates are from the truth, which argues for reporting experimental uncertainty alongside the calibration result.

When No Benchmark Is Available: RoAS Sensitivity and Consensus Diagnostics

Without an experimental benchmark, the RoAS-best specification cannot be identified, and predictive-fit metrics do not resolve this problem. One useful diagnostic is the distribution of RoAS estimates across plausible specifications. The componentwise median RoAS vector is a simple consensus summary: we select the specification whose RoAS vector is closest to it, using the known simulation RoAS only retrospectively to evaluate the choice. In the single-draw scenarios (Appendix Table C.2) this diagnostic chose a model with lower RoAS-MAE than the highest-R2R^2 model in 14 of 16 adstock-included and 13 of 16 adstock-omitted comparisons. The replicated evidence is more sobering: with adstock-included candidates the median-consensus model is only marginally more accurate than the R2R^2-selected model on average (mean RoAS-MAE ratio 0.96, better in 45% of datasets), and with adstock-omitted candidates it is substantially worse (1.69×, better in only 13%), because the median of a set of uniformly biased specifications is itself biased. Consensus summaries therefore cannot substitute for causal validation. When no benchmark is available, analysts should report the median, interquartile range, and full range of RoAS across plausible specifications as a sensitivity statement, not as a selected causal estimate.

Real-world Calibration Illustration for Applied MMM Practice

The empirical application illustrates the proposed evaluation logic in an applied MMM workflow, using anonymized weekly sales and marketing data from a consumer packaged goods advertiser selling through retail channels. Several plausible specifications produce similar predictive fit but materially different RoAS estimates, while experimental RoAS benchmarks provide an external calibration target. The objectives are to quantify specification uncertainty across saturation forms, to examine the relationship between predictive fit and RoAS recovery, and to illustrate the decision implications of specification uncertainty. Because the data are anonymized, some details of the experiments and raw data cannot be disclosed, and because the benchmarks are used as point estimates without uncertainty intervals, the exercise is a calibration illustration; it does not establish statistical dominance of one specification over another.

Data and Business Context

We analyze 156 weekly observations (3 years) across three digital channels: Paid Search, Social Media, and Display Advertising. The dependent variable is normalized log-transformed sales (log sales volume divided by average sales volume per client); the regressors are weekly spend in dollars for each channel, together with controls (Fourier seasonality terms, promotional indicators, pricing variables) and an intercept. Experimental RoAS benchmark estimates for each channel were obtained through causal inference experiments over the same period and serve as calibration targets, consistent with § 5.4.1. Unlike the simulations, the application uses the full dataset for estimation, a common practice when history is limited and the objective is RoAS quantification rather than out-of-sample prediction.

Model Specifications and Estimation

We estimate four candidate specifications (Model 1 = Hill C-shape, Model 2 = Hill S-shape, Model 3 = Weibull C-shape, Model 4 = Weibull S-shape), all sharing the same geometric adstock and control variables and differing only in the saturation form. Each is estimated via constrained maximum likelihood as in § 5, with non-negativity constraints on marketing coefficients.

Results and Model Comparison

Table 5 presents the estimated RoAS for each of the three channels across the four model specifications, along with goodness-of-fit metrics.

Model SpecSaturation FunctionR2R^2Paid Search RoASSocial Media RoASDisplay RoASAvg RoAS
Model 1Hill C-shape0.7822.132.473.812.80
Model 2Hill S-shape0.7781.632.773.812.74
Model 3Weibull C-shape0.7802.212.653.732.86
Model 4Weibull S-shape0.7621.971.583.302.28
Experimental Benchmark——2.242.633.312.73

Table 5. Real-World Application: Model Comparison Across Specifications

Notes: All models estimated on full 156-week dataset. Experimental RoAS benchmark estimates are obtained from causal inference experiments conducted over the same period. Model 1 achieves highest R2R^2 (0.782) but Model 3 achieves best RoAS recovery under the benchmark criterion.

Model Selection and RoAS Recovery

Using the experimental RoAS benchmark estimates, we calculate recovery metrics analogous to the simulation studies. Table 6 reports them for each specification, together with the outcomes of three selection criteria: R2R^2-based, benchmark RoAS-MAE-based, and the median-based heuristic of § 6.5.

Model SpecificationRoAS-MAERoAS-RMSERoAS-MAPER2R^2
Model 1 (Hill C-shape)0.2550.3088.63%0.782
Model 2 (Hill S-shape)0.4170.46415.92%0.778
Model 3 (Weibull C-shape)0.1590.2435.04%0.780
Model 4 (Weibull S-shape)0.4440.62417.46%0.762
Best by R2R^2Model 1RoAS-MAE = 0.255RoAS-MAPE = 8.63%
Best by RoAS-MAEModel 3RoAS-MAE = 0.159RoAS-MAPE = 5.04%
Median-Based SelectionModel 3RoAS-MAE = 0.159RoAS-MAPE = 5.04%
MAE Improvement Factor1.60×\times
Median Improvement Factor1.60×\times

Table 6. RoAS Recovery Performance Against Experimental RoAS Benchmarks

The results mirror the ranking-disagreement pattern observed in simulation. The model with the highest R2R^2 (Model 1: 0.782) is not the model with the best RoAS recovery under the benchmark criterion. Model 3 has a marginally lower R2R^2 (0.780 vs. 0.782) but a substantially lower RoAS-MAE (0.159 vs. 0.255), a 1.60×1.60 \times reduction that would be economically meaningful in media investment decisions and falls within the range observed in the low-complexity simulation scenarios (S1–S2), consistent with the predictive-fit paradox in § 6. The median-based consensus diagnostic (§ 6.5) also selects Model 3; this agreement does not validate the median rule, whose role remains descriptive.

Model 3’s advantage is consistent across channels. Its estimates (Paid Search: 2.21, Social Media: 2.65, Display: 3.73) are close to the experimental benchmark (2.24, 2.63, 3.31), with deviations of 1.3%, 0.8%, and 12.7%, respectively. By contrast, the R2R^2-selected Model 1 overestimates Display RoAS by 15.1%, which could materially affect investment recommendations.

Notes: RoAS-MAE = Mean Absolute Error against experimental RoAS benchmark estimates. RoAS-MAPE = Mean Absolute Percentage Error. Model 1 achieves highest R2R^2 (0.782) but Model 3 achieves best RoAS recovery under the benchmark criterion. The median-based row is retained as an exploratory consensus diagnostic; in this application it selects the same specification as the benchmark RoAS-MAE criterion.

Managerial Implications

Several implications follow. Predictive accuracy alone is insufficient for media investment decisions. A model can fit historical sales while assigning incremental revenue to the wrong channels. Experimental benchmarks should enter calibration after estimand alignment is checked; our replicated simulations indicate that even imprecise experiments are more informative for specification choice than sales fit, so the absence of a perfect experiment is not a reason to fall back on R2R^2. Without benchmarks, RoAS should be reported across plausible specifications rather than anchored to one predictive-fit-selected model. Forecasting and causal diagnostics should be reported separately: R2R^2, MAPE, and RMSE are useful, but not evidence of causal validity.

Concluding Remarks

The results document a practical disconnect between predictive accuracy and causal RoAS recovery in MMM. In the scenarios considered here, the best-fitting sales model is often not the model closest to the causal RoAS benchmark, and replication over hundreds of independent datasets shows that this disagreement is systematic: among structurally sound specifications that differ only in saturation form, out-of-sample R2R^2 is essentially uninformative about RoAS accuracy, and the disagreement persists across levels of media-spend collinearity and noise and under a misspecified carryover shape. Predictive diagnostics remain useful, but they are not sufficient evidence of causal validity for media-investment decisions.

Evaluation criteria should match the model’s use. Predictive metrics such as R2R^2, MAPE, and RMSE remain appropriate for forecasting. For causal RoAS interpretation, candidate MMM specifications should be compared with experimental benchmarks after checking estimand alignment. Without benchmarks, observational fit cannot identify the RoAS-optimal specification; analysts should report specification sensitivity, including ranges and consensus summaries across plausible specifications.

The evidence has limits. The replicated simulations quantify operating characteristics only for the data-generating processes, candidate sets, and spend processes considered; the numerical magnitudes should not be read as universal benchmarks. When all candidates share a severe structural omission, such as missing carryover, predictive fit becomes more informative and the advantage of benchmark-based selection narrows, disappearing when experiments are very imprecise. All candidate models use geometric carryover, and the adstock-form study covers a single alternative kernel; richer carryover representations, such as the rational transfer functions advocated by Palu [24], are a complementary route, but whatever carryover model is used, experimental calibration remains the external check on RoAS. The real-world application treats experimental benchmarks as point estimates and does not report experimental uncertainty intervals. The RoAS estimand used in the paper is an average incremental return over the observed spend path, not a marginal return to the next dollar of spend. The median-consensus diagnostic is exploratory and descriptive; it does not replace experimental calibration. Future research should extend the operating-characteristic analysis to larger channel sets, experiments that cover only a subset of channels, endogenous spend processes, and marginal rather than average RoAS estimands. Other promising extensions include uncertainty-weighted benchmark loss functions, spline-based or machine-learning response curves, cross-channel interactions, multi-product portfolios, and Bayesian posterior calibration against experimental benchmarks.

Data and Code Availability: The Python codes (including the replication code for all simulation studies) and the simulated datasets are available upon request. The real-world dataset is anonymized and cannot be publicly disclosed because of client confidentiality.

Disclosure of interest: The authors declare no competing interests. The views expressed are the authors’ own and do not represent those of their affiliated organizations.

Declaration of funding: No external grants, institutional funding, or sponsorship of any kind supported the preparation of this manuscript.

Details of the Motivating Example

Table A.1 lists the variables of the data-generating process in Equation (1), Table A.2 reports the parameters used to generate the motivating-example data, and Figure A.1 compares the two models on out-of-sample sales metrics.

VariableDescriptionTypeDimension
Response Variable
ylty_{lt}Sales unitsResponseL×TL \times T
Media Variables
xi,ltx_{i,lt}Media spend in channel ii (dollars)PredictorC×L×TC \times L \times T
mi,ltm_{i,lt}Transformed media (adstock + saturation)DerivedC×L×TC \times L \times T
Pricing & Promo Variables
pltp_{lt}Price (dollars)PredictorL×TL \times T
dltd_{lt}Discount percentagePredictorL×TL \times T
Macroeconomic Variables
utu_tUnemployment ratePredictorTT
ctc_tConsumer Confidence IndexPredictorTT
Holiday Indicators
h1,th_{1,t}Independence Day indicatorPredictorTT
h2,th_{2,t}Thanksgiving indicatorPredictorTT
h3,th_{3,t}Christmas indicatorPredictorTT
Seasonality, Trend and Location Variables
ttTime trend (weeks)PredictorTT
sin⁡(2πt52)\sin\left(\frac{2\pi t}{52}\right)Annual seasonality (sine component)PredictorTT
cos⁡(2πt52)\cos\left(\frac{2\pi t}{52}\right)Annual seasonality (cosine component)PredictorTT
1{l=j}\mathbb{1}\{l=j\}Location jj indicatorPredictorLL

Table A.1. Variable Definitions and Descriptions.

Saturation Functions and Estimation Details

Hill and Weibull Saturation Families

Hill Saturation Functions. The Hill function, originally developed in biochemistry to model enzyme kinetics, provides flexible saturation behavior controlled by two parameters:

fHill(z;K,O)=zOKO+zO(11)f_{\mathrm{Hill}}(z; K, O) = \frac{z^{O}}{K^{O} + z^{O}} \tag*{(11)}

where K>0K > 0 is the half-saturation point (i.e., fHill(K;K,O)=0.5f_{\mathrm{Hill}}(K; K, O) = 0.5) and O>0O > 0 controls the curve shape.

ParameterCh 1Ch 2Ch 3Ch 4
Media Channel Parameters
Media coefficient (βi\beta_i)1.150.850.900.55
Adstock decay (αi\alpha_i)0.400.500.300.60
Saturation half-point (KiK_i)8000600050004000
Saturation shape (OiO_i)2.53.02.23.5
Known simulation RoAS (computed)2.482.442.061.46
Base and Location Parameters
Intercept: β0=7.5\beta_0 = 7.5
Pricing and Economic Parameters
Price elasticity: βp=−1.2\beta_p = -1.2Discount: βd=0.015\beta_d = 0.015
Unemployment: βu=−0.05\beta_u = -0.05Consumer confidence: βc=0.01\beta_c = 0.01
Temporal Parameters
Time trend: βt=0.001\beta_t = 0.001
Seasonality: βs(1)=0.08\beta_s^{(1)} = 0.08, βs(2)=0.05\beta_s^{(2)} = 0.05
Holiday effects: γ1=0.10\gamma_1 = 0.10 (Independence), γ2=0.15\gamma_2 = 0.15 (Thanksgiving), γ3=0.20\gamma_3 = 0.20 (Christmas)
Error Term
Standard deviation: σ=0.05\sigma= 0.05

Table A.2. Benchmark Parameters for Data Generation

Sales prediction accuracy comparison for Model A and Model B across R², MAPE, and RMSE

Figure A.1. Sales Prediction Accuracy Comparison. Model B (misspecified) outperforms Model A (correct) on all out-of-sample metrics: R2R^2 (0.93 vs. 0.90), MAPE (9.07% vs. 10.38%), and RMSE (138.55 vs. 169.13). The comparison concerns sales prediction only and should not be interpreted as evidence of causal validity.

  • Hill S-shape (O>1O > 1): Sigmoidal saturation curve

fHill−S(z;K,O)=zOKO+zO,O>1(12)f_{\mathrm{Hill-S}}(z;K,O)=\frac{z^{O}}{K^{O}+z^{O}},\quad O>1 \tag*{(12)}

This specification produces accelerating returns initially, followed by saturation. Larger values of OO increase the sharpness of the transition between the low-spend and high-spend regimes. Common in brand-building campaigns where awareness must reach critical mass.

  • Hill C-shape (0<O≤10 < O \le1): C-shaped saturation curve

fHill−C(z;K,O)=zOKO+zO,0<O≤1(13)f_{\mathrm{Hill-C}}(z;K,O)=\frac{z^{O}}{K^{O}+z^{O}},\quad0<O\le1 \tag*{(13)}

This specification produces strong initial returns that diminish progressively. Smaller values of OO increase concavity, representing rapidly saturating channels where marginal effectiveness declines sharply with spending. Typical in direct response channels with audience saturation dynamics.

The parameter OO determines whether the response is purely concave or sigmoidal. For the Hill function, when 0<O≤10 < O \le1, the curve is everywhere concave. When O>1O > 1, the curve is sigmoidal: it is locally convex at low spend, locally concave at high spend, and has inflection point

zinfHill=K(O−1O+1)1/O.z_{\mathrm{inf}}^{\mathrm{Hill}}=K\left(\frac{O-1}{O+1}\right)^{1/O}.

Weibull Saturation Functions. The Weibull cumulative distribution function, widely used in reliability engineering and survival analysis, offers an alternative parameterization of saturation:

fWeibull(z;K,O)=1−exp⁡(−(zK)O)(14)f_{\mathrm{Weibull}}(z;K,O)=1-\exp\left(-\left(\frac{z}{K}\right)^{O}\right) \tag*{(14)}

where K>0K > 0 is the scale parameter (the point at which fWeibull(K;K,O)≈0.632f_{\mathrm{Weibull}}(K;K,O)\approx0.632) and O>0O > 0 controls the shape.

  • Weibull S-shape (O>1O > 1): S-shaped saturation curve

fWeibull−S(z;K,O)=1−exp⁡(−(zK)O),O>1(15)f_{\mathrm{Weibull-S}}(z;K,O)=1-\exp\left(-\left(\frac{z}{K}\right)^{O}\right),\quad O>1 \tag*{(15)}

This specification produces slow initial response, accelerating growth, then saturation. Larger values of OO create sharper transitions at the inflection point. The Weibull S-shape differs mathematically from the Hill S-shape, offering an alternative functional form for threshold-driven response patterns.

  • Weibull C-shape (0<O≤10 < O \le1): Concave saturation curve

fWeibull−C(z;K,O)=1−exp⁡(−(zK)O),0<O≤1(16)f_{\mathrm{Weibull-C}}(z;K,O)=1-\exp\left(-\left(\frac{z}{K}\right)^O\right),\quad0<O\le1 \tag*{(16)}

This specification produces strong initial returns that diminish progressively. Smaller values of OO increase concavity, representing rapidly saturating channels where marginal effectiveness declines sharply with spending. Common in mature, efficiency-focused channels like search advertising.

For the Weibull function, when 0<O≤10 < O \le1, the curve is everywhere concave. When O>1O > 1, the curve is sigmoidal with inflection point

zinfWeibull=K(O−1O)1/O.z_{\mathrm{inf}}^{\mathrm{Weibull}}=K\left(\frac{O-1}{O}\right)^{1/O}.

Comparative Properties. While both Hill and Weibull functions can produce concave or sigmoidal response curves, they differ in several respects:

  • Mathematical origin: Hill functions arise from rational polynomials, whereas Weibull functions arise from exponential transformations.

  • Half-saturation vs. scale: For the Hill function, fHill(K;K,O)=0.5f_{\mathrm{Hill}}(K;K,O)=0.5, whereas for the Weibull function, fWeibull(K;K,O)=1−e−1≈0.632f_{\mathrm{Weibull}}(K;K,O)=1-e^{-1}\approx0.632.

  • Inflection points: For O>1O>1, the Hill and Weibull sigmoidal curves have different inflection-point locations, given above.

  • Low-spend behavior: The two families can differ materially in sensitivity near zero spend, implying different marginal-RoAS profiles in the low-spend region.

These differences reflect different assumptions about marketing dynamics. Specification choice between these forms should be guided by alignment with observed data patterns and, when available, by agreement with estimand-aligned RoAS benchmarks. Figure 2 in § 5 illustrates the four saturation functions.

Likelihood and Constrained Estimation

To keep the estimation procedure aligned with the model specification in Equation (6), we estimate the model by bound-constrained maximum likelihood. Let

y~lt≡log⁡(ylt),\widetilde{y}_{lt}\equiv\log(y_{lt}),

and define the linear predictor

ηlt(Θ)=β0+∑j=2LβL,j1{l=j}+∑i=1C(βi+∑j=2Lβi,j1{l=j})mi,lt+Xlt′γ,(17)\eta_{lt}(\Theta)=\beta_0+\sum_{j=2}^{L}\beta_{L,j}\mathbb{1}\{l=j\}+\sum_{i=1}^{C}\left(\beta_i+\sum_{j=2}^{L}\beta_{i,j}\mathbb{1}\{l=j\}\right)m_{i,lt}+\mathbf{X}_{lt}^{\prime}\gamma, \tag*{(17)}

where mi,lt=fsat(gadstock(xi,lt;αi);Ki,Oi)m_{i,lt}=f_{\mathrm{sat}}(g_{\mathrm{adstock}}(x_{i,lt};\alpha_i);K_i,O_i).

We assume

y~lt=ηlt(Θ)+εlt,εlt∼iidN(0,σ2).\tilde{y}_{lt}=\eta_{lt}(\Theta)+\varepsilon_{lt},\qquad\varepsilon_{lt}\mathrel{\overset{\mathrm{iid}}{\sim}}\mathcal{N}(0,\sigma^2).

Hence, the conditional density of y~lt\tilde{y}_{lt} given Θ\Theta is

f(y~lt∣Θ)=12πσexp⁡{−(y~lt−ηlt(Θ))22σ2}.(18)f(\tilde{y}_{lt}\mid\Theta)=\frac{1}{\sqrt{2\pi}\sigma}\exp\left\{-\frac{\left(\tilde{y}_{lt}-\eta_{lt}(\Theta)\right)^2}{2\sigma^2}\right\}. \tag*{(18)}

The sample log-likelihood is therefore

ℓ(Θ)=−LT2log⁡(2π)−LTlog⁡σ−12σ2∑l=1L∑t=1T(y~lt−ηlt(Θ))2.(19)\ell(\Theta)=-\frac{LT}{2}\log(2\pi)-LT\log\sigma-\frac{1}{2\sigma^2}\sum_{l=1}^{L}\sum_{t=1}^{T}\left(\tilde{y}_{lt}-\eta_{lt}(\Theta)\right)^2. \tag*{(19)}

We maximize ℓ(Θ)\ell(\Theta) subject to economically motivated sign and shape constraints:

max⁡Θℓ(Θ)subject toβi≥0i=1,…,Cβi+βi,j≥0i=1,…,C, j=2,…,L0≤αi<1i=1,…,CKi>0, Oi>0i=1,…,C.(20)\begin{aligned} \max_{\Theta}\quad&\ell(\Theta)\\ \text{subject to}\quad&\beta_i\geq0 && i=1,\ldots,C\\ &\beta_i+\beta_{i,j}\geq0 && i=1,\ldots,C,\ j=2,\ldots,L\\ &0\leq\alpha_i<1 && i=1,\ldots,C\\ &K_i>0,\ O_i>0 && i=1,\ldots,C. \tag*{(20)} \end{aligned}

The optimization problem is nonlinear and non-convex because of the recursive adstock transformation and the nonlinear saturation functions. We therefore use multi-start bound-constrained optimization, initialized from several plausible starting values, and retain the solution with the highest attained log-likelihood. The nonnegativity constraints on total location-specific media effects, βi+βi,j≥0\beta_i+\beta_{i,j}\geq0, are handled by reparameterization, so that the constrained effects are nonnegative by construction. In our implementation, we use L-BFGS-B as the primary optimizer for the resulting bound-constrained problem and check the solution with Sequential Quadratic Programming (SQP). In practice, initializing transformation parameters near empirically plausible values (e.g., αi∈[0.3,0.6]\alpha_i\in[0.3,0.6], KiK_i near 1.0) and regression coefficients from an unconstrained ordinary least squares fit (with negative coefficients set to small positive values) provides stable starting points for the optimization.

Additional Results for the Single-Draw Simulation Scenarios

DGPScenarioTraditional (R2R^2 Max)Benchmark (RoAS-MAE Min)
R2R^2RoAS-MAERoAS-MAPER2R^2RoAS-MAERoAS-MAPE
Hill CS10.6220.72254.83%0.2340.63646.71%
S20.1831.43547.55%0.1431.19638.24%
S30.26410.95649.00%0.2649.24341.59%
S40.31736.17852.37%0.28927.41843.28%
Average0.34712.32350.94%0.2329.62342.45%
Improvement—1.28×
Hill SS10.6620.83520.34%0.6620.76020.83%
S20.2457.79749.71%0.0645.82336.31%
S30.46368.42334.07%0.31236.58520.80%
S40.45682.34023.38%0.45682.34023.38%
Average0.45739.84931.88%0.37331.37725.33%
Improvement—1.27×
Weibull CS10.4720.69337.85%0.4720.51329.43%
S20.3974.41835.60%0.3923.24728.74%
S30.24340.75628.68%0.24329.83422.57%
S40.31472.38436.24%0.27154.21728.36%
Average0.35629.56334.59%0.34521.95327.27%
Improvement—1.35×
Weibull SS10.4812.57745.55%0.3791.69226.36%
S20.1298.85332.81%0.1296.41324.57%
S30.26362.34731.67%0.26344.82324.83%
S40.14189.21431.15%0.14171.53825.41%
Average0.25440.74835.30%0.22831.11625.29%
Improvement—1.31×

Table C.1. Model Selection Performance: Adstock-Omitted Candidate Models

Notes: All models omit the adstock transformation present in the DGP. Improvement factor = Traditional RoAS-MAE / Benchmark RoAS-MAE.

Spec ConditionDGPScenarioTraditional (R2R^2 Max) Selected ModelTraditional (R2R^2 Max) RoAS-MAEMedian-Based Selected ModelMedian-Based RoAS-MAEImprov. Factor
Adstock-incl.Hill CS1Weibull C-shape0.554Hill C-shape0.6830.81×
Adstock-incl.Hill CS2Hill C-shape1.947Weibull C-shape1.5231.28×
Adstock-incl.Hill CS3Hill C-shape25.114Weibull C-shape9.4872.65×
Adstock-incl.Hill CS4Weibull C-shape33.424Hill C-shape24.8371.35×
Adstock-incl.Hill SS1Weibull S-shape2.235Weibull C-shape2.4180.92×
Adstock-incl.Hill SS2Hill C-shape6.497Hill S-shape4.9831.30×
Adstock-incl.Hill SS3Weibull C-shape47.968Hill S-shape43.5171.10×
Adstock-incl.Hill SS4Weibull C-shape75.979Hill S-shape67.2841.13×
Adstock-incl.Weibull CS1Weibull C-shape1.228Hill S-shape0.9211.33×
Adstock-incl.Weibull CS2Hill C-shape4.213Hill C-shape3.2841.28×
Adstock-incl.Weibull CS3Weibull C-shape50.187Weibull C-shape37.4181.34×
Adstock-incl.Weibull CS4Hill C-shape60.347Hill C-shape59.8731.01×
Adstock-incl.Weibull SS1Weibull S-shape1.624Weibull S-shape1.2741.27×
Adstock-incl.Weibull SS2Weibull C-shape11.041Hill C-shape8.7631.26×
Adstock-incl.Weibull SS3Hill C-shape66.831Weibull C-shape54.9171.22×
Adstock-incl.Weibull SS4Weibull C-shape98.129Weibull C-shape68.3341.44×
Adstock-omit.Hill CS1Hill C-shape0.722Hill C-shape0.5741.26×
Adstock-omit.Hill CS2Hill C-shape1.435Hill S-shape1.8470.78×
Adstock-omit.Hill CS3Hill C-shape10.956Hill C-shape8.4371.30×
Adstock-omit.Hill CS4Hill C-shape36.178Hill C-shape27.3841.32×
Adstock-omit.Hill SS1Weibull C-shape0.835Hill S-shape0.6471.29×
Adstock-omit.Hill SS2Weibull C-shape7.797Hill C-shape5.7061.37×
Adstock-omit.Hill SS3Hill C-shape68.423Weibull S-shape49.3181.39×
Adstock-omit.Hill SS4Hill C-shape82.340Hill S-shape73.2841.12×
Adstock-omit.Weibull CS1Hill C-shape0.693Hill C-shape0.5371.29×
Adstock-omit.Weibull CS2Weibull C-shape4.418Hill S-shape5.1830.85×
Adstock-omit.Weibull CS3Hill C-shape40.756Hill C-shape30.8471.32×
Adstock-omit.Weibull CS4Hill C-shape72.384Weibull C-shape58.7241.23×
Adstock-omit.Weibull SS1Weibull C-shape2.577Weibull S-shape3.1840.81×
Adstock-omit.Weibull SS2Weibull C-shape8.853Hill C-shape6.8471.29×
Adstock-omit.Weibull SS3Hill C-shape62.347Hill C-shape54.8371.14×
Adstock-omit.Weibull SS4Weibull C-shape89.214Hill S-shape67.4831.32×
Adstock-included (16 scenarios)—Median better: 14/161.29×
Adstock-omitted (16 scenarios)—Median better: 13/161.19×

Table C.2. Exploratory Comparison of R2R^2-Based Selection and Median-RoAS Consensus Proximity

Notes: The median-based diagnostic is one example of a specification-consensus rule. It selects the model whose channel-level RoAS vector is closest to the componentwise median RoAS vector across candidate specifications. The median vector is a descriptive summary of model dispersion. It is neither an experimental benchmark nor a substitute for a causal benchmark. Known simulation RoAS values are used only retrospectively to evaluate the selected models in the simulation setting. Improvement factor = R2R^2-based RoAS-MAE / median-consensus RoAS-MAE; values above one indicate that the median-consensus diagnostic selected a model with lower retrospective RoAS-MAE than the highest-R2R^2 model.

Replicated Simulation Studies: Cell-Level Results

This appendix reports the cell-level results of the replicated studies summarized in § 6.3. All studies reuse the DGP distributions, candidate models, optimizer settings (multi-start L-BFGS-B with five starts), 156/52-week split, and RoAS definition of § 6.1; only the replicated quantities are redrawn. Code and raw results are in the additional_runs folder of the replication package. Table D.1 and Figure D.1 report the Monte Carlo replication by DGP and complexity scenario, Tables D.2 and D.3 report the collinearity and noise studies by DGP, and Figure D.2 plots the agreement rate and the RoAS-error penalty of R2R^2 selection against ρ\rho and σ\sigma. Table D.4 reports the adstock-form study by DGP.

CandidatesScen.DGPRRPr(agree) [95% CI]τˉ\bar{\tau}MAE ratio (mean)MAE ratio (median)ΔR2\Delta R^2Exp-25 / R2R^2Median / R2R^2
IncludedS1Hill C400.10 [0.04, 0.23]-0.011.901.850.3020.660.77
IncludedS1Hill S400.15 [0.07, 0.29]-0.201.571.450.1070.750.97
IncludedS1Weibull C400.10 [0.04, 0.23]-0.011.481.260.1880.810.94
IncludedS1Weibull S400.38 [0.24, 0.53]0.021.341.200.1110.871.09
IncludedS2Hill C400.20 [0.10, 0.35]0.101.461.230.3860.820.97
IncludedS2Hill S400.20 [0.10, 0.35]-0.081.291.260.0930.860.97
IncludedS2Weibull C400.20 [0.10, 0.35]0.071.321.190.3200.871.01
IncludedS2Weibull S400.07 [0.03, 0.20]-0.021.391.320.1160.820.94
OmittedS1Hill C400.53 [0.37, 0.67]0.591.251.000.0730.901.81
OmittedS1Hill S400.53 [0.37, 0.67]0.651.151.000.0230.971.54
OmittedS1Weibull C400.57 [0.42, 0.71]0.621.161.000.0360.942.12
OmittedS1Weibull S400.55 [0.40, 0.69]0.611.271.000.0390.891.68
OmittedS2Hill C400.75 [0.60, 0.86]0.711.071.000.0260.991.76
OmittedS2Hill S400.53 [0.37, 0.67]0.601.121.000.0260.961.45
OmittedS2Weibull C400.53 [0.37, 0.67]0.591.101.000.0300.951.80
OmittedS2Weibull S400.45 [0.31, 0.60]0.611.171.050.0350.921.39

Table D.1. Monte Carlo Replication of the Main Design: Results by Cell

Notes: RR = number of independent replications (each redraws DGP parameters, spend, controls and noise). Pr(agree) = share of replications in which the highest-R2R^2 candidate is also the lowest-RoAS-MAE candidate (Wilson 95% interval). τˉ\bar{\tau} = mean Kendall rank correlation between the R2R^2 ranking and the RoAS-accuracy ranking of the four candidates. MAE ratio = RoAS-MAE of the R2R^2-selected model / RoAS-MAE of the RoAS-best model (≥1\ge1). ΔR2\Delta R^2 = mean R2R^2 of the R2R^2-selected model minus that of the RoAS-best model. Exp-25 / R2R^2 = mean ratio of the RoAS-MAE obtained when selecting against a noisy experimental benchmark (25% coefficient of variation per channel) to that of R2R^2 selection; values below one favour the benchmark. Median / R2R^2 = same ratio for the benchmark-free median-consensus diagnostic.

Monte Carlo distribution of the RoAS-error penalty from predictive-fit selection

Figure D.1. Monte Carlo distribution of the RoAS-error penalty from predictive-fit selection: RoAS-MAE of the highest-R2R^2 candidate divided by that of the RoAS-best candidate (log scale; 1×1\times means R2R^2 selected the RoAS-best model). Each point is one of 40 independent datasets per cell; black bars are medians.

ρ\rhoDGPRRPr⁡(agree)\Pr(\text{agree}) [95% CI]τˉ\bar{\tau}MAE ratio (mean)MAE ratio (median)ΔR2\Delta R^2Exp-25 / R2R^2Median / R2R^2
0Hill C150.53 [0.30, 0.75]0.091.171.000.3200.961.05
0Hill S150.20 [0.07, 0.45]-0.131.211.150.1350.900.99
0Weibull C150.40 [0.20, 0.64]-0.021.261.100.3670.901.05
0Weibull S150.20 [0.07, 0.45]-0.091.401.360.1220.851.00
0Pooled600.33 [0.23, 0.46]-0.041.261.120.2360.901.02
0.3Hill C150.47 [0.25, 0.70]0.161.371.060.2360.871.01
0.3Hill S150.20 [0.07, 0.45]-0.091.301.210.1040.851.01
0.3Weibull C150.27 [0.11, 0.52]0.241.341.150.2690.871.02
0.3Weibull S150.27 [0.11, 0.52]-0.111.351.100.0930.871.00
0.3Pooled600.30 [0.20, 0.43]0.051.341.180.1750.861.01
0.6Hill C150.47 [0.25, 0.70]0.071.261.100.3270.900.96
0.6Hill S150.07 [0.01, 0.30]-0.241.541.510.0840.740.98
0.6Weibull C150.33 [0.15, 0.58]0.131.311.030.1850.900.97
0.6Weibull S150.07 [0.01, 0.30]0.071.431.320.1720.800.95
0.6Pooled600.23 [0.14, 0.35]0.011.391.300.1920.830.97
0.9Hill C150.33 [0.15, 0.58]0.041.301.060.2120.880.98
0.9Hill S150.20 [0.07, 0.45]-0.201.371.370.0790.800.97
0.9Weibull C150.20 [0.07, 0.45]0.001.391.210.1740.850.98
0.9Weibull S150.07 [0.01, 0.30]-0.271.371.310.2690.811.06
0.9Pooled600.20 [0.12, 0.32]-0.111.361.300.1840.830.99

Table D.2. Robustness to Media-Spend Collinearity: Results by DGP (S2, adstock-included candidates). Notes: Columns as in Table D.1. The same random seeds are used at every ρ\rho, so differences across rows reflect only the spend correlation; the baseline design of § 6 uses ρ=0.3\rho= 0.3.

σ\sigmaDGPRRPr(agree) [95% CI]τˉ\bar{\tau}MAE ratio (mean)MAE ratio (median)ΔR2\Delta R^2Exp-25 / R2R^2Median / R2R^2
0.02Hill C150.47 [0.25, 0.70]0.041.191.110.3900.911.05
0.02Hill S150.20 [0.07, 0.45]0.041.231.120.0770.880.97
0.02Weibull C150.13 [0.04, 0.38]0.181.341.360.2810.841.04
0.02Weibull S150.13 [0.04, 0.38]−0.111.301.300.1380.860.98
0.02Pooled600.23 [0.14, 0.35]0.041.271.250.2220.871.01
0.05Hill C150.47 [0.25, 0.70]0.201.201.120.2250.911.00
0.05Hill S150.33 [0.15, 0.58]0.091.191.080.0590.910.99
0.05Weibull C150.27 [0.11, 0.52]0.071.271.170.2740.881.05
0.05Weibull S150.20 [0.07, 0.45]−0.131.321.330.0850.860.95
0.05Pooled600.32 [0.21, 0.44]0.061.251.160.1610.891.00
0.1Hill C150.20 [0.07, 0.45]0.091.381.200.3610.840.88
0.1Hill S150.47 [0.25, 0.70]0.181.201.030.0500.931.06
0.1Weibull C150.20 [0.07, 0.45]0.111.291.210.2520.861.02
0.1Weibull S150.33 [0.15, 0.58]0.021.211.070.0790.921.09
0.1Pooled600.30 [0.20, 0.43]0.101.271.140.1850.881.01
0.2Hill C150.33 [0.15, 0.58]0.201.201.080.2550.930.93
0.2Hill S150.27 [0.11, 0.52]0.001.261.250.0730.871.02
0.2Weibull C150.27 [0.11, 0.52]0.311.261.150.1510.910.99
0.2Weibull S150.33 [0.15, 0.58]0.201.201.060.0720.931.06
0.2Pooled600.30 [0.20, 0.43]0.181.231.140.1370.911.00

Table D.3. Robustness to Noise Level: Results by DGP (S2, adstock-included candidates). Notes: Columns as in Table D.1. The same random seeds are used at every σ\sigma, so differences across rows reflect only the noise level; the baseline design of § 6 uses σ=0.02\sigma= 0.02.

DGPRRPr(agree) [95% CI]τˉ\bar{\tau}MAE ratio (mean)MAE ratio (median)ΔR2\Delta R^2Exp-25 / R2R^2Median / R2R^2
Hill C150.33 [0.15, 0.58]0.161.341.180.1830.851.03
Hill S150.27 [0.11, 0.52]0.111.281.310.0450.871.02
Weibull C150.27 [0.11, 0.52]0.091.231.250.2660.911.12
Weibull S150.20 [0.07, 0.45]0.001.261.210.1070.860.97
Pooled600.27 [0.17, 0.39]0.091.281.220.1500.871.04

Table D.4. Adstock-Form Misspecification: Results by DGP (S2; delayed-adstock DGP, geometric-adstock candidates). Notes: Columns as in Table D.1. Sales are generated with a delayed adstock kernel wk∝α(k−θ)2w_k \propto\alpha^{(k-\theta)^2} (k=0,…,12k = 0, \ldots, 12), peak lag θ∈{1,2}\theta\in\{1, 2\} drawn per channel and total carryover matched to the geometric kernel; all candidates use geometric adstock, so carryover is misspecified for every candidate while the true saturation family remains among them.

Collinearity and noise studies, S2 with adstock-included candidates

Figure D.2. Collinearity (left) and noise (right) studies, S2 with adstock-included candidates. Top: share of datasets in which the highest-R2R^2 candidate is also the RoAS-best candidate (pooled line with 95% Wilson band; dashed line = random choice among four candidates). Bottom: mean RoAS-MAE of the R2R^2-selected model relative to the RoAS-best model. Colored lines show individual DGPs; black lines pool the four DGPs.

References

  1. [1]Abadie, A., Diamond, A., Hainmueller, J., 2010. Synthetic control methods for comparative case studies: Estimating the effect of california’s tobacco control program. Journal of the American Statistical Association 105, 493–505.
  2. [2]Barajas, J., Zidar, T., Bay, M., 2020. Advertising incrementality measurement using controlled geo-experiments: The universal app campaign case study. ACM: Washington, DC, USA .
  3. [3]Chan, D., Perry, M., 2017. Challenges and opportunities in media mix modeling. Google Inc 16.
  4. [4]Chen, H., Zhang, M., Han, L., Lim, A., 2021. Hierarchical marketing mix models with sign constraints. Journal of Applied Statistics 48, 2944–2960.
  5. [5]Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., Robins, J., 2018. Double/debiased machine learning for treatment and structural parameters.
  6. [6]Chivukula, V., 2024. The role of adstock and saturation curves in marketing mix models: Implications for accuracy and decision-making. International Journal of Advanced Research in Computer Science & Technology (IJARCST) 7, 10002–10007.DOI
  7. [7]Contreras-Pena, M.F., Victoria-Mas, J.S., Martin-Barreiro, C., Cabezas, X., 2026. A statistical software framework for marketing analytics: Methodology, simulations, case study, and a Python library. Journal of Marketing Analytics , 1–42.
  8. [8]Dew, R., Padilla, N., Shchet kina, A., 2024. Your MMM is broken: Identification of nonlinear and time-varying effects in marketing mix models. arXiv preprint arXiv:2408.07678 .arxiv.org/abs/2408.07678
  9. [9]Ding, P., Li, F., 2018. Causal inference. Statistical Science 33, 214–237.
  10. [10]Ebbes, P., Papies, D., van Heerde, H.J., 2021. Dealing with endogeneity: A nontechnical guide for marketing researchers, in: Handbook of market research. Springer, pp. 181–217.
  11. [11]Farris, P.W., Bendle, N., Pfeifer, P., Reibstein, D., 2010. Marketing metrics: The definitive guide to measuring marketing performance. Pearson Education.DOI
  12. [12]Gaur, J., Bharti, K., 2020. Attribution modelling in marketing: Literature review and research agenda. Academy of Marketing Studies Journal 24, 1–21.
  13. [13]Google, 2024. Meridian. https://github.com/google/meridian. Accessed: 2026-02-28.
  14. [14]Hanssens, D.M., Parsons, L.J., Schultz, R.L., 1990. Market Response Models: Econometric and Time Series Analysis. Kluwer Academic Publishers, Boston, MA.DOI
  15. [15]Hanssens, D.M., Parsons, L.J., Schultz, R.L., 2001. Market response models: Econo-metric and time series analysis. Springer.
  16. [16]Huang, H.C., Xu, J., Lim, A., 2024. Marketing mix optimisation with practical constraints. International Journal of Operational Research 51, 128–149.DOI
  17. [17]Iacobucci, D., Petrescu, M., Krishen, A., Bendixen, M., 2019. The state of marketing analytics in research and practice. Journal of Marketing Analytics 7, 152–181.DOI
  18. [18]Jin, Y., Wang, Y., Sun, Y., Chan, D., Koehler, J., 2017. Bayesian methods for media mix modeling with carryover and shape effects. Google Research , 1–34.
  19. [19]Katiaj, P., Koukოპულos, A., Farmakis, T., Fra idaki, K., 2024. Increasing roas on performance marketing: Leveraging cgc and video content on social media, in: Proceedings of the International Conference on Contemporary Marketing Issues.
  20. [20]Kerman, J., Wang, P., Vaver, J., 2017. Estimating Ad Effectiveness Using Geo Experiments in a Time-Based Regression Framework. Technical Report. Google.
  21. [21]Liu, Y., Laguna, J., Wright, M., He, H., 2014. Media mix modeling—a Monte Carlo simulation study. Journal of Marketing Analytics 2, 173–186.
  22. [22]Meta, 2022. Robyn. https://github.com/facebookexperimental/Robyn. Accessed: 2026-02-28.
  23. [23]Newell, J., Pilotta, J.J., Thomas, J.C., 2008. Mass media displacement and saturation. The International Journal on Media Management 10, 131–138.DOI
  24. [24]Palu, S., 2026. Marketing mix modeling adstock identification is just guesswork. LinkedIn article, 2 November 2026. https://www.linkedin.com/pulse/marketing-mix-modeling-adstock-parameter-just-guesswork-sione-palu-zeppe. Accessed: 2026-11-04.
  25. [25]Rossi, P.E., 2014. Even the rich can make themselves poor: A critical examination of IV methods in marketing applications. Marketing Science 33, 655–672.
  26. [26]Rukh, S., Oziri, S.T., Seyi-Lande, O.B., 2023. Framework for enhancing marketing strategy through predictive and prescriptive analytics. Shodhshauryam, International Scientific Refereed Research Journal 6, 531–569.DOI
  27. [27]Sun, Y., Wang, Y., Jin, Y., Chan, D., Koehler, J., 2017. Geo-level bayesian hierarchical media mix modeling. Google Research technical report.
  28. [28]Vaver, J., Koehler, J., 2011. Measuring ad effectiveness using geo experiments. Google white paper.
  29. [29]Wang, K., Wang, Y., Yao, J., 2005. A comparative study on marketing mix models for digital products, in: International Workshop on Internet and Network Economics, Springer. pp. 660–669.DOI
  30. [30]Wang, Y., Jin, Y., Sun, Y., Chan, D., Koehler, J., 2017. A hierarchical bayesian approach to improve media mix models using category data. Google Inc., research. google. com .
  31. [31]Wigren, R., Cornell, F., 2019. Marketing mix modelling: A comparative study of statistical models.

Paper details

vlnc:W_yfc5bhp2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Not recorded for this paper

View statement
Looking for related papers…
Contents