Introduction
Large-scale customer relationship management (CRM) data recording the behavior of millions or more customers is now commonplace in industries like banking [48 ] , telecommunications [80 ] , and many others. Although increasingly large in length, CRM data often remains highly sparse at the level of individual customers, due in part to the common pattern of attrition by which customers only actively transact for short periods of time, after which they churn or “die” [63 ] . Large customer bases are moreover naturally heterogeneous, with many customers exhibiting a variety of different purchasing patterns [2 ] . This combination of size, sparsity, and heterogeneity motivates statistical models of CRM data that strike a particular balance of being 1) scalable , capable of being efficiently fit to millions of customers, 2) parsimonious , allowing for statistical information to be pooled across sparsely observed units, but also 3) sufficiently expressive , not collapsing salient and predictive heterogeneity in customer behavior.
Probabilistic models following the “Buy ‘Til You Die” (BTYD) framework of [68 ] are a mainstay of statistical marketing [37 , 27 ] , owing in part to their parsimonious representation of customer behavior. This representation splits cleanly into: 1) a transaction process, governing the timing by which a customer “buys”, and 2) a lifetime process, governing the time until a customer “dies”. A small set of such models—e.g., the original Pareto/NBD of [68 ] , or the BG/NBD model of [30 ] —are simple enough to be fit scalably to large datasets using gradient-based optimization schemes which exploit such models’ analytic tractability. While these “workhorse” models account for a large share of practical usage, a large literature has since introduced many new variants which tend to trade analytic and computational tractability for expressivity, introducing more flexible probabilistic assumptions that then necessitate approximate inference schemes like MCMC for parameter estimation [58 , 1 , 59 , 22 ] .
Despite the wide variety of BTYD models to date however, very few depart from the analytically convenient assumption that the event times of an active customer’s transactions follow a Poisson process. The Poisson has long been the canonical model for “pure chance” events in many fields, with departures from it indicating the presence of unaccounted for structure. In marketing, transaction patterns that are either more regular or more “clumpy” than expected under a Poisson process have been highlighted and studied for several decades [15 , 36 , 92 , 60 ] . Nevertheless, the lack of scalable non-Poisson models for heterogeneous patterns of timing in large-scale customer bases remains a major gap.
This paper fills that gap by introducing a family of non-Poisson BTYD models which enjoy almost all the core benefits of the “workhorse” models—namely 1) they can be scalably fit to millions of customers, 2) their key quantities can be efficiently computed, and 3) they can be easily extended to covariates. In particular, this paper shows how to extend amortized variational inference [45 , 65 , 89 ] to such models, which turns the problem of posterior inference into a gradient-based optimization problem, toward which all the tools of modern machine learning—i.e., GPU accelerated computation, automatic differentiation, and neural networks—can be deployed.
This family of models is built around transactions following a Weibull renewal process, an assumption with precedent in the marketing literature [15 , 36 , 3 , 54 ] but one which departs from existing regularity-aware BTYD models which instead assume gamma-distributed renewals [60 , 64 ] . As we show, the shift from gamma to Weibull facilitates the development of gradient-based variational inference, and does so without any appreciable change to such models’ expressivity, interpretation, or performance. We parameterize the Weibull in a nonstandard way so that the model emits a set of customer-specific latent variables with the same interpretation as in previous work, a correspondence which we characterize formally. In addition to the model, we also rely upon the Weibull distribution as the variational family during inference, where its convenient analytic properties contribute to a low-variance stochastic optimization procedure via pathwise or “reparameterization” gradient estimates [39 , 45 ] . Due to this dual reliance on the Weibull distribution, we dub the proposed approach “Wei’ ‘Til You Die” (WTYD).
We illustrate the importance of modeling heterogeneous regularity in large-scale customer bases with two case studies on both proprietary and public data. The first case study is on 5 million customers of an online subscription-based retail firm, whose anonymized data we acquired from the marketing analytics firm Ocurate. The second is on 4 million individual political donors during the run-up to the 2020 US General elections, the data for which is publicly available through the Federal Elections Commission (FEC). In both cases, WTYD reveals interpretable heterogeneous structure in regularity, frequency, and churn that informs and accords with our knowledge of both domains. Fitting WTYD takes 8 minutes on the dataset of 5 million, which takes the current state-of-the-art an estimated 3-4 days. This comes with no appreciable loss in predictive performance, as we demonstrate in empirical comparisons on random subsets of the real data as well as on a suite of simulated datasets. On simulated data, where the ground-truth is known, we also show that WTYD exhibits no degradation in parameter or latent-variable recovery despite its dramatic improvement in computational efficiency. We also provide an illustration on how to extend WTYD to covariates and demonstrate the benefit in doing so.
Roadmap. The contributions of this paper are structured as follows.
In § 5, we then develop an amortized variational inference procedure that fits WTYD by stochastic gradient-based optimization. This section develops the basic procedure based on parameterizing an inference network tailored to point process observations as well as on pathwise gradient estimators. It also introduces a series of algorithmic improvements which are instrumental to making the basic algorithm work in practice, including Rao-Blackwellization, which exploits analytic properties of the model to decrease gradient variance, and “inference compilation” [46 ] which pretrains the inference network on synthetic data. Finally, this section introduces principled variational approximations to key posterior quantities such as “P(alive)”, the probability a customer is alive given their transactions.
In § 6, we report an extensive suite of simulation studies which compares WTYD to Pareto/GGG on recovering ground-truth parameters and latent-variables and forecasting future data across a range of data settings, showing that WTYD obtains a substantial speedup without any appreciable loss in performance.
In § 7 we then provide two case studies on a novel proprietary dataset of 5 million online retail customers and a public dataset of 4 million political donors. In both, we show that WTYD recovers interpretable heterogeneous structure in regularity, frequency and churn, doing so highly efficiently. We also provide an example of how to extend WTYD to covariates.
Customer Relationship Management (CRM) Data
CRM data consists of “which customer bought what and when” micro-records. As the timing pattern of such purchases is often the main object of statistical analysis, we will focus on the scenario where there is only one type of purchase to simplify notation and exposition.
A CRM database grows over time as new customers are added (or “born”) and their transactions recorded. At any current time T T T , there are n n n customers in the database, each of whom is indexed by i ∈ [ n ] i \in[n] i ∈ [ n ] where [ n ] ≔ { 1 , … , n } [n] \coloneqq\{1,\ldots,n\} [ n ] : = { 1 , … , n } . Let t i , 0 < T t_{i,0}<T t i , 0 < T be the time that customer i ∈ [ n ] i\in[n] i ∈ [ n ] is “born”, often defined as the time of their first transaction. The times of their subsequent or “repeat” transactions are then t i ≔ { t i , 1 , … , t i , y i } \mathbf{t}_i \coloneqq\{t_{i,1},\ldots,t_{i,y_i}\} t i : = { t i , 1 , … , t i , y i } where t i , j < T t_{i,j}<T t i , j < T for all j ∈ [ y i ] j\in[y_i] j ∈ [ y i ] . The time of their latest transaction t i , y i t_{i,y_i} t i , y i is called their “recency” while the total count of their repeat transactions during the observation window y i ∈ N y_i\in\mathbb{N} y i ∈ N is called their “frequency”. Note that if y i = 0 y_i=0 y i = 0 then t i = ∅ \mathbf{t}_i=\emptyset t i = ∅ .
For the purpose of evaluation and benchmarking of forecasting methods, we can also define a future holdout period up until time T ⋆ > T T^\star>T T ⋆ > T , such that the observation (or training) window is ( 0 , T ) (0,T) ( 0 , T ) and the holdout (or testing) period is ( T , T ⋆ ) (T,T^\star) ( T , T ⋆ ) . We then denote the transaction times of customer i i i during the holdout period to be t i ⋆ ≔ { t i , 1 ⋆ , … , t i , y i ⋆ ⋆ } \mathbf{t}_i^\star\coloneqq\{t_{i,1}^\star,\ldots,t_{i,y_i^\star}^\star\} t i ⋆ : = { t i , 1 ⋆ , … , t i , y i ⋆ ⋆ } such that t i , j ⋆ ∈ ( T , T ⋆ ) t_{i,j}^\star\in(T,T^\star) t i , j ⋆ ∈ ( T , T ⋆ ) for all j ∈ [ y i ⋆ ] j\in[y_i^\star] j ∈ [ y i ⋆ ] and where y i ⋆ y_i^\star y i ⋆ denotes their number of transactions during the holdout period, which we call the customer’s “future frequency”. Note again that if y i ⋆ = 0 y_i^\star=0 y i ⋆ = 0 then t i ⋆ = ∅ \mathbf{t}_i^\star=\emptyset t i ⋆ = ∅ .
Since customers are “born” into the data at different times, it is common to represent the timing of their transactions relativistically, which simplifies model specification. First, rather than deal in absolute timestamps, we consider the customer’s inter-transaction times (ITTs) , which for y i > 0 y_i > 0 y i > 0 in the observation window are denoted Δ t i ≔ ( Δ t i , 1 , … , Δ t i , y i ) \Delta t_i \coloneqq(\Delta t_{i,1}, \ldots, \Delta t_{i,y_i}) Δ t i : = ( Δ t i , 1 , … , Δ t i , y i ) and defined as
Δ t i , j ≔ t i , j − t i , j − 1 for j ∈ [ y i ] such that t i , j = t i , 0 + ∑ j ′ ≤ j Δ t i , j ′ . (1) \Delta t_{i,j} \coloneqq t_{i,j} - t_{i,j-1} \quad\text{for } j \in[y_i] \text{ such that } t_{i,j} = t_{i,0} + \sum_{j' \le j} \Delta t_{i,j'}. \tag*{(1)} Δ t i , j : = t i , j − t i , j − 1 for j ∈ [ y i ] such that t i , j = t i , 0 + j ′ ≤ j ∑ Δ t i , j ′ . (1)
Similarly, the customer’s ITTs during the holdout period are then Δ t i ⋆ ≔ ( Δ t i , 1 ⋆ , … , Δ t i , y i ⋆ ⋆ ) \Delta t_i^\star\coloneqq(\Delta t_{i,1}^\star, \ldots, \Delta t_{i,y_i^\star}^\star) Δ t i ⋆ : = ( Δ t i , 1 ⋆ , … , Δ t i , y i ⋆ ⋆ ) for y i ⋆ > 0 y_i^\star> 0 y i ⋆ > 0 , where the first of these is the gap between the customer’s last transaction during the observation window and their first transaction during holdout—i.e., Δ t i , 1 ⋆ ≔ t i , 1 ⋆ − t i , y i \Delta t_{i,1}^\star\coloneqq t_{i,1}^\star- t_{i,y_i} Δ t i , 1 ⋆ : = t i , 1 ⋆ − t i , y i .
Considering only the ITTs is called the “customer’s journey ” representation, which allows us to specify models that assume customers are “born” simultaneously at t = 0 t = 0 t = 0 . The length of each customer’s observation window then varies, each of which is defined T i ≔ T − t i , 0 T_i \coloneqq T - t_{i,0} T i : = T − t i , 0 and often called their “total time ”. By extension, T i ⋆ ≔ T ⋆ − t i , 0 T_i^\star\coloneqq T^\star- t_{i,0} T i ⋆ : = T ⋆ − t i , 0 is their total time through the holdout period. For a visual representation of these quantities, see Figure 1 .
Figure 1. Raw CRM data (left) based on absolute transaction timestamps versus the “customer journey” representation (right) based on inter-transaction times and varying total times.
Buy `Til You Die (BTYD) Models
A common pattern of consumer behavior is churn where customers eventually cease purchasing from a firm permanently. Accounting for this behavior is important—e.g., for valuing a customer base or even an entire firm [37 , 53 ] —especially in non-contractual settings where the firm never directly observes a cancellation event [63 ] . Marketers have traditionally done so by modeling each customer as being active (or “alive”) during a set interval ( t i , 0 , τ i ) (t_{i,0}, \tau_i) ( t i , 0 , τ i ) —i.e., between the time of their “birth” t i , 0 t_{i,0} t i , 0 and “death” τ i \tau_i τ i . Critically, while a customer’s birth is observed in CRM data, their death typically is not. Death manifests indirectly though, such as a long-churned customer having “recency” much less than the current time t i , y i ≪ T t_{i,y_i} \ll T t i , y i ≪ T . Thus, statistical models that treat τ i \tau_i τ i explicitly as a latent variable are highly motivated.
A long history of such models in marketing, which collectively form the subfield of customer-base analysis [27 ] , follows the “buy ‘til you die” framework of [68 ] , wherein each model is fundamentally characterized by two processes:
a transaction process for the timing of transactions t i \mathbf{t}_i t i while customers are “alive”, and
a lifetime (or dropout) process for the time τ i \tau_i τ i until customers “die”.
Transaction renewal processes. The vast majority of BTYD models assume transactions follow a Poisson process, or more generally a renewal process [33 , 17 ] , under which ITTs are drawn Δ t i , j ∼ i i d f i \Delta t_{i,j} \overset{\mathrm{iid}}{\sim} f_i Δ t i , j ∼ iid f i from some renewal distribution f i f_i f i specific to customer i i i . For instance, a Poisson transaction process is one where renewals are Exponential—i.e., f i ≔ Exp ( RATE = λ i ) f_i \coloneqq\operatorname{Exp}(\operatorname{RATE} = \lambda_i) f i : = Exp ( RATE = λ i ) . This then induces a continuous-time renewal counting process Y i ( ⋅ ) Y_i(\cdot) Y i ( ⋅ ) :
Δ t i , j ∼ i i d f i and Y i ( t ) ≔ max { m : ∑ j = 1 m Δ t i , j ≤ t } . (2) \Delta t_{i,j} \overset{\mathrm{iid}}{\sim} f_i \quad\text{and} \quad Y_i(t) \coloneqq\max\left\{m : \sum_{j=1}^{m} \Delta t_{i,j} \le t\right\}. \tag*{(2)} Δ t i , j ∼ iid f i and Y i ( t ) : = max { m : j = 1 ∑ m Δ t i , j ≤ t } . (2)
The customer’s “frequency” (as defined in the previous section) is then y i = Y i ( T i ∧ τ i ) y_i = Y_i(T_i \wedge\tau_i) y i = Y i ( T i ∧ τ i ) where a ∧ b ≔ min { a , b } a \wedge b \coloneqq\min\{a,b\} a ∧ b : = min { a , b } . The driving assumption behind BTYD models—i.e., that a customer never buys again after τ i \tau_i τ i —is then expressed as the “customer’s lifetime value ” (CLV)1 being Y i ( ∞ ) = Y i ( τ i ) Y_i(\infty)=Y_i(\tau_i) Y i ( ∞ ) = Y i ( τ i ) . We can also define the counting process on any interval ( s , t ) (s,t) ( s , t ) for s < t s<t s < t to denote Y i ( s , t ) ≔ Y i ( t ) − Y i ( s ) Y_i(s,t) \coloneqq Y_i(t)-Y_i(s) Y i ( s , t ) : = Y i ( t ) − Y i ( s ) . For instance, a customer’s “future frequency” is y i ⋆ = Y i ( T i , T i ⋆ ) y_i^\star=Y_i(T_i,T_i^\star) y i ⋆ = Y i ( T i , T i ⋆ ) .
Key quantities. Practitioners have traditionally sought to query BTYD models in a few key ways, which can be cleanly divided into prior vs posterior quantities, the former being:
(1a) Expected customer lifetime value: E [ Y i ( ∞ ) ] \mathbb{E}[Y_i(\infty)] E [ Y i ( ∞ )] ,
(1b) Marginal likelihood: p ( D i ) = ∫ p ( D i , z i ) d z i p(\mathcal{D}_i)=\int p(\mathcal{D}_i,z_i)\,\mathrm{d}z_i p ( D i ) = ∫ p ( D i , z i ) d z i ,
where D i ≔ { Δ t i , T i , … } \mathcal{D}_i\coloneqq\{\Delta t_i,T_i,\ldots\} D i : = { Δ t i , T i , … } are the data on customer i i i during the observation window, usually their ITTs and total time, but potentially other data (e.g., absolute timestamps t i t_i t i , covariates x i \mathbf{x}_i x i ), and z i ≔ { f i , τ i , … } z_i\coloneqq\{f_i,\tau_i,\ldots\} z i : = { f i , τ i , … } are customer-specific latent variables, usually just their renewal distribution (or parameters thereof) and time of death. The key posterior quantities are then:
(2a) Forecasted future frequency: E [ Y i ( T i , T i ⋆ ) ∣ D i ] \mathbb{E}[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i] E [ Y i ( T i , T i ⋆ ) ∣ D i ] ,
(2b) Posterior predictive distribution: p ( D i ⋆ ∣ D i ) = ∫ p ( D i ⋆ , z i ∣ D i ) d z i p(\mathcal{D}_i^\star\mid\mathcal{D}_i)=\int p(\mathcal{D}_i^\star,z_i\mid\mathcal{D}_i)\,\mathrm{d}z_i p ( D i ⋆ ∣ D i ) = ∫ p ( D i ⋆ , z i ∣ D i ) d z i ,
(2c) Probability that customer i i i is alive “P(ALIVE)”: p ( τ i > T i ∣ D i ) p(\tau_i>T_i\mid\mathcal{D}_i) p ( τ i > T i ∣ D i ) ,
where D i ⋆ ≔ { Δ t i ⋆ , T i ⋆ , … } \mathcal{D}_i^\star\coloneqq\{\Delta t_i^\star,T_i^\star,\ldots\} D i ⋆ : = { Δ t i ⋆ , T i ⋆ , … } are the data on customer i i i during the holdout period.
Being able to efficiently compute these quantities facilitates the use of such models. The expected frequencies in (1a) and (2a) are interpretable summary statistics used by marketers to segment the customer base and make predictions, with expected CLV in (1a) being particularly useful when customer-level covariates are available to help overcome the “cold-start” prediction problem. Tractable computation of the prior and posterior predictive distributions in (1b) and (2b) then facilitates parameter estimation and model selection, as it does with most probabilistic models. Finally, P(ALIVE) in (2c) plays a particularly central role in BTYD models, as computing it is a subroutine for computing most other key quantities, and it is itself an interpretable and useful summary statistic useful.
Whether the quantities above are tractable depends on the probabilistic assumptions of a given model. In the rest of this section, we survey some well-known BTYD models, emphasizing those we build upon, and highlighting the tension between tractability and expressivity that has been a key theme in this literature and which this paper seeks to ameliorate.
Pareto/NBD. The original BTYD model of [68 ] assumed that transactions follow a Poisson process until a Pareto-distributed time-of-death—i.e.,
τ i ∼ Pareto ( s h a p e = s ( τ ) , s c a l e = r ( τ ) ) (3) \tau_i \sim\operatorname{Pareto}(\mathrm{shape}=s^{(\tau)},\mathrm{scale}=r^{(\tau)}) \tag*{(3)} τ i ∼ Pareto ( shape = s ( τ ) , scale = r ( τ ) ) (3)
Δ t i , j ∼ i i d Gamma ( s h a p e = 1 , r a t e = λ i ) (4) \Delta t_{i,j} \mathrel{\overset{\mathrm{iid}}{\sim}} \operatorname{Gamma}(\mathrm{shape}=1,\mathrm{rate}=\lambda_i) \tag*{(4)} Δ t i , j ∼ iid Gamma ( shape = 1 , rate = λ i ) (4)
where we note that a Gamma distribution with shape equal to one is equivalent to an Exponential. Conditional on the customer-specific latent variables—i.e., z i = d e f { τ i , λ i } z_i \mathrel{\mathop{=}^{\mathrm{def}}} \{\tau_i,\lambda_i\} z i = def { τ i , λ i } —this induces a Poisson distribution over the frequency—i.e., y i ∼ Pois ( MEAN = λ i ( τ i ∧ T i ) ) y_i \sim\operatorname{Pois}(\operatorname{MEAN}=\lambda_i(\tau_i \wedge T_i)) y i ∼ Pois ( MEAN = λ i ( τ i ∧ T i )) . The model then further posits a conjugate Gamma prior over the heterogeneous rates:
Table 3.4.
Marginalizing out λ i \lambda_i λ i then induces a Negative Binomial distribution over y i y_i y i , and the model is thus named “Pareto/NBD”. The set of model parameters is θ = d e f { s ( τ ) , r ( τ ) , s ( λ ) , r ( λ ) } \theta\mathrel{\mathop{=}^{\mathrm{def}}} \{s^{(\tau)},r^{(\tau)},s^{(\lambda)},r^{(\lambda)}\} θ = def { s ( τ ) , r ( τ ) , s ( λ ) , r ( λ ) } .
All of the key expressions listed above are analytic. Parameter estimation can thus proceed by directly maximizing the marginal likelihood via gradient ascent2 —e.g.,
θ ← UPDATE ( θ , ∑ i = 1 n ∇ θ log p θ ( D i ) ) . (5) \theta\leftarrow\operatorname{UPDATE}\left(\theta,\sum_{i=1}^{n}\nabla_{\theta}\log p_{\theta}(\mathcal{D}_i)\right). \tag*{(5)} θ ← UPDATE ( θ , i = 1 ∑ n ∇ θ log p θ ( D i ) ) . (5)
Here UPDATE ( w , g ) \operatorname{UPDATE}(\mathbf{w},\mathbf{g}) UPDATE ( w , g ) denotes any single step of a first-order optimizer that moves the parameters w \mathbf{w} w along the gradient g \mathbf{g} g of the training objective (e.g., vanilla gradient ascent, Adam). This is straightforward to implement and scale to large datasets using modern frameworks.
Incorporating covariates. Another benefit of gradient-based parameter estimation is that it makes the model particularly easy to modify so that its parameters are functions of customer-level covariates θ i = d e f θ ( x i ) \theta_i \mathrel{\mathop{=}^{\mathrm{def}}} \theta(\mathbf{x}_i) θ i = def θ ( x i ) . For instance, [26 ] recommend setting
θ i = [ s i ( τ ) r i ( τ ) s i ( λ ) r i ( λ ) ] = θ ( x i ) = [ s 0 ( τ ) r 0 ( τ ) exp ( − x i T β ( τ ) ) s 0 ( λ ) r 0 ( λ ) exp ( − x i T β ( λ ) ) ] , (6) \theta_i =
\begin{bmatrix}
s_i^{(\tau)}\\
r_i^{(\tau)}\\
s_i^{(\lambda)}\\
r_i^{(\lambda)}
\end{bmatrix}
=
\theta(\mathbf{x}_i)=
\begin{bmatrix}
s_0^{(\tau)}\\
r_0^{(\tau)}\exp(-\mathbf{x}_i^{\mathsf{T}}\boldsymbol{\beta}^{(\tau)})\\
s_0^{(\lambda)}\\
r_0^{(\lambda)}\exp(-\mathbf{x}_i^{\mathsf{T}}\boldsymbol{\beta}^{(\lambda)})
\end{bmatrix}, \tag*{(6)} θ i = s i ( τ ) r i ( τ ) s i ( λ ) r i ( λ ) = θ ( x i ) = s 0 ( τ ) r 0 ( τ ) exp ( − x i T β ( τ ) ) s 0 ( λ ) r 0 ( λ ) exp ( − x i T β ( λ ) ) , (6)
where the learnable parameters in θ ( ⋅ ) \theta(\cdot) θ ( ⋅ ) are the weights { β ( τ ) , β ( λ ) } \{\boldsymbol{\beta}^{(\tau)},\boldsymbol{\beta}^{(\lambda)}\} { β ( τ ) , β ( λ ) } and biases { s 0 ( τ ) , r 0 ( τ ) , s 0 ( λ ) , r 0 ( λ ) } \{s_0^{(\tau)},r_0^{(\tau)},s_0^{(\lambda)},r_0^{(\lambda)}\} { s 0 ( τ ) , r 0 ( τ ) , s 0 ( λ ) , r 0 ( λ ) } . Many other settings are possible—e.g., θ ( ⋅ ) \theta(\cdot) θ ( ⋅ ) could be a feedforward neural network—and can be fit via gradient ascent as above provided that θ ( ⋅ ) \theta(\cdot) θ ( ⋅ ) defines a differentiable map.
Pareto/NBD “the easy way” and related Poisson models. While the Pareto/NBD model is entirely analytic, the derivation and implementation of its key expressions can be “rather daunting” [25 ] . [29 ] thus introduced the BG/NBD model—dubbing it “Pareto/NBD the easy way”—which replaces the lifetime process in (3) by assuming the death time coincides exactly with a transaction time τ i ∈ { t i , 1 , t i , 2 , … } \tau_i \in\{t_{i,1},t_{i,2},\ldots\} τ i ∈ { t i , 1 , t i , 2 , … } , with which one being a latent variable drawn according to a Beta-mixture of Geometric distributions—i.e.,
Table 3.7.
Doing this greatly simplifies the model. However, it does so at the expense of tying death time to the transaction times. A simple modification called MBG/NBD [8 ] further allows death to occur before any transaction. In both cases however, by tying the death and transaction times, the (M)BG/NBD model assumes high-volume customers churn faster [43 ] , an assumption not made by Pareto/NBD and one that is often violated in practice.
Due to their analytic tractability, the Pareto/NBD and (M)BG/NBD models have long been the “workhorses” of CRM modeling, with mature implementations in both R [e.g., [24 , 59 ] ] and Python [e.g., [19 , 20 ] ]. A key benefit of these models is that they emit a sufficient statistic representation such that the data on each customer need only consist of their frequency, recency, and total time—i.e., D i ≔ { y i , t i , y i , T i } \mathcal{D}_i \coloneqq\{y_i,t_{i,y_i},T_i\} D i : = { y i , t i , y i , T i } . This further improves these models’ scalability and ease, allowing even for implementations directly in Excel [[29 ] ].
Many extensions of the “workhorse” models have since been proposed—e.g., hierarchical Bayesian variants with covariates [[1 ] ], models with time-varying covariates [[69 ] ], or Bayesian nonparametric models [[22 ] ]—see surveys by [27 ] and [14 ] , among others. A key theme in this literature is that making the “workhorse” models more expressive comes at the cost of tractability and scalability. In particular, the assumption of a Poisson transaction process is key to numerous analytic properties in these models and is thus rarely relaxed.
Non-Poisson transactions. As has been observed for several decades, many real-world transaction patterns are poorly modeled by a Poisson process, exhibiting more or less regularity in their timing than would be expected by Exponentially-distributed renewals [[15 , 36 , 3 ] ]. On one side, subscription-based customers transact with great regularity, exhibiting an ITT distribution that is highly peaked around its mean. On the other side, “binge watching” customers transact many times in short succession followed by long periods of inactivity [[70 ] ], exhibiting a pattern of ITTs known as “clumpiness” [[92 ] ].
Pareto/GGG. Writing the Poisson process assumption as we do in (4) , it is straightforward to modify it to account for regularity by introducing one extra customer-specific latent variable k i k_i k i and defining the ITTs to then be drawn from a fully flexible Gamma distribution as:
Δ t i , j ∼ i i d Gamma ( S H A P E = k i , R A T E = k i λ i ) (7) \Delta t_{i,j} \overset{\mathrm{iid}}{\sim} \operatorname{Gamma}(\mathrm{SHAPE}=k_i,\mathrm{RATE}=k_i\lambda_i) \tag*{(7)} Δ t i , j ∼ iid Gamma ( SHAPE = k i , RATE = k i λ i ) (7)
While the mean remains unchanged E [ Δ t i , j ] = 1 / λ i \mathbb{E}[\Delta t_{i,j}]=1/\lambda_i E [ Δ t i , j ] = 1/ λ i , the concentration of the ITT distribution around its mean is now controlled by k i k_i k i , with k i > 1 k_i>1 k i > 1 and k i < 1 k_i<1 k i < 1 yielding more regular versus “clumpy” timing patterns, respectively. This model was introduced by [60 ] , who additionally posit the following prior over the customer’s “regularity”:
k i ∼ Gamma ( S H A P E = s ( k ) , R A T E = r ( k ) ) (8) k_i \sim\operatorname{Gamma}(\mathrm{SHAPE}=s^{(k)},\mathrm{RATE}=r^{(k)}) \tag*{(8)} k i ∼ Gamma ( SHAPE = s ( k ) , RATE = r ( k ) ) (8)
The model is named “Pareto/GGG” as it retains the Pareto lifetime process and Gamma prior over λ i \lambda_i λ i while adding a Gamma prior over k i k_i k i and making the renewal distribution Gamma. The model retains a sufficient statistic representation, requiring only one extra “regularity” statistic—i.e., D i ≔ { y i , t i , y i , κ i , T i } \mathcal{D}_i \coloneqq\{y_i,t_{i,y_i},\kappa_i,T_i\} D i : = { y i , t i , y i , κ i , T i } —where κ i ≔ ∑ j log Δ t i , j \kappa_i \coloneqq\sum_j \log\Delta t_{i,j} κ i : = ∑ j log Δ t i , j is the sum-of-log-ITTs.
While conceptually simple, the move to a full Gamma renewal process makes all its key expressions non-analytic. In particular, its marginal likelihood p θ ( D i ) p_{\boldsymbol{\theta}}(\mathcal{D}_i) p θ ( D i ) is intractable and thus estimation of its model parameters θ ≔ { s ( τ ) , r ( τ ) , s ( λ ) , r ( λ ) , s ( κ ) , r ( κ ) } \boldsymbol{\theta} \coloneqq\{s^{(\tau)},r^{(\tau)},s^{(\lambda)},r^{(\lambda)},s^{(\kappa)},r^{(\kappa)}\} θ : = { s ( τ ) , r ( τ ) , s ( λ ) , r ( λ ) , s ( κ ) , r ( κ ) } cannot proceed via direct gradient ascent, as it does with Pareto/NBD. Instead, [59 ] provide an MCMC procedure that returns samples of θ \boldsymbol{\theta} θ alongside samples of the customer-specific latent variables z i ≔ { τ i , λ i , κ i } \boldsymbol{z}_i \coloneqq\{\tau_i,\lambda_i,\kappa_i\} z i : = { τ i , λ i , κ i } . The move to MCMC makes parameter estimation substantially more computationally intensive and less scalable. It also makes it harder to incorporate covariates into the model, which is not an available option in BTYDPLUS [59 ] , the main open-source implementation.
Pareto/GGG “the easy way”. In an effort to recover analytic tractability, [64 ] introduced an “easy way” in which the lifetime process is changed to match the (M)BG/NBD model in eq:3.7 and the transaction process is modified so that 1) all customers have the same regularity k i = k k_i = k k i = k where k k k is treated as a global model parameter, and 2) k k k is constrained to only take positive integer values k ∈ { 1 , 2 , … } k \in\{1,2,\ldots\} k ∈ { 1 , 2 , … } , thus inducing an Erlang-k k k renewal distribution, a special case of the Gamma. The model is analytic in more (though still not all) of the key expressions, most notably the marginal likelihood p θ ( D i ) p_{\boldsymbol{\theta}}(\mathcal{D}_i) p θ ( D i ) which can now be optimized directly. However, it is worth noting that the latent variables being marginalized out z i ≔ { τ i , λ i } \boldsymbol{z}_i \coloneqq\{\tau_i,\lambda_i\} z i : = { τ i , λ i } no longer include customer-specific regularities k i k_i k i and that the global regularity is constrained k ≥ 1 k \ge1 k ≥ 1 so that it cannot express “clumpy” timing patterns. Moreover, the resulting model—named the (M)BG/CNBD-k k k for the Condensed Negative Binomial-k k k frequency distribution [? ] that the Erlang-k k k renewals induce—suffers the same drawbacks as the (M)BG/NBD regarding the dependence between the lifetime and transaction processes.
Summary of tradeoffs. A key theme in the BTYD literature has been the tradeoff between analytic tractability and model expressivity. The original “workhorse” models are highly analytic, emitting sufficient statistics, scalable maximum likelihood-based inference, and allowing for the incorporation of covariates. However, these models and most of their extensions are premised on the key assumption that transactions follow a Poisson process. Models can achieve better fit when relaxing this assumption, but inevitably do so at some cost. Table 1 summarizes these tradeoffs. Pareto/GGG generalizes transaction renewals to follow fully flexible Gamma distributions with heterogeneous regularity and does so while retaining sufficiency. However, the model requires MCMC to fit, which is not scalable and hampers incorporation of covariates. (M)BG/CNBD-k k k meanwhile retains sufficiency while recovering a scalable inference scheme, but does so at the cost of constraining regularity to be homogeneous k i = k k_i = k k i = k and non-“clumpy” k > 1 k > 1 k > 1 . In what follows, we introduce a model—Wei’ `Til You Die (WTYD)—that serves as a drop-in replacement for Pareto/GGG, adopting Weibull rather than Gamma renewals. This small change enables a form of scalable gradient-based variational inference (VI) for parameter estimation. The only cost is then to sufficiency: fitting WTYD requires conditioning on all ITTs Δ t i \Delta t_i Δ t i . However, we argue that this tradeoff is highly advantageous over the alternatives, in light of modern datasets with large n n n and modern computational infrastructure with low memory costs.
Table 1. BTYD models trade off different aspects of tractability and expressivity. From left to right, models do ✓ or do not ✗: 1) assume more expressive transaction processes than Poisson, 2) yield sufficient statistics, 3) express heterogeneous regularity, 4) make it easy to incorporate covariates, and 5) adopt scalable inference methods that are easily adapted to stream over minibatches of data.
Wei’ `Til You Die (WTYD)
The model we consider assumes that a customer’s inter-transaction times (ITTs) are drawn
TRANSACTION PROCESS
Δ t i , j ∼ i i d Weibull ( S H A P E = k i , R A T E = λ i Γ ( 1 + 1 k i ) ) (9) \Delta t_{i,j} \overset{\mathrm{iid}}{\sim} \operatorname{Weibull}\left(\mathrm{SHAPE}=k_i,\ \mathrm{RATE}=\lambda_i \Gamma\left(1+\frac{1}{k_i}\right)\right) \tag*{(9)} Δ t i , j ∼ iid Weibull ( SHAPE = k i , RATE = λ i Γ ( 1 + k i 1 ) ) (9)
and thus follow a Weibull renewal process [85 , 88 ] . The Weibull distribution [84 ] has density function Weibull ( w ; S H A P E = s , R A T E = r ) = s r s w s − 1 exp ( − r s w s ) \operatorname{Weibull}(w;\ \mathrm{SHAPE}=s,\ \mathrm{RATE}=r)=s r^s w^{s-1}\exp(-r^s w^s) Weibull ( w ; SHAPE = s , RATE = r ) = s r s w s − 1 exp ( − r s w s ) and expected value E [ w ∣ s , r ] = Γ ( 1 + 1 / s ) r − 1 \mathbb{E}[w\mid s,r]=\Gamma(1+1/s)r^{-1} E [ w ∣ s , r ] = Γ ( 1 + 1/ s ) r − 1 where Γ ( ⋅ ) \Gamma(\cdot) Γ ( ⋅ ) is the gamma function.
Although it has not received much traction, the idea to model transaction processes with Weibull renewals is an old idea dating back at least to [3 ] , who use a Generalized Gamma renewal process (which generalizes the Weibull), or to [54 ] who model transaction frequencies with the Weibull Count distribution which arises from Weibull renewals. The definition of shape and rate in (9) —where both are a function of k i k_i k i —departs from previous work, but will yield an appealing correspondence which we detail below.
We then posit the same lifetime process and priors as Pareto/GGG, repeated here for ease:
LIFETIME PROCESS
τ i ∼ Pareto ( S H A P E = s ( τ ) , S C A L E = r ( τ ) ) (10) \tau_i \sim\operatorname{Pareto}\left(\mathrm{SHAPE}=s^{(\tau)},\ \mathrm{SCALE}=r^{(\tau)}\right) \tag*{(10)} τ i ∼ Pareto ( SHAPE = s ( τ ) , SCALE = r ( τ ) ) (10)
HETEROGENEITY PRIOR
k i ∼ Gamma ( S H A P E = s ( k ) , R A T E = r ( k ) ) λ i ∼ Gamma ( S H A P E = s ( λ ) , R A T E = r ( λ ) ) (11) \begin{aligned}
k_i &\sim\operatorname{Gamma}\left(\mathrm{SHAPE}=s^{(k)},\ \mathrm{RATE}=r^{(k)}\right) \\
\lambda_i &\sim\operatorname{Gamma}\left(\mathrm{SHAPE}=s^{(\lambda)},\ \mathrm{RATE}=r^{(\lambda)}\right) \tag*{(11)}
\end{aligned} k i λ i ∼ Gamma ( SHAPE = s ( k ) , RATE = r ( k ) ) ∼ Gamma ( SHAPE = s ( λ ) , RATE = r ( λ ) ) (11)
λ i ∼ Gamma ( S H A P E = s ( λ ) , R A T E = r ( λ ) ) (12) \lambda_i \sim\operatorname{Gamma}\left(\mathrm{SHAPE}=s^{(\lambda)},\ \mathrm{RATE}=r^{(\lambda)}\right) \tag*{(12)} λ i ∼ Gamma ( SHAPE = s ( λ ) , RATE = r ( λ ) ) (12)
Correspondence to Pareto/GGG. The only difference between WTYD and Pareto/GGG is the Weibull versus Gamma transaction process. The Weibull and Gamma distributions are very similar, to the point of being difficult to distinguish from data [7 , 32 ] . Moreover, our parameterization of the Weibull in (9) matches that of the Gamma in (7) in terms of a mean–concentration parameterization—i.e.,
Weibull ( S H A P E = k i , R A T E = λ i Γ ( 1 + 1 k i ) ) ⟺ Weibull ( M E A N = 1 λ i , C O N C E N T R A T I O N = k i ) , Gamma ( S H A P E = k i , R A T E = k i λ i ) ⟺ Gamma ( M E A N = 1 λ i , C O N C E N T R A T I O N = k i ) . \operatorname{Weibull}\left(\mathrm{SHAPE}=k_i,\ \mathrm{RATE}=\lambda_i\Gamma\left(1+\frac{1}{k_i}\right)\right) \Longleftrightarrow\operatorname{Weibull}\left(\mathrm{MEAN}=\frac{1}{\lambda_i},\ \mathrm{CONCENTRATION}=k_i\right), \\
\operatorname{Gamma}\left(\mathrm{SHAPE}=k_i,\ \mathrm{RATE}=k_i\lambda_i\right) \Longleftrightarrow\operatorname{Gamma}\left(\mathrm{MEAN}=\frac{1}{\lambda_i},\ \mathrm{CONCENTRATION}=k_i\right). Weibull ( SHAPE = k i , RATE = λ i Γ ( 1 + k i 1 ) ) ⟺ Weibull ( MEAN = λ i 1 , CONCENTRATION = k i ) , Gamma ( SHAPE = k i , RATE = k i λ i ) ⟺ Gamma ( MEAN = λ i 1 , CONCENTRATION = k i ) .
Figure 2. Weibull and Gamma renewal models are very similar, with the best-fit Weibull to any given Gamma distribution having the same mean 1 / λ i 1/\lambda_i 1/ λ i and concentration k i k_i k i on the same side of 1. Top: Examples of a Gamma and best-fit Weibull for the three conditions on k i k_i k i , all with λ i = 1 \lambda_i = 1 λ i = 1 . Bottom: Six independent draws from the corresponding Gamma renewal process to illustrate the three conditions.
where 1 / λ i 1/\lambda_i 1/ λ i is the mean of customer i i i ’s ITTs and k i k_i k i is their “concentration”, a term we use to mean any measure of spread which is independent of the mean. For the Gamma in (7) the squared coefficient of variation (CV) is 1 / k i 1/k_i 1/ k i while for the Weibull in (9) it is Γ ( 1 + 2 / k i ) Γ ( 1 + 1 / k i ) 2 − 1 \frac{\Gamma(1+2/k_i)}{\Gamma(1+1/k_i)^2}-1 Γ ( 1 + 1/ k i ) 2 Γ ( 1 + 2/ k i ) − 1 , neither of which depends on λ i \lambda_i λ i .
Both models therefore have the same set of latents z i : = { τ i , λ i , k i } z_i \mathrel{\mathop:}= \{\tau_i,\lambda_i,k_i\} z i : = { τ i , λ i , k i } , with each playing the same qualitative role. This correspondence can be sharpened further by noting that the concentration parameter k i k_i k i acts as an index of regularity in the same way for both models, yielding “clumpy” renewals for k i < 1 k_i < 1 k i < 1 , regular for k i > 1 k_i > 1 k i > 1 , and a Poisson process at k i = 1 k_i = 1 k i = 1 . Moreover, we can show that the best-fit Weibull to any given Gamma, in terms of KL divergence, has equal mean and a concentration on the same side of 1, with a monotone mapping between them.
Proposition 1 (Correspondence of regularity index). Let 1 / λ W ∗ 1/\lambda_W^* 1/ λ W ∗ and k W ∗ k_W^* k W ∗ be the mean and concentration of a Weibull which minimizes the KL divergence to a Gamma with mean 1 / λ G 1/\lambda_G 1/ λ G and concentration k G k_G k G —i.e., k W ∗ , λ W ∗ ∈ argmin k W , λ W KL ( Weibull ( 1 / λ W , k W ) ∥ Gamma ( 1 / λ G , k G ) ) k_W^*,\lambda_W^* \in\operatorname*{argmin}_{k_W,\lambda_W} \operatorname{KL}\left(\operatorname{Weibull}(1/\lambda_W,k_W)\mathbin\Vert\operatorname{Gamma}(1/\lambda_G,k_G)\right) k W ∗ , λ W ∗ ∈ argmin k W , λ W KL ( Weibull ( 1/ λ W , k W ) ∥ Gamma ( 1/ λ G , k G ) ) . The minimizer is unique, and defined below where ψ ( ⋅ ) \psi(\cdot) ψ ( ⋅ ) is the digamma function and γ \gamma γ is Euler’s constant:
λ G = λ W ∗ and k G = k W ∗ ( γ + ψ ( 1 + 1 / k W ∗ ) ) − 1 , (13) \lambda_G = \lambda_W^* \quad\text{and}\quad k_G = k_W^*\left(\gamma+ \psi\left(1 + 1/k_W^*\right)\right)^{-1}, \tag*{(13)} λ G = λ W ∗ and k G = k W ∗ ( γ + ψ ( 1 + 1/ k W ∗ ) ) − 1 , (13)
Moreover the map k G ↦ k W ∗ k_G \mapsto k_W^* k G ↦ k W ∗ is strictly increasing, with equality at 1, and therefore:
k W ∗ < 1 ⟺ k G < 1 (“clumpy”) k W ∗ = 1 ⟺ k G = 1 (random) k W ∗ > 1 ⟺ k G > 1 (regular) . \begin{aligned}
k_W^* < 1 &\Longleftrightarrow k_G < 1 &&\text{(“clumpy”)}\\
k_W^* = 1 &\Longleftrightarrow k_G = 1 &&\text{(random)}\\
k_W^* > 1 &\Longleftrightarrow k_G > 1 &&\text{(regular)}.
\end{aligned} k W ∗ < 1 k W ∗ = 1 k W ∗ > 1 ⟺ k G < 1 ⟺ k G = 1 ⟺ k G > 1 (“clumpy”) (random) (regular) .
See § A.1 for the proof, with the takeaway being that WTYD provides a drop-in replacement for Pareto/GGG with latent variables carrying the same qualitative meaning.
Individual-level likelihood. The likelihood of the model can be written:
p ( D i ∣ z i ) = exp ( − r i k i [ ( T i ∧ τ i ) − t i , y i ] ) [ k i y i r i y i k i ∏ j = 1 y i Δ t i , j k i − 1 exp ( r i k i ∑ j = 1 y i Δ t i , j k i ) ] 1 ( y i > 0 ) (14) p(\mathcal{D}_i \mid z_i) = \exp\left(-r_i^{k_i}\left[(T_i \wedge\tau_i)-t_{i,y_i}\right]\right)\left[\frac{k_i^{y_i}r_i^{y_i k_i}\prod_{j=1}^{y_i}\Delta t_{i,j}^{k_i-1}}{\exp\left(r_i^{k_i}\sum_{j=1}^{y_i}\Delta t_{i,j}^{k_i}\right)}\right]^{\mathbb{1}(y_i>0)} \tag*{(14)} p ( D i ∣ z i ) = exp ( − r i k i [ ( T i ∧ τ i ) − t i , y i ] ) exp ( r i k i ∑ j = 1 y i Δ t i , j k i ) k i y i r i y i k i ∏ j = 1 y i Δ t i , j k i − 1 1 ( y i > 0 ) (14)
where we have aliased the rate parameter r i ≔ λ i Γ ( 1 + 1 / k i ) r_i \coloneqq\lambda_i\Gamma(1+1/k_i) r i : = λ i Γ ( 1 + 1/ k i ) . Like the Pareto/GGG, the marginal likelihood p θ ( D i ) = ∫ p ( D i ∣ z i ) p θ ( z i ) d z i p_\theta(\mathcal{D}_i)=\int p(\mathcal{D}_i\mid z_i)p_\theta(z_i)\,\mathrm{d}z_i p θ ( D i ) = ∫ p ( D i ∣ z i ) p θ ( z i ) d z i is intractable, but can be evaluated approximately via Monte Carlo. Unlike the Pareto/GGG however, the likelihood in (14) involves only elementary functions beyond the gamma function, which every major automatic differentiation framework supports—a fact which will become key in developing scalable inference.
Loss of sufficiency. Note that the likelihood does not emit a sufficient statistic representation and must input the customer’s variable-length set of ITTs D i ≔ { Δ t i , T i } \mathcal{D}_i \coloneqq\{\Delta t_i,T_i\} D i : = { Δ t i , T i } . This is due to the fact that the two-parameter Weibull is not an exponential family distribution. By contrast, the Pareto/GGG model—whose Gamma renewal distribution is an exponential family—conditions only on a fixed-length input D i ≔ { y i , t y i , κ i , T i } \mathcal{D}_i \coloneqq\{y_i,t_{y_i},\kappa_i,T_i\} D i : = { y i , t y i , κ i , T i } . Thus, although the two models are equivalent in expressivity, WTYD incurs this disadvantage, which is practically important when customers have long transaction sequences. Nevertheless, we will show that this loss in sufficiency is made up for by the model becoming amenable to a form of gradient-based inference that is highly scalable.
Expected CLV. The expected customer lifetime value is also intractable, as with Pareto/GGG. However, writing it as E θ [ Y i ( ∞ ) ] = E z i ∼ p θ ( z i ) [ E [ Y i ( τ i ) ∣ z i ] ] \mathbb{E}_\theta[Y_i(\infty)]=\mathbb{E}_{z_i\sim p_\theta(z_i)}[\mathbb{E}[Y_i(\tau_i)\mid z_i]] E θ [ Y i ( ∞ )] = E z i ∼ p θ ( z i ) [ E [ Y i ( τ i ) ∣ z i ]] , the inner expectation is the expectation of a Weibull Count random variable which can be evaluated via a recurrence given by [54 ] . Thus, the Monte Carlo estimator for CLV given below can be Rao-Blackwellized by sampling z i ( s ) ∼ i i d p θ ( z i ) z_i^{(s)}\overset{\mathrm{iid}}{\sim}p_\theta(z_i) z i ( s ) ∼ iid p θ ( z i ) and evaluating the inner expectation:
E ^ θ M C [ Y i ( ∞ ) ] ≔ 1 S ∑ s = 1 S E [ Y i ( τ i ( s ) ) ∣ z i ( s ) ] . (15) \widehat{\mathbb{E}}_\theta^{\mathrm{MC}}[Y_i(\infty)] \coloneqq\frac{1}{S}\sum_{s=1}^{S}\mathbb{E}[Y_i(\tau_i^{(s)})\mid z_i^{(s)}]. \tag*{(15)} E θ MC [ Y i ( ∞ )] : = S 1 s = 1 ∑ S E [ Y i ( τ i ( s ) ) ∣ z i ( s ) ] . (15)
Alternatively, for large τ i \tau_i τ i , we can apply an elementary renewal theorem [33 ] to derive an approximation that is a simple closed-form function of model parameters:
E ^ θ R T [ Y i ( ∞ ) ] ≔ E θ [ τ i λ i ] = r ( τ ) s ( λ ) s ( τ ) − 1 r ( λ ) . (16) \widehat{\mathbb{E}}_\theta^{\mathrm{RT}}[Y_i(\infty)] \coloneqq\mathbb{E}_\theta[\tau_i\lambda_i]=\frac{r^{(\tau)}s^{(\lambda)}}{s^{(\tau)}-1}r^{(\lambda)}. \tag*{(16)} E θ RT [ Y i ( ∞ )] : = E θ [ τ i λ i ] = s ( τ ) − 1 r ( τ ) s ( λ ) r ( λ ) . (16)
Moreover, applying another standard renewal theorem [49 ] , we can derive a simple bound on the error of this estimator, which is particularly small for non-“clumpy” populations:
Proposition 2 (Error of E ^ θ R T [ Y i ( ∞ ) ] \widehat{\mathbb{E}}_\theta^{\mathrm{RT}}[Y_i(\infty)] E θ RT [ Y i ( ∞ )] ). The error of the renewal-theoretic estimator is bounded by
∣ E ^ θ R T [ Y i ( ∞ ) ] − E θ [ Y i ( ∞ ) ] ∣ ≤ E κ i ∼ p θ ( κ i ) [ Γ ( 1 + 2 / κ i ) Γ ( 1 + 1 / κ i ) 2 ] . (17) \left|\widehat{\mathbb{E}}_\theta^{\mathrm{RT}}[Y_i(\infty)]-\mathbb{E}_\theta[Y_i(\infty)]\right|\leq\mathbb{E}_{\kappa_i\sim p_\theta(\kappa_i)}\left[\frac{\Gamma(1+2/\kappa_i)}{\Gamma(1+1/\kappa_i)^2}\right]. \tag*{(17)} E θ RT [ Y i ( ∞ )] − E θ [ Y i ( ∞ )] ≤ E κ i ∼ p θ ( κ i ) [ Γ ( 1 + 1/ κ i ) 2 Γ ( 1 + 2/ κ i ) ] . (17)
where the function c ( k ) ≔ Γ ( 1 + 2 / k ) Γ ( 1 + 1 / k ) 2 c(k)\coloneqq\frac{\Gamma(1+2/k)}{\Gamma(1+1/k)^2} c ( k ) : = Γ ( 1 + 1/ k ) 2 Γ ( 1 + 2/ k ) is decreasing in k k k , with c ( ∞ ) = 1 c(\infty)=1 c ( ∞ ) = 1 and c ( 1 ) = 2 c(1)=2 c ( 1 ) = 2 . Thus, for a non-“clumpy” population—i.e., p θ ( κ i ≥ 1 ) = 1 p_\theta(\kappa_i\geq1)=1 p θ ( κ i ≥ 1 ) = 1 —the error is bounded by 2.
The derivation of the estimator and error bound are given in § A.1. We note that while the estimator only depends on { s ( τ ) , r ( τ ) , s ( λ ) , r ( λ ) } \{s^{(\tau)}, r^{(\tau)}, s^{(\lambda)}, r^{(\lambda)}\} { s ( τ ) , r ( τ ) , s ( λ ) , r ( λ ) } , the error bound depends only on { s ( k ) , r ( k ) } \{s^{(k)}, r^{(k)}\} { s ( k ) , r ( k ) } , which is a consequence of the mean-concentration parameterization we have adopted.
Incorporating covariates θ ( x i ) \theta(x_i) θ ( x i ) . In the next section, we derive a variational inference scheme that enables gradient-based parameter estimation for WTYD. Beyond its scalability, a benefit of this approach is that it makes it easy to modify the WTYD model to incorporate covariates so that the parameters in Eqs. (4.2)–(4.4) become heterogeneous θ i ≔ d e f { s i ( τ ) , r i ( τ ) , s i ( λ ) , r i ( λ ) , s i ( k ) , r i ( k ) } \theta_i \mathrel{\mathop{\coloneqq}^{\mathrm{def}}} \{s_i^{(\tau)}, r_i^{(\tau)}, s_i^{(\lambda)}, r_i^{(\lambda)}, s_i^{(k)}, r_i^{(k)}\} θ i : = def { s i ( τ ) , r i ( τ ) , s i ( λ ) , r i ( λ ) , s i ( k ) , r i ( k ) } , and defined as the output θ i ← θ ( x i ) \theta_i \gets\theta(x_i) θ i ← θ ( x i ) of some differentiable function θ ( ⋅ ) \theta(\cdot) θ ( ⋅ ) —e.g., a feedforward neural network—whose parameters we leave implicit. This is analogous to approaches for incorporating covariates into the classic BTYD models, described in Eq. (3.6), which has been difficult to apply to non-Poisson models when MCMC is used for inference.
Amortized Variational Inference for WTYD
The marginal likelihood p θ ( D 1 : n ) = ∏ i ∫ p θ ( D i , z i ) d z i p_\theta(\mathcal{D}_{1:n}) = \prod_i \int p_\theta(\mathcal{D}_i, z_i)\,\mathrm{d}z_i p θ ( D 1 : n ) = ∏ i ∫ p θ ( D i , z i ) d z i of the model in the previous section is intractable, and thus parameter estimation cannot proceed by directly maximizing it. This section develops an approach based on variational inference (VI) that is highly scalable and exploits modern machine learning frameworks for GPU-accelerated computation, automatic differentiation, and neural network architectures. For background on the statistical foundations of VI and modern advances, see [10 ] and [89 ] , respectively.
Variational EM with amortized Weibull factors
The marginal likelihood (or “evidence”) p θ ( D 1 : n ) = ∏ i ∫ p θ ( D i , z i ) d z i p_\theta(\mathcal{D}_{1:n}) = \prod_i \int p_\theta(\mathcal{D}_i, z_i)\,\mathrm{d}z_i p θ ( D 1 : n ) = ∏ i ∫ p θ ( D i , z i ) d z i is intractable under the model and thus cannot be maximized directly. Instead, we take a variational inference approach which seeks to maximize the following evidence lower bound (ELBO) :
B ( ϕ , θ ) ≔ d e f E z 1 : n ∼ q ϕ [ log p θ ( D 1 : n , z 1 : n ) q ϕ ( z 1 : n ∣ D 1 : n ) ] ≤ log p θ ( D 1 : n ) , (18) \mathcal{B}(\phi,\theta) \mathrel{\mathop{\coloneqq}^{\mathrm{def}}} \mathbb{E}_{z_{1:n}\sim q_\phi}\left[\log\frac{p_\theta(\mathcal{D}_{1:n},z_{1:n})}{q_\phi(z_{1:n}\mid\mathcal{D}_{1:n})}\right] \le\log p_\theta(\mathcal{D}_{1:n}), \tag*{(18)} B ( ϕ , θ ) : = def E z 1 : n ∼ q ϕ [ log q ϕ ( z 1 : n ∣ D 1 : n ) p θ ( D 1 : n , z 1 : n ) ] ≤ log p θ ( D 1 : n ) , (18)
where q ϕ ( z 1 : n ∣ D 1 : n ) q_\phi(z_{1:n}\mid\mathcal{D}_{1:n}) q ϕ ( z 1 : n ∣ D 1 : n ) is the variational distribution, parameterized by parameters ϕ \phi ϕ . For any setting of model parameters θ \theta θ , maximizing the ELBO with respect to ϕ \phi ϕ is equivalent to minimizing the KL divergence from the variational distribution to the exact posterior—i.e., KL ( q ϕ ( z 1 : n ∣ D 1 : n ) ∥ p θ ( z 1 : n ∣ D 1 : n ) ) \operatorname{KL}(q_\phi(z_{1:n}\mid\mathcal{D}_{1:n}) \mathbin{\|} p_\theta(z_{1:n}\mid\mathcal{D}_{1:n})) KL ( q ϕ ( z 1 : n ∣ D 1 : n ) ∥ p θ ( z 1 : n ∣ D 1 : n )) . Thus, the fitted variational distribution approximates the posterior.
Iteratively maximizing the ELBO with respect to ϕ \phi ϕ and θ \theta θ constitutes an extension of the original expectation-maximization (EM) algorithm [21 ] known as variational EM , which is guaranteed to converge to a local optimum of the ELBO [57 ] , [81 ] . This guarantee is retained when generalizing to take gradient steps [87 ] —i.e.,
ϕ ← UPDATE ( ϕ , ∇ ϕ B ( ϕ , θ ) ) (variational E-step) (19) \phi\gets\operatorname{UPDATE}\left(\phi,\nabla_\phi\mathcal{B}(\phi,\theta)\right) \qquad\text{(variational E-step)} \tag*{(19)} ϕ ← UPDATE ( ϕ , ∇ ϕ B ( ϕ , θ ) ) (variational E-step) (19)
θ ← UPDATE ( θ , ∇ θ B ( ϕ , θ ) ) (generalized M-step) (20) \theta\gets\operatorname{UPDATE}\left(\theta,\nabla_\theta\mathcal{B}(\phi,\theta)\right) \qquad\text{(generalized M-step)} \tag*{(20)} θ ← UPDATE ( θ , ∇ θ B ( ϕ , θ ) ) (generalized M-step) (20)
where both steps are often performed simultaneously. While optima of the ELBO do not always match optima of the marginal likelihood, variational EM has been shown to yield consistent parameter estimates in a range of models [e.g., [76 ] , [38 ] ], with extensive success reported for others; we find similar success in the next section.
Variational family. To facilitate optimization, we take a standard mean-field approach under which the variational distribution factorizes across all latent variables—i.e.,
q ϕ ( z 1 : n ∣ D 1 : n ) = ∏ i = 1 n q ϕ i ( k i ∣ D i ) q ϕ i ( λ i ∣ D i ) q ϕ i ( τ i ∣ D i ) . q_{\phi}(z_{1:n} \mid\mathcal{D}_{1:n}) = \prod_{i=1}^{n} q_{\phi_i}(k_i \mid\mathcal{D}_i) q_{\phi_i}(\lambda_i \mid\mathcal{D}_i) q_{\phi_i}(\tau_i \mid\mathcal{D}_i). q ϕ ( z 1 : n ∣ D 1 : n ) = i = 1 ∏ n q ϕ i ( k i ∣ D i ) q ϕ i ( λ i ∣ D i ) q ϕ i ( τ i ∣ D i ) .
We further define the family for all three customer-specific factors to be Weibull, as follows:
q ϕ i ( k i ∣ D i ) = Weibull ( S H A P E = s ~ i ( k ) , R A T E = r ~ i ( k ) ) , (21) q_{\phi_i}(k_i \mid\mathcal{D}_i) = \operatorname{Weibull}\left(\mathrm{SHAPE}=\tilde{s}_i^{(k)},\ \mathrm{RATE}=\tilde{r}_i^{(k)}\right), \tag*{(21)} q ϕ i ( k i ∣ D i ) = Weibull ( SHAPE = s ~ i ( k ) , RATE = r ~ i ( k ) ) , (21)
q ϕ i ( λ i ∣ D i ) = Weibull ( S H A P E = s ~ i ( λ ) , R A T E = r ~ i ( λ ) ) , (22) q_{\phi_i}(\lambda_i \mid\mathcal{D}_i) = \operatorname{Weibull}\left(\mathrm{SHAPE}=\tilde{s}_i^{(\lambda)},\ \mathrm{RATE}=\tilde{r}_i^{(\lambda)}\right), \tag*{(22)} q ϕ i ( λ i ∣ D i ) = Weibull ( SHAPE = s ~ i ( λ ) , RATE = r ~ i ( λ ) ) , (22)
q ϕ i ( τ i ∣ D i ) = Weibull ( S H A P E = s ~ i ( τ ) , R A T E = r ~ i ( τ ) , L O C A T I O N = t i , y i ) , (23) q_{\phi_i}(\tau_i \mid\mathcal{D}_i) = \operatorname{Weibull}\left(\mathrm{SHAPE}=\tilde{s}_i^{(\tau)},\ \mathrm{RATE}=\tilde{r}_i^{(\tau)},\ \mathrm{LOCATION}=t_{i,y_i}\right), \tag*{(23)} q ϕ i ( τ i ∣ D i ) = Weibull ( SHAPE = s ~ i ( τ ) , RATE = r ~ i ( τ ) , LOCATION = t i , y i ) , (23)
where each customer is associated with variational parameters ϕ i = d e f { s ~ i ( k ) , r ~ i ( k ) , s ~ i ( λ ) , r ~ i ( λ ) , s ~ i ( τ ) , r ~ i ( τ ) } \phi_i \mathrel{\mathop{=}^{\mathrm{def}}} \{ \tilde{s}_i^{(k)}, \tilde{r}_i^{(k)}, \tilde{s}_i^{(\lambda)}, \tilde{r}_i^{(\lambda)}, \tilde{s}_i^{(\tau)}, \tilde{r}_i^{(\tau)} \} ϕ i = def { s ~ i ( k ) , r ~ i ( k ) , s ~ i ( λ ) , r ~ i ( λ ) , s ~ i ( τ ) , r ~ i ( τ ) } that define the shapes and rates of their three Weibull factors. The extra location parameter in (23) assures that τ i > t i , y i \tau_i > t_{i,y_i} τ i > t i , y i —i.e., that customer i i i was alive through their latest observed transaction—which has probability 1 under the exact posterior. The choice of Weibull for the factors will be key to low-variance gradient estimators, which we develop later.
Amortization. Fitting ϕ 1 : n \phi_{1:n} ϕ 1 : n for all n n n customers is prohibitive for large datasets. However, we can “amortize ” the cost of inference [[35 ] , [89 ] ] by defining each customer-specific set of variational parameters to be the output of some learnable function ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) of the data—i.e., D i → ϕ ( D i ) ϕ i \mathcal{D}_i \xrightarrow{\phi(\mathcal{D}_i)} \phi_i D i ϕ ( D i ) ϕ i . This amortized variational inference approach was first developed for variational autoencoders (VAEs) [[45 ] , [65 ] ] but has since been applied for large-scale inference in more tailored statistical models, such as topic models [[74 ] , [73 ] ]. An amortized variational distribution generally yields a worse approximation to the exact posterior—a tradeoff known as the “amortization gap” [[18 ] , [52 ] ]—but one that is tractable to compute for large datasets.
Recognition network. Parameterizing ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) as a neural network, it is common to refer to it as the “recognition (or inference) network ”. The variational parameters that are fit during inference are then the weights and biases of this network.3 If the network were defined as a standard feedforward neural network, it would require a fixed-length input. However, in our case, the input data D i \mathcal{D}_i D i includes the variable-length set of ITTs Δ t i \Delta t_i Δ t i , since they do not have a sufficient statistic representation under the model. One option is to follow recent work on neural point processes [e.g., [55 ] , [71 ] , [94 ] , [93 ] ] in defining ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) to be an architecture tailored to sequences, such as a recurrent neural network (RNN) or a transformer. However, a simpler and less “data hungry” approach is to prespecify “pseudo-sufficient” statistics—i.e., a function u ( ⋅ ) u(\cdot) u ( ⋅ ) mapping data to a fixed-length feature representation D i → u ( ⋅ ) u i \mathcal{D}_i \xrightarrow{u(\cdot)} u_i D i u ( ⋅ ) u i . A heuristic that we find works well is defining u ( ⋅ ) u(\cdot) u ( ⋅ ) to be an expansion of the sufficient statistics for the Pareto/GGG model—i.e.,
D i ≔ [ Δ t i , 1 ⋮ Δ t i , y i T i ] ∈ R y i + 1 → u ( D i ) u i ≔ [ y i t i , y i κ i T i ζ i ( 1 ) ⋮ ζ i ( P ) ] ∈ R P + 4 → ϕ ( u i ) ϕ i ≔ [ s ~ i ( τ ) r ~ i ( τ ) s ~ i ( λ ) r ~ i ( λ ) s ~ i ( κ ) r ~ i ( κ ) ] ∈ R 6 , (5.7) (24) \mathcal{D}_i \coloneqq\begin{bmatrix} \Delta t_{i,1} \\ \vdots\\ \Delta t_{i,y_i} \\ T_i \end{bmatrix} \in\mathbb{R}^{y_i+1} \xrightarrow{u(\mathcal{D}_i)} u_i \coloneqq\begin{bmatrix} y_i \\ t_{i,y_i} \\ \kappa_i \\ T_i \\ \zeta_i^{(1)} \\ \vdots\\ \zeta_i^{(P)} \end{bmatrix} \in\mathbb{R}^{P+4} \xrightarrow{\phi(u_i)} \phi_i \coloneqq\begin{bmatrix} \widetilde{s}_i^{(\tau)} \\ \widetilde{r}_i^{(\tau)} \\ \widetilde{s}_i^{(\lambda)} \\ \widetilde{r}_i^{(\lambda)} \\ \widetilde{s}_i^{(\kappa)} \\ \widetilde{r}_i^{(\kappa)} \end{bmatrix} \in\mathbb{R}^{6},
\qquad\text{(5.7)} \tag*{(24)} D i : = Δ t i , 1 ⋮ Δ t i , y i T i ∈ R y i + 1 u ( D i ) u i : = y i t i , y i κ i T i ζ i ( 1 ) ⋮ ζ i ( P ) ∈ R P + 4 ϕ ( u i ) ϕ i : = s i ( τ ) r i ( τ ) s i ( λ ) r i ( λ ) s i ( κ ) r i ( κ ) ∈ R 6 , (5.7) (24)
where ζ i ( p ) ≔ ∑ j ( Δ t i , j ) p / 10 \zeta_i^{(p)} \coloneqq\sum_j (\Delta t_{i,j})^{p/10} ζ i ( p ) : = ∑ j ( Δ t i , j ) p /10 is the sum of fractional-powers of ITTs. The vector u i u_i u i exactly matches the sufficient statistics for Pareto/GGG when P = 0 P=0 P = 0 . We find good performance with P = 25 P=25 P = 25 and with ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) defined as a multilayer perceptron (MLP) with at least two hidden layers. More details can be found in § B. We will sometimes refer to the composition of both maps in (24) as the “recognition network” ϕ ( D i ) ≡ ϕ ( u ( D i ) ) \phi(\mathcal{D}_i) \equiv\phi(u(\mathcal{D}_i)) ϕ ( D i ) ≡ ϕ ( u ( D i )) .
Stochastic optimization with the Weibull ``reparameterization trick''
Evaluating the exact gradients in (19) and (20) involves computation across the entire dataset D 1 : n \mathcal{D}_{1:n} D 1 : n which can be prohibitive when n n n is large. This subsection derives unbiased gradient estimators—i.e., ∇ ^ ϕ B ( ϕ , θ ) \widehat{\nabla}_{\phi}\mathcal{B}(\phi,\theta) ∇ ϕ B ( ϕ , θ ) and ∇ ^ θ B ( ϕ , θ ) \widehat{\nabla}_{\theta}\mathcal{B}(\phi,\theta) ∇ θ B ( ϕ , θ ) —that can be computed instead on minibatches M ⊂ D 1 : n \mathcal{M} \subset\mathcal{D}_{1:n} M ⊂ D 1 : n . In general, gradient ascent with unbiased gradient estimates is guaranteed to converge to a local mode of the objective provided that the step-sizes meet the conditions of Robbins and Monro [66 ] , a fact that has inspired the use of stochastic optimization [? , 12 ] with a broader range of optimizers (e.g., ADAM).
In developing the stochastic optimization procedure in this section, it will be useful to define the “local learning signal ” as b i ( z i ; ϕ , θ ) ≔ log p θ ( D i , z i ) q ϕ ( z i ∣ D i ) b_i(z_i;\phi,\theta) \coloneqq\log\frac{p_\theta(\mathcal{D}_i,z_i)}{q_\phi(z_i\mid\mathcal{D}_i)} b i ( z i ; ϕ , θ ) : = log q ϕ ( z i ∣ D i ) p θ ( D i , z i ) and “local ELBO” as
B ( ϕ , θ ) = ∑ i = 1 n B i ( ϕ , θ ) where B i ( ϕ , θ ) ≔ E z i ∼ q ϕ ( z i ∣ D i ) [ b i ( z i ; ϕ , θ ) ] . (25) \mathcal{B}(\phi,\theta) = \sum_{i=1}^{n} \mathcal{B}_i(\phi,\theta) \quad\text{where} \quad\mathcal{B}_i(\phi,\theta) \coloneqq\mathbb{E}_{z_i\sim q_\phi(z_i\mid\mathcal{D}_i)}[b_i(z_i;\phi,\theta)]. \tag*{(25)} B ( ϕ , θ ) = i = 1 ∑ n B i ( ϕ , θ ) where B i ( ϕ , θ ) : = E z i ∼ q ϕ ( z i ∣ D i ) [ b i ( z i ; ϕ , θ )] . (25)
Given an unbiased estimator for the gradient of the local ELBO ∇ ^ B i ( ϕ , θ ) \widehat{\nabla}\mathcal{B}_i(\phi,\theta) ∇ B i ( ϕ , θ ) , it is straightforward to obtain an unbiased gradient estimate of the full ELBO from a minibatch M ∼ Uniform ( 1 … n ) \mathcal{M}\sim\operatorname{Uniform}(1\ldots n) M ∼ Uniform ( 1 … n ) , since ∇ B ( ϕ , θ ) = ∑ i = 1 n ∇ B i ( ϕ , θ ) \nabla\mathcal{B}(\phi,\theta)=\sum_{i=1}^{n}\nabla\mathcal{B}_i(\phi,\theta) ∇ B ( ϕ , θ ) = ∑ i = 1 n ∇ B i ( ϕ , θ ) and thus n ∣ M ∣ ∑ i ∈ M ∇ ^ B i ( ϕ , θ ) \frac{n}{|\mathcal{M}|}\sum_{i\in\mathcal{M}}\widehat{\nabla}\mathcal{B}_i(\phi,\theta) ∣ M ∣ n ∑ i ∈ M ∇ B i ( ϕ , θ ) is unbiased for ∇ B ( ϕ , θ ) \nabla\mathcal{B}(\phi,\theta) ∇ B ( ϕ , θ ) . The problem thus reduces to deriving unbiased estimators for the local gradients.
Estimating ∇ θ B i ( ϕ , θ ) \nabla_\theta\mathcal{B}_i(\phi,\theta) ∇ θ B i ( ϕ , θ ) . This case is simple, as the gradient pushes into the expectation:
∇ θ B i ( ϕ , θ ) = ∇ θ E z i ∼ q ϕ ( z i ∣ D i ) [ b i ( z i ; ϕ , θ ) ] = E z i ∼ q ϕ ( z i ∣ D i ) [ ∇ θ b i ( z i ; ϕ , θ ) ] , (26) \nabla_\theta\mathcal{B}_i(\phi,\theta) = \nabla_\theta\mathbb{E}_{z_i\sim q_\phi(z_i\mid\mathcal{D}_i)}[b_i(z_i;\phi,\theta)] = \mathbb{E}_{z_i\sim q_\phi(z_i\mid\mathcal{D}_i)}[\nabla_\theta b_i(z_i;\phi,\theta)], \tag*{(26)} ∇ θ B i ( ϕ , θ ) = ∇ θ E z i ∼ q ϕ ( z i ∣ D i ) [ b i ( z i ; ϕ , θ )] = E z i ∼ q ϕ ( z i ∣ D i ) [ ∇ θ b i ( z i ; ϕ , θ )] , (26)
An unbiased estimator is then straightforward to obtain via Monte Carlo:
∇ ^ θ M C B i ( ϕ , θ ) ≔ 1 S ∑ s = 1 S ∇ θ b i ( z i , s ; ϕ , θ ) where z i , s ∼ i i d q ϕ ( z i ∣ D i ) . (27) \widehat{\nabla}^{\mathrm{MC}}_\theta\mathcal{B}_i(\phi,\theta) \coloneqq\frac{1}{S}\sum_{s=1}^{S}\nabla_\theta b_i(z_{i,s};\phi,\theta) \quad\text{where} \quad z_{i,s}\overset{\mathrm{iid}}{\sim}q_\phi(z_i\mid\mathcal{D}_i). \tag*{(27)} ∇ θ MC B i ( ϕ , θ ) : = S 1 s = 1 ∑ S ∇ θ b i ( z i , s ; ϕ , θ ) where z i , s ∼ iid q ϕ ( z i ∣ D i ) . (27)
We also note that for this particular model ∇ θ b i ( z i ; ϕ , θ ) = ∇ θ log p θ ( z i ) \nabla_{\theta} b_i(z_i;\phi,\theta)=\nabla_{\theta}\log p_{\theta}(z_i) ∇ θ b i ( z i ; ϕ , θ ) = ∇ θ log p θ ( z i ) .
Estimating ∇ ϕ B i ( ϕ , θ ) \nabla_{\phi}\mathcal{B}_i(\phi,\theta) ∇ ϕ B i ( ϕ , θ ) . In this case, the gradient does not push in because ϕ \phi ϕ parameterizes the distribution governing the expectation—i.e.,
∇ ϕ B i ( ϕ , θ ) = ∇ ϕ E z i ∼ q ϕ ( z i ∣ D i ) [ b i ( z i ; ϕ , θ ) ] ≠ E z i ∼ q ϕ ( z i ∣ D i ) [ ∇ ϕ b i ( z i ; ϕ , θ ) ] (28) \nabla_{\phi}\mathcal{B}_i(\phi,\theta)=\nabla_{\phi}\mathbb{E}_{z_i\sim q_{\phi}(z_i\mid\mathcal{D}_i)}[b_i(z_i;\phi,\theta)]\ne\mathbb{E}_{z_i\sim q_{\phi}(z_i\mid\mathcal{D}_i)}[\nabla_{\phi}b_i(z_i;\phi,\theta)] \tag*{(28)} ∇ ϕ B i ( ϕ , θ ) = ∇ ϕ E z i ∼ q ϕ ( z i ∣ D i ) [ b i ( z i ; ϕ , θ )] = E z i ∼ q ϕ ( z i ∣ D i ) [ ∇ ϕ b i ( z i ; ϕ , θ )] (28)
Instead, we will exploit the following fact. A Weibull can be simulated as a deterministic function of unit Exponential noise—i.e.,
w ∼ Weibull ( S H A P E = s , R A T E = r , L O C = ℓ ) ⟺ w = ε 1 / s r − 1 + ℓ where ε ∼ Exp ( 1 ) . (29) w\sim\operatorname{Weibull}(\mathrm{SHAPE}=s,\mathrm{RATE}=r,\mathrm{LOC}=\ell)\Longleftrightarrow w=\varepsilon^{1/s}r^{-1}+\ell\quad\text{where }\varepsilon\sim\operatorname{Exp}(1). \tag*{(29)} w ∼ Weibull ( SHAPE = s , RATE = r , LOC = ℓ ) ⟺ w = ε 1/ s r − 1 + ℓ where ε ∼ Exp ( 1 ) . (29)
Simulating z i = d e f [ k i , λ i , τ i ] ∼ q ϕ ( z i ∣ D i ) z_i\mathrel{\mathop{=}^{\mathrm{def}}}[k_i,\lambda_i,\tau_i]\sim q_{\phi}(z_i\mid\mathcal{D}_i) z i = def [ k i , λ i , τ i ] ∼ q ϕ ( z i ∣ D i ) can thus proceed by sampling three Exponential variables ε i = d e f [ ε i ( k ) , ε i ( λ ) , ε i ( τ ) ] ∼ i i d Exp ( 1 ) \varepsilon_i\mathrel{\mathop{=}^{\mathrm{def}}}[\varepsilon_i^{(k)},\varepsilon_i^{(\lambda)},\varepsilon_i^{(\tau)}]\mathrel{\mathop{\sim}^{\mathrm{iid}}}\operatorname{Exp}(1) ε i = def [ ε i ( k ) , ε i ( λ ) , ε i ( τ ) ] ∼ iid Exp ( 1 ) and computing the function z i = z ϕ ( ε i ; D i ) z_i=z_{\phi}(\varepsilon_i;\mathcal{D}_i) z i = z ϕ ( ε i ; D i ) defined as
z ϕ ( ε i ; D i ) = d e f [ ( ε i ( k ) ) 1 / s ~ i ( k ) r ~ i ( k ) , ( ε i ( λ ) ) 1 / s ~ i ( λ ) r ~ i ( λ ) , ( ε i ( τ ) ) 1 / s ~ i ( τ ) r ~ i ( τ ) + t i , y i ] . (30) z_{\phi}(\varepsilon_i;\mathcal{D}_i)\mathrel{\mathop{=}^{\mathrm{def}}}\left[\frac{(\varepsilon_i^{(k)})^{1/\tilde{s}_i^{(k)}}}{\tilde{r}_i^{(k)}},\frac{(\varepsilon_i^{(\lambda)})^{1/\tilde{s}_i^{(\lambda)}}}{\tilde{r}_i^{(\lambda)}},\frac{(\varepsilon_i^{(\tau)})^{1/\tilde{s}_i^{(\tau)}}}{\tilde{r}_i^{(\tau)}}+t_{i,y_i}\right]. \tag*{(30)} z ϕ ( ε i ; D i ) = def [ r ~ i ( k ) ( ε i ( k ) ) 1/ s ~ i ( k ) , r ~ i ( λ ) ( ε i ( λ ) ) 1/ s ~ i ( λ ) , r ~ i ( τ ) ( ε i ( τ ) ) 1/ s ~ i ( τ ) + t i , y i ] . (30)
where the variational parameters are output from the recognition network ϕ ( D i ) \phi(\mathcal{D}_i) ϕ ( D i ) in (24) . This allows us to rewrite ∇ ϕ E z i ∼ q ϕ ( z i ∣ D i ) [ f ( z i ) ] = E ε i [ ∇ ϕ f ( z ϕ ( ε i ; D i ) ) ] \nabla_{\phi}\mathbb{E}_{z_i\sim q_{\phi}(z_i\mid\mathcal{D}_i)}[f(z_i)]=\mathbb{E}_{\varepsilon_i}[\nabla_{\phi}f(z_{\phi}(\varepsilon_i;\mathcal{D}_i))] ∇ ϕ E z i ∼ q ϕ ( z i ∣ D i ) [ f ( z i )] = E ε i [ ∇ ϕ f ( z ϕ ( ε i ; D i ))] for any f ( z i ) f(z_i) f ( z i ) since ϕ \phi ϕ no longer parameterizes the expectation, allowing ∇ ϕ \nabla_{\phi} ∇ ϕ to push in. Applying this to (28) , we can then obtain the following unbiased gradient estimator via Monte Carlo:
∇ ^ ϕ M C B i ( ϕ , θ ) = d e f 1 S ∑ s = 1 S ∇ ϕ b i ( z ϕ ( ε i , s ; D i ) ; ϕ , θ ) where ε i , s ∼ i i d Exp ( 1 ) . (31) \widehat{\nabla}_{\phi}^{\mathrm{MC}}\mathcal{B}_i(\phi,\theta)\mathrel{\mathop{=}^{\mathrm{def}}}\frac{1}{S}\sum_{s=1}^{S}\nabla_{\phi}b_i(z_{\phi}(\varepsilon_{i,s};\mathcal{D}_i);\phi,\theta)\quad\text{where }\varepsilon_{i,s}\mathrel{\mathop{\sim}^{\mathrm{iid}}}\operatorname{Exp}(1). \tag*{(31)} ∇ ϕ MC B i ( ϕ , θ ) = def S 1 s = 1 ∑ S ∇ ϕ b i ( z ϕ ( ε i , s ; D i ) ; ϕ , θ ) where ε i , s ∼ iid Exp ( 1 ) . (31)
This idea was first developed as the “pathwise gradient” estimator [39 ] and later re-introduced as the “reparameterization trick” for inference in variational autoencoders (VAEs) with Gaussian latent variables [45 , 65 , 75 ] . Not all parametric families can be reparameterized explicitly, such as the Gamma distribution, for which approximate schemes for reparameterization gradients are required [34 ] . The Weibull is one of the few non-Gaussian distributions for which the explicit “reparameterization trick” is available, a fact which was exploited first by [90 ] to construct a non-negative VAE, and later by a handful of others [e.g., [72 , 82 , 16 , 23 , 83 , 91 ] ].
Basic algorithm. The core procedure is given in Lines 4–11 of Algorithm 1. Rather than explicitly implement the gradient estimators in (27) and (31) , we compute a Monte Carlo estimate of the ELBO B ^ M C ( ϕ , θ ) \widehat{\mathcal{B}}^{\mathrm{MC}}(\phi,\theta) B MC ( ϕ , θ ) and obtain both gradient estimates via backpropagation (i.e., one pass of reverse-mode automatic-differentiation) which, since the reparameterization trick makes B ( ϕ , θ ) \mathcal{B}(\phi,\theta) B ( ϕ , θ ) a differentiable function of both ϕ \phi ϕ and θ \theta θ , is mathematically equivalent. As displayed, Algorithm 1 computes the Rao–Blackwellized refinement B ^ R B \widehat{\mathcal{B}}^{\mathrm{RB}} B RB of this estimator, given in (33) of § 5.3.
Software implementation. The basic algorithm is easy to implement as both p θ p_{\theta} p θ and q ϕ q_{\phi} q ϕ involve only simple functions, all of which are supported for automatic differentiation in all major Algorithm 1: Amortized VI for WTYD with improvements and optional extensions in green
Input: training data D 1 : n \mathcal{D}_{1:n} D 1 : n ; UPDATE( ⋅ ) (\cdot) ( ⋅ ) ; minibatch size ∣ M ∣ |\mathcal{M}| ∣ M ∣ ; samples S S S ; noise bounds [ a , b ] [a,b] [ a , b ] ; initialized parameters θ \theta θ , recognition network ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) ; (optional) synthetic data size n 0 n_0 n 0 ; (optional) validation data: D n + 1 : n + V \mathcal{D}_{n+1:n+V} D n + 1 : n + V (observation period), D n + 1 : n + V ⋆ \mathcal{D}^{\star}_{n+1:n+V} D n + 1 : n + V ⋆ (holdout)
Output: fitted parameters θ \theta θ , recognition network ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ )
(Optional) Inference compilation, § 5.3
( D ^ i , z ^ i ) i = 1 n 0 ∼ i i d p θ ( D i , z i ) (\widehat{\mathcal{D}}_i,\widehat{z}_i)_{i=1}^{n_0} \overset{\mathrm{iid}}{\sim} p_\theta(\mathcal{D}_i,z_i) ( D i , z i ) i = 1 n 0 ∼ iid p θ ( D i , z i ) ▷ sample synthetic dataset of size n 0 n_0 n 0
repeat until maximum iteration ▷ fit network ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) in supervised manner
ϕ ← UPDATE ( ϕ , ∇ ϕ log q ϕ ( z ^ 1 : n 0 ∣ D ^ 1 : n 0 ) ) \phi\leftarrow\operatorname{UPDATE}(\phi,\nabla_\phi\log q_\phi(\widehat{z}_{1:n_0}\mid\widehat{\mathcal{D}}_{1:n_0})) ϕ ← UPDATE ( ϕ , ∇ ϕ log q ϕ ( z 1 : n 0 ∣ D 1 : n 0 ))
repeat until ELBO converges
M ∼ Uniform ( { 1 , … , n } ) \mathcal{M} \sim\operatorname{Uniform}(\{1,\ldots,n\}) M ∼ Uniform ({ 1 , … , n }) ▷ sample minibatch of size ∣ M ∣ |\mathcal{M}| ∣ M ∣
for i ∈ M i \in\mathcal{M} i ∈ M , s = 1 , … , S s=1,\ldots,S s = 1 , … , S do
ε i , s = [ ε i , s ( k ) , ε i , s ( λ ) , ε i , s ( τ ) ] ∼ i i d TruncExp [ a , b ] ( 1 ) \varepsilon_{i,s} = [\varepsilon^{(k)}_{i,s},\varepsilon^{(\lambda)}_{i,s},\varepsilon^{(\tau)}_{i,s}] \overset{\mathrm{iid}}{\sim} \operatorname{TruncExp}_{[a,b]}(1) ε i , s = [ ε i , s ( k ) , ε i , s ( λ ) , ε i , s ( τ ) ] ∼ iid TruncExp [ a , b ] ( 1 ) ▷ sample truncated Exponential noise
z i , s = [ k i , s , λ i , s , τ i , s ] ← z ϕ ( ε i , s ; D i ) z_{i,s}=[k_{i,s},\lambda_{i,s},\tau_{i,s}] \leftarrow z_\phi(\varepsilon_{i,s};\mathcal{D}_i) z i , s = [ k i , s , λ i , s , τ i , s ] ← z ϕ ( ε i , s ; D i ) ▷ Weibull reparameterization, (30)
B ^ R B ( ϕ , θ ) ← n ∣ M ∣ ∑ i ∈ M B ^ i R B ( ϕ , θ ) \widehat{\mathcal{B}}^{\mathrm{RB}}(\phi,\theta)\leftarrow\frac{n}{|\mathcal{M}|}\sum_{i\in\mathcal{M}}\widehat{\mathcal{B}}^{\mathrm{RB}}_i(\phi,\theta) B RB ( ϕ , θ ) ← ∣ M ∣ n ∑ i ∈ M B i RB ( ϕ , θ ) ▷ Rao–Blackwellized ELBO, (33)
ϕ ← UPDATE ( ϕ , ∇ ϕ B ^ R B ( ϕ , θ ) ) \phi\leftarrow\operatorname{UPDATE}(\phi,\nabla_\phi\widehat{\mathcal{B}}^{\mathrm{RB}}(\phi,\theta)) ϕ ← UPDATE ( ϕ , ∇ ϕ B RB ( ϕ , θ )) ▷ E-step (gradient via backpropagation)
θ ← UPDATE ( θ , ∇ θ B ^ R B ( ϕ , θ ) ) \theta\leftarrow\operatorname{UPDATE}(\theta,\nabla_\theta\widehat{\mathcal{B}}^{\mathrm{RB}}(\phi,\theta)) θ ← UPDATE ( θ , ∇ θ B RB ( ϕ , θ )) ▷ M-step (gradient via backpropagation)
(Optional) Early stopping, § 5.3
Estimate q ϕ ( D n + 1 : n + V ⋆ ∣ D n + 1 : n + V ) q_\phi(\mathcal{D}^{\star}_{n+1:n+V}\mid\mathcal{D}_{n+1:n+V}) q ϕ ( D n + 1 : n + V ⋆ ∣ D n + 1 : n + V ) ▷ posterior predictive density, (38)
if no improvement in recent iterations then
set θ , ϕ ( ⋅ ) \theta,\phi(\cdot) θ , ϕ ( ⋅ ) to best checkpoint and stop
return θ , ϕ ( ⋅ ) \theta,\phi(\cdot) θ , ϕ ( ⋅ )
machine learning packages, such as PyTorch [5 ] , which we use. We note that p θ p_\theta p θ for the Pareto/GGG model involves the incomplete gamma function, which is not supported in most automatic differentiation frameworks, like PyTorch, and is thus why the change to a Weibull transaction process enables this form of scalable inference.
Algorithmic improvements and practical details
This section provides further refinements to the core variational algorithm that improve its practical performance, mainly by reducing the variance of gradient estimates and providing effective initialization to the recognition network. It also provides details on effective optimization schedules, with per-experiment settings further collected in § B.
Rao-Blackwellization. The variance of the Monte Carlo ELBO estimate of Algorithm 1, and thus of the gradients backpropagated through it, can be reduced by exploiting semi-analytic structure that the ELBO admits. Specifically, we can decompose the learning signal into
b i ( z i ; ϕ , θ ) = log p θ ( k i , λ i ) q ϕ ( z i ∣ D i ) ⏟ = d e f b i R B ( z i ; ϕ , θ ) + log p ( D i ∣ z i ) p θ ( τ i ) ⏟ = d e f b i − R B ( z i ; ϕ , θ ) . (32) b_i(z_i;\phi,\theta)=\underbrace{\log\frac{p_\theta(k_i,\lambda_i)}{q_\phi(z_i\mid\mathcal{D}_i)}}_{\mathrel{\mathop{=}^{\mathrm{def}}}b_i^{\mathrm{RB}}(z_i;\phi,\theta)}+\underbrace{\log p(\mathcal{D}_i\mid z_i)p_\theta(\tau_i)}_{\mathrel{\mathop{=}^{\mathrm{def}}}b_i^{-\mathrm{RB}}(z_i;\phi,\theta)}. \tag*{(32)} b i ( z i ; ϕ , θ ) = = def b i RB ( z i ; ϕ , θ ) log q ϕ ( z i ∣ D i ) p θ ( k i , λ i ) + = def b i − RB ( z i ; ϕ , θ ) log p ( D i ∣ z i ) p θ ( τ i ) . (32)
where we can show that the expectation of the first term is analytic. Modifying the algorithm so it computes the exact expectation of the first term is a form of Rao-Blackwellization, which decreases variance [62 ] . The Rao-Blackwellized estimator of the local ELBO is then
B i R B ( ϕ , θ ) = d e f E z i ∼ q ϕ ( z i ∣ D i ) [ b i R B ( z i ; ϕ , θ ) ] + 1 S ∑ s = 1 S b i − R B ( z i , s ; ϕ , θ ) , (33) \mathcal{B}_i^{\mathrm{RB}}(\phi,\theta)\mathrel{\mathop{=}^{\mathrm{def}}}\mathbb{E}_{z_i\sim q_\phi(z_i\mid\mathcal{D}_i)}\left[b_i^{\mathrm{RB}}(z_i;\phi,\theta)\right]+\frac{1}{S}\sum_{s=1}^{S}b_i^{-\mathrm{RB}}(z_{i,s};\phi,\theta), \tag*{(33)} B i RB ( ϕ , θ ) = def E z i ∼ q ϕ ( z i ∣ D i ) [ b i RB ( z i ; ϕ , θ ) ] + S 1 s = 1 ∑ S b i − RB ( z i , s ; ϕ , θ ) , (33)
where the first expectation, which can be formed analytically, equals
E z i ∼ q ϕ ( z i ∣ D i ) [ b i R B ( z i ; ϕ , θ ) ] = H [ q ϕ ( τ i ∣ D i ) ] − KL ( q ϕ ( k i , λ i ∣ D i ) ∥ p θ ( k i , λ i ) ) . (34) \mathbb{E}_{z_i\sim q_\phi(z_i\mid\mathcal{D}_i)}\left[b_i^{\mathrm{RB}}(z_i;\phi,\theta)\right]=\mathbb{H}\left[q_\phi(\tau_i\mid\mathcal{D}_i)\right]-\operatorname{KL}\left(q_\phi(k_i,\lambda_i\mid\mathcal{D}_i)\mathbin{\|}p_\theta(k_i,\lambda_i)\right). \tag*{(34)} E z i ∼ q ϕ ( z i ∣ D i ) [ b i RB ( z i ; ϕ , θ ) ] = H [ q ϕ ( τ i ∣ D i ) ] − KL ( q ϕ ( k i , λ i ∣ D i ) ∥ p θ ( k i , λ i ) ) . (34)
where H \mathbb{H} H is Shannon entropy, which is H [ Weibull ( s ~ , r ~ , ℓ ) ] = γ ( 1 − 1 / s ~ ) − log ( s ~ r ~ ) + 1 \mathbb{H}[\operatorname{Weibull}(\tilde{s},\tilde{r},\ell)]=\gamma(1-1/\tilde{s})-\log(\tilde{s}\tilde{r})+1 H [ Weibull ( s ~ , r ~ , ℓ )] = γ ( 1 − 1/ s ~ ) − log ( s ~ r ~ ) + 1 for a Weibull. Note that it does not depend on location ℓ \ell ℓ . The KL term is the sum of two KL divergences, each from a Weibull variational factor to a Gamma prior, the general form for which follows from the KL between two Generalized Gamma distributions [9 ] —i.e.,
KL ( Weibull ( s ~ , r ~ ) ∥ Gamma ( s , r ) ) = log ( s ~ Γ ( s ) ) + s log ( r ~ / r ) + ( r / r ~ ) Γ ( 1 + 1 / s ~ ) + γ ( s / s ~ − 1 ) − 1. \operatorname{KL}(\operatorname{Weibull}(\tilde{s},\tilde{r})\mathbin{\|}\operatorname{Gamma}(s,r))=\log(\tilde{s}\,\Gamma(s))+s\log(\tilde{r}/r)+(r/\tilde{r})\Gamma(1+1/\tilde{s})+\gamma(s/\tilde{s}-1)-1. KL ( Weibull ( s ~ , r ~ ) ∥ Gamma ( s , r )) = log ( s ~ Γ ( s )) + s log ( r ~ / r ) + ( r / r ~ ) Γ ( 1 + 1/ s ~ ) + γ ( s / s ~ − 1 ) − 1.
Both expressions are differentiable in s ~ , r ~ \tilde{s},\tilde{r} s ~ , r ~ , which correspond to the variational parameters, and contribute zero Monte Carlo variance to the gradient when computed exactly.
Inference compilation. We initialize the recognition network by fitting it in a supervised manner to pairs ( D ^ i , z ^ i ) ∼ i i d p θ ( D i , z i ) (\hat{\mathcal{D}}_i,\hat{z}_i)\stackrel{\mathrm{iid}}{\sim}p_\theta(\mathcal{D}_i,z_i) ( D ^ i , z ^ i ) ∼ iid p θ ( D i , z i ) simulated from the generative model for a fixed initial θ \theta θ . In other words, we initialize ϕ \phi ϕ by maximizing the conditional log-likelihood of simulated data:
max ϕ E ( D i , z i ) ∼ p θ ( D i , z i ) [ log q ϕ ( z i ∣ D i ) ] (35) \max_{\phi}\ \mathbb{E}_{(\mathcal{D}_i,z_i)\sim p_\theta(\mathcal{D}_i,z_i)}\left[\log q_\phi(z_i\mid\mathcal{D}_i)\right] \tag*{(35)} ϕ max E ( D i , z i ) ∼ p θ ( D i , z i ) [ log q ϕ ( z i ∣ D i ) ] (35)
The variational family q ϕ ( z i ∣ D i ) q_\phi(z_i\mid\mathcal{D}_i) q ϕ ( z i ∣ D i ) is thus initialized to a good approximation of the exact posterior p θ ( z i ∣ D i ) p_\theta(z_i\mid\mathcal{D}_i) p θ ( z i ∣ D i ) for the initial θ \theta θ . It therefore produces sensible samples of z i z_i z i when beginning the main training loop which then updates θ \theta θ and ϕ \phi ϕ jointly. This procedure is cheap, as it only requires sampling synthetic data and doing gradient ascent over a fixed dataset. In the Bayesian literature, this idea is known as “inference compilation ” [46 ] . It also corresponds to the sleep phase of the “wake-sleep” algorithm [41 , 11 , 47 ] and is an instance of “forward amortized VI” [4 ] .
Implicit gradient clipping via noise truncation. When the variational shape parameter s ~ \tilde{s} s ~ is small, the reparameterization map ε ↦ ε 1 / s ~ / r ~ \varepsilon\mapsto\varepsilon^{1/\tilde{s}}/\tilde{r} ε ↦ ε 1/ s ~ / r ~ of (30) can yield an extreme sample of z i z_i z i . Extreme values can cause numerical instability and large spikes in the training loss which hinder convergence. We resolve this by replacing the unit exponential noise with a truncated exponential over [ a , b ] [a,b] [ a , b ] with a = log ( 1 / 0.99 ) ≈ 0.01 a=\log(1/0.99)\approx0.01 a = log ( 1/0.99 ) ≈ 0.01 and b = log ( 1 / 0.01 ) ≈ 4.6 b=\log(1/0.01)\approx4.6 b = log ( 1/0.01 ) ≈ 4.6 , discarding only the most extreme 1% of each tail. This acts as an adaptive gradient-clipping scheme, which can coupled with more standard gradient clipping. This trades gradient variance against a small amount of reparameterization bias. A similar scheme is reported by [90 ] .
Early stopping. A small fraction of customers can be held out during training as a validation set to assess overfitting. For V V V of such customers, we split their ITT histories into observation and holdout periods so that the validation data consists of D n + 1 : n + V \mathcal{D}_{n+1:n+V} D n + 1 : n + V and D n + 1 : n + V ⋆ \mathcal{D}^{\star}_{n+1:n+V} D n + 1 : n + V ⋆ . During training, we then estimate the posterior predictive density q ϕ ( D n + 1 : n + V ⋆ ∣ D n + 1 : n + V ) q_{\phi}(\mathcal{D}^{\star}_{n+1:n+V} \mid\mathcal{D}_{n+1:n+V}) q ϕ ( D n + 1 : n + V ⋆ ∣ D n + 1 : n + V ) under the variational distribution (see § 5.4), tracking its running maximum. If a certain number of consecutive iterations fail to increase this maximum, we stop the training loop early and return the most recent checkpoint.
Learning-rate schedule. We use AdamW [50 ] for the gradient UPDATE in alg:1 with a learning rate following a cosine schedule annealed to near zero, with no warmup or restarts. Per-experiment initial rates, batch sizes, and network widths, among other details are given in § B.
Variational approximations of the posterior quantities
The three posterior key quantities listed in § 3 are analytically intractable under the model p θ p_{\theta} p θ . However, the variational posterior q ϕ q_{\phi} q ϕ has convenient analytic properties that enable principled approximations to all three. This section provides an overview of those approximations, with derivations and proofs relegated to § A.2.
P ( A L I V E ) P(\mathrm{ALIVE}) P ( ALIVE ) . Replacing the intractable posterior marginal p θ ( τ i ∣ D i ) p_{\theta}(\tau_i \mid\mathcal{D}_i) p θ ( τ i ∣ D i ) by the Weibull variational factor q ϕ ( τ i ∣ D i ) q_{\phi}(\tau_i \mid\mathcal{D}_i) q ϕ ( τ i ∣ D i ) and integrating over ( T i , ∞ ) (T_i,\infty) ( T i , ∞ ) yields a closed-form approximation—i.e.,
q ϕ ( A L I V E ∣ D i ) ≔ ∫ T i ∞ q ϕ ( τ i ∣ D i ) d τ i = exp ( − [ r ~ i ( τ ) ( T i − t i , y i ) ] s ~ i ( τ ) ) . (36) q_{\phi}(\mathrm{ALIVE} \mid\mathcal{D}_i) \coloneqq\int_{T_i}^{\infty} q_{\phi}(\tau_i \mid\mathcal{D}_i)\,\mathrm{d}\tau_i = \exp\left(-\left[\tilde{r}_i^{(\tau)}(T_i-t_{i,y_i})\right]^{\tilde{s}_i^{(\tau)}}\right). \tag*{(36)} q ϕ ( ALIVE ∣ D i ) : = ∫ T i ∞ q ϕ ( τ i ∣ D i ) d τ i = exp ( − [ r ~ i ( τ ) ( T i − t i , y i ) ] s ~ i ( τ ) ) . (36)
which is a simple function of variational parameters, recency, and total time.
Posterior predictive distribution. Given a training window ( 0 , T i ] (0,T_i] ( 0 , T i ] and a test window ( T i , T i ⋆ ] (T_i,T_i^{\star}] ( T i , T i ⋆ ] with held-out future TTs D i ⋆ \mathcal{D}_i^{\star} D i ⋆ , the posterior predictive density is p θ ( D i ⋆ ∣ D i ) = E p θ ( z i ∣ D i ) [ p ( D i ⋆ ∣ z i ) ] p_{\theta}(\mathcal{D}_i^{\star} \mid\mathcal{D}_i) = \mathbb{E}_{p_{\theta}(z_i \mid\mathcal{D}_i)}[p(\mathcal{D}_i^{\star} \mid z_i)] p θ ( D i ⋆ ∣ D i ) = E p θ ( z i ∣ D i ) [ p ( D i ⋆ ∣ z i )] . Here and below, p ( D i ⋆ ∣ z i ) p(\mathcal{D}_i^{\star} \mid z_i) p ( D i ⋆ ∣ z i ) suppresses an implicit conditioning on D i \mathcal{D}_i D i , which enters through the time already elapsed since the last observed transaction (see § A.2). The plug-in variational approximation is then
q ϕ ( D i ⋆ ∣ D i ) ≔ E q ϕ ( z i ∣ D i ) [ p ( D i ⋆ ∣ z i ) ] = q ϕ ( τ i > t i , y i ⋆ ∣ D i ) ⋅ E q ϕ ( z i ∣ D i , τ i > t i , y i ⋆ ) [ p ( D i ⋆ ∣ z i ) ] , (37) q_{\phi}(\mathcal{D}_i^{\star} \mid\mathcal{D}_i) \coloneqq\mathbb{E}_{q_{\phi}(z_i \mid\mathcal{D}_i)}[p(\mathcal{D}_i^{\star} \mid z_i)]
= q_{\phi}(\tau_i > t_{i,y_i}^{\star} \mid\mathcal{D}_i)\cdot\mathbb{E}_{q_{\phi}(z_i \mid\mathcal{D}_i,\tau_i>t_{i,y_i}^{\star})}[p(\mathcal{D}_i^{\star} \mid z_i)], \tag*{(37)} q ϕ ( D i ⋆ ∣ D i ) : = E q ϕ ( z i ∣ D i ) [ p ( D i ⋆ ∣ z i )] = q ϕ ( τ i > t i , y i ⋆ ∣ D i ) ⋅ E q ϕ ( z i ∣ D i , τ i > t i , y i ⋆ ) [ p ( D i ⋆ ∣ z i )] , (37)
which factors cleanly into a survival probability and a density conditional on survival. The first factor takes the form of q ϕ ( A L I V E ∣ D i ) q_{\phi}(\mathrm{ALIVE} \mid\mathcal{D}_i) q ϕ ( ALIVE ∣ D i ) in (36) while the second is the individual-level likelihood in (14) evaluated on D i ⋆ \mathcal{D}_i^{\star} D i ⋆ and averaged over the variational posterior q ϕ ( z i ∣ D i , τ i > t i , y i ⋆ ) q_{\phi}(z_i \mid\mathcal{D}_i,\tau_i>t_{i,y_i}^{\star}) q ϕ ( z i ∣ D i , τ i > t i , y i ⋆ ) . The following is thus a Monte Carlo estimator
q ^ ϕ M C ( D i ⋆ ∣ D i ) ≔ q ϕ ( τ i > t i , y i ⋆ ∣ D i ) ⋅ [ 1 S ∑ s = 1 S p ( D i ⋆ ∣ z i ( s ) ) ] , z i ( s ) ∼ i i d q ϕ ( z i ∣ D i , τ i > t i , y i ⋆ ) . (38) \widehat{q}_{\phi}^{\mathrm{MC}}(\mathcal{D}_i^{\star} \mid\mathcal{D}_i) \coloneqq q_{\phi}(\tau_i>t_{i,y_i}^{\star} \mid\mathcal{D}_i)\cdot\left[\frac{1}{S}\sum_{s=1}^{S}p(\mathcal{D}_i^{\star}\mid z_i^{(s)})\right], \qquad z_i^{(s)} \overset{\mathrm{iid}}{\sim} q_{\phi}(z_i \mid\mathcal{D}_i,\tau_i>t_{i,y_i}^{\star}). \tag*{(38)} q ϕ MC ( D i ⋆ ∣ D i ) : = q ϕ ( τ i > t i , y i ⋆ ∣ D i ) ⋅ [ S 1 s = 1 ∑ S p ( D i ⋆ ∣ z i ( s ) ) ] , z i ( s ) ∼ iid q ϕ ( z i ∣ D i , τ i > t i , y i ⋆ ) . (38)
which only relies upon drawing i.i.d. samples of z i z_i z i from the conditional variational posterior. Since all three latent variables are independent under q ϕ q_\phi q ϕ , this amounts to drawing k i ( s ) k_i^{(s)} k i ( s ) and λ i ( s ) \lambda_i^{(s)} λ i ( s ) from their Weibull factors and τ i ( s ) \tau_i^{(s)} τ i ( s ) from its Weibull factor left-truncated at t i , y i ⋆ t_{i,y_i}^{\star} t i , y i ⋆ .
Forecasted future frequency. The forecast frequency is the posterior, finite-horizon counterpart of the expected CLV derived in § 4. The plug-in variational approximation is
E ϕ [ Y i ( T i , T i ⋆ ) ∣ D i ] = d e f E z i ∼ q ϕ ( z i ∣ D i ) [ E [ Y i ( T i , T i ⋆ ∧ τ i ) ∣ z i ] ] . (39) \mathbb{E}_\phi\left[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i\right]\mathrel{\overset{\mathrm{def}}{=}}\mathbb{E}_{z_i\sim q_\phi(z_i\mid\mathcal{D}_i)}\left[\mathbb{E}\left[Y_i(T_i,T_i^\star\wedge\tau_i)\mid z_i\right]\right]. \tag*{(39)} E ϕ [ Y i ( T i , T i ⋆ ) ∣ D i ] = def E z i ∼ q ϕ ( z i ∣ D i ) [ E [ Y i ( T i , T i ⋆ ∧ τ i ) ∣ z i ] ] . (39)
where the inner expectation is that of a Weibull Count random variable which can be Monte Carlo approximated or computed exactly via the recurrence of McShane et al. [54 ] . The posterior analogue to the Monte Carlo estimator for expected CLV in (15) is thus
E ^ ϕ M C [ Y i ( T i , T i ⋆ ) ∣ D i ] = d e f 1 S ∑ s = 1 S E [ Y i ( T i , T i ⋆ ∧ τ i ( s ) ) ∣ z i ( s ) ] , z i ( s ) ∼ i i d q ϕ ( z i ∣ D i ) . (40) \widehat{\mathbb{E}}_\phi^{\mathrm{MC}}\left[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i\right]\mathrel{\overset{\mathrm{def}}{=}}\frac{1}{S}\sum_{s=1}^{S}\mathbb{E}\left[Y_i(T_i,T_i^\star\wedge\tau_i^{(s)})\mid z_i^{(s)}\right],\qquad z_i^{(s)}\mathrel{\overset{\mathrm{iid}}{\sim}}q_\phi(z_i\mid\mathcal{D}_i). \tag*{(40)} E ϕ MC [ Y i ( T i , T i ⋆ ) ∣ D i ] = def S 1 s = 1 ∑ S E [ Y i ( T i , T i ⋆ ∧ τ i ( s ) ) ∣ z i ( s ) ] , z i ( s ) ∼ iid q ϕ ( z i ∣ D i ) . (40)
We can also derive an analytic approximation, analogous to (16) , which uses the same renewal theorem [33 ] and the factorization of q ϕ q_\phi q ϕ . For any T i ⋆ > T i T_i^\star>T_i T i ⋆ > T i it is
E ^ ϕ R T [ Y i ( T i , T i ⋆ ) ∣ D i ] = d e f E ϕ [ λ i ] ⋅ q ϕ ( A L I V E ∣ D i ) ⋅ E ϕ [ T i ⋆ ∧ τ i − T i ∣ τ i > T i ] . (41) \widehat{\mathbb{E}}_\phi^{\mathrm{RT}}\left[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i\right]\mathrel{\overset{\mathrm{def}}{=}}\mathbb{E}_\phi[\lambda_i]\cdot q_\phi(\mathrm{ALIVE}\mid\mathcal{D}_i)\cdot\mathbb{E}_\phi\left[T_i^\star\wedge\tau_i-T_i\mid\tau_i>T_i\right]. \tag*{(41)} E ϕ RT [ Y i ( T i , T i ⋆ ) ∣ D i ] = def E ϕ [ λ i ] ⋅ q ϕ ( ALIVE ∣ D i ) ⋅ E ϕ [ T i ⋆ ∧ τ i − T i ∣ τ i > T i ] . (41)
where all three factors are closed-form functions of the variational parameters, as detailed in § A.2. As with CLV, we can further bound the bias of this approximation using a similar argument as used in Proposition 2.
Proposition 3 (Renewal-theoretic bias bound). For any T i ⋆ > T i T_i^\star>T_i T i ⋆ > T i , the renewal-theoretic estimator in (41) of the variational forecast frequency in (39) satisfies
∣ E ^ ϕ R T [ Y i ( T i , T i ⋆ ) ∣ D i ] − E ϕ [ Y i ( T i , T i ⋆ ) ∣ D i ] ∣ ≤ q ϕ ( A L I V E ∣ D i ) ⋅ E ϕ [ Γ ( 1 + 2 / k i ) Γ ( 1 + 1 / k i ) 2 ] . (42) \left|\widehat{\mathbb{E}}_\phi^{\mathrm{RT}}\left[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i\right]-\mathbb{E}_\phi\left[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i\right]\right|\leq q_\phi(\mathrm{ALIVE}\mid\mathcal{D}_i)\cdot\mathbb{E}_\phi\left[\frac{\Gamma(1+2/k_i)}{\Gamma(1+1/k_i)^2}\right]. \tag*{(42)} E ϕ RT [ Y i ( T i , T i ⋆ ) ∣ D i ] − E ϕ [ Y i ( T i , T i ⋆ ) ∣ D i ] ≤ q ϕ ( ALIVE ∣ D i ) ⋅ E ϕ [ Γ ( 1 + 1/ k i ) 2 Γ ( 1 + 2/ k i ) ] . (42)
where the function c ( k ) = d e f Γ ( 1 + 2 / k ) Γ ( 1 + 1 / k ) 2 c(k)\mathrel{\overset{\mathrm{def}}{=}}\frac{\Gamma(1+2/k)}{\Gamma(1+1/k)^2} c ( k ) = def Γ ( 1 + 1/ k ) 2 Γ ( 1 + 2/ k ) is decreasing in k k k , with c ( ∞ ) = 1 c(\infty)=1 c ( ∞ ) = 1 and c ( 1 ) = 2 c(1)=2 c ( 1 ) = 2 . Thus, for a non-“clumpy” customer—i.e., q ϕ ( k i ≥ 1 ∣ D i ) = 1 q_\phi(k_i\geq1\mid\mathcal{D}_i)=1 q ϕ ( k i ≥ 1 ∣ D i ) = 1 —the error is bounded above by 2 events.
Simulation Studies
We first evaluate the proposed model and variational algorithm with a series of synthetic-data studies. § 6.1 assesses the variational approximation in a simplified setting where the exact posterior is known. § 6.2 and 6.3 then assess recovery of ground-truth θ \theta θ and z 1 : n z_{1:n} z 1 : n , respectively, comparing the variational WTYD approach to Pareto/GGG with MCMC. § 6.4 gives a systematic comparison across different synthetic datasets of the held-out predictive performance of WTYD and Pareto/GGG as a function of wall-clock time.
Recovery of conjugate posteriors in simplified sub-model
As an initial check to see whether an amortized VI approach can yield a good approximation to the posterior, we first generated synthetic data according to the Pareto/NBD model, with λ i ∼ Gamma ( s ( λ ) , r ( λ ) ) \lambda_i \sim\operatorname{Gamma}(s^{(\lambda)}, r^{(\lambda)}) λ i ∼ Gamma ( s ( λ ) , r ( λ ) ) , τ i ∼ Pareto ( s ( τ ) , r ( τ ) ) \tau_i \sim\operatorname{Pareto}(s^{(\tau)}, r^{(\tau)}) τ i ∼ Pareto ( s ( τ ) , r ( τ ) ) and k i = 1 k_i = 1 k i = 1 . Each customer’s transaction times were then sampled on the interval ( 0 , T i ∧ τ i ] (0, T_i \wedge\tau_i] ( 0 , T i ∧ τ i ] according to a Poisson process, with the total time fixed at T i = 1 T_i = 1 T i = 1 for all customers. Conditional on τ i \tau_i τ i , the exact posterior over λ i \lambda_i λ i in this case is known:
p θ ( λ i ∣ D i , τ i , k i = 1 ) = Gamma ( λ i ; SHAPE = s ( λ ) + y i , RATE = r ( λ ) + T i ∧ τ i ) p_\theta(\lambda_i \mid\mathcal{D}_i, \tau_i, k_i = 1) = \operatorname{Gamma}(\lambda_i; \text{SHAPE} = s^{(\lambda)} + y_i, \text{RATE} = r^{(\lambda)} + T_i \wedge\tau_i) p θ ( λ i ∣ D i , τ i , k i = 1 ) = Gamma ( λ i ; SHAPE = s ( λ ) + y i , RATE = r ( λ ) + T i ∧ τ i )
Conditioning on the true τ i \tau_i τ i and k i = 1 k_i = 1 k i = 1 , we then fit the Weibull variational posterior q ϕ ( λ i ∣ D i ) q_\phi(\lambda_i \mid\mathcal{D}_i) q ϕ ( λ i ∣ D i ) using the procedure outlined in § 5, with a recognition network ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) whose input is each customer’s variable-length set of ITTs D i = { Δ t i , T i } \mathcal{D}_i = \{\Delta t_i, T_i\} D i = { Δ t i , T i } .
To evaluate the approximation, we use the KL divergence KL ( q ϕ ( λ i ∣ D i ) ∥ p θ ( λ i ∣ D i , τ i , k i = 1 ) ) \operatorname{KL}(q_\phi(\lambda_i \mid\mathcal{D}_i) \mathbin{\|} p_\theta(\lambda_i \mid\mathcal{D}_i, \tau_i, k_i = 1)) KL ( q ϕ ( λ i ∣ D i ) ∥ p θ ( λ i ∣ D i , τ i , k i = 1 )) between the learned Weibull variational distribution and the true Gamma posterior, which can be computed in closed form. The results are given in Figure 3 where the per-customer KL clusters around a median of 0.056 nats. It is only for the very worst case (0.766 nats) where the variational approximation appears clearly wrong, with approximations at the 95th percentile of KL (0.149 nats) still appearing reasonable visually.
Figure 3. The Weibull variational posterior closely matches the exact Gamma posterior in the closed-form sub-model (k i = 1 k_i = 1 k i = 1 , observed τ i \tau_i τ i ). The per-customer KL from the Weibull q q q to the exact Gamma posterior is small across the population (median 0.056 nats). The variational (WTYD) versus exact posteriors over λ i \lambda_i λ i for a typical customer (median KL), the 95th-percentile customer, and the worst case show that the two are nearly indistinguishable except in the tail, and only for the most extreme customers.
Population hyperparameter recovery
We next ask whether variational WTYD recovers known population-level hyperparameters θ = { s ( τ ) , r ( τ ) , s ( λ ) , r ( λ ) , s ( k ) , r ( k ) } \theta= \{s^{(\tau)}, r^{(\tau)}, s^{(\lambda)}, r^{(\lambda)}, s^{(k)}, r^{(k)}\} θ = { s ( τ ) , r ( τ ) , s ( λ ) , r ( λ ) , s ( k ) , r ( k ) } as accurately as Pareto/GGG using MCMC as implemented by [59 ] . We generated panels of n = 1,000 n = 1{,}000 n = 1 , 000 customers, drawing T i ∼ Uniform ( 1 , 10 ) T_i \sim\operatorname{Uniform}(1, 10) T i ∼ Uniform ( 1 , 10 ) , λ i ∼ Gamma ( 2 , 1 / 5 ) \lambda_i \sim\operatorname{Gamma}(2, 1/5) λ i ∼ Gamma ( 2 , 1/5 ) and τ i ∼ Pareto ( 2 , 5 ) \tau_i \sim\operatorname{Pareto}(2, 5) τ i ∼ Pareto ( 2 , 5 ) , and considering three regimes for regularity:
(“regular”) 95% ETI = ( 1.62 , 3.57 ) , k i ∼ Gamma ( 2.5 , 10 ) , (“clumpy”) 95% ETI = ( 0.63 , 1.02 ) , k i ∼ Gamma ( 5 , 80 ) , (“mixed”) 95% ETI = ( 0.70 , 3.26 ) , k i ∼ Gamma ( 7 , 4 ) \begin{aligned}
\text{(“regular”) 95\% ETI} &= (1.62, 3.57), \qquad k_i \sim\operatorname{Gamma}(2.5, 10), \\
\text{(“clumpy”) 95\% ETI} &= (0.63, 1.02), \qquad k_i \sim\operatorname{Gamma}(5, 80), \\
\text{(“mixed”) 95\% ETI} &= (0.70, 3.26), \qquad k_i \sim\operatorname{Gamma}(7, 4)
\end{aligned} (“regular”) 95% ETI (“clumpy”) 95% ETI (“mixed”) 95% ETI = ( 1.62 , 3.57 ) , k i ∼ Gamma ( 2.5 , 10 ) , = ( 0.63 , 1.02 ) , k i ∼ Gamma ( 5 , 80 ) , = ( 0.70 , 3.26 ) , k i ∼ Gamma ( 7 , 4 )
where each is annotated with its 95% equal-tailed interval (ETI). The “regular” prior’s density is concentrated above 1, the “clumpy” prior below 1, and the “mixed” prior evenly spread.
Because the two models assume different renewal families, we generated each regime twice, with Weibull versus Gamma renewals, holding the true generated z 1 : n \mathbf{z}_{1:n} z 1 : n and θ \boldsymbol{\theta} θ fixed, and fitting each model to the variant matching its own likelihood. We trained WTYD for a fixed budget of 80,000 iterations with no early stopping, and fit Pareto/GGG using a single MCMC chain run for 3,100 iterations, with the first 100 discarded as burn-in and every 10th iteration retained thereafter (300 draws); all remaining settings are given in Table B.1 .
Table B.1. Per-experiment settings for the four simulation studies (§ 6). Abbreviations: Ga = Gamma, Pa = Pareto, U = Uniform, Wei = Weibull. All WTYD fits use P = 25 P = 25 P = 25 power features and seed 42; the first study differs by design (a conditioned sub-model with observed τ \tau τ and k ≡ 1 k \equiv1 k ≡ 1 , hence a smaller network, distinct optimizer settings, and no Pareto/GGG comparison). The k k k -regime Uniform priors are stated in Gamma-shape units; Weibull cases use the matched Weibull-k k k equivalents (Proposition 1); “mixture” denotes an equal mixture of the two Uniforms and a point mass at k = 1 k = 1 k = 1 . Bootstrap refits use (0.01, 0.99); the point-estimate fit’s noise bounds were not separately logged. Reference implementation in PyTorch [5 ] ; Pareto/GGG via BTYDplus in R.
The results are shown in Figure 4 where both models do comparably well in almost all settings. The only exception is recovering the regularity hyperparameters { s ( k ) , r ( k ) } \{s^{(k)}, r^{(k)}\} { s ( k ) , r ( k ) } in the “clumpy” regime, which both models do poorly on, with WTYD’s confidence intervals further failing to cover the true value, unlike the credible intervals of Pareto/GGG. This is consistent with variational approximations typically underestimating the posterior variance [10 ] .
Figure 4. WTYD recovers known population hyperparameters and matches Pareto/GGG in the mixed and regular regimes. Each of the hyperparameters is plotted as estimate/truth, so the vertical line marks perfect recovery. For WTYD, we show both the customer-level bootstrap interval and the cheaper asymptotic interval; the Pareto/GGG reference is its posterior credible interval. Both models cover the generating values, except in the clumpy regime, where WTYD underestimates the regularity concentration.
We used two different methods to obtain confidence intervals for WTYD’s estimates, both computed from the single observed dataset rather than across simulation replicates. The first is a customer-level bootstrap: we resampled the n n n customers with replacement and refit WTYD from scratch, repeating this 100 times. The second is an asymptotic approximation which treats θ ^ \hat{\boldsymbol{\theta}} θ ^ as an M-estimator maximizing the ELBO and estimates its covariance by the inverse Hessian at the fitted optimum. While the bootstrap requires full refits, the asymptotic interval requires only one Hessian evaluation; however, this comes at the cost of approximate intervals which often underestimate those of the bootstrap. See § B for more details.
Customer-specific latent variable recovery
We next assess whether variational WTYD recovers the customer-specific latents z i = { τ i , λ i , k i } \mathbf{z}_i = \{\tau_i,\lambda_i,k_i\} z i = { τ i , λ i , k i } as accurately as Pareto/GGG across a range of data settings. We generate twelve settings defined by the cross-product of renewal family, regularity regime and churn level:
{ Gamma , Weibull } × { “regular”, “clumpy”, “mixed” } × { “low”, “high” } \{\text{Gamma}, \text{Weibull}\} \times\{\text{“regular”, “clumpy”, “mixed”}\} \times\{\text{“low”, “high”}\} { Gamma , Weibull } × { “regular”, “clumpy”, “mixed” } × { “low”, “high” }
where the “high” churn level draws τ i ∼ Pareto ( 2 , 2.5 ) \tau_i \sim\operatorname{Pareto}(2, 2.5) τ i ∼ Pareto ( 2 , 2.5 ) so that roughly 1 / 2 1/2 1/2 of customers drop out within the observation window, and “low” draws τ i ∼ Pareto ( 2 , 8.5 ) \tau_i \sim\operatorname{Pareto}(2, 8.5) τ i ∼ Pareto ( 2 , 8.5 ) so that roughly 1 / 5 1/5 1/5 drop out. Unlike the previous study, the regularity regimes here draw k i k_i k i from bounded ranges: “regular” draws k i ∼ Uniform ( 1.5 , 10 ) k_i \sim\operatorname{Uniform}(1.5, 10) k i ∼ Uniform ( 1.5 , 10 ) , “clumpy” draws k i ∼ Uniform ( 0.5 , 0.95 ) k_i \sim\operatorname{Uniform}(0.5, 0.95) k i ∼ Uniform ( 0.5 , 0.95 ) , and “mixed” draws k i k_i k i from an equal-parts mixture of the “regular” and “clumpy” distributions and a point mass at k i = 1 k_i = 1 k i = 1 . These ranges are stated in units of the Gamma concentration where for the Weibull datasets we use the correspondence map of Proposition 1. All settings share λ i ∼ Gamma ( 4 , 3 / 20 ) \lambda_i \sim\operatorname{Gamma}(4, 3/20) λ i ∼ Gamma ( 4 , 3/20 ) and T i = 1 T_i = 1 T i = 1 . For each setting, we generate six datasets of n = 10,000 n = 10{,}000 n = 10 , 000 , and fit both models to each, so that each model is evaluated under both matched and mismatched renewal families.
Since the scales of the latent variables { λ i , k i , τ i } \{\lambda_i, k_i, \tau_i\} { λ i , k i , τ i } may change depending on the renewal family, we assess recovery using Spearman rank correlation between the ground-truth latent variable and the inferred posterior or variational mean—e.g., for τ i \tau_i τ i :
ρ τ = d e f corr ( rank ( τ 1 : n ) , rank ( τ ^ 1 : n ) ) where τ ^ i = d e f E [ τ i ∣ D i ] \rho_\tau\mathrel{\overset{\mathrm{def}}{=}} \operatorname{corr}\left(\operatorname{rank}(\tau_{1:n}), \operatorname{rank}(\widehat{\tau}_{1:n})\right) \quad\text{where} \quad\widehat{\tau}_i \mathrel{\overset{\mathrm{def}}{=}} \mathbb{E}[\tau_i \mid\mathcal{D}_i] ρ τ = def corr ( rank ( τ 1 : n ) , rank ( τ 1 : n ) ) where τ i = def E [ τ i ∣ D i ]
where this is an analytic expectation E q ϕ ( τ i ∣ D i ) [ τ i ∣ D i ] \mathbb{E}_{q_\phi(\tau_i \mid\mathcal{D}_i)}[\tau_i \mid\mathcal{D}_i] E q ϕ ( τ i ∣ D i ) [ τ i ∣ D i ] for WTYD and an empirical average of MCMC samples 1 S ∑ s = 1 S τ i ( s ) \frac{1}{S}\sum_{s=1}^{S}\tau_i^{(s)} S 1 ∑ s = 1 S τ i ( s ) for Pareto/GGG. While the exact scales may differ, we expect both models to rank order customers correctly for all three variables.
Figure 5 visualizes the results where both models perform comparably. Pooled across settings and random seeds, the median correlation for WTYD versus Pareto/GGG is 0.867 versus 0.881 for λ i \lambda_i λ i , 0.725 versus 0.765 for k i k_i k i , and 0.617 versus 0.655 for τ i \tau_i τ i . We do not expect WTYD to recover latents better than Pareto/GGG, as both models are equally expressive (Proposition 1), and amortized VI introduces approximation error. However, the gap between models is much smaller than the variation across settings, suggesting that the approximation error introduced by VI is minor. For instance, recovery of τ i \tau_i τ i is largely governed by churn level for both models with ρ τ \rho_\tau ρ τ falling from 0.76–0.86 under “high” churn to 0.45–0.48 under “low” churn, where 80% of customers outlive the window and thus reveal little about their lifetimes. Similarly, recovery of k i k_i k i is worst in the “clumpy” and “high”-churn setting—i.e., 0.42 for WTYD, 0.48 for Pareto/GGG—where short, irregular histories contain few ITTs.
Figure 5. WTYD recovers customer-level latents as well as Pareto/GGG. Rank correlation between true and inferred λ i \lambda_i λ i , k i k_i k i , and τ i \tau_i τ i across the synthetic grid. Lifetime recovery degrades when most customers remain alive at T i T_i T i (no dropout signal); regularity recovery is hardest in the clumpy, high-churn corner.
The results of the preceding simulation studies suggest that variational WTYD and Pareto/GGG with MCMC perform comparably, with WTYD incurring minor approximation error. Here we assess the tradeoff that VI offers by measuring the two models’ predictive performance as a function of wall-clock time. We re-use the synthetic datasets created in the previous study, this time further creating future holdout periods on which to assess forecasting performance. We again fit both models to each training dataset, this time tracking an estimate of the posterior predictive density of the held-out data. For WTYD, this is simply q ϕ ( D i ∗ ∣ D i ) q_\phi(\mathcal{D}_i^* \mid\mathcal{D}_i) q ϕ ( D i ∗ ∣ D i ) as estimated by (38) at each iteration, while for Pareto/GGG we use the accumulated set of MCMC samples at each iteration to estimate p θ ( D i ∗ ∣ D i ) p_\theta(\mathcal{D}_i^* \mid\mathcal{D}_i) p θ ( D i ∗ ∣ D i ) by averaging the analogous per-draw held-out likelihood; see § A.2 for more details. When reporting wall-clock time, we do not include the time spent on computing these quantities.
Figure 6 visualizes the results. We see that at this size n = 10,000 n=10{,}000 n = 10 , 000 , WTYD typically converges before the MCMC chain for Pareto/GGG has burned in. At convergence, the performance of the two models is broadly comparable with small differences being consistent with renewal family mismatch as WTYD typically converges to better performance on the Weibull datasets and Pareto/GGG better on the Gamma datasets.
Figure 6. WTYD obtains comparable predictive performance at a fraction of Pareto/GGG’s computational cost. Each curve shows the held-out posterior predictive density for WTYD (blue) and Pareto/GGG (red) over wall-clock training time on one of the six synthetic datasets we generated for each setting. We subtract from both curves the final value of Pareto/GGG, so that all red curves end at zero, and blue curves above-vs-below zero denote WTYD obtaining better-vs-worse predictive performance than Pareto/GGG at convergence.
Case Studies
This section presents two case studies applying WTYD to predict and explore structure in two large-scale real-world datasets of 1) online retail transactions at an anonymous subscription-based firm whose CRM data we obtained from the company Ocurate and 2) political donations during the run-up to the 2020 U.S. Presidential Election as recorded by the U.S. Federal Election Commission (FEC). These case studies both highlight the need for scalable non-Poisson models that can account for heterogeneity in regularity.
Case Study 1: 5 Million Subscription-Based Online Retail Customers
Much of the literature on BTYD models focuses on Poisson models for the non-contractual setting. However, such models often fail to appropriately capture regularity in the contractual setting, which has been the primary motivation for the few non-Poisson models that have thus far been introduced [e.g., [40 ] , [15 ] , [67 ] , [60 ] , [64 ] ]. This case study centers around a subscription-based online retail firm, where many customers are contractually prompted to transact monthly or bimonthly, but can skip or delay transactions, and can otherwise transact at will, yielding ITT histories that exhibit regularity but are not trivially predictable.
Data and preprocessing. We obtained anonymized CRM data from the marketing analytics company Ocurate4 . The dataset records 20.8 million transactions from 5.1 million unique customers over the course of five years from January 1, 2019 to February 14, 2024. As is typical, there is a high churn rate with a heavy tail of loyal customers, with 40% of customers never making any repeat purchases, 25% making five or more, and 2% making 20 or more.
We processed the data into the standard “customer’s journey” representation (Figure 1 ), shifting and rescaling each customer’s history so that all customers are “born” at t = 0 t = 0 t = 0 and are observed for different total times T i T_i T i . For the purpose of predictive evaluation, we also created training-holdout splits with two-thirds of each customer’s true observation window designated for training D i \mathcal{D}_i D i and the remaining held-out D i ∗ \mathcal{D}_i^* D i ∗ .
Predictive evaluation. We first repeat the same kind of forecasting experiments reported in § 6.4, this time on real-world data, comparing the performance of variational WTYD to Pareto/GGG as a function of wall-clock time. Since MCMC with Pareto/GGG does not scale computationally to the size of the full dataset, we selected random subsets of n ∈ { 1000 , 2500 , 10000 } n \in\{1000, 2500, 10000\} n ∈ { 1000 , 2500 , 10000 } customers, splitting each into a train-holdout split as described above. Both models were fit to each dataset using the same settings as in § 6.4, computing the posterior predictive density of held-out data over training time. In addition, we also calculated the mean-squared error (MSE) of the true versus forecasted future frequency 1 n ∑ i = 1 n ( y i ∗ − E [ y i ∗ ∣ D i ] ) 2 \frac{1}{n}\sum_{i=1}^{n}(y_i^*-\mathbb{E}[y_i^* \mid\mathcal{D}_i])^2 n 1 ∑ i = 1 n ( y i ∗ − E [ y i ∗ ∣ D i ] ) 2 using the variational approximation of the forecasted future frequency E [ y i ∗ ∣ D i ] \mathbb{E}[y_i^* \mid\mathcal{D}_i] E [ y i ∗ ∣ D i ] given in (40) for WTYD, and using Monte Carlo averages for Pareto/GGG—i.e., by simulating a Gamma renewal process at each posterior sample of z i z_i z i . An estimator caveat affecting roughly 1% of customers per subset in Pareto/GGG’s held-out means is noted in § B.
Figure 7 visualizes the posterior predictive and mean-squared error results for all three dataset sizes as a function of training time. For the smallest size of n = 1,000 n = 1{,}000 n = 1 , 000 , both models converge comparably quickly to very similar values of both held-out posterior predictive density and mean-squared error. However, as the size increases to n = 10,000 n = 10{,}000 n = 10 , 000 , WTYD converges before Pareto/GGG has even burned in, just like in § 6.4. Moreover, at convergence, both models exhibit comparable forecasting performance as measured by posterior predictive density, but with WTYD further exhibiting substantially lower mean-squared error in its approximate frequency forecasts.
Figure 7. WTYD matches Pareto/GGG’s forecast performance at a fraction of its wall-clock cost on real data. Following the conventions of Figure 6 , each curve shows a model’s performance relative to Pareto/GGG’s final value over wall-clock training time, here on random retail subsets of size n ∈ { 1000 , 2500 , 10000 } n \in\{1000, 2500, 10000\} n ∈ { 1000 , 2500 , 10000 } (columns). The top row reports the held-out posterior predictive density and the bottom row the mean-squared error of the forecasted future frequency, for which blue curves below zero denote WTYD forecasting more accurately than Pareto/GGG at convergence.
We note that while Pareto/GGG’s performance curve is already dominated by WTYD at this size, n = 10,000 n=10{,}000 n = 10 , 000 still represents only 0.2% of all 5 million customers. While we estimate fitting Pareto/GGG to the whole dataset would take around 3-4 days, WTYD is fit in under 8 minutes.
Exploratory analysis. Fitting WTYD to the full dataset, we then explored the inferred latent structure, as given by the variational means of z i = d e f { k i , λ i , τ i } z_i \mathrel{\overset{\mathrm{def}}{=}}\{k_i,\lambda_i,\tau_i\} z i = def { k i , λ i , τ i } . The first figure in Figure 8 shows the distribution of inferred regularity k i k_i k i , with a large peak around k i = 2.5 k_i=2.5 k i = 2.5 and substantial mass extending even beyond k i > 5 k_i>5 k i > 5 . Only a very small fraction of 2.5% customers are “clumpy” or “random” with k i ≤ 1 k_i\leq1 k i ≤ 1 , as would be expected in a contractual setting.
Figure 8. Left: The distribution of inferred k i k_i k i , where the dashed line marks k i = 1 k_i = 1 k i = 1 , shows most customers are “regular”. Middle: The joint distribution of inferred k i k_i k i and λ i \lambda_i λ i with a LOWESS summary shows that more regular customers are also more frequent. Right: The joint distribution of k i k_i k i against the empirical coefficient of variation of ITTs C V ^ i \widehat{\mathrm{CV}}_i CV i then shows high correspondence with the model’s predicted relationship, suggesting that inferred relationship between regularity and frequency is not driven by model or approximate inference artifacts.
Although nothing in the model assumes this, we also find that more regular customers purchase more frequently. The middle plot of Figure 8 shows the joint distribution of the inferred k i k_i k i and λ i \lambda_i λ i where increasing k i k_i k i is highly predictive of larger λ i \lambda_i λ i for k i < 4 k_i<4 k i < 4 with the average transaction rate among “random” (k i = 1 k_i=1 k i = 1 ) customers being almost half that of more regular ones with k i = 3 k_i=3 k i = 3 . This relationship tapers off after k i > 4 k_i>4 k i > 4 with transaction rates saturating at around λ i = 20 \lambda_i=20 λ i = 20 . One might still wonder whether this inferred relationship is an artifact of approximate inference, and that larger k i k_i k i is partly reflecting larger frequencies rather than simply the regularity of transaction timings. To examine this, we visualize in the rightmost plot of Figure 8 the joint distribution of k i k_i k i with the empirical coefficient of variation of the customer’s ITTs—i.e., C V ^ i = d e f s i / m i \widehat{\mathrm{CV}}_i \mathrel{\overset{\mathrm{def}}{=}}s_i/m_i CV i = def s i / m i , where m i m_i m i and s i s_i s i are the sample mean and standard deviation of customer i i i ’s observed ITTs Δ t i , j \Delta t_{i,j} Δ t i , j , defined for customers with y i ≥ 2 y_i\geq2 y i ≥ 2 . The empirical relationship is just as predicted under the model, with larger k i k_i k i strongly associated with smaller C V ^ i \widehat{\mathrm{CV}}_i CV i . We also overlay the theoretical relationship under the model, as given by C V i = d e f Γ ( 1 + 2 / k i ) Γ ( 1 + 1 / k i ) 2 − 1 \mathrm{CV}_i \mathrel{\overset{\mathrm{def}}{=}}\sqrt{\frac{\Gamma(1+2/k_i)}{\Gamma(1+1/k_i)^2}-1} CV i = def Γ ( 1 + 1/ k i ) 2 Γ ( 1 + 2/ k i ) − 1 , which aligns with a LOWESS summary of the empirical relationship.
We also explore the inferred structure of churn and expected future purchasing. The leftmost panel of Figure 9 visualizes average q ϕ ( A L I V E ∣ D i ) q_{\phi}(\mathrm{ALIVE}\mid\mathcal{D}_i) q ϕ ( ALIVE ∣ D i ) among subsets of customers with similar values of λ i \lambda_i λ i and k i k_i k i . We see that only a small segment of the customer base has q ϕ ( A L I V E ∣ D i ) > 0.5 q_{\phi}(\mathrm{ALIVE}\mid\mathcal{D}_i)>0.5 q ϕ ( ALIVE ∣ D i ) > 0.5 . This is concentrated in two regions: low frequency and low regularity (“low engagement”), where τ i \tau_i τ i is most difficult to infer, and high frequency and high regularity (“high-high”), which comprises the firm’s heaviest or most loyal customers.
Figure 9. Inferred churn and expected future purchasing on the full customer base. Left: The average of q ϕ ( A L I V E ∣ D i ) q_{\phi}(\mathrm{ALIVE}\mid\mathcal{D}_i) q ϕ ( ALIVE ∣ D i ) over the ( k i , λ i ) (k_i,\lambda_i) ( k i , λ i ) plane shows that only a small segment of the base is inferred to be still alive, concentrated among “low-low” and “high-high” customers. Middle: The expected number of purchases over the next 24 months shows that expected future purchasing concentrates among “medium-medium” customers. Right: The observed frequencies of customers observed for at least four years also concentrate around “medium-medium” as well as “high-high” customers.
We then compare this in the middle panel to the expected number of purchases over the next 24 months, computed using the estimator (41) . Here the bulk of expected future purchases are concentrated among customers with medium frequency and medium regularity (“medium-medium”), who are less likely to be alive than low-low customers but transact faster when they are. We compare these forecasts in the rightmost panel to the observed empirical frequencies among the roughly 300,000 customers that were acquired early enough to be observed for at least four years. We see that the bulk of observed frequencies is also concentrated among the medium-medium customers, as well as the high-high, as expected.
Case Study 2: 4 Million Donors in the 2020 US General Elections
This case study focuses on individual political contributions 2019–2020 during the run-up to the US General elections. In this setting, there is a mix of contractual and non-contractual contributions. Moreover, funding campaigns, deadlines, and other structural features modulate the timing of contributions, inducing a range of “clumpy” and “regular” timing patterns, as we will show. Political donation dynamics have previously been shown to respond to filing deadlines, debates, and other salient events [51 , 13 , 77 ] , and non-political charitable giving has been studied as a canonical application of BTYD models [58 , 31 ] including Pareto/GGG [60 , 79 ] .
Data and preprocessing. We obtained data from the public bulk files of the U.S. Federal Election Commission (FEC) [78 ] , which record every itemized contribution by an individual to a federal committee and include the conduit records through which the partisan companies ActBlue and WinRed route small-dollar gifts. We filtered to individual donors with valid transaction codes, positive amounts, and gifts inside the cycle window from January 1, 2019 to election day, November 3, 2020, and further collapse multiple gifts on the same day into “donation-days”. Since the FEC assigns no persistent donor identifier, we identify donors by hashed name and five-digit ZIP code. This results in a dataset of 24.5 million donation-days from 4.1 million unique donors over a window of 673 days. The distribution of donor activity closely mirrors the retail panel in the previous case study with 40% of donors giving on only a single day, 30% giving on five or more days, and only 6% on twenty or more.
Exploratory analysis. Fitting WTYD to the full dataset, we again explored the inferred latent structure. Figure 10 shows the distribution of inferred regularity k i k_i k i where unlike in the retail panel where regularity dominated, the donor base here is bimodal, with one mode near k i = 1 k_i = 1 k i = 1 and another near k i = 1.75 k_i = 1.75 k i = 1.75 . There is again a heavy right tail of highly “regular” donors and, unlike in the retail setting, a sizeable number of “clumpy” donors, accounting for nearly 30% of the donor base. As in the previous case study, the inferred k i k_i k i closely tracks the empirical coefficient of variation C V ^ i \widehat{\mathrm{CV}}_i CV i , with rank correlation ρ = − 0.79 \rho= -0.79 ρ = − 0.79 , suggesting that the split reflects real differences in donors’ timing patterns rather than artifacts of approximate inference. In the middle plot of Figure 10 we also show the distribution of ITTs among the most regular decile of donors where we see that 82% of gaps fall at exactly 7, 14, 30, or 31 days, as would be expected under automated recurring contributions.
Figure 10. Inferred regularity structure on the full donor base. Left: A bimodal distribution of inferred k i k_i k i . Middle: The distribution of ITTs among the most and least “regular” decile of donors. Right: Share of each week’s donors that are “clumpy”, smoothed by a 3-week moving average, with 95% confidence intervals.
The rightmost plot of Figure 10 then shows when in the cycle the most “clumpy” donors were giving. We assign each donor to the week containing the median date of their gifts, and then plot the share of each week’s donors that are “clumpy”, doing so separately for ActBlue, WinRed, and direct donors. Among ActBlue donors, this share spikes to 43% around Giving Tuesday and to 69% around Super Tuesday, the latter of which coincided with the largest single day of the 2020 Democratic presidential primary. Among WinRed donors, the share of “clumpy” instead spikes to around 60% in the week spanning the end of the second fundraising quarter on June 30, a reporting deadline that campaigns target with a burst of appeals, and July 4th, a natural focal point for Republican fundraising. All three groups then also exhibit a spike in “clumpiness” just before Election Day, during which campaigns often concentrate appeals.
Incorporating covariates for “cold-start” prediction. We can associate with each donor a set of static covariates x i \mathbf{x}_i x i —i.e., party, US state, and first gift amount—where party is inferred from whether the donor gave through ActBlue ( p a r t y i = D E M \mathrm{party}_i = \mathrm{DEM} party i = DEM ), WinRed ( p a r t y i = R E P U B \mathrm{party}_i = \mathrm{REPUB} party i = REPUB ), or directly ( p a r t y i = N O N E \mathrm{party}_i = \mathrm{NONE} party i = NONE )5 . Such covariates may be useful in improving “cold start” prediction for newly observed donors. As noted in § 4, WTYD’s gradient-based inference algorithm makes incorporating covariates into the prior hyperparameters easy. We define the map to be log-linear—i.e., θ ( x i ) ≔ exp ( W x i ) \boldsymbol{\theta}(\mathbf{x}_i) \coloneqq\exp(W\mathbf{x}_i) θ ( x i ) : = exp ( W x i ) —where the exponential applies elementwise and x i \mathbf{x}_i x i includes a constant. The model is then fit exactly as before, with gradients flowing through θ ( x i ) \boldsymbol{\theta}(\mathbf{x}_i) θ ( x i ) to the coefficients W W W .
To see whether covariates improve “cold start” prediction, we constructed train-test splits at the donor level, holding out all of the ITTs for 200,000 test donors and then fitting both the original and various covariate-conditional WTYD models to the remaining donor pool. We then evaluated the prior-predictive (i.e., marginal likelihood) density of the held-out donors’ ITTs under the fitted models, along with the MSE of their true versus expected frequencies, as approximated by (16) .
Figure 11 visualizes the improvement in “cold start” prediction of various covariate-conditional models over the original. While conditioning on the US state of the donor yielded no improvement, both party and first gift size improve predictions independently. Moreover, we fit two further variants that localize covariates to specific parts of the model. In one case, the churn hyperparameters { s ( τ ) , r ( τ ) } \{s^{(\tau)}, r^{(\tau)}\} { s ( τ ) , r ( τ ) } are kept global and not covariate-conditioned while in the other the frequency hyperparameters { s ( λ ) , r ( λ ) } \{s^{(\lambda)}, r^{(\lambda)}\} { s ( λ ) , r ( λ ) } are kept global. While fixing the churn prior to be global has no effect, fixing the frequency prior substantially reduces the gain, suggesting that a donor’s first gift and party is highly informative about how frequently they will donate when active, but carries little information about how long they will remain active.
Figure 11. Cold-start prediction of held-out donors, where positive values favor the covariate prior. Left: The gain in held-out log-likelihood of each covariate specification over the global prior. Middle: The analogous reduction in squared error of the held-out frequency forecast. Right: The gain log-likelihood gain by each donor’s eventual number of donation-days, separately by conduit.
The gains in performance are unevenly distributed across the donor pool, with the best-performing covariate-conditional model only improving in held-out log-likelihood for about two-thirds of test donors who are concentrated among more frequent ActBlue donors, as shown in the rightmost panel of Figure 11 . One explanation for why the impact of covariates is so uneven is that ActBlue donors are more heterogeneous, spanning the full range of giving styles, from large single-time gifts to sustained recurring schedules. For instance, 44% of held-out ActBlue donors give on four or more donation-days compared to only 9.5% of WinRed donors. First gift size is thus useful in predicting the subtype of ActBlue donor, with smaller first gifts suggesting a sustained recurring donor, while such information does not improve “cold start” prediction for the more homogeneous WinRed donor pool.
Discussion
We found that heterogeneous regularity was indeed important to account for in two real-world settings of subscription-based retail and political donations, consistent with work over the last several decades that details departures from Poisson timing in transaction data [15 , 36 , 92 , 59 ] . There are few existing non-Poisson BTYD models that account for such heterogeneity, and these models do not scale computationally to the size of modern datasets. Our approach breaks ground on this problem by pairing a BTYD model based on Weibull renewals with amortized variational inference. The proposed approach fits a dataset of 5 million customers in 8 minutes compared to an estimated 3-4 days that would have been required by the existing state-of-the-art, and does so without any appreciable loss in predictive performance or model interpretability.
Alongside a recent preprint by [56 ] , this paper is among the first to adapt amortized VI to BTYDs. [56 ] introduce the Customer Lifetime Variational Autoencoder (CLVAE) which adapts amortized VI to fit Pareto/NBD. Adapting amortized VI beyond Pareto/NBD to non-Poisson models, the challenge that our paper addresses, is hampered by the form of the likelihood in more complex models, and specifically the inability to take gradients through it via automatic differentiation. In general, for any renewal family f i f_i f i , a BTYD likelihood takes the following form:
p ( D i ∣ z i ) = S i ( ( T i ∧ τ i ) − t i , y i ) ∏ j = 1 y i f i ( Δ t i , j ) , (43) p(\mathcal{D}_i \mid z_i) = S_i\left((T_i \wedge\tau_i) - t_{i,y_i}\right) \prod_{j=1}^{y_i} f_i(\Delta t_{i,j}), \tag*{(43)} p ( D i ∣ z i ) = S i ( ( T i ∧ τ i ) − t i , y i ) j = 1 ∏ y i f i ( Δ t i , j ) , (43)
where S i ( Δ t ) : = ∫ Δ t ∞ f i ( u ) d u S_i(\Delta t) \mathrel{\mathop:}= \int_{\Delta t}^{\infty} f_i(u)\,\mathrm{d}u S i ( Δ t ) : = ∫ Δ t ∞ f i ( u ) d u is the survival function —i.e., the probability of an ITT greater than Δ t \Delta t Δ t . For the Gamma renewals, S i ( Δ t ) = Γ ( k i , k i λ i Δ t ) Γ ( k i ) S_i(\Delta t) = \frac{\Gamma(k_i,k_i\lambda_i\Delta t)}{\Gamma(k_i)} S i ( Δ t ) = Γ ( k i ) Γ ( k i , k i λ i Δ t ) , which involves the upper incomplete gamma function Γ ( s , x ) : = ∫ x ∞ u s − 1 e − u d u \Gamma(s,x) \mathrel{\mathop:}= \int_x^{\infty} u^{s-1}e^{-u}\,\mathrm{d}u Γ ( s , x ) : = ∫ x ∞ u s − 1 e − u d u , which is not supported in most automatic differentiation frameworks, like PYTORCH. Replacing Gamma renewals with Weibull as we do then instead yields S i ( Δ t ) = exp ( − ( r i Δ t ) k i ) S_i(\Delta t) = \exp(-(r_i\Delta t)^{k_i}) S i ( Δ t ) = exp ( − ( r i Δ t ) k i ) and with it the likelihood in (14) , which involves only elementary functions beyond the gamma function, which all automatic differentiation frameworks support.
In changing to a Weibull renewal model, we took care to adopt a mean–concentration parameterization which admits latent variables λ i \lambda_i λ i and k i k_i k i that retain their qualitative interpretation under Pareto/GGG, and moreover have a formal correspondence, as detailed in Proposition 1. The two models are thus only very subtly different. In § 6.4, each model attained slightly better held-out performance when the synthetic data were generated under its own renewal family, but with the gap between the two always being small. In § 7.1, WTYD seemed to fit the online retail data slightly better than Pareto/GGG. However, this was again with a small gap and moreover with there being no a priori reason to expect one model to fit better than the other. The main benefit is computational, with WTYD requiring orders of magnitude less time to fit, and with the additional benefit of the ease at which covariates can be incorporated, as illustrated in § 7.2.
A major tradeoff of the proposed approach is its lack of a sufficient statistic representation, which thus requires the recognition network to consume each customer’s variable-length sequence of ITTs Δ t i \Delta t_i Δ t i . We adopt a simple approach to this, mapping each sequence to a fixed-length vector of prescribed “pseudo-sufficient” statistics, as in (24) . A more sophisticated approach that future follow-up work could consider would define the recognition network to be a recurrent network or transformer, among other architectures tailored to sequences, drawing upon the emerging literature on neural point processes [55 , 71 , 94 , 93 ] .
Future work might also seek to enrich the lifetime process. WTYD retains the simple Pareto lifetime process of its predecessors, but nothing in the inference procedure depends on this choice, as the lifetime density, like the renewal survival function in (43) , need only be automatically differentiable. One avenue of future work that amortized VI thus enables is to parameterize the survival function directly as a deep neural network, drawing upon recent work on neural survival analysis [e.g., [86 , 44 ] ]. One could also generalize beyond simple alive-versus-dead states via Markov-modulated renewal processes [[61 ] ] such that customers transition through multiple transacting modes, drawing upon BTYD models inspired by hidden Markov models and other discrete-time processes [e.g., [3 , 58 , 6 ] ]. Amortized VI enables exploration of many such extensions as the only requirement it imposes is that the complete-data likelihood be automatically differentiable; any such model satisfying this can then be conceivably fit to large-scale datasets with covariates, as demonstrated here.
Acknowledgements
AS helped develop a precursor to this work called “Py(Torch) ‘Til You Die” while he was a research scientist 2020–2022 at Ocurate (since acquired by Fenix Commerce), during which time he benefitted greatly from close collaboration and support from Gio Fernandes, Harlan Holt, Vivian Ellis, Sabrina Margetic, David Rothschild, and Tobias Konitzer, among others at Ocurate, as well as from useful feedback from Eva Ascarza and Peter Fader.
References
M. Abe. “Counting your customers” one by one: A hierarchical Bayes extension to the Pareto/NBD model. Marketing Science , 28(3):541–553, 2009.
G. M. Allenby and P. E. Rossi. Marketing models of consumer heterogeneity. Journal of Econometrics , 89(1–2):57–78, 1998.
G. M. Allenby, R. P. Leone, and L. Jen. A dynamic model of purchase timing with application to direct marketing. Journal of the American Statistical Association , 94(446):365–374, 1999.
L. Ambrogioni, U. Güçlü, J. Berezutskaya, E. van den Borne, Y. Güçlütürk, M. Hinne, E. Maris, and M. van Gerven. Forward amortized inference for likelihood-free variational marginalization. In AISTATS , pages 777–786, 2019.
J. Ansel et al. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In ASPLOS , 2024.
E. Ascarza, O. Netzer, and B. G. S. Hardie. Some customers would rather leave without saying goodbye. Marketing Science , 37(1):54–77, 2018.
L. J. Bain and M. Engelhardt. Probability of correct selection of Weibull versus gamma based on livelihood ratio. Communications in statistics-theory and methods , 9(4):375–381, 1980.
E. P. Batislam, M. Denizel, and A. Filiztekin. Empirical validation and comparison of models for customer base analysis. International Journal of Research in Marketing , 24(3):201–209, 2007.
C. Bauckhage. Computing the Kullback-Leibler divergence between two generalized gamma distributions. arXiv preprint arXiv:1401.6853 , 2014.
Proofs and derivations
Model properties and prior key quantities
Proof of Proposition 1 (correspondence of regularity index). In the mean–concentration parameterization the reverse KL from the Weibull (mean 1 / λ W 1/\lambda_W 1/ λ W , concentration k W k_W k W ) to the Gamma (mean 1 / λ G 1/\lambda_G 1/ λ G , concentration k G k_G k G ) is
KL = − γ − 1 + log k W − k G log ( k G λ G ) + log Γ ( k G ) + k G λ G λ W + k G log ( λ W Γ ( 1 + 1 k W ) ) + k G γ k W , (44) \operatorname{KL} = -\gamma- 1 + \log k_W - k_G \log(k_G \lambda_G) + \log\Gamma(k_G) + \frac{k_G \lambda_G}{\lambda_W} + k_G \log\left(\lambda_W \Gamma\left(1 + \frac{1}{k_W}\right)\right) + \frac{k_G \gamma}{k_W}, \tag*{(44)} KL = − γ − 1 + log k W − k G log ( k G λ G ) + log Γ ( k G ) + λ W k G λ G + k G log ( λ W Γ ( 1 + k W 1 ) ) + k W k G γ , (44)
where we used that the Weibull and Gamma rates in this parameterization are λ W Γ ( 1 + 1 / k W ) \lambda_W \Gamma(1 + 1/k_W) λ W Γ ( 1 + 1/ k W ) and k G λ G k_G\lambda_G k G λ G . Setting ∂ / ∂ λ W = 0 \partial/\partial\lambda_W = 0 ∂ / ∂ λ W = 0 gives − k G λ G / λ W 2 + k G / λ W = 0 -k_G\lambda_G/\lambda_W^2 + k_G/\lambda_W = 0 − k G λ G / λ W 2 + k G / λ W = 0 , so λ W ∗ = λ G \lambda_W^* = \lambda_G λ W ∗ = λ G ; the second derivative k G / λ G 2 > 0 k_G/\lambda_G^2 > 0 k G / λ G 2 > 0 there confirms a minimum. Setting ∂ / ∂ k W = 0 \partial/\partial k_W = 0 ∂ / ∂ k W = 0 gives
k G = k W ∗ ( γ + ψ ( 1 + 1 k W ∗ ) ) − 1 , k_G = k_W^*\left(\gamma+ \psi\left(1 + \frac{1}{k_W^*}\right)\right)^{-1}, k G = k W ∗ ( γ + ψ ( 1 + k W ∗ 1 ) ) − 1 ,
which is (13) . Since ψ \psi ψ is increasing and 1 + 1 / k W 1 + 1/k_W 1 + 1/ k W is decreasing in k W k_W k W , the denominator γ + ψ ( 1 + 1 / k W ) \gamma+ \psi(1 + 1/k_W) γ + ψ ( 1 + 1/ k W ) is decreasing, so the map k W ∗ ↦ k G k_W^* \mapsto k_G k W ∗ ↦ k G is strictly increasing (as is its inverse k G ↦ k W ∗ k_G \mapsto k_W^* k G ↦ k W ∗ ); at k W ∗ = 1 k_W^* = 1 k W ∗ = 1 , ψ ( 2 ) = 1 − γ \psi(2) = 1 - \gamma ψ ( 2 ) = 1 − γ gives k G = 1 k_G = 1 k G = 1 . Hence k W ∗ k_W^* k W ∗ and k G k_G k G always lie on the same side of 1. Uniqueness follows because the stationary point is unique—λ W ∗ = λ G \lambda_W^* = \lambda_G λ W ∗ = λ G , with a unique k W ∗ k_W^* k W ∗ for each k G k_G k G by strict monotonicity—while the KL in (44) diverges as k W k_W k W or λ W \lambda_W λ W tends to 0 or ∞ \infty ∞ , so this stationary point is the global minimum.
Derivation of (14) (individual-level likelihood). While alive, the ITTs are i.i.d. Weibull( k i , r i ) (k_i,r_i) ( k i , r i ) with r i = λ i Γ ( 1 + 1 / k i ) r_i = \lambda_i\Gamma(1 + 1/k_i) r i = λ i Γ ( 1 + 1/ k i ) , density f ( w ) = k i r i k i w k i − 1 e − r i k i w k i f(w) = k_i r_i^{k_i} w^{k_i-1} e^{-r_i^{k_i}w^{k_i}} f ( w ) = k i r i k i w k i − 1 e − r i k i w k i and survival S ( w ) = e − r i k i w k i S(w) = e^{-r_i^{k_i}w^{k_i}} S ( w ) = e − r i k i w k i . The history D i \mathcal{D}_i D i consists of the y i y_i y i observed ITTs followed by a censored gap from the last transaction t i , y i t_{i,y_i} t i , y i to the end of observation T i ∧ τ i T_i \wedge\tau_i T i ∧ τ i , during which no transaction occurs, so
p ( D i ∣ z i ) = [ ∏ j = 1 y i f ( Δ t i , j ) ] S ( T i ∧ τ i − t i , y i ) . p(\mathcal{D}_i \mid z_i) = \left[\prod_{j=1}^{y_i} f(\Delta t_{i,j})\right] S\left(T_i \wedge\tau_i - t_{i,y_i}\right). p ( D i ∣ z i ) = [ j = 1 ∏ y i f ( Δ t i , j ) ] S ( T i ∧ τ i − t i , y i ) .
Substituting f f f and S S S and collecting the exponentials yields (14) ; the indicator 1 ( y i > 0 ) \mathbb{1}(y_i > 0) 1 ( y i > 0 ) drops the transaction-density factor for never-purchasing customers, leaving the survival term alone.
Derivation of (16) (expected CLV estimator). The mean ITT is μ i = E [ Δ t i , j ] = Γ ( 1 + 1 / k i ) / r i = 1 / λ i \mu_i = \mathbb{E}[\Delta t_{i,j}] = \Gamma(1 + 1/k_i)/r_i = 1/\lambda_i μ i = E [ Δ t i , j ] = Γ ( 1 + 1/ k i ) / r i = 1/ λ i , so the long-run rate of the renewal process is exactly λ i \lambda_i λ i . By the elementary renewal theorem E [ Y i ( t ) ∣ z i ] → t / μ i = λ i t \mathbb{E}[Y_i(t) \mid z_i] \to t/\mu_i = \lambda_i t E [ Y i ( t ) ∣ z i ] → t / μ i = λ i t , hence E [ Y i ( τ i ) ∣ z i ] ≈ λ i τ i \mathbb{E}[Y_i(\tau_i) \mid z_i] \approx\lambda_i\tau_i E [ Y i ( τ i ) ∣ z i ] ≈ λ i τ i for large τ i \tau_i τ i . Taking the prior expectation and using prior independence of λ i \lambda_i λ i and τ i \tau_i τ i ,
E θ [ Y i ( ∞ ) ] ≈ E θ [ λ i τ i ] = E θ [ λ i ] E θ [ τ i ] = s ( λ ) r ( λ ) ⋅ r ( τ ) s ( τ ) − 1 , \mathbb{E}_{\theta}[Y_i(\infty)] \approx\mathbb{E}_{\theta}[\lambda_i\tau_i] = \mathbb{E}_{\theta}[\lambda_i]\mathbb{E}_{\theta}[\tau_i] = \frac{s^{(\lambda)}}{r^{(\lambda)}} \cdot\frac{r^{(\tau)}}{s^{(\tau)}-1}, E θ [ Y i ( ∞ )] ≈ E θ [ λ i τ i ] = E θ [ λ i ] E θ [ τ i ] = r ( λ ) s ( λ ) ⋅ s ( τ ) − 1 r ( τ ) ,
the product of the Gamma( s ( λ ) , r ( λ ) ) (s^{(\lambda)},r^{(\lambda)}) ( s ( λ ) , r ( λ ) ) and Pareto( s ( τ ) , r ( τ ) ) (s^{(\tau)},r^{(\tau)}) ( s ( τ ) , r ( τ ) ) prior means, which is (16) . (Throughout, Pareto( S H A P E = s , S C A L E = r ) (\mathrm{SHAPE}=s,\mathrm{SCALE}=r) ( SHAPE = s , SCALE = r ) denotes the Pareto type-II, or Lomax, distribution with density s r ( 1 + τ / r ) − ( s + 1 ) \frac{s}{r}(1+\tau/r)^{-(s+1)} r s ( 1 + τ / r ) − ( s + 1 ) on τ > 0 \tau> 0 τ > 0 , whose mean r s − 1 \frac{r}{s-1} s − 1 r requires s > 1 s > 1 s > 1 .)
Proof of eq:2 (error bound of CLV estimator). Lorden’s upper bound on the renewal function [49 ] , combined with the elementary lower bound E [ Y i ( t ) ∣ z i ] ≥ t / μ i − 1 \mathbb{E}[Y_i(t) \mid z_i] \ge t/\mu_i - 1 E [ Y i ( t ) ∣ z i ] ≥ t / μ i − 1 , gives ∣ E [ Y i ( t ) ∣ z i ] − t / μ i ∣ ≤ E [ Δ t i 2 ∣ z i ] / μ i 2 \left|\mathbb{E}[Y_i(t) \mid z_i] - t/\mu_i\right| \le\mathbb{E}[\Delta t_i^2 \mid z_i]/\mu_i^2 ∣ E [ Y i ( t ) ∣ z i ] − t / μ i ∣ ≤ E [ Δ t i 2 ∣ z i ] / μ i 2 for all t t t . For the Weibull ITT, E [ Δ t i 2 ∣ z i ] / μ i 2 = Γ ( 1 + 2 / k i ) / Γ ( 1 + 1 / k i ) 2 = : c ( k i ) \mathbb{E}[\Delta t_i^2 \mid z_i]/\mu_i^2 = \Gamma(1 + 2/k_i)/\Gamma(1 + 1/k_i)^2 =: c(k_i) E [ Δ t i 2 ∣ z i ] / μ i 2 = Γ ( 1 + 2/ k i ) /Γ ( 1 + 1/ k i ) 2 =: c ( k i ) , which is scale-free (independent of r i r_i r i ). Evaluating at t = τ i t = \tau_i t = τ i , where t / μ i = λ i τ i t/\mu_i = \lambda_i\tau_i t / μ i = λ i τ i , and taking the prior expectation gives
∣ E θ [ Y i ( ∞ ) ] − E ^ θ R T [ Y i ( ∞ ) ] ∣ ≤ E p θ ( k i ) [ c ( k i ) ] , \left|\mathbb{E}_{\theta}[Y_i(\infty)] - \widehat{\mathbb{E}}_{\theta}^{\mathrm{RT}}[Y_i(\infty)]\right| \le\mathbb{E}_{p_{\theta}(k_i)}[c(k_i)], E θ [ Y i ( ∞ )] − E θ RT [ Y i ( ∞ )] ≤ E p θ ( k i ) [ c ( k i )] ,
which is (17) ; the bound depends only on { s ( k ) , r ( k ) } \{s^{(k)},r^{(k)}\} { s ( k ) , r ( k ) } because c c c is scale-free. Because c ( k ) c(k) c ( k ) grows super-exponentially as k → 0 k \to0 k → 0 , the expectations in (17) and (42) are finite only when the distribution of k i k_i k i is supported away from zero; if k i ≥ k 0 k_i \ge k_0 k i ≥ k 0 almost surely, monotonicity of c c c gives the finite bound c ( k 0 ) c(k_0) c ( k 0 ) .
Variational inference and posterior key quantities
Derivation of (36) (q ϕ ( A L I V E ∣ D i ) q_\phi(\mathrm{ALIVE}\mid\mathcal{D}_i) q ϕ ( ALIVE ∣ D i ) ). q ϕ ( A L I V E ∣ D i ) q_\phi(\mathrm{ALIVE}\mid\mathcal{D}_i) q ϕ ( ALIVE ∣ D i ) is the variational probability that τ i > T i \tau_i>T_i τ i > T i . The τ \tau τ -factor is Weibull ( s ~ i ( τ ) , r ~ i ( τ ) , L O C = t i , y i ) \operatorname{Weibull}(\tilde{s}_i^{(\tau)},\tilde{r}_i^{(\tau)},\mathrm{LOC}=t_{i,y_i}) Weibull ( s ~ i ( τ ) , r ~ i ( τ ) , LOC = t i , y i ) , whose survival for τ > t i , y i \tau>t_{i,y_i} τ > t i , y i is exp [ − { r ~ i ( τ ) ( τ − t i , y i ) } s ~ i ( τ ) ] \exp\left[-\left\{\tilde{r}_i^{(\tau)}(\tau-t_{i,y_i})\right\}^{\tilde{s}_i^{(\tau)}}\right] exp [ − { r ~ i ( τ ) ( τ − t i , y i ) } s ~ i ( τ ) ] . Evaluating at τ = T i \tau=T_i τ = T i gives (36) .
Derivation of (37) (q ϕ ( D i ⋆ ∣ D i ) q_\phi(\mathcal{D}_i^\star\mid\mathcal{D}_i) q ϕ ( D i ⋆ ∣ D i ) ). The plug-in substitutes q ϕ ( z i ∣ D i ) q_\phi(z_i\mid\mathcal{D}_i) q ϕ ( z i ∣ D i ) for the exact posterior in p θ ( D i ⋆ ∣ D i ) = E [ p ( D i ⋆ ∣ z i ) ] p_\theta(\mathcal{D}_i^\star\mid\mathcal{D}_i)=\mathbb{E}[p(\mathcal{D}_i^\star\mid z_i)] p θ ( D i ⋆ ∣ D i ) = E [ p ( D i ⋆ ∣ z i )] . A held-out history requires the customer to survive past their last held-out transaction, so conditioning on the event { τ i > t i , y i ⋆ ⋆ } \{\tau_i>t_{i,y_i^\star}^\star\} { τ i > t i , y i ⋆ ⋆ } ,
q ϕ ( D i ⋆ ∣ D i ) = q ϕ ( τ i > t i , y i ⋆ ⋆ ∣ D i ) E q ϕ ( z i ∣ D i , τ i > t i , y i ⋆ ⋆ ) [ p ( D i ⋆ ∣ z i ) ] . q_\phi(\mathcal{D}_i^\star\mid\mathcal{D}_i)=q_\phi(\tau_i>t_{i,y_i^\star}^\star\mid\mathcal{D}_i)\mathbb{E}_{q_\phi(z_i\mid\mathcal{D}_i,\tau_i>t_{i,y_i^\star}^\star)}[p(\mathcal{D}_i^\star\mid z_i)]. q ϕ ( D i ⋆ ∣ D i ) = q ϕ ( τ i > t i , y i ⋆ ⋆ ∣ D i ) E q ϕ ( z i ∣ D i , τ i > t i , y i ⋆ ⋆ ) [ p ( D i ⋆ ∣ z i )] .
the survival factor from (36) and the likelihood (14) evaluated on D i ⋆ \mathcal{D}_i^\star D i ⋆ conditional on survival. This is (37) . The conditional expectation is estimated by Monte Carlo (38) ; we evaluate the bracketed Monte Carlo average in log space via the log-sum-exp trick, which keeps it numerically stable.
Evaluation of (38) . The per-draw held-out likelihood is the renewal complete-data likelihood under z i ( s ) = ( τ i , λ i , k i ) z_i^{(s)}=(\tau_i,\lambda_i,k_i) z i ( s ) = ( τ i , λ i , k i ) , with the first held-out inter-transaction time conditioned on no purchase having occurred between the last observed transaction and the end of the training window. Writing f f f and S S S for the ITT density and survival under z i ( s ) z_i^{(s)} z i ( s ) , and Δ t i , 1 ⋆ , … , Δ t i , y i ⋆ ⋆ \Delta t_{i,1}^\star,\ldots,\Delta t_{i,y_i^\star}^\star Δ t i , 1 ⋆ , … , Δ t i , y i ⋆ ⋆ for the held-out ITTs, a draw with τ i ≥ T i \tau_i\ge T_i τ i ≥ T i contributes
p ( D i ⋆ ∣ z i ( s ) ) = 1 S ( T i − t i , y i ) [ ∏ j = 1 y i ⋆ f ( Δ t i , j ⋆ ) ] S ( min ( T i ⋆ , τ i ) − t i , y i ⋆ ⋆ ) . p(\mathcal{D}_i^\star\mid z_i^{(s)})=\frac{1}{S(T_i-t_{i,y_i})}\left[\prod_{j=1}^{y_i^\star}f(\Delta t_{i,j}^\star)\right]S\left(\min(T_i^\star,\tau_i)-t_{i,y_i^\star}^\star\right). p ( D i ⋆ ∣ z i ( s ) ) = S ( T i − t i , y i ) 1 j = 1 ∏ y i ⋆ f ( Δ t i , j ⋆ ) S ( min ( T i ⋆ , τ i ) − t i , y i ⋆ ⋆ ) .
where the leading factor is the left-truncation correction—the reciprocal of the survival factor already included in the training likelihood of (14) —the product runs over held-out purchases, and the final survival term encodes no further purchase between the last held-out transaction and the earlier of the window end T i ⋆ T_i^\star T i ⋆ and death τ i \tau_i τ i . For y i ⋆ = 0 y_i^\star=0 y i ⋆ = 0 the product is empty and t i , y i ⋆ ⋆ = t i , y i t_{i,y_i^\star}^\star=t_{i,y_i} t i , y i ⋆ ⋆ = t i , y i , so only the survival terms remain. A draw with τ i < T i \tau_i<T_i τ i < T i is instead handled exactly: a customer dead before the end of the training window makes no held-out purchases with probability one, so p ( D i ⋆ ∣ z i ( s ) ) = 1 ( y i ⋆ = 0 ) p(\mathcal{D}_i^\star\mid z_i^{(s)})=\mathbb{1}(y_i^\star=0) p ( D i ⋆ ∣ z i ( s ) ) = 1 ( y i ⋆ = 0 ) . Under the truncated sampler of (38) such draws arise only when y i ⋆ = 0 y_i^\star=0 y i ⋆ = 0 , since for y i ⋆ > 0 y_i^\star>0 y i ⋆ > 0 every draw satisfies τ i > t i , y i ⋆ ⋆ > T i \tau_i>t_{i,y_i^\star}^\star>T_i τ i > t i , y i ⋆ ⋆ > T i .
Derivation of (41) (forecasted future frequency estimator). This is the posterior, finite-horizon analog of (16) . Conditional on z i z_i z i and survival to T i T_i T i , the residual process over ( T i , T i ⋆ ∧ τ i ] (T_i,T_i^\star\wedge\tau_i] ( T i , T i ⋆ ∧ τ i ] accrues transactions at rate λ i \lambda_i λ i , so E [ Y i ( T i , T i ⋆ ∧ τ i ) ∣ z i ] ≈ λ i ( T i ⋆ ∧ τ i − T i ) 1 ( τ i > T i ) \mathbb{E}[Y_i(T_i,T_i^\star\wedge\tau_i)\mid z_i]\approx\lambda_i(T_i^\star\wedge\tau_i-T_i)\mathbb{1}(\tau_i>T_i) E [ Y i ( T i , T i ⋆ ∧ τ i ) ∣ z i ] ≈ λ i ( T i ⋆ ∧ τ i − T i ) 1 ( τ i > T i ) . Taking E q ϕ \mathbb{E}_{q_\phi} E q ϕ and using the mean-field factorization ( λ i ⊥ τ i under q ϕ ) (\lambda_i\perp\tau_i\text{ under }q_\phi) ( λ i ⊥ τ i under q ϕ ) ,
E ^ ϕ [ Y i ( T i , T i ⋆ ) ∣ D i ] = E q ϕ [ λ i ] E q ϕ [ ( T i ⋆ ∧ τ i − T i ) 1 ( τ i > T i ) ] = E q ϕ [ λ i ] q ϕ ( A L I V E ∣ D i ) E q ϕ [ T i ⋆ ∧ τ i − T i ∣ τ i > T i ] , \begin{aligned}
\widehat{\mathbb{E}}_\phi[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i]=\mathbb{E}_{q_\phi}[\lambda_i]\mathbb{E}_{q_\phi}[(T_i^\star\wedge\tau_i-T_i)\mathbb{1}(\tau_i>T_i)] \\
&=\mathbb{E}_{q_\phi}[\lambda_i]q_\phi(\mathrm{ALIVE}\mid\mathcal{D}_i)\mathbb{E}_{q_\phi}[T_i^\star\wedge\tau_i-T_i\mid\tau_i>T_i],
\end{aligned} E ϕ [ Y i ( T i , T i ⋆ ) ∣ D i ] = E q ϕ [ λ i ] E q ϕ [( T i ⋆ ∧ τ i − T i ) 1 ( τ i > T i )] = E q ϕ [ λ i ] q ϕ ( ALIVE ∣ D i ) E q ϕ [ T i ⋆ ∧ τ i − T i ∣ τ i > T i ] ,
which is (41) ; the k i k_i k i -marginal drops because λ i \lambda_i λ i alone sets the rate. The rate factor E q ϕ [ λ i ] \mathbb{E}_{q_\phi}[\lambda_i] E q ϕ [ λ i ] and the survival factor q ϕ ( A L I V E ∣ D i ) q_\phi(\mathrm{ALIVE}\mid\mathcal{D}_i) q ϕ ( ALIVE ∣ D i ) are elementary Weibull moments of the variational marginals. The residual-lifetime factor E q ϕ [ T i ⋆ ∧ τ i − T i ∣ τ i > T i ] \mathbb{E}_{q_\phi}[T_i^\star\wedge\tau_i-T_i\mid\tau_i>T_i] E q ϕ [ T i ⋆ ∧ τ i − T i ∣ τ i > T i ] is not an elementary moment, but a regularized incomplete-gamma expression in closed form: writing ( s ~ , r ~ ) (\tilde{s},\tilde{r}) ( s ~ , r ~ ) for ( s ~ i ( τ ) , r ~ i ( τ ) ) (\tilde{s}_i^{(\tau)},\tilde{r}_i^{(\tau)}) ( s ~ i ( τ ) , r ~ i ( τ ) ) and P P P for the regularized lower incomplete gamma function, it equals Γ ( 1 + 1 / s ~ ) r ~ [ P ( 1 / s ~ , [ r ~ ( T i ⋆ − t i , y i ) ] s ~ ) − P ( 1 / s ~ , [ r ~ ( T i − t i , y i ) ] s ~ ) ] q ϕ ( A L I V E ∣ D i ) \frac{\Gamma(1+1/\tilde{s})}{\tilde{r}}\left[P\left(1/\tilde{s},[\tilde{r}(T_i^\star-t_{i,y_i})]^{\tilde{s}}\right)-P\left(1/\tilde{s},[\tilde{r}(T_i-t_{i,y_i})]^{\tilde{s}}\right)\right]q_\phi(\mathrm{ALIVE}\mid\mathcal{D}_i) r ~ Γ ( 1 + 1/ s ~ ) [ P ( 1/ s ~ , [ r ~ ( T i ⋆ − t i , y i ) ] s ~ ) − P ( 1/ s ~ , [ r ~ ( T i − t i , y i ) ] s ~ ) ] q ϕ ( ALIVE ∣ D i ) .
Proof of Proposition 3 (error bound of forecast estimator). Since − 1 ≤ E [ Y i ( t ) ∣ z i ] − λ i t ≤ c ( k i ) − 1 -1\leq\mathbb{E}[Y_i(t)\mid z_i]-\lambda_i t\leq c(k_i)-1 − 1 ≤ E [ Y i ( t ) ∣ z i ] − λ i t ≤ c ( k i ) − 1 for all t t t (elementary-renewal lower bound and Lorden upper bound), the count over ( T i , T i ⋆ ∧ τ i ] (T_i,T_i^\star\wedge\tau_i] ( T i , T i ⋆ ∧ τ i ] deviates from its rate ( λ i ) (\lambda_i) ( λ i ) value by at most the band width c ( k i ) c(k_i) c ( k i ) : conditional on τ i > T i \tau_i>T_i τ i > T i , ∣ E [ Y i ( T i , T i ⋆ ∧ τ i ) ∣ z i ] − λ i ( T i ⋆ ∧ τ i − T i ) ∣ ≤ c ( k i ) \left|\mathbb{E}[Y_i(T_i,T_i^\star\wedge\tau_i)\mid z_i]-\lambda_i(T_i^\star\wedge\tau_i-T_i)\right|\leq c(k_i) ∣ E [ Y i ( T i , T i ⋆ ∧ τ i ) ∣ z i ] − λ i ( T i ⋆ ∧ τ i − T i ) ∣ ≤ c ( k i ) , with c c c as in Proposition 2, and 0 when τ i ≤ T i \tau_i\leq T_i τ i ≤ T i . Taking E q ϕ \mathbb{E}_{q_\phi} E q ϕ and factorizing the survival event under the mean-field q ϕ ( k i ⊥ τ i ) q_\phi(k_i\perp\tau_i) q ϕ ( k i ⊥ τ i ) yields
∣ E ϕ R T [ Y i ( T i , T i ⋆ ) ∣ D i ] − E ϕ [ Y i ( T i , T i ⋆ ) ∣ D i ] ∣ ≤ q ϕ ( A L I V E ∣ D i ) E q ϕ ( k i ∣ D i ) [ c ( k i ) ] \left|\mathbb{E}_{\phi}^{\mathrm{RT}}[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i]-\mathbb{E}_{\phi}[Y_i(T_i,T_i^\star)\mid\mathcal{D}_i]\right|\leq q_\phi(\mathrm{ALIVE}\mid\mathcal{D}_i)\mathbb{E}_{q_\phi(k_i\mid\mathcal{D}_i)}[c(k_i)] E ϕ RT [ Y i ( T i , T i ⋆ ) ∣ D i ] − E ϕ [ Y i ( T i , T i ⋆ ) ∣ D i ] ≤ q ϕ ( ALIVE ∣ D i ) E q ϕ ( k i ∣ D i ) [ c ( k i )]
which is (42) .
Algorithmic and implementational details
Recognition network. The summary u ( D i ) u(\mathcal{D}_i) u ( D i ) stacks ( y i , t i , y i , κ i , T i ) (y_i,t_{i,y_i},\kappa_i,T_i) ( y i , t i , y i , κ i , T i ) with the moment features ζ i ( p ) = ∑ j ( Δ t i , j ) p / 10 \zeta_i^{(p)}=\sum_j(\Delta t_{i,j})^{p/10} ζ i ( p ) = ∑ j ( Δ t i , j ) p /10 for p = 1 , … , P p=1,\ldots,P p = 1 , … , P , with P = 25 P=25 P = 25 in all runs. The recognition network ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) is built from an MLP with two-to-three hidden layers (widths 256–64–32 for the synthetic studies, (8,8) for the sub-model study of § 6.1, and 256–128–64 for the full-panel fit) and ReLU activations; its six outputs are mapped to positive shapes and rates and clamped to a numerically stable range. Concretely, the recognition network maps the summary to variational parameters as ϕ i = exp ( g + f ψ ( u i ) ) \phi_i=\exp(g+f_\psi(u_i)) ϕ i = exp ( g + f ψ ( u i )) , where f ψ f_\psi f ψ is the MLP with weights ψ \psi ψ and g ∈ R 6 g\in\mathbb{R}^{6} g ∈ R 6 is a shared, per-dimension global offset applied in log-space—the shared global variational parameters. The full variational parameter set is ϕ d e f = { ψ , g } \phi^{\mathrm{def}}=\{\psi,g\} ϕ def = { ψ , g } .
Optimizer and learning-rate schedule. We use Adam/AdamW with two parameter groups—the recognition-network weights, and everything global (the shared global variational parameters together with the prior hyperparameters)—each with its own weight decay (10 − 4 10^{-4} 1 0 − 4 on the network weights, 0 on the global group) and, in principle, its own step size, though all reported runs share one initial rate. The learning rate follows a cosine schedule to near zero with no warmup and no restarts; initial rates are 3 × 10 − 4 3\times10^{-4} 3 × 1 0 − 4 (synthetic studies; 10 − 3 10^{-3} 1 0 − 3 for the sub-model study) and 5 × 10 − 5 5\times10^{-5} 5 × 1 0 − 5 (full-panel fit).
Gradient stability. Reparameterization noise is the truncated exponential ϵ = − log u \epsilon=-\log u ϵ = − log u with u ∼ Uniform ( 0.01 , 0.99 ) u\sim\operatorname{Uniform}(0.01,0.99) u ∼ Uniform ( 0.01 , 0.99 ) (§ 5.3), and gradients are additionally clipped to global norm 10 at every step.
Inference compilation and early stopping. The recognition network is pretrained on 5,000 (20,000 for the full panel) synthetic customers for 500 (1,000) Adam steps. When early stopping is enabled we hold out 20% of customers, stratified by test-window activity with a fixed seed, and stop when their held-out predictive log-likelihood has not improved for 100 evaluations, restoring the best checkpoint; population-parameter recovery (§ 6.2) and the full-panel fit instead run a fixed iteration budget.
Confidence intervals for θ ^ \hat{\theta} θ ^ in § 6.2. We computed 95% confidence intervals for WTYD’s hyperparameter estimate θ ^ \hat{\theta} θ ^ in two ways, both from the single observed dataset rather than across simulation replicates. The first is a customer-level bootstrap: we resampled the n n n customers with replacement, refit WTYD from scratch on the resampled panel with the same settings and iteration budget as the original fit, repeated this 100 times, and took the 2.5th and 97.5th percentiles of the resulting estimates. The second is an asymptotic approximation. Treating θ ^ \hat{\theta} θ ^ as an M-estimator, i.e., as the maximizer of the ELBO with the recognition network held fixed at its fitted value ϕ ^ \hat{\phi} ϕ ^ , we estimated its covariance by the inverse Hessian
Cov ^ ( θ ^ ) = [ − ∇ θ 2 B ( ϕ ^ , θ ) ∣ θ = θ ^ ] − 1 \widehat{\operatorname{Cov}}(\hat{\theta}) = \left[ -\nabla_{\theta}^{2} \mathcal{B}(\hat{\phi},\theta)\bigg|_{\theta=\hat{\theta}} \right]^{-1} Cov ( θ ^ ) = [ − ∇ θ 2 B ( ϕ ^ , θ ) θ = θ ^ ] − 1
and formed intervals on the log scale to respect positivity. Across the 18 regime-hyperparameter combinations of Figure 4 , the asymptotic standard error is between 0.40 and 0.80 times the bootstrap standard deviation, with the greatest shortfall on the regular-regime k k k hyperparameters. The asymptotic approximation conditions on the fitted recognition network and uses the curvature of the ELBO in place of that of the marginal likelihood, and both simplifications plausibly understate variability. We therefore recommend the bootstrap whenever calibrated intervals matter. Pareto/GGG’s interval in Figure 4 is its native 95% equal-tailed posterior credible interval, computed from the same MCMC draws as its point estimate.
Software and reproducibility. The reference implementation is in PyTorch [5 ] , with all subsampling, reparameterization noise, and Monte Carlo sampling driven by a single fixed seed (42) per run. One estimator caveat: Pareto/GGG’s held-out means average over customers with a finite Monte Carlo estimate. For roughly 1% of customers per subset, every retained MCMC draw places τ \tau τ before the customer’s last held-out transaction, so the finite-sample estimate is zero probability (− ∞ -\infty − ∞ log-likelihood) even though the exact model probability is positive; these customers are excluded from Pareto/GGG’s mean (their count is reported alongside each run’s outputs), while WTYD’s means average all customers. A second implementation note: WTYD’s training loop skips any iteration whose minibatch loss is non-finite rather than halting. Instrumented re-runs of eight representative fits (four of the n = 1,000 n=1{,}000 n = 1 , 000 retail subsets and four contrasting synthetic n = 10,000 n=10{,}000 n = 10 , 000 cells) measured how often this fires: never in five of the eight fits, and between 0.02% and 0.22% of iterations in the remaining three (at most 7 iterations in any single fit, concentrated in heavy-tailed and clumpy regimes); the companion guard against runtime errors never fired at all. Skip counts are logged with every fit. Table B.1 shows the per-experiment settings for each figure in the simulation studies. Note that, for the second study, held-out data is generated but unused, and the clumpy regime is additionally swept to 3.2 × 10 5 3.2 \times10^{5} 3.2 × 1 0 5 iterations to check that its s ( k ) s^{(k)} s ( k ) underestimate is not a training-budget artifact. The figures for the third and fourth studies report the n = 10,000 n=10{,}000 n = 10 , 000 datasets.