main.pdf
Introduction
Data recorded over an ordered index often behave smoothly for long stretches punctuated by a handful of sharp, isolated changes: a policy shock in a price series, a phase transition in a physical process, a saturation threshold in a demand curve. Cubic smoothing splines [4 ] (Wecker and Ansley 1983), and their equivalent state-space signal-extraction formulation (Harvey and Koopman, 2000), recover the smooth part of such signals well, but they are twice continuously differentiable by construction: the roughness penalty that makes them smooth also prevents them from reproducing a discontinuity, which is either washed out over several observations or, if the penalty is relaxed everywhere, accommodated at the price of overfitting the noise elsewhere. What is needed is an estimator that is smooth almost everywhere and sharp exactly where the data demand it, without the analyst having to specify in advance where.
This paper proposes such an estimator. The idea is to leave the classical roughness penalty unchanged and to deform the domain over which it is computed. Every pair of consecutive observations is allowed a non-negative domain warp γ i ≥ 0 \gamma_i \ge0 γ i ≥ 0 , an amount of fictitious domain length inserted between them, subject to a total budget M M M , and the roughness of the spline is penalized on the warped domain. Where the signal is smooth, warping buys nothing, the optimal γ i \gamma_i γ i is zero and the fit coincides with an ordinary smoothing spline. Where the signal jumps, the same vertical change can be spread over a longer warped interval at a much lower curvature cost, so the optimizer concentrates the budget on the few intervals where a jump is present. Strictly speaking, the fitted function remains continuous: it approximates a discontinuity much as a steep sigmoid approximates a step function. Because the mechanism is a reparameterization of the domain, the estimator inherits the state-space representation of smoothing splines, which we use for maximum likelihood estimation via the Kalman filter, and it also admits a fast alternating algorithm that avoids numerical likelihood maximization altogether.
The idea of penalizing discontinuities directly, rather than smoothing through them, goes back to the weak rod and weak plate models of Blake and Zisserman (1987) and to the Mumford–Shah functional (Mumford and Shah, 1989), which penalize the roughness of a piecewise-smooth function together with the number of its discontinuities and lead to a combinatorial search over candidate discontinuity sets. The estimator closest to ours, and our benchmark throughout, is the cubic smoothing spline with discontinuities (CSSD) of Storath and Weinmann (2024), which specializes the weak-rod functional to cubic splines and non-equidistant data and solves it exactly by dynamic programming. Wavelet shrinkage (Donoho and Johnstone, 1994) denoises jumpy signals by thresholding multiresolution coefficients, but it targets equidistant designs and does not produce a piecewise-smooth curve. The fused lasso (Tibshirani et al., 2005) and trend filtering (Kim et al., 2009; Tibshirani, 2014) penalize the ℓ 1 \ell_1 ℓ 1 norm of successive differences of the fitted values or of their derivatives, a convex problem with automatic knot placement, but the knots are a by-product of a coefficient-level penalty rather than of an explicit, interpretable deformation. Free-knot splines (Koo, 1997) and two-stage procedures, which first locate breaks with a changepoint detector such as PELT (Killick et al., 2012) and then smooth between them, decouple detection from estimation. Our formulation keeps the roughness penalty unchanged and reparameterizes the metric in which it is computed, so that detection and estimation are solved jointly, through a single continuous, non-negative vector γ \gamma γ .
Section 2 introduces the domain-warping formulation, its state-space form, the alternating algorithm and the tuning of the method. Section 3 compares the estimators with CSSD and with a classical smoothing spline by Monte Carlo simulation, and Section 4 applies them to two energy-related data sets, chosen to cover the two kinds of data the method is meant for. The first application investigates the temperature response of electricity demand, a cross-sectional relation estimated on hourly observations ordered by temperature rather than by time, irregularly spaced and with many ties, whose discontinuities are thresholds of the physical system and are not known in advance. The second uses weekly time series of retail fuel prices, serially dependent and close to a stochastic trend, in which changes in the excise duty shift the level of the prices at dates that are a matter of public record, so that the estimated discontinuities can be checked against a ground truth that real data rarely offer. Section 5 concludes. Three supplements accompany the paper: the Computational Supplement collects derivations and implementation details, the Calibration Supplement the tuning of the method, and the Application Supplement additional results for the two applications.
Methodology
The main idea
Let ( x i , y i ) (x_i,y_i) ( x i , y i ) , i = 1 , … , n i=1,\ldots,n i = 1 , … , n , be the observations, with x 1 < x 2 < ⋯ < x n x_1 < x_2 < \cdots< x_n x 1 < x 2 < ⋯ < x n , and let δ i = x i + 1 − x i \delta_i = x_{i+1}-x_i δ i = x i + 1 − x i be the length of the i i i -th interval of the physical domain (i.e., the physical interval between consecutive knots). We introduce a vector of non-negative domain warps γ = ( γ 1 , … , γ n − 1 ) T \boldsymbol{\gamma} = (\gamma_1,\ldots,\gamma_{n-1})^{\mathsf{T}} γ = ( γ 1 , … , γ n − 1 ) T and define the length of the i i i -th interval in a latent domain as τ i = δ i + γ i \tau_i = \delta_i + \gamma_i τ i = δ i + γ i .
If γ i = 0 \gamma_i = 0 γ i = 0 the physical and latent intervals coincide; if γ i > 0 \gamma_i > 0 γ i > 0 the interval [ x i , x i + 1 ) [x_i,x_{i+1}) [ x i , x i + 1 ) is stretched. On each interval the function is a cubic polynomial f i ( s ) f_i(s) f i ( s ) in the latent coordinate s ∈ [ 0 , τ i ] s \in[0,\tau_i] s ∈ [ 0 , τ i ] , which is linked to the physical coordinate by the linear dilation s i ( x ) = ( x − x i ) ( 1 + γ i / δ i ) s_i(x) = (x-x_i)(1+\gamma_i/\delta_i) s i ( x ) = ( x − x i ) ( 1 + γ i / δ i ) , so that the function observed in the physical domain is f ( x ) = f i ( s i ( x ) ) f(x) = f_i(s_i(x)) f ( x ) = f i ( s i ( x )) for x i ≤ x < x i + 1 x_i \le x < x_{i+1} x i ≤ x < x i + 1 , $i=1,\ldots,n-1. A function that is smooth in the latent domain can thus be steep in the physical one: the dilation compresses a smooth latent transition into a short physical interval, where it appears as a jump if γ i \gamma_i γ i is large.
The estimator minimizes
J ( f , γ ) = ∑ i = 1 n ( y i − f ( x i ) ) 2 + λ ∑ i = 1 n − 1 ∫ 0 τ i ( d 2 f i ( s ) d s 2 ) 2 d s s.t. γ i ≥ 0 , ∑ i = 1 n − 1 γ i ≤ M , (1) J(f,\gamma)=\sum_{i=1}^{n}(y_i-f(x_i))^2+\lambda\sum_{i=1}^{n-1}\int_0^{\tau_i}\left(\frac{d^2f_i(s)}{ds^2}\right)^2\,ds
\qquad\text{s.t.}\qquad\gamma_i\ge0,\quad\sum_{i=1}^{n-1}\gamma_i\le M, \tag*{(1)} J ( f , γ ) = i = 1 ∑ n ( y i − f ( x i ) ) 2 + λ i = 1 ∑ n − 1 ∫ 0 τ i ( d s 2 d 2 f i ( s ) ) 2 d s s.t. γ i ≥ 0 , i = 1 ∑ n − 1 γ i ≤ M , (1)
which balances the distance from the observations against the roughness of the function measured in the latent domain; λ \lambda λ is the smoothing constant and M M M the warping budget.
The mechanism becomes transparent when one looks at the roughness penalty of a single interval, which we call its energy by analogy with the bending energy of an elastic beam. For a cubic segment covering a height difference Δ i = f i + 1 − f i \Delta_i=f_{i+1}-f_i Δ i = f i + 1 − f i , the energy is dominated, for short intervals, by a proportional term such that
∫ 0 τ i ( f i ′ ′ ( s ) ) 2 d s ∝ ( Δ y i ) 2 τ i 3 = ( Δ y i ) 2 ( δ i + γ i ) 3 \int_0^{\tau_i}(f_i''(s))^2\,ds \propto\frac{(\Delta y_i)^2}{\tau_i^3}=\frac{(\Delta y_i)^2}{(\delta_i+\gamma_i)^3} ∫ 0 τ i ( f i ′′ ( s ) ) 2 d s ∝ τ i 3 ( Δ y i ) 2 = ( δ i + γ i ) 3 ( Δ y i ) 2
(see § 2.1.2 and the exact statement is in § 6 of the Computational Supplement).
The key intuition behind the proposed approach is that a discontinuity is difficult for a classical cubic smoothing spline because the spline is forced to explain a large change in the response over a very short interval of the physical domain. On the one hand, in regions of discontinuity, if ∣ Δ y i ∣ \lvert\Delta y_i \rvert ∣ Δ y i ∣ is large while δ i \delta_i δ i is small, the standard energy term would explode (→ ∞ \to\infty → ∞ ); a large change over a short physical interval is therefore very expensive for an ordinary smoothing spline, which is forced to over-smooth it. Instead, our algorithm allocates a large positive γ i \gamma_i γ i that dilates the denominator ( δ i + γ i ) 3 (\delta_i+\gamma_i)^3 ( δ i + γ i ) 3 and lets the curve complete the same transition as a smooth, low-energy curve in the latent domain (the penalty cost is then drastically reduced), so that a discontinuity in the physical domain is represented as a smooth transition in a locally stretched domain (Figure 1). On the other hand, in regions of continuity where the signal evolves gradually, extra domain length brings little reduction of energy and the optimal γ i \gamma_i γ i stays at or near zero, the latent interval equals the physical interval (τ i ≈ δ i \tau_i \approx\delta_i τ i ≈ δ i ), so that the fit behaves as a standard cubic smoothing spline. The budget constraint ∑ i = 1 n − 1 γ i ≤ M \sum_{i=1}^{n-1}\gamma_i \le M ∑ i = 1 n − 1 γ i ≤ M is what makes the solution sparse: it prevents the whole domain from being stretched uniformly and forces the warping onto the few intervals where it is most valuable, so that large values of γ ^ i \hat{\gamma}_i γ ^ i are data-driven indicators of jumps or sharp regime changes.
The design points are taken to be distinct. A tie would make δ i \delta_i δ i vanish, and with it the latent length of an interval that carries no warp, so that neither the roughness penalty of that interval nor the transition of the state-space form of § 2.1.1 would be defined; more fundamentally, a warp on an interval of zero length would be a discontinuity between responses recorded at the same value of the covariate, which no data can identify. Tied observations, common when the covariate is recorded at a coarse resolution, are therefore collapsed into their mean at each distinct value, weighted by the number of replicates. The reduction leaves the fitted curve unchanged, but the within-group scatter that it discards still informs the observation variance, and the information criterion of § 2.1.3 counts the candidate locations on the distinct values and the observations on the raw data (§ 11 of the Computational Supplement).
Figure 1. Domain warping. The same observations on the physical domain (a) and on the latent warped domain (b), at the same horizontal scale. Away from the jump the intervals are unchanged and the grey connectors are vertical; at the jump, positive warps stretch the interval (τ i = δ i + γ i \tau_i = \delta_i + \gamma_i τ i = δ i + γ i , in yellow, with a lighter stretching of the neighboring intervals in blue), so that the same vertical change is completed over a longer latent interval at a lower curvature cost. The latent domain is longer than the physical one by the total warping ∑ i γ i \sum_i \gamma_i ∑ i γ i .
A signal extraction formulation
Smoothing splines can also be obtained by signal extraction (Wahba, 1978; Wecker and Ansley, 1983): the cubic smoothing spline is f ^ t = E [ f ( x t ) ∣ Y 1 = y 1 , … , Y n = y n ] \hat{f}_t = \mathrm{E}[f(x_t) \mid Y_1 = y_1, \ldots, Y_n = y_n] f ^ t = E [ f ( x t ) ∣ Y 1 = y 1 , … , Y n = y n ] when the data are generated by Y t = f ( x t ) + ε t Y_t = f(x_t) + \varepsilon_t Y t = f ( x t ) + ε t , ε t ∼ NID ( 0 , λ σ 2 ) \varepsilon_t \sim\operatorname{NID}(0,\lambda\sigma^2) ε t ∼ NID ( 0 , λ σ 2 ) , d 2 f / d x 2 = σ w ( x ) d^2f/dx^2 = \sigma w(x) d 2 f / d x 2 = σ w ( x ) , with w ( x ) w(x) w ( x ) a continuous-time standard white noise and diffuse initial conditions for f ( x 1 ) f(x_1) f ( x 1 ) and its derivative. This model has the state-space form (see, e.g., Harvey and Koopman, 2000)
Y t = f ( x t ) + ε t , ε t ∼ NID ( 0 , λ σ 2 ) Y_t = f(x_t) + \varepsilon_t,\quad\varepsilon_t \sim\operatorname{NID}(0,\lambda\sigma^2) Y t = f ( x t ) + ε t , ε t ∼ NID ( 0 , λ σ 2 )
\begin{bmatrix} f(x_{t+1}) \\ f'(x_{t+1}) \end{bmatrix}
=
\begin{bmatrix} 1 & \delta_t \\ 0 & 1 \end{bmatrix}
\begin{bmatrix} f(x_t) \\ f'(x_t) \end{bmatrix}
+
\begin{bmatrix} \eta_t \\ \zeta_t \end{bmatrix},
\quad
\begin{bmatrix} \eta_t \\ \zeta_t \end{bmatrix}
\sim\operatorname{NID}\left(
\begin{bmatrix} 0 \\ 0 \end{bmatrix},
\sigma^2
\begin{bmatrix}
\delta_t^3/3 & \delta_t^2/2 \\
\delta_t^2/2 & \delta_t
\end{bmatrix}
\right) \tag*{(2)}\end{bmatrix}
with ( f ( x 1 ) , f ′ ( x 1 ) ) T ∼ N ( 0 , κ I 2 ) (f(x_1),f'(x_1))^\mathsf{T} \sim\mathrm{N}(0,\kappa I_2) ( f ( x 1 ) , f ′ ( x 1 ) ) T ∼ N ( 0 , κ I 2 ) , κ → ∞ \kappa\to\infty κ → ∞ . The Kalman filter delivers the likelihood of ( σ , λ ) (\sigma,\lambda) ( σ , λ ) , and the state smoother the smoothing spline itself.
Adapting this representation to the latent domain only requires replacing the physical increments δ t \delta_t δ t with their warped counterparts τ t = δ t + γ t \tau_t = \delta_t + \gamma_t τ t = δ t + γ t . The likelihood is then maximized with respect to ( σ , λ , γ 1 , … , γ n − 1 ) (\sigma,\lambda,\gamma_1,\ldots,\gamma_{n-1}) ( σ , λ , γ 1 , … , γ n − 1 ) subject to γ t ≥ 0 \gamma_t \ge0 γ t ≥ 0 and ∑ t γ t ≤ M \sum_t \gamma_t \le M ∑ t γ t ≤ M , a problem for which the algorithm of Svanberg (2002), implemented in the NLopt library (Johnson, 2007), is effective. The analytical scores are available in closed form from one pass of the Kalman filter and smoother, so that a gradient of dimension n + 1 n+1 n + 1 never has to be approximated by finite differences; for γ t \gamma_t γ t they contain an additional term, because γ t \gamma_t γ t enters the transition matrix as well as the disturbance covariance (§ 10 of the Computational Supplement). Since the likelihood is not convex, good starting values for γ \gamma γ are important, and values close to zero are generally a good choice. We call this estimator SSJ–MLE (smoothing splines with jumps, maximum likelihood); when λ \lambda λ is fixed at a supplied value and the likelihood is maximized with respect to the remaining parameters, we call it SSJ–KFS.
An alternating domain-warping algorithm
As an alternative to the numerical maximization of a non-convex likelihood, we minimize (1) for given λ \lambda λ and M M M by block coordinate descent, alternating a spline step, in which γ \gamma γ is fixed and f f f solves a convex quadratic problem, with a warping step, in which f f f is fixed and γ \gamma γ takes a projected-gradient step. We call the resulting estimator SSJ–ATW (alternating domain warping). Each iteration costs O ( n log n ) O(n \log n) O ( n log n ) operations.
Spline step (f f f -step). In this step, γ \gamma γ is fixed. Let z = ( f 1 , f 1 ′ , f 2 , f 2 ′ , … , f n , f n ′ ) T z = (f_1,f'_1,f_2,f'_2,\ldots,f_n,f'_n)^\mathsf{T} z = ( f 1 , f 1 ′ , f 2 , f 2 ′ , … , f n , f n ′ ) T collect the values and first derivatives of the function at the observations. By a well-known argument based on Hermite polynomials (see Section 2 of Dougherty et al., 1989), the energy of interval i i i (i.e., the contribution of the interval i i i to the total roughness energy) is the quadratic form
k i : = ∫ 0 τ i ( d 2 f i ( s ) d s 2 ) 2 d s = v i T K i v i , v i = ( f i , f i ′ , f i + 1 , f i + 1 ′ ) T , (3) k_i := \int_0^{\tau_i} \left(\frac{d^2 f_i(s)}{ds^2}\right)^2 ds = v_i^\mathsf{T}K_i v_i,\qquad v_i = (f_i,f'_i,f_{i+1},f'_{i+1})^\mathsf{T}, \tag*{(3)} k i := ∫ 0 τ i ( d s 2 d 2 f i ( s ) ) 2 d s = v i T K i v i , v i = ( f i , f i ′ , f i + 1 , f i + 1 ′ ) T , (3)
with the local stiffness matrix
K i = 1 τ i 3 [ 12 6 τ i − 12 6 τ i 6 τ i 4 τ i 2 − 6 τ i 2 τ i 2 − 12 − 6 τ i 12 − 6 τ i 6 τ i 2 τ i 2 − 6 τ i 4 τ i 2 ] K_i = \frac{1}{\tau_i^3}
\begin{bmatrix}
12 & 6\tau_i & -12 & 6\tau_i \\
6\tau_i & 4\tau_i^2 & -6\tau_i & 2\tau_i^2 \\
-12 & -6\tau_i & 12 & -6\tau_i \\
6\tau_i & 2\tau_i^2 & -6\tau_i & 4\tau_i^2
\end{bmatrix} K i = τ i 3 1 12 6 τ i − 12 6 τ i 6 τ i 4 τ i 2 − 6 τ i 2 τ i 2 − 12 − 6 τ i 12 − 6 τ i 6 τ i 2 τ i 2 − 6 τ i 4 τ i 2
Adding each K i K_i K i to the 4 × 4 4 \times4 4 × 4 diagonal block of rows and columns 2 i − 1 , … , 2 i + 2 2i-1,\ldots,2i+2 2 i − 1 , … , 2 i + 2 of a 2 n × 2 n 2n \times2n 2 n × 2 n matrix of zeros gives the roughness penalty matrix R R R , which is symmetric and block tridiagonal. Following the classical approach that goes back to Strang and Fix (1973), the objective function becomes ( y ∗ − S z ) T ( y ∗ − S z ) + λ z T R z (y^* - Sz)^\mathsf{T}(y^* - Sz) + \lambda z^\mathsf{T}Rz ( y ∗ − S z ) T ( y ∗ − S z ) + λ z T R z , where y ∗ y^* y ∗ has y i y_i y i in position 2 i − 1 2i-1 2 i − 1 and zeros elsewhere and the selection matrix S = diag ( 1 , 0 , 1 , 0 , … , 1 , 0 ) S = \operatorname{diag}(1,0,1,0,\ldots,1,0) S = diag ( 1 , 0 , 1 , 0 , … , 1 , 0 ) retains the function values, and it is minimized by the solution of the banded linear system
( S + λ R ) z = y ∗ . (S+\lambda R)z = y^*. ( S + λ R ) z = y ∗ .
Because S + λ R S+\lambda R S + λ R is block tridiagonal, a block Cholesky decomposition solves the system in O ( n ) O(n) O ( n ) operations (§ 7 of the Computational Supplement), an idea that goes back to de Boor (1978) and that the recent work of Strøth and Weinmann (2024) also exploits. The output of this step is the updated f f f (odd elements of z z z ) and f ′ f' f ′ (even elements of z z z ).
Recall that A = ( S + λ R ) A = (S+\lambda R) A = ( S + λ R ) . To evaluate the complexity of the model (useful for generalized cross-validation or information criteria of § 2.1.3 ), calculate the trace of the smoothing matrix A − 1 S = ( S + λ R ) − 1 S A^{-1}S = (S+\lambda R)^{-1}S A − 1 S = ( S + λ R ) − 1 S (or equivalently the sum of the odd diagonal elements of A − 1 A^{-1} A − 1 , which provides the effective degrees of freedom (EDF):
E D F = ∑ i = 1 , 3 , … , 2 n − 1 a ˉ i i , \mathrm{EDF} = \sum_{i=1,3,\ldots,2n-1} \bar{a}_{ii}, EDF = i = 1 , 3 , … , 2 n − 1 ∑ a ˉ ii ,
where a ˉ i j \bar{a}_{ij} a ˉ ij denotes the generic element of the matrix A − 1 A^{-1} A − 1 (see Section 5.4 of Hastie et al., 2009).
Warping step (γ \gamma γ -step). In this step, f f f and f ′ f' f ′ are fixed, and we want to update the domain warp vector γ \gamma γ to reduce the current objective function value. First, we update γ \gamma γ to reduce the total roughness energy. With z z z fixed, the energy of interval i i i has the reduced form
k i = 12 Δ i 2 τ i 3 − 12 Δ i S i τ i 2 + 4 P i τ i , Δ i = f i + 1 − f i , S i = f i ′ + f i + 1 ′ , P i = ( f i ′ ) 2 + f i ′ f i + 1 ′ + ( f i + 1 ′ ) 2 , k_i = \frac{12\Delta_i^2}{\tau_i^3} - \frac{12\Delta_i S_i}{\tau_i^2} + \frac{4P_i}{\tau_i}, \qquad\Delta_i = f_{i+1}-f_i,\quad S_i=f'_i+f'_{i+1},\quad P_i=(f'_i)^2+f'_if'_{i+1}+(f'_{i+1})^2, k i = τ i 3 12 Δ i 2 − τ i 2 12 Δ i S i + τ i 4 P i , Δ i = f i + 1 − f i , S i = f i ′ + f i + 1 ′ , P i = ( f i ′ ) 2 + f i ′ f i + 1 ′ + ( f i + 1 ′ ) 2 ,
that is, it is a Laurent polynomial in τ i \tau_i τ i whose coefficients do not depend on τ i \tau_i τ i (see Section A1 of the Computational Supplement for derivation and analytical details). Its derivative with respect to the warp,
∇ i = ∂ k i ∂ γ i = − 36 Δ i 2 τ i 4 + 24 Δ i S i τ i 3 − 4 P i τ i 2 , (4) \nabla_i = \frac{\partial k_i}{\partial\gamma_i} = -36\frac{\Delta_i^2}{\tau_i^4}+24\frac{\Delta_iS_i}{\tau_i^3}-4\frac{P_i}{\tau_i^2}, \tag*{(4)} ∇ i = ∂ γ i ∂ k i = − 36 τ i 4 Δ i 2 + 24 τ i 3 Δ i S i − 4 τ i 2 P i , (4)
gives the gradient step γ − η ∇ \gamma-\eta\nabla γ − η ∇ , where the learning rate η \eta η absorbs the factor λ \lambda λ . Second, to ensure that γ \gamma γ respects the constraints, the result of the f f f -step is projected onto the feasible set { γ : γ i ≥ 0 , ∑ i γ i ≤ M } \{\gamma:\gamma_i\geq0,\ \sum_i\gamma_i\leq M\} { γ : γ i ≥ 0 , ∑ i γ i ≤ M } , exactly and in O ( n log n ) O(n\log n) O ( n log n ) operations, by the sort-based algorithm of Duchi et al. (2008) (Section A3 of the Computational Supplement for details on the projection onto the Simplex). The projection has a statistical interpretation: once the gradient has indicated where additional domain length would reduce the energy, the projection redistributes the available budget and sets many components exactly to zero, so that only a few intervals are stretched, and these are expected to be those where a smoothing spline would otherwise pay an excessive curvature penalty.
The two steps are repeated until the change in the variables falls below a tolerance level.
Tuning and implementation
Three constants have to be chosen: the smoothing constant λ \lambda λ , the warping budget M M M and the constant ξ \xi ξ of the information criterion that selects M M M . We summarize the choices here; the Calibration Supplement gives the details.
Smoothing constant λ \lambda λ . SSJ–MLE estimates λ \lambda λ by maximum likelihood together with the other parameters. SSJ–KFS and SSJ–ATW take it as given, either from a preliminary SSJ–MLE fit or by K K K -fold cross-validation: the held-out squared error C V ( λ , M ) \mathrm{CV}(\lambda,M) CV ( λ , M ) is computed on a grid of pairs, λ ^ \hat{\lambda} λ ^ minimizes min M C V ( λ , M ) \min_M\mathrm{CV}(\lambda,M) min M CV ( λ , M ) , and the budget is then selected afresh, on the full sample, by the criterion below. When the observations are serially dependent the folds are contiguous blocks rather than random draws ([3 ] ): a held-out observation whose neighbors stay in the training sample is nearly predicted by them, and the criterion is then won by a fit that all but interpolates. The minimization over M M M is a profiling step, not a selection. Only the few observations near a transition carry information on the budget, and prediction error is nearly indifferent to a few spurious warps, so selecting M M M by cross-validation declares many more discontinuities than the criterion below at the same accuracy (Section 1 of the Calibration Supplement).
Warping budget. M M M is selected on a grid by the extended BIC of Chen and Chen [1 ] ,
E B I C ( ξ ) = − 2 ℓ ^ + d f log n + 2 ξ k log ( n − 1 ) , ξ ∈ [ 0 , 1 ] , (5) \mathrm{EBIC}(\xi)=-2\hat{\ell}+\mathrm{df}\log n+2\xi k\log(n-1),\qquad\xi\in[0,1], \tag*{(5)} EBIC ( ξ ) = − 2 ℓ ^ + df log n + 2 ξ k log ( n − 1 ) , ξ ∈ [ 0 , 1 ] , (5)
where ℓ ^ \hat{\ell} ℓ ^ is the maximized log-likelihood, d f \mathrm{df} df the effective degrees of freedom, k k k the number of active warps, and ξ = 0 \xi=0 ξ = 0 returns the BIC. A budget with k k k active warps also decides which of the n − 1 n-1 n − 1 intervals carry them, so the model space at size k k k has ( n − 1 k ) \binom{n-1}{k} ( k n − 1 ) members and grows with the sample; a criterion that charges only for dimension, such as the AIC or the BIC, reads the gain from searching over this space as evidence and keeps adding discontinuities. The extended term charges for where the discontinuities are placed, as distinct from how many there are.
The constant ξ \xi ξ . The consistency result of Chen and Chen [1 ] supplies a floor rather than a value, ξ ≥ 1 − log n / { 2 log ( n − 1 ) } ≈ 0.5 \xi\geq1-\log n/\{2\log(n-1)\}\approx0.5 ξ ≥ 1 − log n / { 2 log ( n − 1 )} ≈ 0.5 , and it is stated for a criterion in which the extended term alone pays for the search, whereas here the effective degrees of freedom grow with every activated warp and the d f log n \mathrm{df}\log n df log n term already pays part of it. We therefore calibrate ξ \xi ξ by simulation, on a design separate from the one used for evaluation in Section 3, with 200 replications per cell. The design has three scenarios: a fixed smooth signal observed with independent errors on an irregular design (the reference case), serially dependent errors, and a signal that is partly or entirely a stochastic trend, namely an integrated Wiener process, the prior that the smoothing spline assumes. Each scenario is crossed with the sample size (n n n from 50 to 400), the noise level and the number of true discontinuities (0 to 5). The constant is read off two minimax regret rules over the design cells,1 one on the root mean squared error (RMSE) of the fitted curve and one on the absolute error in the number of discontinuities declared, and Table 1 reports their midpoint.
Table 1. Calibrated values of ξ \xi ξ by scenario, estimator and source of λ \lambda λ (from SSJ–MLE, or by cross-validation with K = 5 K = 5 K = 5 or K = 20 K = 20 K = 20 folds drawn at random or as contiguous blocks). Each value is the midpoint, rounded up to the grid, of the minimax regret rules in RMSE and in the count error, each pooled over the levels of the scenario (ψ ≥ 0.6 \psi\ge0.6 ψ ≥ 0.6 under dependent errors, ω = 1 \omega= 1 ω = 1 under a stochastic trend; Section 7 of the Calibration Supplement).
Three regularities matter in practice (Section 7 of the Calibration Supplement). First, the calibrated values lie below the nominal floor without contradicting the theory, because part of the charge per discontinuity is levied through df; for the same reason SSJ–KFS with λ \lambda λ from SSJ–MLE, whose smoothing constant is fixed before the budget is chosen and whose search over the budget can therefore buy less, needs a milder penalty than SSJ–MLE itself in every scenario. Second, serial dependence and stochastic trends both call for a heavier penalty than the reference case, through different channels: persistence acts through λ \lambda λ , and the penalty responds to it when λ \lambda λ is cross-validated on contiguous blocks but not with random folds, whereas a stochastic trend leaves the reconstruction unaffected and inflates the number of discontinuities declared. The value depends on the estimator as much as on the data. Third, a single constant is a compromise across sample sizes: Table 5 of the Calibration Supplement reports the calibration at each n n n , and Table 7 a closed-form rule ξ ( n ) \xi(n) ξ ( n ) that extends it beyond the sample sizes of the design for SSJ–KFS with λ \lambda λ from SSJ–MLE or from contiguous blocks. The values adopted are largely robust to the rule that synthesizes the two criteria (Table 16 of the Calibration Supplement).
Table 5: The calibration by sample size, in both scenarios: minimax regret in RMSE, for seven representative arms. The first column is the reference case; then the persistence ψ \psi ψ of the errors and the weight ω \omega ω of the stochastic-trend component. Beneath each value, in small type, the central half of the same criterion over the twelve cells of that sample size.
Table 5.
Table 7. The rule of Equation (9) for SSJ–KFS, with λ \lambda λ from maximum likelihood and from contiguous blocks at K = 20 K=20 K = 20 : C ^ \hat{C} C ^ is the mean optimal charge over the 36 cells with q ≥ 1 q \ge1 q ≥ 1 , and the columns give ( C ^ − s log n ) / { 2 log ( n − 1 ) } (\hat{C}-s\log n)/\{2\log(n-1)\} ( C ^ − s log n ) / { 2 log ( n − 1 )} at the four sample sizes of the design and at two beyond it.
Table 16: Robustness to the decision rule: the values of ξ \xi ξ adopted for each scenario and arm (cross-validation at K = 5 K = 5 K = 5 , 10 and 20) under the four syntheses of the two pooled rules — the count rule alone, the RMSE rule alone, their midpoint rounded up to the grid (the synthesis of Table 15 ) and the larger of the two. Columns: the reference case, dependent errors (ψ ≥ 0.6 \psi\ge0.6 ψ ≥ 0.6 ) and a stochastic trend (ω = 1 \omega= 1 ω = 1 ).
Table 16.
Declaring discontinuities. The estimated warps are continuous, and an interval is declared a discontinuity when its warp is large both relative to the total warping of the fit and in absolute terms. Unless otherwise stated, a warp is declared when it exceeds both 5% of the total warping and 30% of the mean sampling interval, a rule that filters out numerically small warps while retaining genuine jumps of very different sizes.
Software. Both algorithms are implemented in the R package jumps, version 2.0.0 or later, available from CRAN (Pelagatti and Maranzano, 2026); the computationally intensive parts are written in C++ and interfaced through Rcpp (Eddelbuettel and François, 2011).
Simulation experiment
To assess how well the estimators recover a piecewise-smooth signal from noisy, irregularly sampled data, we replicate and extend the illustrative simulation of Storath and Weinmann (2024). The test signal is
g ( x ) = J 1 ( 20 x ) + x ⋅ 1 [ 0.3 , 0.4 ] ( x ) − x ⋅ 1 [ 0.6 , 1 ] ( x ) , g(x) = J_1(20x) + x \cdot1_{[0.3,0.4]}(x) - x \cdot1_{[0.6,1]}(x), g ( x ) = J 1 ( 20 x ) + x ⋅ 1 [ 0.3 , 0.4 ] ( x ) − x ⋅ 1 [ 0.6 , 1 ] ( x ) ,
where J 1 J_1 J 1 is the Bessel function of the first kind of order one; it has three discontinuities, at x = 0.3 x = 0.3 x = 0.3 (jump +0.3), x = 0.4 x = 0.4 x = 0.4 (jump −0.4) and x = 0.6 x = 0.6 x = 0.6 (jump −0.6), on an oscillating smooth background. For each of three noise levels, σ ∈ { 0.05 , 0.10 , 0.15 } \sigma\in\{0.05, 0.10, 0.15\} σ ∈ { 0.05 , 0.10 , 0.15 } , we generate 1,000 replications, each with N = 100 N = 100 N = 100 abscissas drawn uniformly on [ 0 , 1 ] [0,1] [ 0 , 1 ] and y i = g ( x i ) + ϵ i y_i = g(x_i) + \epsilon_i y i = g ( x i ) + ϵ i , ϵ i ∼ N ( 0 , σ 2 ) \epsilon_i \sim N(0,\sigma^2) ϵ i ∼ N ( 0 , σ 2 ) . Every estimator is evaluated on a grid of 500 equispaced points on [ 0 , 1 ] [0,1] [ 0 , 1 ] , clamped to the observed range of each replication, so that extrapolation is not scored.
Four estimators are compared. CSSD has the two tuning parameters of its weak-rod penalty chosen by 5-fold cross-validation, as recommended by Storath and Weinmann (2024, Section 3.4), and returns its discontinuity set directly, as part of the solution of the underlying combinatorial problem.2 SSJ–MLE estimates ( σ , λ ) (\sigma,\lambda) ( σ , λ ) by maximum likelihood and SSJ–ATW takes λ \lambda λ from 5-fold cross-validation (§ 2.1.3 ); both select M M M by the extended BIC at ξ = 0.30 \xi= 0.30 ξ = 0.30 and declare discontinuities by the rule of § 2.1.3 . This is the reference-case value of Table 1 for SSJ–ATW with λ \lambda λ from random 5-fold cross-validation; for SSJ–MLE it is one grid step above the value of the table and equal to the one that the RMSE rule selects on its own (Table 1 of the Calibration Supplement), and it keeps the two estimators at an equal penalty. The naive benchmark, which ignores discontinuities altogether, is a cubic smoothing spline with smoothing parameter chosen by generalized cross-validation.
At every noise level SSJ–ATW and SSJ–MLE reconstruct g g g more accurately than either benchmark (Table 2), and they stay close to each other: their RMSEs differ by at most 4%, with SSJ–ATW ahead at σ = 0.05 \sigma= 0.05 σ = 0.05 and SSJ–MLE at σ = 0.15 \sigma= 0.15 σ = 0.15 . The two-step algorithm of § 2.1.2 thus attains the accuracy of the maximum likelihood fit it approximates. Both improve on the classical smoothing spline, by 7% to 11% at σ = 0.05 \sigma= 0.05 σ = 0.05 and by 3% to 7% at the other two levels.
Table 2. RMSE of the fitted curve against the true signal g g g , pooled over 1,000 replications and the 500-point evaluation grid, at three noise levels. All values are multiplied by 10 3 10^3 1 0 3 . CSSD is reported twice: over all replications, and excluding those on which its fit diverges between the design points; n e x n_{\mathrm{ex}} n ex counts the latter.
The comparison with CSSD requires a qualification, which is why Table 2 reports it twice. Pooled over all replications CSSD is the least accurate of the four estimators, trailing even the smoothing spline, and SSJ–ATW is 33% more accurate at σ = 0.05 \sigma= 0.05 σ = 0.05 ; but nearly all of the gap is contributed by four replications out of 3,000, two at σ = 0.05 \sigma= 0.05 σ = 0.05 and two at σ = 0.15 \sigma= 0.15 σ = 0.15 . On those draws a declared discontinuity isolates two nearly coincident design points into a segment of their own; the spline on that segment is the straight line through them, and the linear extension used to evaluate the fit carries its slope, of order 10 3 10^3 1 0 3 , across the segment. The fitted values at the design points remain sane, so the failure appears only between them; we verified it against the reference implementation of Storath and Weinmann (2024), and it is a property of the estimator in this sampling regime. These replications, flagged because their RMSE exceeds five times the median of the same estimator at the same noise level (they sit at 8 to 23 times the median, the largest unflagged one at 3.7), produce the two spikes in CSSD’s mean squared error near x = 0.47 x = 0.47 x = 0.47 and x = 0.98 x = 0.98 x = 0.98 in Figure 3, far from any jump; away from the three discontinuities, the two divergent replications at σ = 0.05 \sigma= 0.05 σ = 0.05 account for 97% of CSSD’s excess mean squared error over SSJ–ATW. With them set aside, SSJ–ATW and SSJ–MLE are between 5% and 12% more accurate than CSSD, which is then on a par with the smoothing spline; the medians of the per-replication errors, which no tail can move, agree (59.5 for CSSD and 60.0 for the smoothing spline at σ = 0.05 \sigma= 0.05 σ = 0.05 , against 53.5 and 52.9 for SSJ–ATW and SSJ–MLE). At the discontinuities themselves the estimators behave alike: at x = 0.6 x = 0.6 x = 0.6 , the largest jump, the bias of CSSD is − 0.285 -0.285 − 0.285 against − 0.286 -0.286 − 0.286 for SSJ–ATW, the unavoidable cost of smoothing across a jump, and its mean squared error is larger by a factor of about 1.6.
The picture changes when the question is where the discontinuities are. The lower panels of Figure 2 show the locations declared over the replications, and Table 3 the detection rates within ± 0.02 \pm0.02 ± 0.02 of each true break and the false positives. CSSD raises far fewer false alarms, 0.55 per replication at σ = 0.05 \sigma= 0.05 σ = 0.05 against 2.23 for SSJ–ATW and 2.92 for SSJ–MLE, but it also finds fewer of the breaks: at σ = 0.05 \sigma= 0.05 σ = 0.05 it recovers the jump at x = 0.3 x = 0.3 x = 0.3 in 45% of the replications, against 83% and 84% for the SSJ estimators, and the one at x = 0.4 x = 0.4 x = 0.4 in 58%, against 73% and 79%; only at the largest jump, x = 0.6 x = 0.6 x = 0.6 , are the three comparable. As the noise rises, detection power declines for every estimator and the ordering ceases to be clear-cut: by σ = 0.15 \sigma= 0.15 σ = 0.15 no estimator recovers the two smaller jumps reliably. CSSD is conservative and the SSJ estimators are liberal, consistently with a warping vector that is optimized to reduce roughness globally rather than to solve a detection problem, and with a calibration of ξ \xi ξ that weighs the accuracy of the curve as well as the count.
In short, CSSD is the more conservative estimator of where the discontinuities are, declaring few and being right about nearly all of them, while SSJ–ATW and SSJ–MLE are the more accurate estimators of what the signal is doing, and the
Figure 2. For each of the three discontinuity-aware estimators, the upper panel shows the mean fitted curve (solid) and the pointwise 95% Monte Carlo interval (shaded band) against the true signal (dashed), and the lower panel the histogram of the locations declared to be discontinuities.
Figure 3. Pointwise bias (top row) and mean squared error (bottom row) of the four estimators, over 1,000 replications, for σ = 0.05 \sigma= 0.05 σ = 0.05 (blue), 0.10 (green) and 0.15 (orange). Vertical dotted lines mark the true discontinuities. The two spikes in CSSD’s mean squared error near x = 0.47 x = 0.47 x = 0.47 and x = 0.98 x = 0.98 x = 0.98 are the divergent replications discussed in the text.
Table 3. Discontinuity detection by estimator and noise level. Detection rate: fraction of the 1,000 replications in which a discontinuity is declared within ± 0.02 \pm0.02 ± 0.02 of the true break. False positives: mean number, per replication, of declared locations outside all three windows, and the same figure as a percentage of the 87.0 candidate locations that lie outside them on average (discontinuities can be declared only between consecutive design points, N − 1 = 99 N - 1 = 99 N − 1 = 99 candidates).
more sensitive detectors when the signal-to-noise ratio permits. When the discontinuities are few, isolated and large relative to the noise, the three estimators agree, as they do on the fuel prices of § 3.1.1 .
Applications
Temperature and electricity demand in Northern Italy
We estimate the relation between the hourly electricity load of Northern Italy, from the day-ahead market data published by the Gestore Mercati Energetici (GME, https://www.mercatoelettrico.org/ ), and the temperature in Milan, from Open Meteo (https://open-meteo.com/ ), between 1 January 2023 and 30 September 2025. To obtain homogeneous consumption behavior we keep the hours from 10 a.m. to 3 p.m. of Tuesdays to Thursdays and exclude the months with typical Italian vacations, keeping February, March, May, June, July, September, October and November; after dropping the few hours with a demand below 15,000 MWh or a temperature above 35 ∘ C 35^\circ\mathrm{C} 3 5 ∘ C , the sample has 1,691 hourly observations between 2.5 and 34.8 ∘ C 34.8^\circ\mathrm{C} 34. 8 ∘ C . The temperatures are recorded to the nearest 0.1 ∘ C 0.1^\circ\mathrm{C} 0. 1 ∘ C , so that many hours share the same value; these ties are collapsed into weighted means at the distinct temperatures, as described in § 2.1 and, in detail, in § 11 of the Computational Supplement. The abscissa is the temperature, irregularly spaced, and the relation is a fixed curve observed with errors, the reference case of the calibration; both estimators are therefore run at ξ = 0.30 \xi= 0.30 ξ = 0.30 , as in Section 3. We discuss the warps using the rule of § 2.1.3 , even though in Figure 4 all warps are depicted (in gray those not matching the rule).
SSJ–MLE selects a budget with three active warps, at 11.9 ∘ C 11.9^\circ\mathrm{C} 11. 9 ∘ C , 27.1 ∘ C 27.1^\circ\mathrm{C} 27. 1 ∘ C , and 30.1 ∘ C 30.1^\circ\mathrm{C} 30. 1 ∘ C , while SSJ–ATW (with cross-validated λ \lambda λ ) detects only 11.9 ∘ C 11.9^\circ\mathrm{C} 11. 9 ∘ C and 27.1 ∘ C 27.1^\circ\mathrm{C} 27. 1 ∘ C . Figure 4 shows that the two fits agree on the shape of the curve and on where it breaks, and differ mainly in how much warping they spend: 3.24 for SSJ–MLE and 1.62 for SSJ–ATW. The state-space form of SSJ-MLE also delivers standard errors.
Figure 4. Temperature and electricity demand, with the signal extracted at ξ = 0.30 \xi= 0.30 ξ = 0.30 by (a) SSJ–MLE, (b) SSJ–ATW with λ \lambda λ from 5-fold cross-validation. The band in (a) is ± 2 \pm2 ± 2 standard errors of the extracted signal, which the state-space form returns and the alternating algorithm does not. The bottom plots show γ t \gamma_t γ t for every active warp.
The discontinuities mark temperatures at which the physical system behind the load changes state. The large jump detected by both methods at 27.1 ∘ C 27.1^\circ\mathrm{C} 27. 1 ∘ C is associated with the temperature at which air conditioning is generally turned on. The break at 11.9 ∘ C 11.9^\circ\mathrm{C} 11. 9 ∘ C is an order of magnitude smaller than the one at 27 ∘ C 27^\circ\mathrm{C} 2 7 ∘ C and it has the opposite sign, demand falling as the temperature rises through it. A natural reading is the mirror image of the air-conditioning threshold: space heating in Lombardy is overwhelmingly gas-fired and so largely invisible in electricity demand, but the supplementary electric heating that is used in the shoulder season — reversible split units run as heat pumps, and electric convectors in commercial premises — is switched on below roughly this temperature. The same behavioral mechanism as at 27 ∘ C 27^\circ\mathrm{C} 2 7 ∘ C would then act on a much smaller share of the load, which is what the relative size of the two steps suggests. The discontinuity at 30.1 ∘ C 30.1\,^{\circ}\mathrm{C} 30.1 ∘ C , detected only by SSJ–MLE, appears to be a change in slope and could be due to many air conditioners already operating at their maximum power, thus, unable to respond to temperatures above 30.1 ∘ C 30.1\,^{\circ}\mathrm{C} 30.1 ∘ C .
None of these features can be expressed by the degree-day construct that energy systems conventionally use to summarize the temperature response, two straight lines joined at a base temperature fixed by convention (20 ∘ C 20\,^{\circ}\mathrm{C} 20 ∘ C in Italy). The fitted curve attains its minimum at 18.9 ∘ C 18.9\,^{\circ}\mathrm{C} 18.9 ∘ C but stays within 1% of it over the whole interval [ 16.0 , 23.4 ] ∘ C [16.0,\,23.4]\,^{\circ}\mathrm{C} [ 16.0 , 23.4 ] ∘ C , so there is no sharp vertex for the data to locate; its two arms are asymmetric, with about 270 MWh per degree of cold below 14 ∘ C 14\,^{\circ}\mathrm{C} 14 ∘ C against 320 per degree of heat above 24 ∘ C 24\,^{\circ}\mathrm{C} 24 ∘ C , which a single base temperature cannot serve; and, being continuous, a degree-day representation can only absorb the steps into a steeper slope. The estimator does not have to choose between a kink and a step.
We also applied the CSSD algorithm of Storath and Weinmann (2024) to the same data, but no discontinuities were detected, so the extracted signal was the same as that of a classical smoothing spline.
Retail fuel prices and excise-duty changes in Italy
The second application is a time series in which the location of the discontinuities is known in advance. This is unusual and valuable: change-point procedures are almost always assessed on simulated data, because in real series the truth is not observed. Retail fuel prices are an exception. The excise duty on motor fuels is a per-liter amount fixed by decree, it enters the pump price mechanically, and the date on which a new amount takes effect is a matter of public record, so a change in the duty is a discontinuity whose timing, although not its size, is known exactly.
The last years have provided two such interventions of unusual size. After the invasion of Ukraine the average price of gasoline rose by more than 0.23 euros per liter in the second week of March 2022, to an all-time high above 2.18 euros, and the government cut the excise duty by 0.25 euros per liter from 22 March 2022; the cut was reduced by 0.10 euros on 1 December 2022 and withdrawn on 1 January 2023. A second cut, of 0.20 euros per liter, entered the prices on 23 March 2026 and was likewise followed by a staggered restoration. In between, the duties on gasoline and diesel were realigned in two small steps until both reached 0.6729 euros per liter in January 2026. Table 4 lists the ten weeks of the sample in which the duty changed, dated at the first weekly survey in which the new amount appears.3 Seven dates affect both fuels and three only one, for seventeen events; the two realignment steps are an order of magnitude smaller than the emergency measures and are included in the ground truth without any expectation that they be detectable.
We use the weekly survey of the national average consumer prices of petroleum products published as open data by the Italian Ministry of Environment and Energy Security (MASE, https://sisen.mase.gov.it/dgsaie/open-data ), from 5 January 2015 to 27 July 2026, that is, n = 591 n = 591 n = 591 weeks, for gasoline and automotive diesel. For each fuel we smooth two series: the pump price, which a change in the duty moves by construction, and the industrial price, which excludes the excise duty and the value-added tax and therefore reacts to a fiscal measure only to the extent that the measure is not passed on.4 We treat the observations as equally spaced, x t = t x_t = t x t = t , so that γ t \gamma_t γ t is measured in weeks of fictitious time and so is the grid of budgets; a location is declared a discontinuity when γ ^ t > 1 \hat{\gamma}_t > 1 γ ^ t > 1 , that is, when the latent interval is more than twice the physical one, and a break is dated at the first observation of the new regime. Weekly prices leave serially dependent residuals, and their level behaves like a stochastic trend rather than a fixed curve, since crude oil, the exchange rate and refining margins accumulate shocks; λ \lambda λ is therefore estimated by maximum likelihood and, for SSJ–ATW, also by cross-validation on K = 20 K = 20 K = 20 contiguous blocks of about thirty weeks (Section 2.5 of the Application Supplement).
These are the two departures of the calibration, for which Table 1 gives values of ξ \xi ξ above the reference case. Because the dates of the interventions are known, the choice can also be checked on the data (see § 25.1 of the Application Supplement). The declared sets are nested in ξ \xi ξ , and at the reference-case value the estimators declare a long tail of dates that are not policy events: SSJ–ATW at the maximum likelihood λ \lambda λ reports 16 such weeks on the two pump-price series alongside 11 of the 17 events. Raising ξ \xi ξ to 0.45 removes much of the tail at little cost, leaving 10 spurious weeks and 9 events, and beyond 0.45 nothing more is gained. We therefore set ξ = 0.45 \xi= 0.45 ξ = 0.45 , the upper end of the values that Table 1 gives under the two departures, which a weekly price series combines; § 25.4 of the Application Supplement repeats the analysis at ξ = 0.25 \xi= 0.25 ξ = 0.25 , 0.35 and 0.50. Four of the seventeen events are recovered by no estimator at any ξ \xi ξ : the two realignment steps on either fuel, which move the duty by 1.5 and 4 cents against a weekly standard deviation of the pump price several times larger.
Figure 5 reports the four series and the signals extracted by SSJ–MLE, together with the warping sequences of SSJ–MLE and SSJ–ATW and the discontinuities declared by the two and by CSSD. The estimators are mutually consistent: on gasoline the three return the same dates on both price concepts, and on diesel they differ only by one or two locations at which SSJ–ATW is slightly more liberal. The discontinuities are few, and they sit on the policy dates. On the pump price the largest warp of the whole sample falls on 28 March 2022, the first survey after the emergency cut, with γ ^ t = 3.6 \hat{\gamma}_t = 3.6 γ ^ t = 3.6 for gasoline and 5.1 for diesel, and the second largest on 9 January 2023, the first survey after the duty was restored in full. The one further discontinuity that the estimators isolate on the pump price is not a fiscal event but the market shock of 14 March 2022, two weeks earlier, and it is separated cleanly from the tax change that followed it (Figure S3 of the Application Supplement magnifies the two episodes).
Figure S3. Zoom-in of the two excise-duty episodes. Rows: price paid at the pump and industrial price; columns: the two fuels. Points are the weekly observations and the blue line the signal extracted by SSJ–MLE; vertical rules as in Figure 5 of the paper, with the sideways nudge reduced in proportion to the horizontal scale. Panel (a): the cut of 28 March 2022, its partial withdrawal in December 2022, and its full restoration in January 2023. Panel (b): the cut of 23 March 2026 and its staggered restoration.
The estimated jumps are economically interpretable. A cut of ∣ Δ A ∣ |\Delta A| ∣Δ A ∣ in the duty lowers the pump price by ∣ Δ A ∣ ( 1 + φ ) |\Delta A|(1 + \varphi) ∣Δ A ∣ ( 1 + φ ) if it is passed on in full, so the 0.25 euro cut of March 2022 corresponds to a mechanical reduction of 0.305 euro per liter. The estimated discontinuity is −0.278 (SSJ–MLE), −0.267 (SSJ–ATW) and −0.267 (CSSD) for gasoline, and −0.284, −0.263 and −0.267 for diesel, which is close to nine tenths of the mechanical effect. The complement is visible in the other series: SSJ–MLE declares no discontinuity on the industrial price at that date, when the industrial price rose by only 0.031 euro per liter. The restoration of January 2023 gives estimates of +0.137 to +0.168 against a mechanical +0.183, so the transmission is of comparable size in the two directions. The 2026 episode inverts the pattern. The pump price barely reacts, falling by 0.040 euro per liter for gasoline and 0.010 for diesel against a mechanical 0.244, and no discontinuity is declared on it, while a large one appears on the industrial price, estimated at +0.157 to +0.164 for gasoline and +0.179 to +0.191 for diesel. Because the cut came on top of a violent market surge, the size of the pass-through is uncertain, but the conclusion that the second intervention reached the pump far less completely than the first is robust (§ 25.2 of the Application Supplement).
Finally, we remark that a change point in the middle of a long sample is an easy target: near the boundary the spline has almost no data on one side, and the roughness penalty alone must arbitrate between a discontinuity and a steep continuous transition. Because the date of the March 2022 cut is known, this can be put to the test by re-estimating the three procedures on 41 nested windows in which the edge of the sample is moved toward the cut, from the left and from the right for the cut of March 2022 and from the right for that of March 2026 (§ 25.3 of the Application Supplement). On the pump price both SSJ estimators recover the March 2022 cut in every window, from either side, and date it exactly; its estimated size stays at −0.267 euro per liter from the full sample down to the shortest window, so the boundary neither displaces nor attenuates the jump. CSSD loses the jump in one short window on each side, where it detects a move of about 0.03 to 0.05 euro per liter instead. On the industrial price, where the 2022 cut leaves no trace, the estimators find nothing at the policy date in any window, which doubles as a placebo check.
Conclusion
We have generalized the cubic smoothing spline to signals with an unknown, finite number of discontinuities by deforming the domain over which smoothness is measured, rather than by searching over discontinuity sets or penalizing coefficients. Because the mechanism only reparameterizes the metric of the classical roughness penalty, the model inherits the state-space form of smoothing splines, which gives maximum likelihood estimation via the Kalman filter (SSJ–MLE), and it admits a fast alternating algorithm (SSJ–ATW) that alternates a banded linear solve with a projected-gradient update of the warps, without maximizing a non-convex likelihood numerically.
The simulation study and the two applications point to a consistent division of labor with the CSSD benchmark of Storath and Weinmann (2024). SSJ–ATW and SSJ–MLE reconstruct the signal more accurately than CSSD and than a classical smoothing spline, and they detect more of the genuine breaks when the signal-to-noise ratio permits; CSSD declares fewer discontinuities and is nearly always right about those it declares. On the fuel prices, where the timing of the discontinuities is a matter of public record, the three estimators agree on the policy dates, and on the pump price the
Figure 5. Consumer prices of gasoline (left) and automotive diesel (right) in Italy. Rows one and three: the price paid at the pump (excise duty and VAT included) and the industrial price (taxes excluded); observations (gray lines) and the signal extracted by SSJ–MLE (blue line). Rows two and four: the warping sequences on a log ( 1 + γ ^ t ) \log(1+\hat{\gamma}_t) log ( 1 + γ ^ t ) scale of SSJ–MLE (left) and SSJ–ATW (right); the dashed rule is the threshold γ ^ t > 1 \hat{\gamma}_t > 1 γ ^ t > 1 above which a location is declared a discontinuity. Bottom row: the excise duty. Colored vertical rules mark the discontinuities found by each estimator, gray dotted rules the weeks in which a change in the excise duty entered the surveyed prices.
SSJ estimators recover the March 2022 cut, with a stable size, in every boundary window, while CSSD occasionally loses it near the edge of the sample.
Several limitations suggest directions for future work. The optimization over γ \gamma γ is non-convex for both SSJ–MLE and SSJ–ATW, so neither is guaranteed to find a global optimum; starting values close to γ = 0 \gamma= 0 γ = 0 work well in our experiments but are not backed by a formal convergence guarantee. The budget is selected by an extended BIC whose constant is calibrated by simulation and depends on the estimator and on the character of the data; a joint, likelihood-based selection of ( σ , λ , M ) (\sigma, \lambda, M) ( σ , λ , M ) remains open. The declaration of a discontinuity is a threshold rule on γ ^ t \hat{\gamma}_t γ ^ t ; a sampling distribution for γ ^ \hat{\gamma} γ ^ , and hence a formal test for a jump at a given location, would put this step on firmer ground and is a natural extension of the analytical scores of the Computational Supplement. Finally, the domain-warping idea is not tied to a one-dimensional domain or to a Gaussian signal: extensions to surfaces, in the spirit of the weak plate model of Blake and Zisserman (1987), and to non-Gaussian observation models within the same state-space machinery are directions we leave for future research.
Supplementary material
Computational Supplement: Reduced form of the roughness energy of an interval and its gradient, assembly and block-tridiagonal solution of the spline step, projection onto the feasible set of the warps, analytical scores of the Kalman-filter log-likelihood, and the treatment of repeated values of the covariate.
Calibration Supplement: Tuning of the smoothing constant and of the warping budget, and design, criteria and complete results of the calibration of the extended-BIC constant ξ \xi ξ .
Application Supplement: Additional results for the two applications: the temperature–demand relation over the grid of ξ \xi ξ and the comparison with degree days; for the fuel prices, the check of the calibration on the policy dates, the 2026 episode, the boundary experiment, and the sensitivity to ξ \xi ξ and to the number of cross-validation folds.
R package jumps: Implementation of SSJ–MLE and SSJ–ATW, version 2.0.0 or later, available from CRAN (Pelagatti and Maranzano, 2026).
Replication package: Code that reproduces every table and figure of the paper and of the supplements, available from the authors upon request. The data are publicly available from the sources given in Section 4.
supplement-A-computations.pdf
This document collects the derivations and the implementation details of the two estimators of Section 2 of the paper. Section A1 derives the reduced form of the roughness energy of one interval and its derivative with respect to the domain warp, on which the warping step of the alternating algorithm (SSJ–ATW) rests. Section A2 shows how the spline step is assembled and solved in O ( n ) O(n) O ( n ) operations, and Section A3 how the warps are projected onto their feasible set. Section A4 derives the analytical scores of the log-likelihood of the state-space form, which make the maximum likelihood estimator (SSJ–MLE) efficient, and Section A5 explains how repeated values of the covariate are treated. Equation numbers without the prefix A refer to the paper.
The roughness energy of one interval
With v i = ( f i , f i ′ , f i + 1 , f i + 1 ′ ) T \mathbf{v}_i=(f_i,f_i',f_{i+1},f_{i+1}')^{\mathsf{T}} v i = ( f i , f i ′ , f i + 1 , f i + 1 ′ ) T and the local stiffness matrix
K i = 1 τ i 3 [ 12 6 τ i − 12 6 τ i 6 τ i 4 τ i 2 − 6 τ i 2 τ i 2 − 12 − 6 τ i 12 − 6 τ i 6 τ i 2 τ i 2 − 6 τ i 4 τ i 2 ] \mathbf{K}_i=\frac{1}{\tau_i^3}
\begin{bmatrix}
12 & 6\tau_i & -12 & 6\tau_i \\
6\tau_i & 4\tau_i^2 & -6\tau_i & 2\tau_i^2 \\
-12 & -6\tau_i & 12 & -6\tau_i \\
6\tau_i & 2\tau_i^2 & -6\tau_i & 4\tau_i^2
\end{bmatrix} K i = τ i 3 1 12 6 τ i − 12 6 τ i 6 τ i 4 τ i 2 − 6 τ i 2 τ i 2 − 12 − 6 τ i 12 − 6 τ i 6 τ i 2 τ i 2 − 6 τ i 4 τ i 2
the roughness energy of the i i i -th interval is k i = v i T K i v i k_i=\mathbf{v}_i^{\mathsf{T}}\mathbf{K}_i\mathbf{v}_i k i = v i T K i v i ((19) of the paper). Expanding the quadratic form gives
k i = 1 τ i 3 [ 12 ( f i 2 − 2 f i f i + 1 + f i + 1 2 ) + 12 τ i ( f i f i ′ + f i f i + 1 ′ − f i + 1 f i ′ − f i + 1 f i + 1 ′ ) + 4 τ i 2 ( ( f i ′ ) 2 + f i ′ f i + 1 ′ + ( f i + 1 ′ ) 2 ) ] . k_i=\frac{1}{\tau_i^3}\left[12(f_i^2-2f_if_{i+1}+f_{i+1}^2)+12\tau_i(f_if_i'+f_if_{i+1}'-f_{i+1}f_i'-f_{i+1}f_{i+1}')+4\tau_i^2\left((f_i')^2+f_i'f_{i+1}'+(f_{i+1}')^2\right)\right]. k i = τ i 3 1 [ 12 ( f i 2 − 2 f i f i + 1 + f i + 1 2 ) + 12 τ i ( f i f i ′ + f i f i + 1 ′ − f i + 1 f i ′ − f i + 1 f i + 1 ′ ) + 4 τ i 2 ( ( f i ′ ) 2 + f i ′ f i + 1 ′ + ( f i + 1 ′ ) 2 ) ] .
The three brackets are respectively ( f i + 1 − f i ) 2 = Δ i 2 (f_{i+1}-f_i)^2=\Delta_i^2 ( f i + 1 − f i ) 2 = Δ i 2 , ( f i − f i + 1 ) ( f i ′ + f i + 1 ′ ) = − Δ i S i (f_i-f_{i+1})(f_i'+f_{i+1}')=-\Delta_iS_i ( f i − f i + 1 ) ( f i ′ + f i + 1 ′ ) = − Δ i S i and P i P_i P i , where
Δ i = f i + 1 − f i , S i = f i ′ + f i + 1 ′ , P i = ( f i ′ ) 2 + f i ′ f i + 1 ′ + ( f i + 1 ′ ) 2 , (6) \Delta_i=f_{i+1}-f_i,\qquad S_i=f_i'+f_{i+1}',\qquad P_i=(f_i')^2+f_i'f_{i+1}'+(f_{i+1}')^2, \tag*{(6)} Δ i = f i + 1 − f i , S i = f i ′ + f i + 1 ′ , P i = ( f i ′ ) 2 + f i ′ f i + 1 ′ + ( f i + 1 ′ ) 2 , (6)
so that the energy of the interval admits the reduced form
k i = 12 Δ i 2 τ i 3 − 12 Δ i S i τ i 2 + 4 P i τ i . (7) k_i=\frac{12\Delta_i^2}{\tau_i^3}-\frac{12\Delta_iS_i}{\tau_i^2}+\frac{4P_i}{\tau_i}. \tag*{(7)} k i = τ i 3 12 Δ i 2 − τ i 2 12 Δ i S i + τ i 4 P i . (7)
Since Δ i \Delta_i Δ i , S i S_i S i and P i P_i P i depend on the nodal values only, k i k_i k i is a Laurent polynomial in τ i \tau_i τ i whose coefficients do not depend on τ i \tau_i τ i . Differentiating it term by term and using ∂ τ i / ∂ γ i = 1 \partial\tau_i/\partial\gamma_i=1 ∂ τ i / ∂ γ i = 1 gives ∇ i \nabla_i ∇ i at once. Notice that the derivative is taken at z z z fixed, which is what the γ \gamma γ -step of the block coordinate descent requires, and it is the derivative of the roughness energy alone: the corresponding derivative of the objective function is λ ∇ i \lambda\nabla_i λ ∇ i , so that λ \lambda λ is absorbed into the learning rate η \eta η .
Gradient with respect to the warps. Since Δ i \Delta_i Δ i , S i S_i S i and P i P_i P i depend on the nodal values only, k i k_i k i is a Laurent polynomial in τ i \tau_i τ i whose coefficients do not depend on τ i \tau_i τ i . Differentiating it term by term and using ∂ τ i / ∂ γ i = 1 \partial\tau_i/\partial\gamma_i=1 ∂ τ i / ∂ γ i = 1 gives
∇ i = ∂ k i ∂ γ i = − 36 Δ i 2 τ i 4 + 24 Δ i S i τ i 3 − 4 P i τ i 2 . (8) \nabla_i=\frac{\partial k_i}{\partial\gamma_i}=-36\frac{\Delta_i^2}{\tau_i^4}+24\frac{\Delta_iS_i}{\tau_i^3}-4\frac{P_i}{\tau_i^2}. \tag*{(8)} ∇ i = ∂ γ i ∂ k i = − 36 τ i 4 Δ i 2 + 24 τ i 3 Δ i S i − 4 τ i 2 P i . (8)
which is Equation (4) of the paper. The derivative is taken at z z z fixed, which is what the warping step of the block coordinate descent requires, and it is the derivative of the roughness energy alone: the corresponding derivative of the objective function (1) of the paper is λ ∇ i \lambda\nabla_i λ ∇ i , so that λ \lambda λ is absorbed into the learning rate η \eta η .
The energy of a short interval. Section 2.1 of the paper states that the energy of an interval is dominated, for short intervals, by a term proportional to Δ i 2 / τ i 3 \Delta_i^2/\tau_i^3 Δ i 2 / τ i 3 , by analogy with the bending energy of an elastic beam. Two precise statements lie behind it, and both follow from Equation (7) , with Δ i \Delta_i Δ i the height difference that the fitted curve actually covers on the interval. If the segment is required to arrive and to leave with zero slope, then k i = 12 Δ i 2 / τ i 3 k_i = 12\Delta_i^2/\tau_i^3 k i = 12 Δ i 2 / τ i 3 exactly, which is the classical stiffness of a beam clamped at both ends. For arbitrary but fixed endpoint derivatives and Δ i ≠ 0 \Delta_i \ne0 Δ i = 0 , the remaining two terms of k i k_i k i are of order τ i − 2 \tau_i^{-2} τ i − 2 and τ i − 1 \tau_i^{-1} τ i − 1 , so that k i = 12 Δ i 2 τ i − 3 { 1 + O ( τ i ) } k_i = 12\Delta_i^2\tau_i^{-3}\{1+O(\tau_i)\} k i = 12 Δ i 2 τ i − 3 { 1 + O ( τ i )} as τ i → 0 \tau_i \to0 τ i → 0 . The proportionality describes how the energy grows as the interval shrinks and is not a lower bound over all cubics through the two points: the minimum of k i k_i k i over the endpoint derivatives is zero, attained by the straight line.
Units of the smoothing constant. The reduced form also shows that the smoothing constant is not scale-free. Each term of Equation (7) has the units of y 2 / x 3 y^2/x^3 y 2 / x 3 and the fidelity term of the objective function has the units of y 2 y^2 y 2 , so λ \lambda λ carries the units of x 3 x^3 x 3 . This is why the grids of λ \lambda λ are specified relative to the cubed range of the abscissa (Calibration Supplement), and it is the same fact that lets λ \lambda λ be absorbed into the learning rate of the warping step.
The spline step
In the spline step γ \gamma γ is fixed and the objective function is minimized with respect to the state vector z = ( f 1 , f 1 ′ , f 2 , f 2 ′ , … , f n , f n ′ ) T z = (f_1, f'_1, f_2, f'_2, \ldots, f_n, f'_n)^\mathsf{T} z = ( f 1 , f 1 ′ , f 2 , f 2 ′ , … , f n , f n ′ ) T of size 2 n 2n 2 n , which contains the values and the first derivatives of the function at the observations. Following the classical approach that goes back to Strang and Fix (1973), let y ∗ y^* y ∗ be the vector of size 2 n 2n 2 n with the y y y values in the odd positions and zeros in the even positions, and let S S S be the 2 n × 2 n 2n \times2n 2 n × 2 n diagonal selection matrix with diagonal ( 1 , 0 , 1 , 0 , … , 1 , 0 ) (1,0,1,0,\ldots,1,0) ( 1 , 0 , 1 , 0 , … , 1 , 0 ) : applied to z z z , it retains the function values and zeros out the derivatives, as the data inform f f f but not f ′ f' f ′ .
Assembly of the penalty matrix. The roughness penalty matrix R R R is a sparse matrix assembled from the local stiffness matrices K i K_i K i of Section A1. Since the elements ( f i , f i ′ ) (f_i,f'_i) ( f i , f i ′ ) of z z z , i = 2 , … , n − 1 i=2,\ldots,n-1 i = 2 , … , n − 1 , play the role of both the ending point of interval i − 1 i-1 i − 1 and the starting point of interval i i i , R R R is built as follows.
Set all the elements of the 2 n × 2 n 2n \times2n 2 n × 2 n matrix R R R equal to zero.
For i = 1 , 2 , … , n − 1 i=1,2,\ldots,n-1 i = 1 , 2 , … , n − 1 , set
R [ ( 2 i − 1 ) : ( 2 i + 2 ) , ( 2 i − 1 ) : ( 2 i + 2 ) ] = R [ ( 2 i − 1 ) : ( 2 i + 2 ) , ( 2 i − 1 ) : ( 2 i + 2 ) ] + K i , R[(2i-1):(2i+2),(2i-1):(2i+2)] =
R[(2i-1):(2i+2),(2i-1):(2i+2)] + K_i, R [( 2 i − 1 ) : ( 2 i + 2 ) , ( 2 i − 1 ) : ( 2 i + 2 )] = R [( 2 i − 1 ) : ( 2 i + 2 ) , ( 2 i − 1 ) : ( 2 i + 2 )] + K i ,
where R [ a : b , c : d ] R[a:b,c:d] R [ a : b , c : d ] refers to the submatrix of R R R with (integer) row indices ranging from a a a to $b and column indices from c c c to d d d .
The matrix R R R is symmetric and block tridiagonal. The objective function can then be written as
J ( z , γ ) = ( S y ∗ − S z ) T ( S y ∗ − S z ) + λ z T R z , J(z,\gamma) = (Sy^* - Sz)^{\mathsf{T}}(Sy^* - Sz) + \lambda z^{\mathsf{T}}Rz, J ( z , γ ) = ( S y ∗ − S z ) T ( S y ∗ − S z ) + λ z T R z ,
which is minimized by the solution of the linear system
( S + λ R ) z = y ∗ . (9) (S+\lambda R)z = y^*. \tag*{(9)} ( S + λ R ) z = y ∗ . (9)
The output of the step is the updated f f f (odd elements of z z z ) and f ′ f' f ′ (even elements of z z z ).
Block-tridiagonal Cholesky decomposition
The matrix A = S + λ R A = S + \lambda R A = S + λ R has a block-tridiagonal structure with 2 × 2 2 \times2 2 × 2 blocks,
A = [ D 1 U 1 0 ⋯ U 1 T D 2 U 2 ⋯ 0 U 2 T D 3 ⋯ ⋮ ⋮ ⋮ ⋱ ] A = \begin{bmatrix}
D_1 & U_1 & 0 & \cdots\\
U_1^{\mathsf{T}} & D_2 & U_2 & \cdots\\
0 & U_2^{\mathsf{T}} & D_3 & \cdots\\
\vdots& \vdots& \vdots& \ddots
\end{bmatrix} A = D 1 U 1 T 0 ⋮ U 1 D 2 U 2 T ⋮ 0 U 2 D 3 ⋮ ⋯ ⋯ ⋯ ⋱
so that, partitioning z z z and y ∗ y^* y ∗ conformably into the 2 × 1 2 \times1 2 × 1 blocks z i = ( f i , f i ′ ) T z_i = (f_i, f_i')^{\mathsf{T}} z i = ( f i , f i ′ ) T and y i = ( y i , 0 ) T y_i = (y_i, 0)^{\mathsf{T}} y i = ( y i , 0 ) T , system (A4) is a chain of small systems, each linking a block to its two neighbors. Due to this chain structure, it can be solved in two linear passes, one forward and one backward (see Section 4.5.1 of Golub and Van Loan (2013)); this is essentially Gaussian elimination organized by blocks, the idea exploited by Storath and Weinmann (2024) and that goes back to de Boor (1978).
Forward step. Set D ~ 1 = D 1 \widetilde{D}_1 = D_1 D 1 = D 1 and y ~ 1 = y 1 \widetilde{y}_1 = y_1 y 1 = y 1 and, for i = 2 , … , n i = 2,\ldots,n i = 2 , … , n , compute the modified diagonal blocks, which are Schur complements, and the modified right-hand sides
D ~ i = D i − U i − 1 T D ~ i − 1 − 1 U i − 1 , y ~ i = y i − U i − 1 T D ~ i − 1 − 1 y ~ i − 1 . \widetilde{D}_i = D_i - U_{i-1}^{\mathsf{T}}\widetilde{D}_{i-1}^{-1}U_{i-1}, \qquad\widetilde{y}_i = y_i - U_{i-1}^{\mathsf{T}}\widetilde{D}_{i-1}^{-1}\widetilde{y}_{i-1}. D i = D i − U i − 1 T D i − 1 − 1 U i − 1 , y i = y i − U i − 1 T D i − 1 − 1 y i − 1 .
This operation transforms the system into a block upper-triangular one, which can be solved block by block.
Backward step. Set z n = D ~ n − 1 y ~ n z_n = \widetilde{D}_n^{-1}\widetilde{y}_n z n = D n − 1 y n and, for i = n − 1 , … , 1 i = n-1,\ldots,1 i = n − 1 , … , 1 , compute
z ~ i = D ~ i − 1 ( y ~ i − U i z i + 1 ) . \widetilde{z}_i = \widetilde{D}_i^{-1}(\widetilde{y}_i - U_i z_{i+1}). z i = D i − 1 ( y i − U i z i + 1 ) .
Each operation involves 2 × 2 2 \times2 2 × 2 matrices only, so the whole solution costs O ( n ) O(n) O ( n ) operations.
Projection onto the simplex
In the warping step f f f and f ′ f' f ′ are fixed, and the warps are first moved by a gradient step, γ t e m p = γ c u r r e n t − η ∇ \gamma_{\mathrm{temp}} = \gamma_{\mathrm{current}} - \eta\nabla γ temp = γ current − η ∇ , with ∇ = ( ∇ 1 , … , ∇ n − 1 ) T \nabla= (\nabla_1,\ldots,\nabla_{n-1})^{\mathsf{T}} ∇ = ( ∇ 1 , … , ∇ n − 1 ) T given by Equation (8) and learning rate η \eta η . To ensure that the constraints γ n e w ≥ 0 \gamma_{\mathrm{new}} \ge0 γ new ≥ 0 and ∑ i γ n e w , i ≤ M \sum_i \gamma_{\mathrm{new},i} \le M ∑ i γ new , i ≤ M hold, the result is then projected onto the simplex by the following algorithm, due to Duchi et al. (2008).
Let u = γ t e m p \boldsymbol{u} = \gamma_{\mathrm{temp}} u = γ temp and set any negative components to 0.
If ∑ u i ≤ M \sum u_i \le M ∑ u i ≤ M , return u \boldsymbol{u} u .
Sort u \boldsymbol{u} u in descending order: u 1 ≥ u 2 ≥ … u_1 \ge u_2 \ge\ldots u 1 ≥ u 2 ≥ … .
Compute the cumulative sum of the sorted u \boldsymbol{u} u : w k = ∑ j = 1 k u j w_k = \sum_{j=1}^{k} u_j w k = ∑ j = 1 k u j .
Find the index ρ \rho ρ that is the largest k k k such that:
u k − w k − M k > 0 u_k - \frac{w_k - M}{k} > 0 u k − k w k − M > 0
Calculate the threshold θ \theta θ :
θ = w ρ − M ρ \theta= \frac{w_\rho- M}{\rho} θ = ρ w ρ − M
Final update: γ i = max ( u i − θ , 0 ) \gamma_i = \max(u_i - \theta, 0) γ i = max ( u i − θ , 0 ) for i = 1 , … , n − 1 i = 1,\ldots,n-1 i = 1 , … , n − 1 .
The cost is dominated by the sort, O ( n log n ) O(n \log n) O ( n log n ) operations. The spline step and the warping step are repeated until the change in the variables falls below a tolerance level.
Analytical scores of the Kalman-filter log-likelihood
Since model (2) of the paper is estimated by exploiting the Kalman filter, in this section we provide the analytical scores of the (penalized) log-likelihood ℓ \ell ℓ required to make the computations both efficient and precise. We also show that, unlike σ \sigma σ , the domain-warp parameters γ t \gamma_t γ t require an additional score term beyond the standard formula, because they affect the transition matrix of the state-space form and not only its disturbance covariances.
Scores with respect to the log-likelihood One can, in fact, adopt the solution presented in Koopman and Shephard (1992, result 3.2). By adapting it to our scenario, given a generic component θ \theta θ of the parameter vector θ \boldsymbol{\theta} θ we can write
∂ ℓ ( θ ) ∂ θ = 1 2 ∑ t = 1 n tr ( ( r t r t T − N t ) ∂ Q t ∂ θ ) + ( ε t 2 − d t ) ∂ H t ∂ θ (10) \frac{\partial\ell(\boldsymbol{\theta})}{\partial\theta}
=
\frac{1}{2}\sum_{t=1}^{n}
\operatorname{tr}\left((\mathbf{r}_t\mathbf{r}_t^{\mathsf{T}}-\mathbf{N}_t)\frac{\partial\mathbf{Q}_t}{\partial\theta}\right)
+
(\varepsilon_t^2-d_t)\frac{\partial H_t}{\partial\theta} \tag*{(10)} ∂ θ ∂ ℓ ( θ ) = 2 1 t = 1 ∑ n tr ( ( r t r t T − N t ) ∂ θ ∂ Q t ) + ( ε t 2 − d t ) ∂ θ ∂ H t (10)
where H t \mathbf{H}_t H t is the variance matrix of the observation error, and Q t \mathbf{Q}_t Q t is the variance matrix of the state error. Formula (A5) is valid for any θ \theta θ that enters H t \mathbf{H}_t H t and/or Q t \mathbf{Q}_t Q t only, leaving the system matrices that drive the state recursion unaffected; we will return to this restriction below.
In our case, the parameter vector is defined as
θ : = [ σ γ 1 ⋯ γ n − 1 ] T , \boldsymbol{\theta}:=[\sigma\ \ \gamma_1\ \ \cdots\ \ \gamma_{n-1}]^{\mathsf{T}}, θ := [ σ γ 1 ⋯ γ n − 1 ] T ,
and we write the state and state-disturbance vectors as α t : = [ f ( x t ) f ′ ( x t ) ] T \boldsymbol{\alpha}_t:=[f(x_t)\ \ f'(x_t)]^{\mathsf{T}} α t := [ f ( x t ) f ′ ( x t ) ] T and η t : = [ η t ζ t ] T \boldsymbol{\eta}_t:=[\eta_t\ \ \zeta_t]^{\mathsf{T}} η t := [ η t ζ t ] T , so that the state equation in (2) of the paper reads α t + 1 = T t α t + η t \boldsymbol{\alpha}_{t+1}=\mathbf{T}_t\boldsymbol{\alpha}_t+\boldsymbol{\eta}_t α t + 1 = T t α t + η t with observation matrix Z : = [ 1 0 ] \mathbf{Z}:=[1\ \ 0] Z := [ 1 0 ] and transition matrix
T t : = [ 1 τ t 0 1 ] , τ t : = δ t + γ t . \mathbf{T}_t:=
\begin{bmatrix}
1 & \tau_t\\
0 & 1
\end{bmatrix},
\qquad
\tau_t:=\delta_t+\gamma_t. T t := [ 1 0 τ t 1 ] , τ t := δ t + γ t .
The derivatives of the disturbance variance matrices
H t = λ σ 2 , Q t = σ 2 Δ t , Δ t : = [ 1 3 τ t 3 1 2 τ t 2 1 2 τ t 2 τ t ] H_t=\lambda\sigma^2,
\qquad
\mathbf{Q}_t=\sigma^2\boldsymbol{\Delta}_t,
\qquad
\boldsymbol{\Delta}_t:=
\begin{bmatrix}
\frac{1}{3}\tau_t^3 & \frac{1}{2}\tau_t^2\\
\frac{1}{2}\tau_t^2 & \tau_t
\end{bmatrix} H t = λ σ 2 , Q t = σ 2 Δ t , Δ t := [ 3 1 τ t 3 2 1 τ t 2 2 1 τ t 2 τ t ]
with respect to θ \boldsymbol{\theta} θ ’s components can be computed promptly: for all t t t and s ≠ t s\ne t s = t
∂ H t ∂ σ = 2 λ σ , ∂ H t ∂ γ t = 0 , ∂ H s ∂ γ t = 0 \frac{\partial H_t}{\partial\sigma}=2\lambda\sigma,
\qquad
\frac{\partial H_t}{\partial\gamma_t}=0,
\qquad
\frac{\partial H_s}{\partial\gamma_t}=0 ∂ σ ∂ H t = 2 λσ , ∂ γ t ∂ H t = 0 , ∂ γ t ∂ H s = 0
and
∂ Q t ∂ σ = 2 σ Δ t , ∂ Q t ∂ γ t = σ 2 [ τ t 2 τ t τ t 1 ] , ∂ Q s ∂ γ t = 0. \frac{\partial\mathbf{Q}_t}{\partial\sigma}=2\sigma\boldsymbol{\Delta}_t,
\qquad
\frac{\partial\mathbf{Q}_t}{\partial\gamma_t}
=
\sigma^2
\begin{bmatrix}
\tau_t^2 & \tau_t\\
\tau_t & 1
\end{bmatrix},
\qquad
\frac{\partial\mathbf{Q}_s}{\partial\gamma_t}=\mathbf{0}. ∂ σ ∂ Q t = 2 σ Δ t , ∂ γ t ∂ Q t = σ 2 [ τ t 2 τ t τ t 1 ] , ∂ γ t ∂ Q s = 0 .
Since σ \sigma σ enters only H t \mathbf{H}_t H t and Q t \mathbf{Q}_t Q t , formula (A5) applies to it unchanged. However γ t \gamma_t γ t , through τ t \tau_t τ t , also enters the transition matrix T t \mathbf{T}_t T t :
∂ T t ∂ γ t = T ˙ : = [ 0 1 0 0 ] , ∂ T s ∂ γ t = 0 ( s ≠ t ) , \frac{\partial\mathbf{T}_t}{\partial\gamma_t}
=
\dot{\mathbf{T}}
:=
\begin{bmatrix}
0 & 1\\
0 & 0
\end{bmatrix},
\qquad
\frac{\partial\mathbf{T}_s}{\partial\gamma_t}=\mathbf{0}
\quad(s\ne t), ∂ γ t ∂ T t = T ˙ := [ 0 0 1 0 ] , ∂ γ t ∂ T s = 0 ( s = t ) ,
so formula (A5) by itself is incomplete for γ t \gamma_t γ t : it must be supplemented with an extra term accounting for ∂ T t / ∂ γ t \partial\mathbf{T}_t/\partial\gamma_t ∂ T t / ∂ γ t .
The additional score term induced by the transition matrix The Kalman filter likelihood is the marginal likelihood of y \mathbf{y} y obtained after integrating out the latent states α : = ( α 1 , … , α n ) \boldsymbol{\alpha}:=(\boldsymbol{\alpha}_1,\ldots,\boldsymbol{\alpha}_n) α := ( α 1 , … , α n ) from the complete-data log-likelihood
log p ( α , y ; θ ) = − 1 2 ∑ t = 1 n [ log ∣ H t ∣ + ε t 2 H t ] − 1 2 ∑ t = 1 n − 1 [ log ∣ Q t ∣ + η t T Q t − 1 η t ] + const , \log p(\boldsymbol{\alpha},\mathbf{y};\boldsymbol{\theta})
=
-\frac{1}{2}\sum_{t=1}^{n}
\left[\log|H_t|+\frac{\varepsilon_t^2}{H_t}\right]
-\frac{1}{2}\sum_{t=1}^{n-1}
\left[\log|\mathbf{Q}_t|+\boldsymbol{\eta}_t^{\mathsf{T}}\mathbf{Q}_t^{-1}\boldsymbol{\eta}_t\right]
+\operatorname{const}, log p ( α , y ; θ ) = − 2 1 t = 1 ∑ n [ log ∣ H t ∣ + H t ε t 2 ] − 2 1 t = 1 ∑ n − 1 [ log ∣ Q t ∣ + η t T Q t − 1 η t ] + const ,
with ε t : = y t − Z α t \varepsilon_t := y_t - Z\alpha_t ε t := y t − Z α t and η t = α t + 1 − T t α t \eta_t = \alpha_{t+1} - T_t\alpha_t η t = α t + 1 − T t α t . By the Fisher (or Louis) identity that underlies all EM-type score computations,
∂ ℓ ( θ ) ∂ θ = E [ ∂ log p ( α , y ; θ ) ∂ θ ∣ y ] , \frac{\partial\ell(\theta)}{\partial\theta} = \mathbb{E}\left[\left.\frac{\partial\log p(\boldsymbol{\alpha},\boldsymbol{y};\theta)}{\partial\theta}\right| \boldsymbol{y}\right], ∂ θ ∂ ℓ ( θ ) = E [ ∂ θ ∂ log p ( α , y ; θ ) y ] ,
where the expectation is taken at the current value of θ \theta θ . Formula (A5) is precisely the result of applying this identity when θ \theta θ affects only H t , Q t H_t,Q_t H t , Q t . For θ = γ t \theta=\gamma_t θ = γ t , only the summand at index t t t depends on θ \theta θ , and expanding η t ⊤ Q t − 1 η t = α t + 1 ⊤ Q t − 1 α t + 1 − 2 α t + 1 ⊤ Q t − 1 T t α t + α t ⊤ T t ⊤ Q t − 1 T t α t \eta_t^\top Q_t^{-1}\eta_t = \alpha_{t+1}^\top Q_t^{-1}\alpha_{t+1} - 2\alpha_{t+1}^\top Q_t^{-1}T_t\alpha_t + \alpha_t^\top T_t^\top Q_t^{-1}T_t\alpha_t η t ⊤ Q t − 1 η t = α t + 1 ⊤ Q t − 1 α t + 1 − 2 α t + 1 ⊤ Q t − 1 T t α t + α t ⊤ T t ⊤ Q t − 1 T t α t and differentiating with respect to γ t \gamma_t γ t before taking the conditional expectation yields, after simplification,
∂ ℓ ( θ ) ∂ γ t = 1 2 tr ( ( r t r t ⊤ − N t ) ∂ Q t ∂ γ t ) + tr ( Q t − 1 T ˙ M t ) , (11) \frac{\partial\ell(\theta)}{\partial\gamma_t}
= \frac{1}{2}\operatorname{tr}\left((r_t r_t^\top-N_t)\frac{\partial Q_t}{\partial\gamma_t}\right)
+\operatorname{tr}(Q_t^{-1}\dot{T}M_t), \tag*{(11)} ∂ γ t ∂ ℓ ( θ ) = 2 1 tr ( ( r t r t ⊤ − N t ) ∂ γ t ∂ Q t ) + tr ( Q t − 1 T ˙ M t ) , (11)
where M t : = E [ α t η t ⊤ ∣ y ] = P ~ t + 1 , t ⊤ − P ~ t T ~ ⊤ M_t := \mathbb{E}[\alpha_t\eta_t^\top\mid\boldsymbol{y}] = \widetilde{P}_{t+1,t}^\top-\widetilde{P}_t\widetilde{T}^\top M t := E [ α t η t ⊤ ∣ y ] = P t + 1 , t ⊤ − P t T ⊤ , P ~ t : = E [ α t α t ⊤ ∣ y ] \widetilde{P}_t := \mathbb{E}[\alpha_t\alpha_t^\top\mid\boldsymbol{y}] P t := E [ α t α t ⊤ ∣ y ] , P ~ t + 1 , t : = E [ α t + 1 α t ⊤ ∣ y ] \widetilde{P}_{t+1,t} := \mathbb{E}[\alpha_{t+1}\alpha_t^\top\mid\boldsymbol{y}] P t + 1 , t := E [ α t + 1 α t ⊤ ∣ y ] .
The second term in (A6) is exactly the contribution missed by treating γ t \gamma_t γ t as if it only entered Q t Q_t Q t .
To evaluate it in closed form we need the smoothed cross-moment M t M_t M t , equivalently the smoothed cross-covariance Cov ( α t , η t ∣ y ) = M t − α ^ t η ^ t ⊤ \operatorname{Cov}(\alpha_t,\eta_t\mid\boldsymbol{y}) = M_t-\widehat{\alpha}_t\widehat{\eta}_t^\top Cov ( α t , η t ∣ y ) = M t − α t η t ⊤ , where α ^ t : = a t + P t r t − 1 \widehat{\alpha}_t := a_t+P_t r_{t-1} α t := a t + P t r t − 1 and η ^ t : = Q t r t \widehat{\eta}_t := Q_t r_t η t := Q t r t are the usual smoothed state and smoothed disturbance (a t , P t a_t,P_t a t , P t being the one-step-ahead predicted mean and MSE matrix produced by the Kalman filter). Writing K t K_t K t for the Kalman gain and L t : = T t − K t Z L_t := T_t-K_tZ L t := T t − K t Z for the disturbance-smoothing matrix associated with the backward recursions for r t , N t r_t,N_t r t , N t , the smoothed cross-covariance admits the closed form
Cov ( α t , η t ∣ y ) = − P t L t ⊤ N t Q t , (12) \operatorname{Cov}(\alpha_t,\eta_t\mid\boldsymbol{y})=-P_tL_t^\top N_tQ_t, \tag*{(12)} Cov ( α t , η t ∣ y ) = − P t L t ⊤ N t Q t , (12)
which can be obtained from the disturbance- and state-smoothing recursions (cf. the lag-one smoothed state covariance of Shumway and Stoffer, 1982, here re-expressed without recourse to the filtered-variance sequence) and which we have additionally verified by direct computation against the exact, non-recursive Gaussian conditional covariance of the finite-sample model. Substituting M t = ( α ^ t r t ⊤ − P t L t ⊤ N t ) Q t M_t=(\widehat{\alpha}_t r_t^\top-P_tL_t^\top N_t)Q_t M t = ( α t r t ⊤ − P t L t ⊤ N t ) Q t —obtained from (A7) together with η ^ t = Q t r t \widehat{\eta}_t=Q_t r_t η t = Q t r t —into the second term of (A6), the factors Q t Q_t Q t and Q t − 1 Q_t^{-1} Q t − 1 cancel by the cyclic property of the trace:
tr ( Q t − 1 T ˙ M t ) = tr ( T ˙ ( α ^ t r t ⊤ − P t L t ⊤ N t ) ) = α ^ t ( 2 ) r t ( 1 ) − ( P t L t ⊤ N t ) 21 , \operatorname{tr}(Q_t^{-1}\dot{T}M_t)=\operatorname{tr}\left(\dot{T}(\widehat{\alpha}_t r_t^\top-P_tL_t^\top N_t)\right)=\widehat{\alpha}_t^{(2)}r_t^{(1)}-(P_tL_t^\top N_t)_{21}, tr ( Q t − 1 T ˙ M t ) = tr ( T ˙ ( α t r t ⊤ − P t L t ⊤ N t ) ) = α t ( 2 ) r t ( 1 ) − ( P t L t ⊤ N t ) 21 ,
the last equality following from T ˙ \dot{T} T ˙ having a single nonzero entry in position ( 1 , 2 ) (1,2) ( 1 , 2 ) , which selects the ( 2 , 1 ) (2,1) ( 2 , 1 ) entry of the matrix it multiplies. No new filter or smoother recursion is required: every quantity above (P t , L t , N t , r t , α ^ t P_t,L_t,N_t,r_t,\widehat{\alpha}_t P t , L t , N t , r t , α t ) is already produced by the Kalman filter and smoother used for ∂ ℓ / ∂ σ \partial\ell/\partial\sigma ∂ ℓ / ∂ σ .
The complete score for γ t \gamma_t γ t Granted the notation
r t : = [ r t ( 1 ) r t ( 2 ) ] , N t : = [ n t ( 11 ) n t ( 12 ) n t ( 12 ) n t ( 22 ) ] , a t : = [ a t ( 1 ) a t ( 2 ) ] , P t : = [ p t ( 11 ) p t ( 12 ) p t ( 12 ) p t ( 22 ) ] , r_t := \begin{bmatrix}r_t^{(1)}\\r_t^{(2)}\end{bmatrix},\quad
N_t := \begin{bmatrix}n_t^{(11)}&n_t^{(12)}\\n_t^{(12)}&n_t^{(22)}\end{bmatrix},\quad
a_t := \begin{bmatrix}a_t^{(1)}\\a_t^{(2)}\end{bmatrix},\quad
P_t := \begin{bmatrix}p_t^{(11)}&p_t^{(12)}\\p_t^{(12)}&p_t^{(22)}\end{bmatrix}, r t := [ r t ( 1 ) r t ( 2 ) ] , N t := [ n t ( 11 ) n t ( 12 ) n t ( 12 ) n t ( 22 ) ] , a t := [ a t ( 1 ) a t ( 2 ) ] , P t := [ p t ( 11 ) p t ( 12 ) p t ( 12 ) p t ( 22 ) ] ,
and writing k t ( 1 ) , k t ( 2 ) k_t^{(1)}, k_t^{(2)} k t ( 1 ) , k t ( 2 ) for the two components of the Kalman gain K t K_t K t (so that L t = [ 1 − k t ( 1 ) τ t − k t ( 2 ) 1 ] L_t=\begin{bmatrix}1-k_t^{(1)}&\tau_t\\-k_t^{(2)}&1\end{bmatrix} L t = [ 1 − k t ( 1 ) − k t ( 2 ) τ t 1 ] ), the score with respect to σ \sigma σ is unaffected by the above and remains as in (A5), while the complete score with respect to γ t \gamma_t γ t combines the term already given by (A5) with the new term derived above:
∂ ℓ ( θ ) ∂ σ = σ ∑ t = 1 n [ ( r t ( 1 ) r t ( 1 ) − n t ( 11 ) ) 1 3 τ t 3 + ( r t ( 1 ) r t ( 2 ) − n t ( 12 ) ) τ t 2 + ( r t ( 2 ) r t ( 2 ) − n t ( 22 ) ) τ t ] + λ σ ∑ t = 1 n ( ε t 2 − d t ) \frac{\partial\ell(\theta)}{\partial\sigma}
= \sigma\sum_{t=1}^{n} \left[ \left(r_t^{(1)}r_t^{(1)}-n_t^{(11)}\right)\frac{1}{3}\tau_t^3+\left(r_t^{(1)}r_t^{(2)}-n_t^{(12)}\right)\tau_t^2+\left(r_t^{(2)}r_t^{(2)}-n_t^{(22)}\right)\tau_t \right]+\lambda\sigma\sum_{t=1}^{n}\left(\varepsilon_t^2-d_t\right) ∂ σ ∂ ℓ ( θ ) = σ t = 1 ∑ n [ ( r t ( 1 ) r t ( 1 ) − n t ( 11 ) ) 3 1 τ t 3 + ( r t ( 1 ) r t ( 2 ) − n t ( 12 ) ) τ t 2 + ( r t ( 2 ) r t ( 2 ) − n t ( 22 ) ) τ t ] + λσ t = 1 ∑ n ( ε t 2 − d t )
∂ ℓ ( θ ) ∂ γ t = σ 2 2 [ ( r t ( 1 ) r t ( 1 ) − n t ( 11 ) ) τ t 2 + 2 ( r t ( 1 ) r t ( 2 ) − n t ( 12 ) ) τ t + ( r t ( 2 ) r t ( 2 ) − n t ( 22 ) ) ] + α ^ t ( 2 ) r t ( 1 ) − p t ( 12 ) [ ( 1 − k t ( 1 ) ) n t ( 11 ) − k t ( 2 ) n t ( 12 ) ] − p t ( 22 ) [ τ t n t ( 11 ) + n t ( 12 ) ] \frac{\partial\ell(\theta)}{\partial\gamma_t}
= \frac{\sigma^2}{2}\left[\left(r_t^{(1)}r_t^{(1)}-n_t^{(11)}\right)\tau_t^2+2\left(r_t^{(1)}r_t^{(2)}-n_t^{(12)}\right)\tau_t+\left(r_t^{(2)}r_t^{(2)}-n_t^{(22)}\right)\right]+\widehat{\alpha}_t^{(2)}r_t^{(1)}-p_t^{(12)}\left[\left(1-k_t^{(1)}\right)n_t^{(11)}-k_t^{(2)}n_t^{(12)}\right]-p_t^{(22)}\left[\tau_t n_t^{(11)}+n_t^{(12)}\right] ∂ γ t ∂ ℓ ( θ ) = 2 σ 2 [ ( r t ( 1 ) r t ( 1 ) − n t ( 11 ) ) τ t 2 + 2 ( r t ( 1 ) r t ( 2 ) − n t ( 12 ) ) τ t + ( r t ( 2 ) r t ( 2 ) − n t ( 22 ) ) ] + α t ( 2 ) r t ( 1 ) − p t ( 12 ) [ ( 1 − k t ( 1 ) ) n t ( 11 ) − k t ( 2 ) n t ( 12 ) ] − p t ( 22 ) [ τ t n t ( 11 ) + n t ( 12 ) ]
∀ t = 1 , … , n − 1 \forall t=1,\ldots,n-1 ∀ t = 1 , … , n − 1 ,
where α ^ t ( 2 ) = a t ( 2 ) + p t − 1 ( 12 ) r t − 1 ( 1 ) + p t ( 22 ) r t − 1 ( 2 ) \widehat{\alpha}_t^{(2)}=a_t^{(2)}+p_{t-1}^{(12)}r_{t-1}^{(1)}+p_t^{(22)}r_{t-1}^{(2)} α t ( 2 ) = a t ( 2 ) + p t − 1 ( 12 ) r t − 1 ( 1 ) + p t ( 22 ) r t − 1 ( 2 ) is the smoothed slope at x t x_t x t .
Scores with respect to the constraint function Since the constraint function is defined as
g ( θ ) : = ∑ t = 1 n − 1 γ t g(\theta):=\sum_{t=1}^{n-1}\gamma_t g ( θ ) := t = 1 ∑ n − 1 γ t
its derivatives with respect to θ \theta θ ’s components are trivial:
∂ g ( θ ) ∂ σ = 0 , ∂ g ( θ ) ∂ γ t = 1 ∀ t = 1 , … , n − 1 \frac{\partial g(\theta)}{\partial\sigma}=0,\qquad\frac{\partial g(\theta)}{\partial\gamma_t}=1\quad\forall t=1,\ldots,n-1 ∂ σ ∂ g ( θ ) = 0 , ∂ γ t ∂ g ( θ ) = 1 ∀ t = 1 , … , n − 1
With estimation of lambda By model parameterization we have E [ ε t ε t T ] : = λ σ 2 \mathbb{E}[\varepsilon_t\varepsilon_t^\mathsf{T}]:=\lambda\sigma^2 E [ ε t ε t T ] := λ σ 2 . We can also directly define the variance of the error term ε t \varepsilon_t ε t :
σ ε 2 : = E [ ε t ε t T ] \sigma_{\varepsilon}^{2}:=\mathbb{E}[\varepsilon_t\varepsilon_t^\mathsf{T}] σ ε 2 := E [ ε t ε t T ]
This allows us to define the vector of parameters to be estimated as
θ : = [ σ σ ε γ 1 ⋯ γ n − 1 ] T , \theta:=\left[\sigma\ \ \sigma_{\varepsilon}\ \ \gamma_1\ \ \cdots\ \ \gamma_{n-1}\right]^\mathsf{T}, θ := [ σ σ ε γ 1 ⋯ γ n − 1 ] T ,
This means that one can now carry out the estimation without a fixed value for λ \lambda λ . Such estimation returns the estimate for σ ε \sigma_{\varepsilon} σ ε and then (if necessary) one can estimate λ \lambda λ by
λ ^ = σ ^ ε 2 σ ^ 2 \widehat{\lambda}=\frac{\widehat{\sigma}_{\varepsilon}^{2}}{\widehat{\sigma}^{2}} λ = σ 2 σ ε 2
In order to perform the estimation in question, the derivatives of the disturbance variance matrices must be computed. The only change affecting these matrices with respect to the previous case regards H t \boldsymbol{H}_t H t , which is now defined as
H t = σ ε 2 \boldsymbol{H}_t=\sigma_{\varepsilon}^{2} H t = σ ε 2
Consequently, the derivatives are almost unaffected: for all t t t and s ≠ t s\ne t s = t
∂ H t ∂ σ = 0 , ∂ H t ∂ σ ε = 2 σ ε , ∂ H t ∂ γ t = 0 , ∂ H s ∂ γ t = 0 \frac{\partial\boldsymbol{H}_t}{\partial\sigma}=0,\qquad\frac{\partial\boldsymbol{H}_t}{\partial\sigma_{\varepsilon}}=2\sigma_{\varepsilon},\qquad\frac{\partial\boldsymbol{H}_t}{\partial\gamma_t}=0,\qquad\frac{\partial\boldsymbol{H}_s}{\partial\gamma_t}=0 ∂ σ ∂ H t = 0 , ∂ σ ε ∂ H t = 2 σ ε , ∂ γ t ∂ H t = 0 , ∂ γ t ∂ H s = 0
and
∂ Q t ∂ σ = 2 σ Δ t , ∂ Q t ∂ σ ε = 0 , ∂ Q t ∂ γ t = σ 2 [ τ t 2 τ t τ t 1 ] , ∂ Q s ∂ γ t = 0 \frac{\partial\boldsymbol{Q}_t}{\partial\sigma}=2\sigma\Delta_t,\qquad\frac{\partial\boldsymbol{Q}_t}{\partial\sigma_{\varepsilon}}=0,\qquad\frac{\partial\boldsymbol{Q}_t}{\partial\gamma_t}=\sigma^2\begin{bmatrix}\tau_t^2&\tau_t\\\tau_t&1\end{bmatrix},\qquad\frac{\partial\boldsymbol{Q}_s}{\partial\gamma_t}=0 ∂ σ ∂ Q t = 2 σ Δ t , ∂ σ ε ∂ Q t = 0 , ∂ γ t ∂ Q t = σ 2 [ τ t 2 τ t τ t 1 ] , ∂ γ t ∂ Q s = 0
As above, σ ε \sigma_{\varepsilon} σ ε enters only H t \mathbf{H}_t H t , so its score follows directly from (A5); γ t \gamma_t γ t still requires the additional term derived above, which is unaffected by the reparameterization of H t \mathbf{H}_t H t . The gradient turns out to be
∂ ℓ ( θ ) ∂ σ = σ ∑ t = 1 n [ ( r t ( 1 ) r t ( 1 ) − n t ( 11 ) ) 1 3 τ t 3 + ( r t ( 1 ) r t ( 2 ) − n t ( 12 ) ) τ t 2 + ( r t ( 2 ) r t ( 2 ) − n t ( 22 ) ) τ t ] \frac{\partial\ell(\boldsymbol{\theta})}{\partial\sigma}
= \sigma\sum_{t=1}^{n} \left[
\left(r_t^{(1)}r_t^{(1)}-n_t^{(11)}\right)\frac{1}{3}\tau_t^3
+\left(r_t^{(1)}r_t^{(2)}-n_t^{(12)}\right)\tau_t^2
+\left(r_t^{(2)}r_t^{(2)}-n_t^{(22)}\right)\tau_t
\right] ∂ σ ∂ ℓ ( θ ) = σ t = 1 ∑ n [ ( r t ( 1 ) r t ( 1 ) − n t ( 11 ) ) 3 1 τ t 3 + ( r t ( 1 ) r t ( 2 ) − n t ( 12 ) ) τ t 2 + ( r t ( 2 ) r t ( 2 ) − n t ( 22 ) ) τ t ]
∂ ℓ ( θ ) ∂ σ ε = σ ε ∑ t = 1 n ( e t 2 − d t ) \frac{\partial\ell(\boldsymbol{\theta})}{\partial\sigma_{\varepsilon}}
= \sigma_{\varepsilon}\sum_{t=1}^{n}\left(e_t^2-d_t\right) ∂ σ ε ∂ ℓ ( θ ) = σ ε t = 1 ∑ n ( e t 2 − d t )
∂ ℓ ( θ ) ∂ γ t = σ 2 2 [ ( r t ( 1 ) r t ( 1 ) − n t ( 11 ) ) τ t 2 + 2 ( r t ( 1 ) r t ( 2 ) − n t ( 12 ) ) τ t + ( r t ( 2 ) r t ( 2 ) − n t ( 22 ) ) ] + α ^ t ( 2 ) r t ( 1 ) − p t ( 12 ) [ ( 1 − k t ( 1 ) ) n t ( 11 ) − k t ( 2 ) n t ( 12 ) ] − p t ( 22 ) [ τ t n t ( 11 ) + n t ( 12 ) ] \frac{\partial\ell(\boldsymbol{\theta})}{\partial\gamma_t}
= \frac{\sigma^2}{2}\left[
\left(r_t^{(1)}r_t^{(1)}-n_t^{(11)}\right)\tau_t^2
+2\left(r_t^{(1)}r_t^{(2)}-n_t^{(12)}\right)\tau_t
+\left(r_t^{(2)}r_t^{(2)}-n_t^{(22)}\right)
\right]
+\hat{\alpha}_t^{(2)}r_t^{(1)}
-p_t^{(12)}\left[(1-k_t^{(1)})n_t^{(11)}-k_t^{(2)}n_t^{(12)}\right]
-p_t^{(22)}\left[\tau_t n_t^{(11)}+n_t^{(12)}\right] ∂ γ t ∂ ℓ ( θ ) = 2 σ 2 [ ( r t ( 1 ) r t ( 1 ) − n t ( 11 ) ) τ t 2 + 2 ( r t ( 1 ) r t ( 2 ) − n t ( 12 ) ) τ t + ( r t ( 2 ) r t ( 2 ) − n t ( 22 ) ) ] + α ^ t ( 2 ) r t ( 1 ) − p t ( 12 ) [ ( 1 − k t ( 1 ) ) n t ( 11 ) − k t ( 2 ) n t ( 12 ) ] − p t ( 22 ) [ τ t n t ( 11 ) + n t ( 12 ) ]
∀ t = 1 , … , n − 1 \forall t=1,\ldots,n-1 ∀ t = 1 , … , n − 1
and
∂ g ( θ ) ∂ σ = 0 , ∂ g ( θ ) ∂ σ ε = 0 , ∂ g ( θ ) ∂ γ t = 1 ∀ t = 1 , … , n − 1 \frac{\partial g(\boldsymbol{\theta})}{\partial\sigma}=0,\qquad
\frac{\partial g(\boldsymbol{\theta})}{\partial\sigma_{\varepsilon}}=0,\qquad
\frac{\partial g(\boldsymbol{\theta})}{\partial\gamma_t}=1
\quad\forall t=1,\ldots,n-1 ∂ σ ∂ g ( θ ) = 0 , ∂ σ ε ∂ g ( θ ) = 0 , ∂ γ t ∂ g ( θ ) = 1 ∀ t = 1 , … , n − 1
Repeated values of the covariate
The construction of Section 2 of the paper is stated for distinct design points, and a great deal of it stops working if they are not; yet ties are common in practice, for instance when the covariate is recorded at a coarse resolution, as the temperatures of § 3.1 of the paper are. Write the sample as ( x i , y i j ) (x_i,y_{ij}) ( x i , y ij ) , j = 1 , … , m i j=1,\ldots,m_i j = 1 , … , m i , i = 1 , … , n i=1,\ldots,n i = 1 , … , n , where x 1 < x 2 < ⋯ < x n x_1<x_2<\cdots<x_n x 1 < x 2 < ⋯ < x n are the distinct values of the covariate, m i ≥ 1 m_i\ge1 m i ≥ 1 is the number of replicates at x i x_i x i , and N = ∑ i m i N=\sum_i m_i N = ∑ i m i is the total number of data points. Each may carry a positive weight w i j w_{ij} w ij , a point of weight w i j w_{ij} w ij being treated as the mean of w i j w_{ij} w ij independent observations; unit weights are the default. In this section n n n is the number of distinct values, so that every δ i = x i + 1 − x i \delta_i=x_{i+1}-x_i δ i = x i + 1 − x i is positive.
Why ties must be removed. The warping is indexed by the intervals δ i = x i + 1 − x i \delta_i=x_{i+1}-x_i δ i = x i + 1 − x i between consecutive design points, and a tie makes one of them vanish. Two distinct things then break. Mechanically, the latent length τ i = δ i + γ i \tau_i=\delta_i+\gamma_i τ i = δ i + γ i of such an interval is zero whenever it carries no warp, which at the optimum is the case for all but a few intervals. The state-space form (2) of the paper then has Var ( η i ) = σ 2 τ i 3 / 3 \operatorname{Var}(\eta_i)=\sigma^2\tau_i^3/3 Var ( η i ) = σ 2 τ i 3 /3 , Cov ( η i , ζ i ) = σ 2 τ i 2 / 2 \operatorname{Cov}(\eta_i,\zeta_i)=\sigma^2\tau_i^2/2 Cov ( η i , ζ i ) = σ 2 τ i 2 /2 and Var ( ζ i ) = σ 2 τ i \operatorname{Var}(\zeta_i)=\sigma^2\tau_i Var ( ζ i ) = σ 2 τ i , all of which are zero, so the transition is degenerate; and the stiffness matrix K i \mathbf{K}_i K i of § 6 , proportional to τ i − 3 \tau_i^{-3} τ i − 3 , and the gradient (A3) of the warping step are not defined. More fundamentally, the model is not identified there: a warp placed on an interval of zero length is a discontinuity between two responses recorded at the same value of the covariate, and no feature of the data can say which replicate lies on which side of it. Collapsing ties is therefore not a computational convenience but the removal of coordinates that carry no information.
The reduction is exact. Let W i = ∑ j w i j W_i=\sum_j w_{ij} W i = ∑ j w ij be the total weight at x i x_i x i , y ˉ i = W i − 1 ∑ j w i j y i j \bar{y}_i=W_i^{-1}\sum_j w_{ij}y_{ij} y ˉ i = W i − 1 ∑ j w ij y ij the weighted group mean, and
S i = ∑ j = 1 m i w i j ( y i j − y ˉ i ) 2 , S = ∑ i = 1 n S i (13) S_i=\sum_{j=1}^{m_i}w_{ij}(y_{ij}-\bar{y}_i)^2,\qquad S=\sum_{i=1}^{n}S_i \tag*{(13)} S i = j = 1 ∑ m i w ij ( y ij − y ˉ i ) 2 , S = i = 1 ∑ n S i (13)
the within-group scatter, written S S S to distinguish it from the sums S i S_i S i of § 6 and from the selection matrix S S S of § 7 . For every i i i the elementary identity ∑ j w i j ( y i j − f i ) 2 = \sum_j w_{ij}(y_{ij}-f_i)^2= ∑ j w ij ( y ij − f i ) 2 = W i ( y ˉ i − f i ) 2 + S i W_i(\bar{y}_i-f_i)^2+S_i W i ( y ˉ i − f i ) 2 + S i holds, so that the weighted version of the objective function (1) of the paper splits as
∑ i = 1 n ∑ j = 1 m i w i j ( y i j − f ( x i ) ) 2 + λ ∑ i = 1 n − 1 k i = ∑ i = 1 n W i ( y ˉ i − f ( x i ) ) 2 + λ ∑ i = 1 n − 1 k i + S , (14) \sum_{i=1}^{n}\sum_{j=1}^{m_i}w_{ij}(y_{ij}-f(x_i))^2+\lambda\sum_{i=1}^{n-1}k_i
=
\sum_{i=1}^{n}W_i(\bar{y}_i-f(x_i))^2+\lambda\sum_{i=1}^{n-1}k_i+S, \tag*{(14)} i = 1 ∑ n j = 1 ∑ m i w ij ( y ij − f ( x i ) ) 2 + λ i = 1 ∑ n − 1 k i = i = 1 ∑ n W i ( y ˉ i − f ( x i ) ) 2 + λ i = 1 ∑ n − 1 k i + S , (14)
with k i k_i k i the energy of interval i i i (Equation (3) of the paper). The penalty is untouched, because it depends on f f f , on the warps and on the distinct x i x_i x i alone, and S S S involves neither f f f nor γ \gamma γ . The pair ( y ˉ i , W i ) (\bar{y}_i,W_i) ( y ˉ i , W i ) is therefore sufficient for the signal: at a fixed smoothing constant, the weighted fit on the n n n collapsed points is identical to the fit on the N N N raw ones, not an approximation to it.
In the state-space representation the weight enters through the observation variance, H i = λ σ 2 / W i H_i=\lambda\sigma^2/W_i H i = λ σ 2 / W i , which is to say that a point of weight W i W_i W i is treated as the mean of W i W_i W i independent observations. In the spline step of § 7 it enters through the data term ∑ i W i ( y ˉ i − f ( x i ) ) 2 \sum_i W_i(\bar{y}_i-f(x_i))^2 ∑ i W i ( y ˉ i − f ( x i ) ) 2 : the selection matrix S S S is replaced by S W = diag ( W 1 , 0 , W 2 , 0 , … , W n , 0 ) S_W=\operatorname{diag}(W_1,0,W_2,0,\ldots,W_n,0) S W = diag ( W 1 , 0 , W 2 , 0 , … , W n , 0 ) , and system (A4) becomes
( S W + λ R ) z = S W y ˉ ∗ , (S_W+\lambda R)z=S_W\bar{y}^{*}, ( S W + λ R ) z = S W y ˉ ∗ ,
with y ˉ ∗ \bar{y}^{*} y ˉ ∗ the vector of size 2 n 2n 2 n carrying y ˉ i \bar{y}_i y ˉ i in position 2 i − 1 2i-1 2 i − 1 and zeros elsewhere. The system is still block tridiagonal, and of order 2 n 2n 2 n with n n n the number of distinct values, so the solution of § 8 applies unchanged. So does the warping step, since the energies k i k_i k i and their gradient depend on the distinct x i x_i x i only.
What the reduction discards, and when it matters. Equation (14) says that S S S is irrelevant to f f f . It does not say that S S S is irrelevant. The within-group scatter is not ancillary for the observation variance σ ε 2 = λ σ 2 \sigma_{\varepsilon}^2=\lambda\sigma^2 σ ε 2 = λ σ 2 : it is an estimate of σ ε 2 \sigma_{\varepsilon}^2 σ ε 2 in its own right, carrying
ν = ∑ i = 1 n ( m i − 1 ) = N − n (15) \nu=\sum_{i=1}^{n}(m_i-1)=N-n \tag*{(15)} ν = i = 1 ∑ n ( m i − 1 ) = N − n (15)
degrees of freedom, one for every data point beyond the first at each site. Should the collapsed data be supplied directly, with S S S computed from the underlying raw observations rather than from the points passed to the fitter, ν \nu ν counts those observations instead and becomes ∑ i W i − n \sum_i W_i-n ∑ i W i − n .
The consequence is a distinction between the two kinds of fitter. Where the fit does not involve the observation variance, as in the alternating algorithm at a supplied λ \lambda λ , there is nothing to restore and the collapsed problem is the whole problem. Where σ ε 2 \sigma_{\varepsilon}^2 σ ε 2 is estimated by maximum likelihood, as in SSJ–MLE, the log-likelihood of the collapsed data is augmented by
− 1 2 ( ν log σ ε 2 + S σ ε 2 ) , (16) -\frac{1}{2}\left(\nu\log\sigma_{\varepsilon}^2+\frac{S}{\sigma_{\varepsilon}^2}\right), \tag*{(16)} − 2 1 ( ν log σ ε 2 + σ ε 2 S ) , (16)
together with the corresponding constant, so that the reported likelihood is that of all the data points and σ ^ ε 2 \hat{\sigma}_{\varepsilon}^2 σ ^ ε 2 uses the within-group scatter as well as the between-group residuals. In the scores of § 10 , H t H_t H t becomes λ σ 2 / W t \lambda\sigma^2/W_t λ σ 2 / W t , or σ ε 2 / W t \sigma_{\varepsilon}^2/W_t σ ε 2 / W t when λ \lambda λ is estimated, so that the terms in e t 2 − d t e_t^2-d_t e t 2 − d t are divided by W t W_t W t ; and the correction (A11) adds − ν / σ ε + S / σ ε 3 -\nu/\sigma_{\varepsilon}+S/\sigma_{\varepsilon}^3 − ν / σ ε + S / σ ε 3 to the score with respect to σ ε \sigma_{\varepsilon} σ ε , or − ν / σ + S / ( λ σ 3 ) -\nu/\sigma+S/(\lambda\sigma^3) − ν / σ + S / ( λ σ 3 ) to the score with respect to σ \sigma σ when λ \lambda λ is fixed.
SSJ–KFS and the fixed-variance fitter do not apply (A11): they report the likelihood of the collapsed means, and their σ ^ ε 2 \hat{\sigma}_{\varepsilon}^2 σ ^ ε 2 is formed from the between-group residuals alone. Such an estimate remains consistent as n n n grows, since the collapsed means have variance σ ε 2 / W i \sigma_{\varepsilon}^2/W_i σ ε 2 / W i and the weighted filter identifies σ ε 2 \sigma_{\varepsilon}^2 σ ε 2 from them; but it throws away y y y data, points’ worth of information, and under heavy replication, with many replicates at each of moderately many sites, that is most of the information about σ ε 2 \sigma_{\varepsilon}^2 σ ε 2 in the sample. A log-likelihood reported by SSJ–MLE and one reported by SSJ–KFS are therefore on different footings whenever ties are present, and are not comparable as they stand.
Consequences for the selection criterion. Every count entering the criteria is read off the collapsed problem. The candidate locations for a discontinuity are the n − 1 n-1 n − 1 intervals between distinct values of the covariate, so the dimension of the model space entering the extended BIC (5) of the paper is p = n − 1 p=n-1 p = n − 1 , with n n n the number of distinct values and not N N N , and the extended term is 2 ξ k log ( n − 1 ) 2\xi k\log(n-1) 2 ξ k log ( n − 1 ) . The sample size whose logarithm multiplies the degrees of freedom is n n n likewise. The consistency floor of Chen and Chen [1 ] is therefore 1 − log n / { 2 log ( n − 1 ) } 1-\log n/\{2\log(n-1)\} 1 − log n / { 2 log ( n − 1 )} , the value of about 0.5 recalled in § 2.1.3 of the paper, unchanged by the presence of replicates.
The effective degrees of freedom are the trace of the n × n n\times n n × n weighted smoothing matrix, which equals that of the N × N N\times N N × N smoothing matrix of the raw observations; in the alternating algorithm it is ∑ i W i { ( S W + λ R ) − 1 } 2 i − 1 , 2 i − 1 \sum_i W_i\{(S_W+\lambda R)^{-1}\}_{2i-1,2i-1} ∑ i W i {( S W + λ R ) − 1 } 2 i − 1 , 2 i − 1 . The residual sum of squares ∑ i W i ( y ˉ i − f ^ i ) 2 \sum_i W_i(\bar y_i-\hat f_i)^2 ∑ i W i ( y ˉ i − f ^ i ) 2 that the criteria are built on holds only between-point variation, S S S having been removed by the collapse, so it carries n − edf n-\operatorname{edf} n − edf degrees of freedom and the residual variance is estimated by ∑ i W i ( y ˉ i − f ^ i ) 2 / ( n − edf ) \sum_i W_i(\bar y_i-\hat f_i)^2/(n-\operatorname{edf}) ∑ i W i ( y ˉ i − f ^ i ) 2 / ( n − edf ) — one degree of freedom per fitted point, not one per data point. Fitted values are returned on the n n n distinct, sorted values of the covariate, so a fit and the data it came from have different lengths whenever ties are present, and any comparison of the two has to be made against y ˉ i \bar y_i y ˉ i .
Cross-validation. Held-out error is weighted in the same way as the fit, so that a fold is scored by the observations it actually represents: the cross-validation criterion of the Calibration Supplement becomes
CV ( λ , M ) = ∑ k = 1 K ∑ i ∈ F k W i ( y ˉ i − f ^ λ , M ( − k ) ( x i ) ) 2 ∑ k = 1 K ∑ i ∈ F k W i . \operatorname{CV}(\lambda,M)=\frac{\sum_{k=1}^{K}\sum_{i\in\mathcal{F}_k}W_i\left(\bar y_i-\hat f_{\lambda,M}^{(-k)}(x_i)\right)^2}{\sum_{k=1}^{K}\sum_{i\in\mathcal{F}_k}W_i}. CV ( λ , M ) = ∑ k = 1 K ∑ i ∈ F k W i ∑ k = 1 K ∑ i ∈ F k W i ( y ˉ i − f ^ λ , M ( − k ) ( x i ) ) 2 .
Replication requires a choice about the partition that is easy to overlook. If folds are drawn over the N N N data points, replicates of the same site land on both sides of the split, and a held-out point is then evaluated at a covariate value that the training sample also contains: the criterion estimates the error of predicting a fresh observation at an already observed x x x , and it is optimistic as an estimate of the error at a new one. If folds are drawn over the n n n distinct sites, all the replicates at a site are held out together and the criterion estimates the latter; it then differs from the held-out error of the raw observations only by the constant S / ∑ i W i S/\sum_i W_i S / ∑ i W i , and has the same minimiser. The second is the relevant target when λ \lambda λ is being chosen to recover the signal, and it is what we use; the distinction is immaterial when the design has no ties.
Calibration Supplement — Tuning of the smoothing constant and of the warping budget, and calibration of the extended-BIC constant ξ \xi ξ , for Smoothing splines with some discontinuities
Matteo Pelagatti; Paolo Maranzano; Giulio Cesare Gatto Ronchieri
This document reports the tuning of the smoothing constant λ \lambda λ and of the warping budget M M M and, in full, the calibration of the extended-BIC constant ξ \xi ξ , all summarized in Section 2.4 of the paper: the cross-validation of λ \lambda λ and the reasons for selecting M M M by an information criterion, the data generating process and its three scenarios in closed form, the design and the criteria, every table and figure, and the numbers behind every reading. The experiment rests on 200 replications per design cell; the code that computes it and the code that produces every table and figure of this document are in the replication package.
supplement-C-calibration.pdf
Tuning the smoothing constant and the warping budget
This section gives the details of the tuning of λ \lambda λ and M M M summarized in § 2.1.3 of the paper; the rest of the document is devoted to the constant ξ \xi ξ of the extended BIC.
An information criterion for the budget. In the literature and in the applications of classical cubic smoothing splines the smoothing constant is generally determined by cross-validation, and with the state-space form it can also be estimated by Gaussian maximum likelihood. Cross-validation is not a valid way to select the warping budget M M M , however: given the small number of discontinuities expected in the data, most held-out subsamples reveal none of them. M M M must therefore be chosen by an information criterion, and with a number of candidate parameters of the same order as the number of observations the classical criteria are, as Chen and Chen [1 ] point out, “usually too liberal”.
Selecting the budget is not a choice of model dimension. A budget admits some number k k k of positive warps, but it also decides which of the n − 1 n-1 n − 1 inter-observation intervals carry them, and the estimator arrives at that subset by searching over all of them. The relevant model space at size k k k therefore has ( n − 1 k ) \binom{n-1}{k} ( k n − 1 ) members and grows with the sample: the number of candidate locations is p = n − 1 p=n-1 p = n − 1 , so p ≍ n p\asymp n p ≍ n . The Akaike and Bayesian criteria charge only for dimension. Both have the form − 2 ℓ ^ + c ( n ) d f -2\hat{\ell}+c(n)\,\mathrm{df} − 2 ℓ ^ + c ( n ) df , with c ( n ) = 2 c(n)=2 c ( n ) = 2 and c ( n ) = log n c(n)=\log n c ( n ) = log n respectively, and neither term depends on how many configurations were examined before the reported one was chosen. That is harmless when the model space is fixed, because the omitted cost is then a constant, but it is not harmless here. The best-fitting arrangement of k k k warps among ( n − 1 k ) \binom{n-1}{k} ( k n − 1 ) possibilities fits appreciably better than a k k k -warp arrangement nominated in advance, and the gap widens with p p p ; a criterion blind to it reads the selection gain as evidence and keeps adding discontinuities. Formally, the consistency of the Bayesian criterion rests on p p p being fixed, and it is lost once p p p grows with n n n . Chen and Chen [1 ] restore it by replacing the prior that is uniform over models with one that is uniform over models of the same size , which adds the log-cardinality of the class searched. With log ( n − 1 k ) ≈ k log ( n − 1 ) \log\binom{n-1}{k}\approx k\log(n-1) log ( k n − 1 ) ≈ k log ( n − 1 ) for small k k k , this yields the extended BIC of Equation (2) in Section 2, whose additional term is a charge for where the discontinuities were placed, as distinct from how many there are.
Two remarks on our use of these results, that ξ \xi ξ is calibrated on a design separate from the one used for evaluation and that calibrated values below the nominal consistency floor do not contradict the theory, because the d f log n \mathrm{df}\log n df log n term already pays part of the search cost, are developed in Section 2 and § 20.5 . A third concerns the way λ \lambda λ is supplied, which also decides how much the extended term has left to charge. When SSJ–KFS or SSJ–ATW take λ \lambda λ from a preliminary fit, one parameter of the model has been fixed before the budget is chosen: the fit is stiffer than one in which λ \lambda λ is free and re-estimated at every budget, part of its degrees of freedom have already been given up, and the search over the budget can buy less, so the additional penalty governed by ξ \xi ξ has to be milder. The calibration confirms this in every scenario (§ 20.1 and § 21 ).
Cross-validation of the smoothing constant. In the implementation based on the state-space form, SSJ–MLE, λ \lambda λ is estimated together with the other parameters by maximum likelihood and the budget is selected by minimizing the extended BIC on a grid of values of M M M . SSJ–KFS and SSJ–ATW instead take the smoothing constant as given. A simple way of supplying a good value is a preliminary SSJ–MLE fit, but this makes the faster estimator depend on the slower one it is meant to approximate; the following K K K -fold cross-validation removes that dependence at a comparable cost. Write f ^ λ , M ( − k ) \hat{f}_{\lambda,M}^{(-k)} f ^ λ , M ( − k ) for the fit obtained from the observations outside fold k k k , at smoothing constant λ \lambda λ and warping budget M M M . With the n n n observations split into K K K folds F 1 , … , F K \mathcal{F}_1,\ldots,\mathcal{F}_K F 1 , … , F K of nearly equal size, which partition the sample, the cross-validation criterion on a grid Λ × M \Lambda\times\mathcal{M} Λ × M of candidate pairs is
CV ( λ , M ) = 1 n ∑ k = 1 K ∑ i ∈ F k ( y i − f ^ λ , M ( − k ) ( x i ) ) 2 . (17) \operatorname{CV}(\lambda,M)=\frac{1}{n}\sum_{k=1}^{K}\sum_{i\in\mathcal{F}_k}\left(y_i-\hat{f}_{\lambda,M}^{(-k)}(x_i)\right)^2. \tag*{(17)} CV ( λ , M ) = n 1 k = 1 ∑ K i ∈ F k ∑ ( y i − f ^ λ , M ( − k ) ( x i ) ) 2 . (17)
Held-out points falling outside the range of their training sample are evaluated at the nearest training abscissa rather than by extrapolation: beyond the data the fit is only extended linearly, and one or two boundary points would otherwise dominate a fold. The estimator is then run on the full sample at λ ^ = arg min λ min M CV ( λ , M ) \hat{\lambda} = \arg\min_{\lambda}\min_{M}\operatorname{CV}(\lambda, M) λ ^ = arg min λ min M CV ( λ , M ) , and the budget is chosen afresh by the extended BIC.
The minimization over M M M in Equation (1) is a profiling step, not a selection: its argument is discarded. This asymmetry is deliberate and reflects what the two constants do. The smoothing constant λ \lambda λ governs a global bias–variance trade-off to which every held-out observation contributes, and there cross-validation is competitive with maximum likelihood: on the design of Section 3 of the paper the two choices of λ \lambda λ agree closely and the resulting root mean squared errors (RMSEs) differ by 1% to 4%. The budget M M M governs a sparse, local feature. Only the observations lying in the transition regions carry information about whether it is right, so the effective sample size for that decision grows with the number of discontinuities and not with n n n ; and prediction error is nearly indifferent to a few spurious warps, since each displaces the fitted curve over a short stretch of the domain. Selecting M M M by Equation (1) therefore declares markedly more discontinuities than the information criterion at the same accuracy, which is why it is not used for that purpose.
The folds can be drawn at random or as contiguous blocks. To respect the temporal sequence of the observations, the blocked scheme holds out contiguous stretches of the series and so preserves its temporal structure [3 ] . With serially dependent errors this matters: a held-out observation whose neighbors remain in the training sample is nearly predicted by them, the random-fold criterion is then minimized by a fit that all but interpolates, and the selected λ \lambda λ collapses (§ 20.2 ). The simulations of Section 3 of the paper use K = 5 K = 5 K = 5 random folds and the applications of Section 4 use K = 20 K = 20 K = 20 ; with contiguous blocks K = 20 K = 20 K = 20 is preferable, since holding out a fifth of the series in one stretch over-smooths (§ 20.2 ).
Grids and computational cost. The smoothing constant is not scale-free: it carries the units of x 3 x^{3} x 3 , because the roughness energy of an interval has the units of y 2 / x 3 y^{2}/x^{3} y 2 / x 3 while the fidelity term of the objective function has the units of y 2 y^{2} y 2 (the reduced form of the energy, which shows it, is derived in the Computational Supplement). The default grid is therefore specified for the scale-free ratio λ / ( x ( n ) − x ( 1 ) ) 3 \lambda/(x_{(n)} - x_{(1)})^{3} λ / ( x ( n ) − x ( 1 ) ) 3 , nine logarithmically spaced values from 10 − 6 10^{-6} 1 0 − 6 to 10 − 2 10^{-2} 1 0 − 2 , which are multiplied by ( x ( n ) − x ( 1 ) ) 3 (x_{(n)} - x_{(1)})^{3} ( x ( n ) − x ( 1 ) ) 3 before use. The warps γ i \gamma_i γ i , and with them the budget M M M , are lengths of the domain, so the grid of budgets is expressed in the units of x x x and is the same one used for the final selection. The procedure costs K ∣ Λ ∣ ∣ M ∣ K|\Lambda||\mathcal{M}| K ∣Λ∣∣ M ∣ fits, which is affordable only because the spline step of the alternating algorithm solves a banded system in O ( n ) O(n) O ( n ) operations; within each fold and each λ \lambda λ the budgets are traversed in increasing order, each fit warm-started from the previous one.
The constant
All three estimators that carry an information criterion, namely SSJ–MLE, SSJ–KFS and SSJ–ATW, select the warping budget M M M by the extended Bayesian information criterion of Chen and Chen [1 ] ,
EBIC ( ξ ) = df log ( n ) − 2 ℓ ^ + 2 ξ k log ( n − 1 ) , (18) \operatorname{EBIC}(\xi) = \operatorname{df}\log(n) - 2\hat{\ell} + 2\xi k\log(n - 1), \tag*{(18)} EBIC ( ξ ) = df log ( n ) − 2 ℓ ^ + 2 ξ k log ( n − 1 ) , (18)
with ℓ ^ \hat{\ell} ℓ ^ the maximized log-likelihood, df \operatorname{df} df the effective degrees of freedom, k k k the number of active warps and n − 1 n - 1 n − 1 the number of candidate discontinuity locations. The strength ξ ∈ [ 0 , 1 ] \xi\in[0,1] ξ ∈ [ 0 , 1 ] is a free constant.
Three considerations rule out the obvious procedure of sweeping a grid of ξ \xi ξ , reporting the minimizer and adopting it:
If ξ \xi ξ were chosen on the design of Section 3 of the paper and that same design were then used to report performance, the comparison against CSSD would be contaminated: CSSD’s constants are cross-validated within each replication, ours would have been chosen by inspecting the truth across all of them. The calibration design must be separate from the evaluation design.
The root mean squared error (RMSE) as a function of ξ \xi ξ is nearly flat near its minimum, so a point minimizer is largely Monte Carlo noise. A rule is needed that reports what the design resolves rather than the location of a shallow argmin.
The consistency condition of Chen and Chen [1 ] for p n ≍ n κ p_n \asymp n^\kappa p n ≍ n κ candidates is ξ ≥ 1 − 1 / ( 2 κ ) \xi\ge1 - 1/(2\kappa) ξ ≥ 1 − 1/ ( 2 κ ) . Here the candidates are the n − 1 n-1 n − 1 inter-observation intervals, so κ = 1 \kappa=1 κ = 1 and the nominal floor is ξ ≥ 1 − log ( n ) / { 2 log ( n − 1 ) } ≈ 0.50 \xi\ge1 - \log(n)/\{2\log(n-1)\} \approx0.50 ξ ≥ 1 − log ( n ) / { 2 log ( n − 1 )} ≈ 0.50 . Every value calibrated below lies under that floor, and the discrepancy has to be explained rather than left for the reader to notice.
The three issues are addressed as follows. The first is met by the design: ξ \xi ξ is calibrated on a data generating process of its own, described in Section 3 and never used to evaluate the estimators, so that the comparison with CSSD in Section 3 of the paper is made with a constant fixed in advance. The second is met by the criteria of Section 5: the point minimizer is replaced by two minimax regret rules, whose output is the value of ξ \xi ξ whose worst regret over the design cells is smallest, and every calibrated value is reported with the spread of the cell-wise optima it summarizes, so that a flat valley is visible as such. The third is met in § 20.5 . df is the trace of the smoothing matrix, and that trace itself grows with every activated warp, so the df log n \log n log n term of Equation (2) already charges part of the cost of each declared discontinuity. What the theory constrains is the total charge per discontinuity; the extended term supplies only what is left of it, and a calibrated ξ \xi ξ below the nominal floor is consistent with the theory once the part levied through df is counted.
Data generating process and scenarios
The constant is calibrated by simulation because 1) the data do not identify ξ \xi ξ , since it enters only the selection among fits and never a fit; 2) the theory of Chen and Chen [1 ] supplies a floor rather than a value; and 3) only on simulated series are the two quantities a choice of ξ \xi ξ has to be scored against, the true signal and the true number of discontinuities, available. The design has to cover the kinds of data the estimators will meet, and it does so with one data generating process and two departure parameters, run as three scenarios.
Scenario 1 is the common boundary of the other two and is reached by the same code. Scenario 2 describes series whose errors are persistent and Scenario 3 series whose level accumulates shocks; the fuel prices of § 3.1.1 of the paper have both, so the two bracket the answer rather than either supplying it.
The three experiments differ only in two parameters of one data generating process. With b b b a fixed smooth background, r r r a draw from an integrated Wiener process and e e e the errors,
x ( 1 ) ≤ ⋯ ≤ x ( n ) order statistics of n draws from U ( 0 , 1 ) , c j ∼ U ( 0.12 , 0.88 ) , s j = s j v j , s j ∼ U { − 1 , 1 } , v j ∼ U ( 0.30 , 0.70 ) , b i = sin ( 2 π x i ) + 1 2 sin ( 5 π x i ) , [ r i + 1 r i + 1 ′ ] = [ 1 δ i 0 1 ] [ r i r i ′ ] + v i , v i ∼ N ( 0 , [ δ i 3 / 3 δ i 2 / 2 δ i 2 / 2 δ i ] ) , ( r 1 , r 1 ′ ) = ( 0 , 0 ) , g i = 1 − ω b i − b ˉ sd ( b ) + ω r i − r ˉ sd ( r ) , f i = g i − g ˉ sd ( g ) , e 1 = σ u 1 , e i + 1 = ϕ i e i + 1 − ϕ i 2 σ u i + 1 , ϕ i = ψ δ i / δ ˉ , y i = f i + ∑ j = 1 q s j 1 [ c j , 1 ) ( x i ) + e i , (19) \begin{aligned}
x_{(1)} &\le\cdots\le x_{(n)} \quad\text{order statistics of } n \text{ draws from } \mathrm{U}(0,1),\\
c_j &\sim\mathrm{U}(0.12,0.88), \qquad s_j=s_jv_j,\qquad s_j\sim\mathrm{U}\{-1,1\},\quad v_j\sim\mathrm{U}(0.30,0.70),\\
b_i &= \sin(2\pi x_i)+\frac{1}{2}\sin(5\pi x_i),\qquad
\begin{bmatrix}r_{i+1}\\r'_{i+1}\end{bmatrix}
=
\begin{bmatrix}1&\delta_i\\0&1\end{bmatrix}
\begin{bmatrix}r_i\\r'_i\end{bmatrix}
+v_i,\\
v_i &\sim\mathrm{N}\left(0,
\begin{bmatrix}\delta_i^3/3&\delta_i^2/2\\\delta_i^2/2&\delta_i\end{bmatrix}\right),
\qquad(r_1,r'_1)=(0,0),\\
g_i &= \sqrt{1-\omega}\frac{b_i-\bar b}{\operatorname{sd}(b)}
+\sqrt{\omega}\frac{r_i-\bar r}{\operatorname{sd}(r)},\qquad
f_i=\frac{g_i-\bar g}{\operatorname{sd}(g)},\\
e_1 &= \sigma u_1,\qquad e_{i+1}=\phi_i e_i+\sqrt{1-\phi_i^2}\,\sigma u_{i+1},\qquad\phi_i=\psi^{\delta_i/\bar\delta},\\
y_i &= f_i+\sum_{j=1}^{q}s_j\mathbf{1}_{[c_j,1)}(x_i)+e_i, \tag*{(19)}
\end{aligned} x ( 1 ) c j b i v i g i e 1 y i ≤ ⋯ ≤ x ( n ) order statistics of n draws from U ( 0 , 1 ) , ∼ U ( 0.12 , 0.88 ) , s j = s j v j , s j ∼ U { − 1 , 1 } , v j ∼ U ( 0.30 , 0.70 ) , = sin ( 2 π x i ) + 2 1 sin ( 5 π x i ) , [ r i + 1 r i + 1 ′ ] = [ 1 0 δ i 1 ] [ r i r i ′ ] + v i , ∼ N ( 0 , [ δ i 3 /3 δ i 2 /2 δ i 2 /2 δ i ] ) , ( r 1 , r 1 ′ ) = ( 0 , 0 ) , = 1 − ω sd ( b ) b i − b ˉ + ω sd ( r ) r i − r ˉ , f i = sd ( g ) g i − g ˉ , = σ u 1 , e i + 1 = ϕ i e i + 1 − ϕ i 2 σ u i + 1 , ϕ i = ψ δ i / δ ˉ , = f i + j = 1 ∑ q s j 1 [ c j , 1 ) ( x i ) + e i , (19)
with u i u_i u i independent standard normal, δ i = x i + 1 − x i \delta_i = x_{i+1} - x_i δ i = x i + 1 − x i and δ ˉ = ( x n − x 1 ) / ( n − 1 ) \bar{\delta} = (x_n - x_1)/(n - 1) δ ˉ = ( x n − x 1 ) / ( n − 1 ) . The abscissas x x x are uniform draws, sorted, so the spacing is irregular (thus consistent with both cross-sectional and continuous-time objects). At the same increment δ i \delta_i δ i enters the transition of the simulated error and the transition matrix of the state-space form of § 2.1.1 of the paper.
Discontinuity positions and signed sizes are redrawn every replication, kept away from the edges so that a break is a detection problem rather than a boundary problem, and bounded away from zero so that a miss is a failure of the method rather than of the signal-to-noise ratio. Signal, discontinuities and errors are additive in the last line of Equation (3), and the discontinuities enter as steps that persist to the end of the sample, so the size s j s_j s j is the level shift at c j c_j c j and not a transient.
The covariance [ δ 3 / 3 , δ 2 / 2 ; δ 2 / 2 , δ ] [\delta^3/3,\ \delta^2/2;\ \delta^2/2,\ \delta] [ δ 3 /3 , δ 2 /2 ; δ 2 /2 , δ ] is that of an integrated Wiener process over a gap δ \delta δ , which by the correspondence of Wahba [4 ] is the prior the smoothing spline itself places over the signal.
The signal g g g is a weighted linear combination of the deterministic component b b b with weight 1 − ω \sqrt{1-\omega} 1 − ω and the integrated Wiener stochastic process r r r with weight ω \sqrt{\omega} ω . Since the weights are the square roots of two shares that sum to one, ω ∈ [ 0 , 1 ] \omega\in[0,1] ω ∈ [ 0 , 1 ] is the share of the signal’s variance carried by the stochastic trend: at ω = 0 \omega= 0 ω = 0 the signal is the fixed curve and only the jumps are random, at ω = 1 \omega= 1 ω = 1 the curve is replaced entirely by a path of the process, and in between the signal has both a recurring shape and a level that wanders. The two components are independent by construction, b b b being a fixed function of x x x and r r r being driven by innovations that do not depend on it.
The signal is standardized twice. The first standardization, of b b b and r r r separately, makes ω \omega ω a share of variance: the two components have no common scale (i.e., b b b is a fixed sinusoid, r r r a random walk in the slope whose variance grows along the sample and with n n n ) so raw weights would confound the mixing proportion with the accident of their relative sizes. The second standardization acts on the mixture g g g and holds the amplitude constant across ω \omega ω : sd ( g ) \operatorname{sd}(g) sd ( g ) is not 1 unless the two parts happen to be uncorrelated in the realized sample. It matters because the jump sizes s j s_j s j and the noise level σ \sigma σ are not rescaled, so without it the jump-to-signal and signal-to-noise ratios would drift with ω \omega ω and a change in the calibrated ξ \xi ξ could not be attributed to the departure rather than to the jumps having become relatively smaller.
In the error recursion the two coefficients do separate jobs. The factor ϕ i = ψ δ i / δ ˉ \phi_i = \psi^{\delta_i/\bar{\delta}} ϕ i = ψ δ i / δ ˉ is the correlation carried across a gap of length δ i \delta_i δ i , equal to ψ \psi ψ exactly at the mean spacing, larger for closer pairs and smaller for farther ones; correlation therefore decays in distance. The factor 1 − ϕ i 2 \sqrt{1-\phi_i^2} 1 − ϕ i 2 holds the marginal variance fixed: by induction Var ( e i ) = σ 2 \operatorname{Var}(e_i) = \sigma^2 Var ( e i ) = σ 2 for every i i i and every ψ \psi ψ , while Corr ( e i , e i + 1 ) = ϕ i \operatorname{Corr}(e_i,e_{i+1}) = \phi_i Corr ( e i , e i + 1 ) = ϕ i . Persistence here is therefore not extra noise but the same noise arranged differently, which is the case in which it is easiest to overlook. In symbols, if Var ( e i ) = σ 2 \operatorname{Var}(e_i) = \sigma^2 Var ( e i ) = σ 2 then
Var ( e i + 1 ) = ϕ i 2 σ 2 + ( 1 − ϕ i 2 ) σ 2 = σ 2 , Cov ( e i , e i + 1 ) = ϕ i σ 2 , \operatorname{Var}(e_{i+1}) = \phi_i^2\sigma^2 + (1-\phi_i^2)\sigma^2 = \sigma^2,\qquad\operatorname{Cov}(e_i,e_{i+1}) = \phi_i\sigma^2, Var ( e i + 1 ) = ϕ i 2 σ 2 + ( 1 − ϕ i 2 ) σ 2 = σ 2 , Cov ( e i , e i + 1 ) = ϕ i σ 2 ,
so that σ \sigma σ is the standard deviation of the errors at every i i i and every ψ \psi ψ , and ϕ i \phi_i ϕ i is exactly their correlation.
Because ψ \psi ψ is defined at the mean spacing while the design is irregular, the autocorrelation on a realized series is not ψ \psi ψ itself. The gaps of n n n sorted uniform draws are, to a close approximation, exponentially distributed with mean δ ˉ \bar{\delta} δ ˉ , and since ϕ i \phi_i ϕ i is convex in the gap the expected lag-one autocorrelation lies above ψ \psi ψ , in closed form
E [ ACF 1 ] = E [ ψ Δ / δ ˉ ] = 1 1 − log ψ , Δ / δ ˉ ∼ Exp ( 1 ) , (20) \mathbb{E}[\operatorname{ACF}_1] = \mathbb{E}\left[\psi^{\Delta/\bar{\delta}}\right] = \frac{1}{1-\log\psi},\qquad\Delta/\bar{\delta} \sim\operatorname{Exp}(1), \tag*{(20)} E [ ACF 1 ] = E [ ψ Δ/ δ ˉ ] = 1 − log ψ 1 , Δ/ δ ˉ ∼ Exp ( 1 ) , (20)
so that the ten levels of ψ \psi ψ span realized autocorrelations from 0 to about 0.9. Over 300 draws at n = 200 n = 200 n = 200 the realized mean autocorrelation was 0.44, 0.64 and 0.88 for ψ = 0.3 , 0.6 , 0.9 \psi= 0.3, 0.6, 0.9 ψ = 0.3 , 0.6 , 0.9 , against 0.454, 0.662 and 0.905 from Equation (4).
The three scenarios.
Scenario 1 sets ψ = ω = 0 \psi= \omega= 0 ψ = ω = 0 : every ϕ i \phi_i ϕ i is zero so e i = σ u i e_i = \sigma u_i e i = σ u i , the stochastic part drops out of g g g , and what remains is the standardized deterministic component b b b observed with independent errors on an irregular design. This is the reference case, and it describes a cross-sectional experimental design;
Scenario 2 sets ω = 0 \omega= 0 ω = 0 and sweeps ψ ∈ { 0 , 0.1 , … , 0.9 } \psi\in\{0, 0.1, \ldots, 0.9\} ψ ∈ { 0 , 0.1 , … , 0.9 } , keeping the fixed signal and turning on the error dependence. It describes anything recorded sequentially, and, because the dependence is in distance rather than in index, also designs that are not temporal, such as a spatial transect;
Scenario 3 sets ψ = 0 \psi= 0 ψ = 0 and sweeps ω ∈ { 0 , 0.25 , 0.5 , 0.75 , 1 } \omega\in\{0, 0.25, 0.5, 0.75, 1\} ω ∈ { 0 , 0.25 , 0.5 , 0.75 , 1 } , keeping independent errors and making the signal itself a draw from the prior the spline assumes. By the equivalence of Wahba [4 ] the smoothing spline is the posterior mean under an integrated Wiener prior. Here it is used in the other direction, to generate signals from exactly what the estimator believes a signal looks like before it has seen any data, so that a moving signal is tested without confounding it with a signal the estimator was never designed for. At ω = 0 \omega= 0 ω = 0 the signal is a fixed curve and only the jumps are random. For 0 < ω < 1 0 < \omega< 1 0 < ω < 1 part of the variance is a recurring shape and part a trend that never repeats, which is what a stable seasonal or diurnal pattern on a slow-moving level looks like. At ω = 1 \omega= 1 ω = 1 the signal is an integrated random walk ([2 ] , i.e., the local linear trend of unobserved-components models and the model implicit in the Hodrick–Prescott filter) so it is non stationary and follows no fixed function, thus a second realization would produce a different curve, and only its scale and the location of the jumps are left to estimate (§ 16.3 ). Also, at ω = 1 \omega= 1 ω = 1 the estimator’s view of the signal is therefore correct , which makes it the well-specified case and gives the results of § 20.3 their force.
Closed forms of each scenario are given in § 16.1 to § 16.3 .
Scenario 1: a fixed smooth signal with independent errors
Setting ψ = 0 \psi= 0 ψ = 0 the error recursion of Equation (3) collapse (i.e., ϕ i = ψ δ i / δ ˉ = 0 \phi_i = \psi^{\delta_i/\bar{\delta}} = 0 ϕ i = ψ δ i / δ ˉ = 0 ) for every i i i , so the error recursion reduces to e i = σ u i e_i = \sigma u_i e i = σ u i ; and with ω = 0 \omega= 0 ω = 0 the stochastic part and the mixture drop out of g g g , leaving the standardized background. Equation (3) becomes
y i = b i − b ˉ sd ( b ) + ∑ j = 1 q s j 1 [ c j , 1 ] ( x i ) + ε i , ε i ∼ i i d N ( 0 , σ 2 ) , b i = sin ( 2 π x i ) + 1 2 sin ( 5 π x i ) . (21) \begin{aligned}
y_i &= \frac{b_i-\bar{b}}{\operatorname{sd}(b)}+\sum_{j=1}^{q}s_j\mathbf{1}_{[c_j,1]}(x_i)+\varepsilon_i,\qquad\varepsilon_i \overset{\mathrm{iid}}{\sim} \mathcal{N}(0,\sigma^2),\\
b_i &= \sin(2\pi x_i)+\frac{1}{2}\sin(5\pi x_i). \tag*{(21)}
\end{aligned} y i b i = sd ( b ) b i − b ˉ + j = 1 ∑ q s j 1 [ c j , 1 ] ( x i ) + ε i , ε i ∼ iid N ( 0 , σ 2 ) , = sin ( 2 π x i ) + 2 1 sin ( 5 π x i ) . (21)
This is the reference case (i.e., common boundary of the other two scenarios): a fixed deterministic curve, observed with independent errors, on an irregular design. It describes a designed experiment, or a cross-section in which the abscissa orders the observations without inducing dependence among them.
Figure 1 shows what it looks like. The scenario has no departure parameter to raise, so the rows carry the one dimension of the factorial that decides how hard the reference problem is: the noise level. At σ = 0.05 \sigma= 0.05 σ = 0.05 the jumps are unmistakable and the curve is traced almost exactly; at σ = 0.15 \sigma= 0.15 σ = 0.15 the smaller of the five is comparable to the scatter, which is what makes the count criterion of Section 5 bite in the cells that matter.
Scenario 2: serially dependent errors
Setting ω = 0 \omega= 0 ω = 0 and letting ψ \psi ψ vary keeps the fixed signal and turns on the error dependence:
Figure 1 (PDF p. 31). The reference case: independent errors, a fixed signal, three noise levels. n = 200 n = 200 n = 200 , ψ = ω = 0 \psi= \omega= 0 ψ = ω = 0 , one replication. Blue: the truth. Dashed: the true jump locations. Down a column the seed is fixed, so only the noise level changes. Scenario 1, Equation (5). Rows: the noise level σ \sigma σ ; columns: the number of true discontinuities. Points are the observations, the blue line is the signal, and the dashed rules are the true jump locations. The seed does not depend on σ \sigma σ , so within a column the abscissas, the signal and the jump configuration are identical going down and only the scatter about the curve changes.
y i = b i − b ˉ sd ( b ) + ∑ j = 1 q s j 1 [ c j , 1 ] ( x i ) + e i , (22) y_i = \frac{b_i-\bar{b}}{\operatorname{sd}(b)} + \sum_{j=1}^{q} s_j \mathbf{1}_{[c_j,1]}(x_i) + e_i, \tag*{(22)} y i = sd ( b ) b i − b ˉ + j = 1 ∑ q s j 1 [ c j , 1 ] ( x i ) + e i , (22)
e 1 = σ u 1 , e i + 1 = ϕ i e i + 1 − ϕ i 2 σ u i + 1 , ϕ i = ψ δ i / δ ˉ , Var ( e i ) = σ 2 for all i , Corr ( e i , e i + 1 ) = ϕ i . \begin{aligned}
e_1 &= \sigma u_1,\qquad e_{i+1} = \phi_i e_i + \sqrt{1-\phi_i^2}\,\sigma u_{i+1},\qquad\phi_i = \psi^{\delta_i/\bar{\delta}},\\
\operatorname{Var}(e_i) &= \sigma^2\quad\text{for all }i,\qquad\operatorname{Corr}(e_i,e_{i+1}) = \phi_i.
\end{aligned} e 1 Var ( e i ) = σ u 1 , e i + 1 = ϕ i e i + 1 − ϕ i 2 σ u i + 1 , ϕ i = ψ δ i / δ ˉ , = σ 2 for all i , Corr ( e i , e i + 1 ) = ϕ i .
ψ \psi ψ is swept over { 0 , 0.1 , … , 0.9 } \{0,0.1,\ldots,0.9\} { 0 , 0.1 , … , 0.9 } , ten levels, for a realized lag-one autocorrelation running from 0 to about 0.87. The scenario describes the normal condition of anything recorded sequentially (e.g., weekly prices, daily pollutant concentrations, hourly load, instrument readings) where what is left over once a signal has been removed is itself persistent. Because the dependence is specified in distance rather than in index, it covers designs that are not temporal at all, such as a spatial transect where adjacency means physical proximity.
Locating a real series on the ψ \psi ψ axis. It is tempting to read the persistence of a series off the autocorrelation of the residuals of a fitted model, and Equation (4) would then map that number onto ψ \psi ψ . On these data this reading does not work, and the reason is worth recording because it recurs whenever smoothing and dependence are estimated together. The residual autocorrelation is not a property of the errors alone: it moves with the amount of smoothing. A fit that over-smooths leaves part of the signal in the residuals and inherits its persistence; a fit that follows the data closely absorbs the persistence into the fitted curve and leaves residuals that are negatively correlated. On the pump price of § 3.1.1 of the paper the same series gives a lag-one residual autocorrelation of about + 0.87 +0.87 + 0.87 at a fit with 24 effective degrees of freedom and of about − 0.20 -0.20 − 0.20 at the fit actually selected, which carries 309. The selected fit is the flexible one, so what its residuals report is not the absence of dependence in the errors but the presence of enough flexibility to have taken it up. Two consequences follow. Persistence in the errors and flexibility in the signal are not separately identified from a single series without further assumptions, so no single number read off one fit can place a series on the ψ \psi ψ axis. The experiment therefore records, at each ψ \psi ψ and at each estimator’s own calibrated ξ \xi ξ , the residual autocorrelation of the selected fit so that a value measured on real data is matched against simulated values rather than against ψ \psi ψ itself.
Figure 2 depicts some examples drawn from this DGP. Because the marginal variance is held at σ 2 \sigma^2 σ 2 whatever ψ \psi ψ , the scatter does not grow down the rows. Persistence here is not extra noise but the same noise arranged differently. What changes is that the cloud collapses from a symmetric swarm into a slow wobble that hugs the curve, and that is the mechanism behind the failure of random-fold cross-validation under dependence: a held-out point is then nearly predicted by the neighbors left in the training set.
Scenario 3: a mixture of deterministic and stochastic trend signal
Setting ψ = 0 \psi= 0 ψ = 0 and letting ω \omega ω vary keeps the errors independent and makes the signal itself random:
y i = f i + ∑ j = 1 q s j 1 [ c j , 1 ] ( x i ) + ϵ i , ϵ i ∼ i i d N ( 0 , σ 2 ) , f i = g i − g ˉ sd ( g ) , g i = 1 − ω b i − b ˉ sd ( b ) + ω r i − r ˉ sd ( r ) , [ r i + 1 r i + 1 ′ ] = [ 1 δ i 0 1 ] [ r i r i ′ ] + ν i , ν i ∼ N ( 0 , [ δ i 3 / 3 δ i 2 / 2 δ i 2 / 2 δ i ] ) . (23) \begin{aligned}
y_i &= f_i + \sum_{j=1}^{q} s_j \mathbf{1}_{[c_j,1]}(x_i) + \epsilon_i,\qquad\epsilon_i \mathrel{\overset{\mathrm{iid}}{\sim}} \mathrm{N}(0,\sigma^2),\\
f_i &= \frac{g_i-\bar{g}}{\operatorname{sd}(g)},\qquad
g_i = \sqrt{1-\omega}\,\frac{b_i-\bar{b}}{\operatorname{sd}(b)} + \sqrt{\omega}\,\frac{r_i-\bar{r}}{\operatorname{sd}(r)},\\
\begin{bmatrix}r_{i+1}\\r'_{i+1}\end{bmatrix}
&=
\begin{bmatrix}1 & \delta_i\\0 & 1\end{bmatrix}
\begin{bmatrix}r_i\\r'_i\end{bmatrix}
+\nu_i,\qquad
\nu_i \sim\mathrm{N}\left(0,
\begin{bmatrix}
\delta_i^3/3 & \delta_i^2/2\\
\delta_i^2/2 & \delta_i
\end{bmatrix}\right). \tag*{(23)}
\end{aligned} y i f i [ r i + 1 r i + 1 ′ ] = f i + j = 1 ∑ q s j 1 [ c j , 1 ] ( x i ) + ϵ i , ϵ i ∼ iid N ( 0 , σ 2 ) , = sd ( g ) g i − g ˉ , g i = 1 − ω sd ( b ) b i − b ˉ + ω sd ( r ) r i − r ˉ , = [ 1 0 δ i 1 ] [ r i r i ′ ] + ν i , ν i ∼ N ( 0 , [ δ i 3 /3 δ i 2 /2 δ i 2 /2 δ i ] ) . (23)
ω \omega ω is swept over { 0 , 0.25 , 0.5 , 0.75 , 1 } \{0,0.25,0.5,0.75,1\} { 0 , 0.25 , 0.5 , 0.75 , 1 } .
Figure 2 (PDF p. 33). Dependent errors: the same signal and the same jumps, only the errors change. n = 200 n = 200 n = 200 , σ = 0.10 \sigma= 0.10 σ = 0.10 , one replication. Blue: the truth. Dashed: the true jump locations. Down a column the seed is fixed. Columns: J = 0 J = 0 J = 0 , J = 1 J = 1 J = 1 , J = 5 J = 5 J = 5 . Rows: y = 0 y = 0 y = 0 (ACF1 = 0.02 _1 = 0.02 1 = 0.02 ), y = 0.3 y = 0.3 y = 0.3 (ACF1 = 0.40 _1 = 0.40 1 = 0.40 ), y = 0.6 y = 0.6 y = 0.6 (ACF1 = 0.60 _1 = 0.60 1 = 0.60 ), y = 0.9 y = 0.9 y = 0.9 (ACF1 = 0.85 _1 = 0.85 1 = 0.85 ). Figure 2: Scenario 2, Equation (22) with σ = 0.10 \sigma= 0.10 σ = 0.10 . Rows: the persistence ψ \psi ψ , with the lag-one autocorrelation it realizes on average. Columns: the number of true discontinuities. Points are the observations, the blue line is the signal, and the dashed rules are the true jump locations. Within a column the seed is fixed, so the abscissas, the standardized shocks and the jump configuration are identical going down and only the correlation of the errors changes.
The three regimes.
At ω = 0 \omega= 0 ω = 0 the signal is the deterministic curve and the only randomness in the signal is where the jumps fall (e.g., the case of a dose–response relation: a stable link between a covariate and a response, the same whenever and wherever it is measured, which a large enough sample would pin down exactly); there the estimator treats as random something that is fixed, so that the smoothing constant it selects measures how much data are needed to resolve a curve that is genuinely there;
At ω = 1 \omega= 1 ω = 1 the signal is entirely a draw from the stochastic process: in continuous time an integrated Wiener process, in discrete time an integrated random walk (i.e., the local linear trend of unobserved-components models and the model implicit in the Hodrick–Prescott filter Maranzano and Pelagatti, [2 ] , so that every series that has ever been HP-filtered is a series someone chose to describe as ω = 1 \omega= 1 ω = 1 ). It is not stationary and there is no fixed function that it follows (i.e., a second realization of the same experiment would produce a different curve). When ω = 1 \omega= 1 ω = 1 the signal is a draw from the process the spline assumes, so the estimator’s view of it is correct : this is the well-specified case rather than a misspecified one (i.e., whatever goes wrong there cannot be blamed on the model being wrong about the signal); the only things left to estimate being the scale λ \lambda λ , that is, how much of the observed variation is signal, and the location of the jumps.
Two cautions on ω \omega ω . First, it is not a roughness axis: an integrated Wiener path is continuously differentiable and, standardized, carries no more curvature than b b b — the mean absolute second difference runs 0.52, 0.69, 0.52 at ω = 0 , 0.5 , 1 \omega= 0, 0.5, 1 ω = 0 , 0.5 , 1 . Second, it induces no dependence: the residual autocorrelation at the calibrated value stays negative and moves toward zero as ω \omega ω rises, from − 0.23 -0.23 − 0.23 to − 0.10 -0.10 − 0.10 on average over the seven arms of Table 15 , so Scenario 2 is not reachable from Scenario 3.
Table 15. Calibrated values of ξ \xi ξ for each scenario and arm. The upper panel reports the value adopted for each source of λ \lambda λ (i.e., from SSJ–MLE or by cross-validation at K = 5 K=5 K = 5 , 10 and 20, with folds drawn at random or as contiguous blocks): the midpoint of minimax regret in RMSE and in the count error, each pooled over the levels of the setting, rounded up to the grid. The values are pooled over the four sample sizes of the design and depend on the length of the series. The lower panel reports the two pooled rules in the reference case, over ψ ≥ 0.6 \psi\ge0.6 ψ ≥ 0.6 under dependent errors, and at ω = 1 \omega=1 ω = 1 under a stochastic trend.
Why a stochastic trend is the right alternative. The roughness penalty ∫ ( f ′ ′ ) 2 \int(f^{\prime\prime})^2 ∫ ( f ′′ ) 2 is not a neutral measure of wiggliness: by the equivalence of Wahba [4 ] , the cubic smoothing spline is the posterior mean of a signal-extraction problem in which f f f is an integrated Wiener stochastic process (i.e., a process whose second derivative is white noise) observed with independent Gaussian error. Penalizing squared second derivatives and placing that prior on f f f are the same act seen from two sides; the smoothing constant is the ratio of the variance of the observation error to the variance of the innovations driving the process, and the fitted curve is the conditional expectation of the signal given the data. § 2.1.1 of the paper uses this equivalence to obtain the estimator from the Kalman filter. Here it is used to ask whether the estimator believes a signal looks like before it has seen any data, and then to generate signals from exactly that belief. Any other rough alternative (e.g., an arbitrary jagged function, a rougher deterministic curve) would confound two things: whether the calibration survives a moving signal and whether it survives a signal the estimator was never designed for. Drawing from the prior separates them, because on that class the estimator is right. Noticeable, a draw from an integrated Wiener process is continuously differentiable (i.e., its slope is a Brownian motion, so the path has no kinks) but it is nowhere twice differentiable, and the roughness penalty evaluated at such a draw is infinite. This is the ordinary situation in nonparametric smoothing: the penalty is a shrinkage device, not a constraint, and the posterior mean is far smoother than the prior draws from which it is computed. What ω \omega ω moves is the class the signal belongs to, from a function with a bounded second derivative to one with none.
The transition equation. The third line of Equation (23) is the exact transition of the state-space model of § 2.1.1 of the paper at unit process scale, carrying level and slope; using the exact transition, rather than shocking the slope at unit spacing, is what makes r r r a legitimate path of the process on an irregular design rather than only on a grid. Indexing the departure by λ \lambda λ directly would seem more natural, since a small value indicates a flexible fit in the estimator and would indicate a wandering signal here. However, it does not work: the process has no stationary distribution and its variance grows with the range of x x x , so there is no fixed relationship between λ \lambda λ and the amplitude of f f f , and amplitude and persistence would move together along the grid. The mixture of Equation (3) separates them, and a λ \lambda λ consistent with a given draw can be read off afterward from a fit rather than imposed as an input.
Figure 3 shows selected examples of Scenario 3.
A joint representation of the three DGPs
Figure 4 depicts the three scenarios side by side. First column: at ψ = 0 \psi= 0 ψ = 0 and ω = 0 \omega= 0 ω = 0 the data are byte-identical (i.e., the first column repeats unchanged), so the top row is one experiment drawn three times, and then move apart in different directions. In the second column the signal never changes and the observations stop scattering symmetrically about it, so that by ψ = 0.9 \psi= 0.9 ψ = 0.9 they lie in runs on one side of the curve and then the other. In the third column the errors never change and the signal itself loses its shape, so that by ω = 1 \omega= 1 ω = 1 what is left is a wandering level on which the three discontinuities are the only sharp features.
Design of the experiment
The factorial is n ∈ { 50 , 100 , 200 , 400 } n \in\{50, 100, 200, 400\} n ∈ { 50 , 100 , 200 , 400 } , σ ∈ { 0.05 , 0.10 , 0.15 } \sigma\in\{0.05, 0.10, 0.15\} σ ∈ { 0.05 , 0.10 , 0.15 } and q ∈ { 0 , 1 , 3 , 5 } q \in\{0, 1, 3, 5\} q ∈ { 0 , 1 , 3 , 5 } , for 48 cells, crossed with ten levels of ψ \psi ψ or five of ω \omega ω . Each cell is replicated 200 times. The allocation is flat across n n n rather than tapered, and flatly for a reason specific to the choice rule: minimax regret takes the maximum of the regret over cells, so the noisiest cell is the one most likely to decide the answer, and tapering the allocation with n n n would hand that decision to the large-n n n cells.
Here n n n is the length of the series: 50 is short enough that a single discontinuity is a substantial fraction of the information in the sample, 400 long enough that the asymptotic arguments begin to hold. σ \sigma σ is the standard deviation of the errors, and since the signal is standardized to unit variance it is also the reciprocal of the signal-to-noise ratio: at σ = 0.05 \sigma= 0.05 σ = 0.05 the discontinuities, whose sizes lie between 0.30 and 0.70, stand six to fourteen error standard deviations clear of the noise, and at σ = 0.15 \sigma= 0.15 σ = 0.15 only two to five, so the three levels run from an easy detection problem to a hard one. Finally, q q q is the number of true discontinuities in the series. q = 0 q = 0 q = 0 is used to measure false discoveries where there is nothing to find, while q = { 1 , 3 , 5 } q = \{1, 3, 5\} q = { 1 , 3 , 5 } range from a single break to a number large enough that the estimator must spread a fixed budget over several of them.
ξ \xi ξ enters only the selection among grid points and never a fit, so the regularization path over a 25-point grid of M M M running to twice the range of x x x is computed once per replication and every ξ ∈ { 0 , 0.05 , … , 1 } \xi\in\{0, 0.05, \ldots, 1\} ξ ∈ { 0 , 0.05 , … , 1 } re-selects from it. All values of ξ \xi ξ are therefore compared on identical fits, and the standard error of a difference between two of them is an order of magnitude smaller than that of either level.
SSJ–KFS and SSJ–ATW take λ \lambda λ as given, supplied in several ways:
from an SSJ–MLE fit taken at a fixed reference ξ = 0.25 \xi= 0.25 ξ = 0.25 , held constant while ξ \xi ξ is calibrated so that the exercise is not circular,
by cross-validation under two fold layouts (Section 1), that is, folds are drawn at random and contiguous blocks at K ∈ { 5 , 10 , 20 } K \in\{5, 10, 20\} K ∈ { 5 , 10 , 20 } each. Random folds leave a held-out point surrounded by its own neighbors, which once the errors are persistent nearly determine it, so the criterion is minimized by a fit that all but interpolates and the selected λ \lambda λ collapses. Contiguous blocks remove that adjacency.
Figure 3 (PDF p. 36). Scenario 3, Equation (23) with σ = 0.10 \sigma= 0.10 σ = 0.10 . Rows: the share ω \omega ω of the signal’s variance carried by the integrated Wiener path; columns: the number of true discontinuities. Within a column the seed is fixed, so the errors and the jump configuration are identical going down and only the signal changes.
Figure 4 (PDF p. 37). The three scenarios in one frame, at q = 3 q = 3 q = 3 simulated discontinuities, n = 200 n = 200 n = 200 and σ = 0.15 \sigma= 0.15 σ = 0.15 , the largest noise level of the factorial, so that the change in the texture of the errors down the second column is visible at all. Columns are the scenario, rows the departure, matched by the fraction of each swept range, since ψ \psi ψ stops at 0.9 and ω \omega ω at 1; each panel is labeled with the value it uses. Points are the observations, the blue line is the signal, and the dashed rules are the true jump locations. The seed is the same in every panel, so the abscissas, the standardized shocks and the jump configuration are common to all fifteen; the vertical scale is common as well, since the signal is standardized to unit variance in every scenario.
Criteria
Two rules are used to identify calibration values, both of the minimax-regret form. Within a cell, the regret of a value of ξ \xi ξ is the loss it attains there minus the smallest loss attainable in that cell by any ξ \xi ξ ; the rule selects the ξ \xi ξ whose largest regret over the cells is smallest, so that no cell of the design is served badly, which is the guarantee a single constant meant for data of unknown difficulty needs. The two rules differ in the loss: the first is built on the reconstruction of the signal, the second on the number of discontinuities declared. Both are reported because they answer different questions, and because under the departures of Scenarios 2 and 3 they part company; a third criterion, the zero of the signed count error, is used as a check on the second and is defined at the end of this section.
Minimax regret in RMSE. Writing r ( ξ , c ) r(\xi,c) r ( ξ , c ) for the RMSE of ξ \xi ξ in cell c c c , measured against the true signal on a common 500-point grid, the regret is ρ ( ξ , c ) = r ( ξ , c ) − min ξ ′ r ( ξ ′ , c ) \rho(\xi,c)=r(\xi,c)-\min_{\xi'}r(\xi',c) ρ ( ξ , c ) = r ( ξ , c ) − min ξ ′ r ( ξ ′ , c ) and the reported value is arg min ξ max c ρ ( ξ , c ) \arg\min_{\xi}\max_{c}\rho(\xi,c) arg min ξ max c ρ ( ξ , c ) .
The cell is summarized by the mean over replications. Reporting the per-cell minimizer would overstate what the design resolves and averaging across cells would let the easiest dominate. The replication distribution within a cell is mildly right skewed, so the median was computed as well: at the baseline it moves the calibrated value by at most one grid step on fourteen of the fifteen arms and by three on the fifteenth (SSJ–ATW with contiguous blocks at K = 5 K=5 K = 5 ), never downward, which is why it is recorded here and not tabulated.
Minimax regret in the count error. The rule above is built on reconstruction error and does not validate whether adding or removing one discontinuity in a long series moves the error very little, so the regret valley is shallow and becomes shallower as the departure grows, until the minimizer is a point drawn out of a plateau. We therefore also apply the minimax regret rule to a loss built on the count. Writing ℓ ( ξ , c ) = E ∣ k ^ − q ∣ \ell(\xi,c)=\mathbb{E}\lvert\hat{k}-q\rvert ℓ ( ξ , c ) = E ∣ k ^ − q ∣ for the mean absolute error in the number declared in cell c c c , over the 36 cells with q ∈ { 1 , 3 , 5 } q\in\{1,3,5\} q ∈ { 1 , 3 , 5 } (the cells with q = 0 q=0 q = 0 are excluded because there the count error is simply the false-discovery count, which is reported on its own), the calibrated value is arg min ξ max c { ℓ ( ξ , c ) − min ξ ′ ℓ ( ξ ′ , c ) } \arg\min_{\xi}\max_{c}\{\ell(\xi,c)-\min_{\xi'}\ell(\xi',c)\} arg min ξ max c { ℓ ( ξ , c ) − min ξ ′ ℓ ( ξ ′ , c )} .
We stress that averaging a signed count error over q q q would let over-declaration at q = 1 q=1 q = 1 cancel under-declaration at q = 5 q=5 q = 5 , and under strong dependence the average can then be brought to zero by cancellation, or never reach it at all. The obvious alternative, used below as a check, is the zero of the signed count error : writing e ˉ ( ξ ) \bar{e}(\xi) e ˉ ( ξ ) for the mean, over the 36 cells with q > 0 q>0 q > 0 , of the number declared minus q q q , it is the value of ξ \xi ξ at which e ˉ \bar{e} e ˉ crosses zero from above, interpolated linearly between grid points, that is, the ξ \xi ξ at which the estimator declares on average the right number; it is undefined when e ˉ ( ξ ) < 0 \bar{e}(\xi)<0 e ˉ ( ξ ) < 0 at every ξ \xi ξ , which happens when the estimator cannot reach the true number at any penalty. The absolute error is defined in every case, and where the signed error does have a zero the two agree to within a step of the grid (over the 209 combinations of arm and departure level in which the zero exists, out of 225, the correlation between the two is 0.92, the median difference 0.03, and 171 agree to within one step of the ξ \xi ξ grid).
Why not locations. Neither rule asks whether the declared discontinuities are in the right places. While a reader may expect the calibration to rest on that criterion, it cannot: any summary of the confusion matrix between declared and true locations has a denominator that involves the number of true positives, and with q ∈ { 0 , 1 , 3 , 5 } q\in\{0,1,3,5\} q ∈ { 0 , 1 , 3 , 5 } that number takes so few values that the summary’s optimum wanders over the whole grid from one arm to the next. A location score also tolerates a single discontinuity being declared twice at adjacent intervals, which might occur at these values of ξ \xi ξ ; the count criteria charge for it. Location accuracy is therefore reported in Section 3 of the paper as a consequence of the calibration, not used as an input to it.
Results
Scenario 1: the baseline
Table 1 (PDF p. 39). Scenario 1. The calibrated ξ \xi ξ for each of the fifteen arms under the minimax regret in RMSE and minimax regret in the count error rules, each with the worst-case regret it attains (i.e., on the RMSE scale for the first, in discontinuities for the second). Beneath each calibrated value, in small type, the central half of the same criterion computed cell by cell: it is the spread of the best ξ \xi ξ over the design cells that one pooled number has to cover, not a confidence interval.
The two rules agree to within one grid step on every arm but SSJ–MLE, for which the count rule sits two steps below the reconstruction rule, at 0.20 against 0.30, and the alternative ways of supplying λ \lambda λ are nearly indistinguishable. For reconstruction the cells put the best ξ \xi ξ inside a band of median width 0.15, three steps of the grid, so the pooled value is close to what a typical cell would have chosen on its own.
The count loss is a different matter: its band has median width 0.56, more than half the grid, because the cells with one discontinuity want a large penalty and those with five a small one, and the pooled value is the compromise between them; under the departures the bands widen further. Notice that SSJ–KFS settles below SSJ–MLE, at 0.15 against 0.20 and 0.30, for a reason that is general rather than procedural. At a given λ \lambda λ and budget the two fit the same model, but SSJ–KFS takes λ \lambda λ as given, from a preliminary SSJE–MLE fit, while SSJ–MLE re-estimates it at every point of the budget grid. Supplying λ \lambda λ fixes one parameter before the budget is chosen: the fit is stiffer, part of its degrees of freedom have already been given up, and the search over the budget can buy less, so the additional penalty that ξ \xi ξ governs has to be milder. The same ordering holds between the sources of λ \lambda λ within an estimator: under the count rule, with λ \lambda λ from SSJ–MLE the calibrated value is never above the one obtained with a cross-validated λ \lambda λ , 0.15 against 0.15–0.20 for SSJ–KFS and 0.25 against 0.25–0.30 for SSJ–ATW, since a cross-validated constant is chosen on the data at hand and leaves the budget more to buy.
Every value lies between 0.10 and 0.30, so ξ = 1 \xi= 1 ξ = 1 , the top of the admissible range, is rejected throughout, and so is the nominal floor of 0.50 derived from Chen and Chen [1 ] and discussed in § 20.5 .
Scenario 2: serially dependent errors
Table 2 (PDF p. 40). Scenario 2. The calibrated ξ \xi ξ against the persistence ψ \psi ψ of the errors under the two rules, minimax regret in RMSE and minimax regret in the count error, both pooled over sample size. Beneath each, in small type, the central half of the same criterion computed cell by cell.
Persistence substantially raises the value that the count rule asks for: SSJ–ATW with contiguous blocks at K = 20 K=20 K = 20 moves from 0.25 at ψ = 0 \psi= 0 ψ = 0 to 0.55 at ψ = 0.9 \psi= 0.9 ψ = 0.9 and SSJ–KFS from 0.15 to 0.30 by ψ = 0.3 \psi= 0.3 ψ = 0.3 , where it settles (0.25 at ψ = 0.9 \psi= 0.9 ψ = 0.9 ), and the direction is the same for every arm. The reconstruction rule moves in the same direction, from 0.30 to 0.45 for SSJ–ATW and from 0.15 to 0.45 for SSJ–KFS.
Only contiguous blocks preserve that response. With random folds the movement of the count rule is smaller (SSJ–ATW goes from 0.30 to 0.40 and SSJ–KFS from 0.20 to 0.25), and it is with random folds that SSJ–ATW stops reaching the true number of discontinuities at all over the middle of the range, whatever the penalty. The mechanism is visible in the selected smoothing constant, which collapses under random folds as ψ \psi ψ grows while the blocked layouts hold it: a held-out point is then nearly predicted by its neighbors in the training set, the criterion rewards a curve that follows the errors, and once λ \lambda λ has collapsed the fitted curve absorbs the discontinuities itself, so that no penalty on the count can recover what the smoothing constant has given away. The same collapse is visible in the reconstruction: from moderate persistence onwards the blocked layouts attain the lowest error of any way of choosing λ \lambda λ , for both estimators and at every ξ \xi ξ , after costing a little under independence.
Regardless of the fold layout, SSJ–KFS suppresses false discoveries at a markedly lower ξ \xi ξ than SSJ–ATW, whose cross-validated arms do not bring the false-discovery rate down anywhere on the grid. The two estimators solve the same problem but their warping sequences are not equally sparse, and this is the simulated counterpart of what § 3.1.1 of the paper finds on the industrial prices, where SSJ–MLE never raises the false alarm at the 2022 policy date and some SSJ–ATW variants do.
The numbers behind these readings are the following:
Only contiguous blocks respond to the dependence. With random folds the movement of the count rule is smaller (i.e., SSJ–ATW goes from 0.30 to 0.40 and SSJ–KFS from 0.20 to 0.25, against 0.25 to 0.55 and 0.15 to 0.30 for the corresponding block layouts) and it is with random folds that SSJ–ATW stops reaching the true number of discontinuities at all between ψ = 0.5 \psi= 0.5 ψ = 0.5 and ψ = 0.8 \psi= 0.8 ψ = 0.8 , whatever the penalty (Section 5). The mechanism is visible in the selected smoothing constant, whose median falls from 10 − 5 10^{-5} 1 0 − 5 at ψ = 0 \psi= 0 ψ = 0 to 10 − 7 10^{-7} 1 0 − 7 by ψ = 0.4 \psi= 0.4 ψ = 0.4 under random folds while the blocked layouts at K = 10 K = 10 K = 10 and K = 20 K = 20 K = 20 hold at 10 − 5 10^{-5} 1 0 − 5 up to ψ = 0.6 \psi= 0.6 ψ = 0.6 and reach only 10 − 6 10^{-6} 1 0 − 6 from ψ = 0.7 \psi= 0.7 ψ = 0.7 . Once λ \lambda λ has collapsed the fitted curve absorbs the discontinuities itself, nothing is left for the budget to buy, and no penalty on the count can recover what the smoothing constant has given away.
The same collapse is visible in the reconstruction. From ψ = 0.3 \psi= 0.3 ψ = 0.3 onwards the blocked layouts at K = 10 K = 10 K = 10 and K = 20 K = 20 K = 20 attain a lower RMSE than any other way of choosing λ \lambda λ , for both estimators and at every ξ \xi ξ : at ψ = 0.5 \psi= 0.5 ψ = 0.5 , 0.086 against 0.098 with random folds, a gap of more than 10%. The ordering reverses at ψ = 0 \psi= 0 ψ = 0 , where blocks cost about 2%, so contiguous folds are best understood as insurance. Blocks at K = 5 K = 5 K = 5 are the exception at both ends: a fifth of the sample is too long a stretch to hold out, and the criterion is then minimized by a smoothing constant that over-smooths (Figure 8 ).
Figure 8. Scenario 2, RMSE against the true signal, one level of ψ \psi ψ per row, on a common vertical scale. From ψ = 0.2 \psi= 0.2 ψ = 0.2 onwards the two block-fold curves at K = 10 K = 10 K = 10 and K = 20 K = 20 K = 20 lie below every other layout, for both estimators and at every ξ \xi ξ . This is the clearest single statement of what the fold layout is worth once the errors are dependent.
SSJ–MLE loses the ability to detect rather than needing a different penalty. Its calibrated value in the count error stays between 0.15 and 0.25 at every ψ \psi ψ , and its minimax-regret value in RMSE between 0.25 and 0.35, but its signed count error stops reaching zero from ψ = 0.7 \psi= 0.7 ψ = 0.7 onwards. At high persistence the estimator cannot reach the true number at any ξ \xi ξ , so there is no trade for ξ \xi ξ to make and the count target becomes unattainable. Its row is a diagnosis, not a recommendation.
SSJ–KFS suppresses false discoveries at a markedly lower ξ \xi ξ than SSJ–ATW, whatever the fold layout. Taking one spurious discontinuity per twenty replications as the threshold, SSJ–KFS reaches it between 0.25 and 0.29 at ψ = 0 \psi= 0 ψ = 0 with λ \lambda λ from maximum likelihood, from random folds and from blocks at K = 20 K = 20 K = 20 (at 0.35 with blocks at K = 10 K = 10 K = 10 , and not before ξ = 0.8 \xi= 0.8 ξ = 0.8 with blocks at K = 5 K = 5 K = 5 , whose constant over-smooths), and between 0.33 and 0.51 at ψ = 0.9 \psi= 0.9 ψ = 0.9 ; SSJ–MLE needs 0.85; and SSJ–ATW reaches it only at ψ = 0 \psi= 0 ψ = 0 , at 0.52 with λ \lambda λ from maximum likelihood and at 0.74 and 0.95 with random folds at K = 20 K = 20 K = 20 and K = 5 K = 5 K = 5 , while none of its arms reaches it anywhere on ξ ≤ 1 \xi\le1 ξ ≤ 1 once $\psi > 0 ({xref}Figure 10). The two estimators solve the same problem but their warping sequences are not equally sparse, and no admissible penalty makes SSJ–ATW as conservative as SSJ–KFS. {xref}Section 6.7` shows that at the highest levels of persistence it declares fewer discontinuities than there are at every ξ \xi ξ .
The zero of the signed count error moves below the cell-wise zeros as the persistence grows, which identifies cancellation. At ψ = 0 \psi= 0 ψ = 0 the zero of the signed count error defined in Section 5 lies within one grid step of the count rule on every one of the fifteen arms, with a median gap of 0.02. As ψ \psi ψ grows the pooled zero falls below the zeros computed
within the individual cells: for SSJ–MLE it is 0.10 at ψ = 0.4 \psi= 0.4 ψ = 0.4 and 0.02 at ψ = 0.6 \psi= 0.6 ψ = 0.6 , whereas the central half of the cells in which a zero exists lies at 0.16–0.31 and 0.10–0.28. A pooled zero below the cell-wise zeros cannot be a low optimum; it is produced by cancellation, the mean count error being brought to zero by over-declaration in the cells with q = 1 q = 1 q = 1 offsetting under-declaration in the cells with q = 5 q = 5 q = 5 . The same mechanism removes the zero altogether two levels later, at ψ ≥ 0.7 \psi\ge0.7 ψ ≥ 0.7 , and it is the reason the rule is built on the absolute count error, which cannot cancel (Figure 11 ).
Figure 11. Scenario 2, declared minus true over the cells with q ∈ { 1 , 3 , 5 } q \in\{1, 3, 5\} q ∈ { 1 , 3 , 5 } ; color is ψ \psi ψ , and the circles are the zeros of the curves, not the count rule of Table 2, which sits within a grid step of them where both exist. At ψ = 0 \psi= 0 ψ = 0 the zero sits almost exactly at 0.25 for SSJ–ATW and rises monotonically toward 0.50 as ψ \psi ψ grows. The SSJ–MLE panel shows the power collapse directly: at ψ = 0.9 \psi= 0.9 ψ = 0.9 the curve starts below zero at ξ = 0 \xi= 0 ξ = 0 , so the estimator cannot reach the true number at any penalty.
The dependence reverses the sample-size pattern. Under independence the calibrated value falls with n n n ; under strong persistence it rises with n n n , because a longer series supplies more observations but not proportionally more independent information, so the penalty required to hold the false-discovery rate rises with the length of the series instead of falling. § 20.4 reports the values at each sample size.
Figure 5 (PDF p. 43). Scenario 2: the calibrated ξ against ψ under five criteria. Rows one and two apply minimax regret to the mean and to the median cell summary; row three takes the maximum of exact recovery; row four the zero of the signed count error, kept as the check of Section 5; row five the minimax regret in the count error, which is the second rule of the tables. Rows four and five move together and are visibly smoother than the first two: a criterion built on the count is better conditioned than one that locates the minimum of a flat valley. A gap in row four means the signed error has no zero — the arm under-declares at every ξ — where row five is still defined.
Figure 6. Scenario 2, worst-case regret, one level of ψ \psi ψ per row. The valley flattens as ψ \psi ψ grows, which is why the minimax rule becomes weakly determined there and why the count criterion was introduced.
Figure 7. Scenario 2, worst-case regret; color is ψ \psi ψ .
Figure 9. Scenario 2, RMSE; color is ψ \psi ψ .
Figure 10. Scenario 2, discontinuities declared where there are none; color is ψ \psi ψ . Circles mark where a curve falls below 0.05 declarations per replication. SSJ–KFS reaches that level between ξ = 0.25 \xi= 0.25 ξ = 0.25 and 0.29 at ψ = 0 \psi= 0 ψ = 0 (later with contiguous blocks at K = 10 K = 10 K = 10 and K = 5 K = 5 K = 5 , whose constant over-smooths) and between 0.33 and 0.51 at ψ = 0.9 \psi= 0.9 ψ = 0.9 ; SSJ–MLE needs 0.85; and SSJ–ATW reaches it only at ψ = 0 \psi= 0 ψ = 0 , at 0.52 with λ \lambda λ from maximum likelihood and at 0.74 and 0.95 with random folds at K = 20 K = 20 K = 20 and K = 5 K = 5 K = 5 , while none of its arms reaches it anywhere on ξ ≤ 1 \xi\le1 ξ ≤ 1 once ψ > 0 \psi> 0 ψ > 0 . Regardless of the fold layout, SSJ–KFS therefore needs a markedly lower ξ \xi ξ than SSJ–ATW to drive the false-discovery rate to zero, and no admissible penalty makes SSJ–ATW as conservative as SSJ–KFS.
Figure 12. Scenario 2, the same, one level of ψ \psi ψ per row. At ψ = 0 \psi= 0 ψ = 0 the seven curves lie on top of one another — every fold layout behaves identically under independence — and from ψ = 0.5 \psi= 0.5 ψ = 0.5 onwards the block curves lie above the random ones throughout, for both SSJ–KFS and SSJ–ATW. The reading is that with random folds the fitted curve has already absorbed the discontinuities, so fewer are left to declare at any ξ \xi ξ ; the block curves stay higher because the smoothing constant they select leaves the discontinuities in the residual, where the budget can still buy them.
Figure 13. Scenario 2, exact recovery over the whole design, one level of ψ \psi ψ per row. The apparent improvement at large ψ \psi ψ is largely the composition effect described above: the estimator declares almost nothing and so is right in the q = 0 q=0 q = 0 cells.
Scenario 3: a stochastic-trend signal
Table 3 (PDF p. 52). Scenario 3. The calibrated ξ \xi ξ against the weight ω \omega ω of the stochastic-trend component under the two rules, minimax regret in RMSE and minimax regret in the count error, both pooled over sample size. Beneath each, in small type, the central half of the same criterion computed cell by cell.
Here the two criteria disagree. On reconstruction the baseline calibration transfers unchanged: across all fifteen arms and all five levels of ω \omega ω the minimax value stays between 0.15 and 0.30 for SSJ–ATW, 0.10 and 0.20 for SSJ–KFS and 0.30 and 0.45 for SSJ–MLE, so the calibration is not an artifact of the particular curve used, and only SSJ–MLE (upward) drifts. The reconstruction even improves at each arm’s calibrated value: a signal drawn from the estimator’s own prior is one it reconstructs well.
The declared count meanwhile deteriorates markedly. At ω = 1 \omega=1 ω = 1 , with nothing to find, the estimators with λ \lambda λ from SSJ–MLE declare several discontinuities per replication where at ω = 0 \omega=0 ω = 0 they declared almost none, and the same happens where there is something to find, so this is inflation of the count and not a failure confined to the null cells. The count rule therefore moves a long way: from 0.25 to 0.50 for SSJ–ATW at the maximum likelihood λ \lambda λ , from 0.15 to 0.45 for SSJ–KFS and from 0.20 to 0.40 for SSJ–MLE; with a cross-validated λ \lambda λ the movement is smaller, to 0.35–0.45 and to 0.20–0.30, but in the same direction for fourteen of the fifteen arms (SSJ–KFS with random folds at K = 20 K=20 K = 20 does not move). The rise is not gradual: flat to ω = 0.5 \omega=0.5 ω = 0.5 and steep thereafter, so a series only partly trend-like needs little correction.
Exact recovery improves, but only at a higher ξ \xi ξ . Its maximum over ξ \xi ξ rises for nearly every arm, and the value at which it is attained roughly doubles, to about 0.5. This is not a composition effect of the q = 0 q=0 q = 0 cells becoming easier: restricted to the cells where there is something to find, the probability of recovering the count exactly at the calibrated value is substantially higher at ω = 1 \omega= 1 ω = 1 than at ω = 0 \omega= 0 ω = 0 , in every arm.
The general reading is that in the presence of stochastic trends a single fixed constant might be wrong. A criterion built on reconstruction error is nearly blind to a change that only affects how a fitted warping sequence is turned into a list of discontinuities, and at ω = 1 \omega= 1 ω = 1 the estimator’s view of the signal is exactly right, so nothing can be blamed on misspecification. What is left is the warping budget and the rule that reads it. At ω = 1 \omega= 1 ω = 1 the criterion buys more warping than at ω = 0 \omega= 0 ω = 0 : in the cells with q = 0 q = 0 q = 0 , at each arm’s calibrated value, the selected budget rises from 0.05 to 0.14 for SSJ–MLE and from 0.14 to 0.22 for SSJ–ATW at λ \lambda λ from SSJ–MLE, the number of active warps from about one and a half to two and from three and a half to six, and for SSJ–KFS at that λ \lambda λ from under one to more than six, because a wandering trend is cheaper to follow by deforming the domain at a few places than by paying curvature everywhere (i.e., the effective degrees of freedom actually fall , from 0.19n n n to about 0.09n n n ). The declaration rule then asks only whether a warp holds more than a fixed share of the total warping, and one or two of those warps do, so they are reported as discontinuities on a signal that has none. The rule cannot distinguish warping bought to track a wandering trend from warping bought to represent a jump, and that is the whole of the effect.
The numbers behind these readings are the following.
Figure 16. Scenario 3, the same. The ordering of the fold layouts is the reverse of Figure 8 : with independent errors the random-fold curves lie below the block ones at every ω \omega ω , and blocks at K = 5 K = 5 K = 5 are the worst of the seven — a fifth of the sample is too long a stretch to hold out, and the criterion is then minimized by a constant that over-smooths.
The declared count deteriorates markedly with λ \lambda λ from SSJ–MLE. At ω = 1 \omega= 1 ω = 1 , with nothing to find, SSJ–KFS at a smoothing constant from SSJ–MLE declares 2.19 discontinuities per replication against 0.28 at ω = 0 \omega= 0 ω = 0 , SSJ–ATW at the same constant 1.46 against 0.64, and SSJ–MLE 0.94 against 0.41. The same happens where there is something to find, so this is inflation of the count and not a failure confined to the null cells (Figure 17 ).
Figure 17. Scenario 3, the same against ω \omega ω . The ω = 1 \omega= 1 ω = 1 curve sits well above the rest and decays far more slowly: SSJ–KFS at λ \lambda λ from maximum likelihood still declares about one spurious discontinuity per replication at ξ = 1 \xi= 1 ξ = 1 , where at ω ≤ 0.5 \omega\le0.5 ω ≤ 0.5 it is below one in twenty by ξ = 0.5 \xi= 0.5 ξ = 0.5 .
Exact recovery improves, but only at a higher ξ \xi ξ . The maximum over ξ \xi ξ of the probability of declaring exactly the right number rises from 0.29–0.36 at ω = 0 \omega= 0 ω = 0 to 0.32–0.39 at ω = 1 \omega= 1 ω = 1 for thirteen of the fifteen arms, and the value at which it is attained moves from 0.20–0.35 to 0.40–0.60 (0.85 for SSJ–KFS at λ \lambda λ from SSJ–MLE). This is not a composition effect of the q = 0 q = 0 q = 0 cells becoming easier: restricted to the cells where there is something to find, the probability at ξ = 0.45 \xi= 0.45 ξ = 0.45 is between 1.7 and 2.4 times higher at ω = 1 \omega= 1 ω = 1 than at ω = 0 \omega= 0 ω = 0 , in all fifteen arms without exception. What changes is the shape rather than the height (at ω = 0 \omega= 0 ω = 0 the curve falls steeply from a maximum near ξ = 0 \xi= 0 ξ = 0 , at ω = 1 \omega= 1 ω = 1 it is nearly flat) so the two are comparable at small ξ \xi ξ and far apart at the values the count criterion recommends (Figure 20 to Figure 22 ).
Figure 20. Scenario 3, the probability of declaring exactly the right number over the whole design, one level of ω \omega ω per row. Read the position of the maximum rather than its height: a quarter of the cells have q = 0 q = 0 q = 0 and become easier as the departure smooths the fit, so the level is not comparable across rows. The position moves steadily to the right as ω \omega ω grows.
Figure 22. Scenario 3, exact recovery restricted to the cells with q = 0 q = 0 q = 0 : the specificity half, which rises monotonically in ξ \xi ξ . The interior optimum of the combined criterion in Figure 20 exists only because this half and the previous one pull in opposite directions.
The source of λ \lambda λ matters more than the fold layout. Cross-validating λ \lambda λ protects SSJ–ATW entirely: its count at q = 0 q = 0 q = 0 is 0.37 at both ω = 0 \omega= 0 ω = 0 and ω = 1 \omega= 1 ω = 1 with random folds at K = 20 K = 20 K = 20 and falls from 0.50 to 0.50 with contiguous blocks, against 0.64 to 1.46 when the constant is inherited from SSJ–MLE. Between the two layouts there is little to choose, and in RMSE the ordering is the exact reverse of Scenario 2: here random folds lie below blocks at every ω \omega ω , which is the premium referred to above.
Figure 14. Scenario 3: the same five criteria against ω \omega ω . The first two rows are nearly flat and the last two are not, which is the central observation of § 20.3 : what ω \omega ω costs is detection accuracy, not reconstruction, and a criterion built on RMSE cannot see it.
Figure 15. Scenario 3, worst-case regret, one level of ω \omega ω per row. At ω = 1 \omega= 1 ω = 1 the SSJ–MLE valley is essentially level between 0.35 and 0.55: the argmin there is a point drawn out of a plateau.
Figure 18. Scenario 3, one level of ω \omega ω per row. The zero of the curve barely moves up to ω = 0.5 \omega= 0.5 ω = 0.5 and then travels sharply to the right. At ω = 1 \omega= 1 ω = 1 SSJ–ATW crosses zero at about 0.36 with a cross-validated λ \lambda λ and at 0.46 with λ \lambda λ from SSJ–MLE, SSJ–KFS at about 0.30 and 0.45, and SSJ–MLE at 0.43. Notice that the count rule of Table 3 is not the same quantity, since it protects the worst cell rather than the average, and at ω = 1 \omega= 1 ω = 1 it gives 0.35 to 0.50 for SSJ–ATW, 0.40 for SSJ–MLE and 0.20 to 0.45 for SSJ–KFS depending on the source of λ \lambda λ . At ω = 0.75 \omega= 0.75 ω = 0.75 , by contrast, a value of 0.30 or a little below is still adequate for every arm.
Figure 19. Scenario 3, declared minus true; color is ω \omega ω .
Figure 21. Scenario 3, exact recovery restricted to the cells with q > 0 q > 0 q > 0 : the power half of the criterion. At ω = 0 \omega= 0 ω = 0 it falls steeply from a maximum at ξ = 0 \xi= 0 ξ = 0 and has no interior optimum; at ω = 1 \omega= 1 ω = 1 it is nearly flat and develops a mild interior bump. This is the panel that shows the improvement at ω = 1 \omega= 1 ω = 1 is real rather than a composition effect — at ξ = 0.45 \xi= 0.45 ξ = 0.45 the curve is between 1.7 and 2.4 times higher at ω = 1 \omega= 1 ω = 1 than at ω = 0 \omega= 0 ω = 0 , in every arm.
Dependence on the length of the series
The calibrated value is not the same at every sample size. Table 4 reports it at each n n n for the baseline, for every arm and under both rules. Table 5 does the same for seven representative arms at every level of the two departures, under the reconstruction rule.
Table 4. Weeks of the estimation sample in which a change in the excise duty on motor fuels entered the surveyed prices, and the size of the change in euros per liter, positive for an increase. A dash means that the duty on that fuel did not change in that week.
The count rule is not shown there because at a single sample size it is decided by one cell: with nine cells left instead of thirty-six, the maximum is attained by a q = 5 q=5 q = 5 cell in which the estimator under-declares at every ξ \xi ξ , and the rule then returns the floor of the grid. Those entries at the floor are consistent with the rule of Section 6.5, of which they are the limit: none of them occurs at n = 50 n=50 n = 50 and 25 of the 30 occur at n = 200 n=200 n = 200 and n = 400 n=400 n = 400 , where the df log n n n term already levies most of the charge per discontinuity and, on the nine cells of one sample size, nothing is left for the extended term to supply, so the most permissive penalty minimizes the worst-case count error.
Table 4 (PDF p. 63). Scenario 1, by sample size, for every arm: minimax regret in RMSE on the left and in the count error on the right, each computed over the cells of one sample size. Beneath each value, in small type, the central half of the same criterion computed cell by cell. A 0.00 under the count rule is the floor of the grid: at that n n n the worst cell is one with five discontinuities in which the estimator under-declares at every ξ \xi ξ .
At the baseline the two rules disagree about the direction. The reconstruction rule falls with n n n : for SSJ–ATW from 0.40 at n = 50 n=50 n = 50 to 0.10 at n = 400 n=400 n = 400 , for SSJ–KFS from 0.20 to 0.10, for SSJ–MLE from 0.30 to 0.25. The count rule rises with n n n for SSJ–KFS, from 0.15 to 0.25, and drifts down for SSJ–ATW, from 0.30 to 0.20, until at n = 400 n=400 n = 400 its worst cell becomes one that under-declares and the rule drops to the floor.
A single constant is therefore a compromise in n n n under either rule, and in opposite directions for the two.
Table 5 (continued).
The two departures act on that pattern in opposite ways.
The stochastic trend leaves the baseline pattern untouched: at every level of ω \omega ω the value falls with n n n as in the reference column, and the rows of the right-hand block of Table 5 are nearly indistinguishable from one another (i.e., SSJ–ATW with blocks 0.35, 0.25, 0.15, 0.10 in the reference case and 0.25, 0.20, 0.15, 0.10 at ω = 1 \omega=1 ω = 1 ; SSJ–KFS at the maximum likelihood λ \lambda λ 0.20, 0.20, 0.15, 0.10 and 0.15, 0.10, 0.15, 0.15), and the rows at intermediate ω \omega ω differ from these by at most one grid step, save at n = 50 n=50 n = 50 for SSJ–ATW at the maximum likelihood λ \lambda λ , where the value drifts down from 0.40 to 0.25 across the range. The departure that moves the count criterion a long way moves this one nowhere at any sample size. The one exception is again SSJ–MLE, which shifts upward at ω = 1 \omega=1 ω = 1 , to 0.45, 0.45, 0.40, 0.35 against 0.30, 0.30, 0.25, 0.25, while still falling with the sample size.
The dependence reverses that pattern. Under strong persistence the fall of the reconstruction optimum turns into a rise: a longer series supplies more observations but not proportionally more independent information, so the penalty has to increase with n n n to compensate. Under independence the calibrated value falls with the length of the series, for the reason Section 6.5 makes precise (i.e., the plain BIC term charges more per discontinuity as n n n grows and less is left for the extended term to supply): SSJ–ATW with contiguous blocks runs 0.35, 0.25, 0.15, 0.10 over n = 50 n=50 n = 50 , 100, 200, 400 at ψ = 0 \psi=0 ψ = 0 . At ψ = 0.4 \psi=0.4 ψ = 0.4 the fall is already shallower, 0.40, 0.40, 0.30, 0.30, and at ψ = 0.9 \psi=0.9 ψ = 0.9 it has turned into a rise from n = 100 n=100 n = 100 onward, 0.35, 0.40, 0.50, the entry at n = 50 n=50 n = 50 being the floor of the grid; SSJ–KFS with contiguous blocks goes the same way, from 0.20, 0.20, 0.20, 0.15 at ψ = 0 \psi=0 ψ = 0 to 0.30, 0.30, 0.40, 0.55 at ψ = 0.9 \psi=0.9 ψ = 0.9 . With λ \lambda λ from maximum likelihood the rise is weaker, and for SSJ–ATW it is absent at ψ = 0.9 \psi=0.9 ψ = 0.9 , where the estimator has lost most of its power. Thus, a longer series supplies more observations but not proportionally more independent information, so the penalty has to increase to compensate rather than fall as the charge argument alone would predict. The two effects work against each other, and which of them dominates is a property of the data rather than of the estimator.
Table 6: The calibration by sample size under minimax regret in the count error, over the nine cells with q > 0 q > 0 q > 0 of each sample size. A 0.00 is the floor of the grid: at that sample size and level the worst cell is one with five discontinuities in which the estimator under-declares at every ξ \xi ξ , so the most permissive penalty minimizes the worst-case count error.
Table 6.
Table 6 (continued).
What the EBIC charges per discontinuity
Two of the results above look like anomalies and are not. Every value calibrated in this document sits below the nominal consistency floor of about 0.50 recalled in Section 2, and at the baseline the value that minimizes reconstruction error is not a constant but falls with the length of the series. Both dissolve once one asks what the criterion charges per discontinuity in total, rather than what ξ \xi ξ alone charges.
Because df grows with the number of active warps, the plain BIC term of (18) already charges each declared discontinuity before the extended term is added. Writing s s s for the increase in df that one activated warp produces (small for SSJ–MLE, larger for SSJ–KFS, largest for SSJ–ATW) the total charge per discontinuity is
Φ ( n , ξ ) = s log ( n ) + 2 ξ log ( n − 1 ) , (24) \Phi(n,\xi)=s\log(n)+2\xi\log(n-1), \tag*{(24)} Φ ( n , ξ ) = s log ( n ) + 2 ξ log ( n − 1 ) , (24)
and if what the data ask for is a charge C C C that does not depend on n n n , then the constant that delivers it is
ξ ( n ) = C − s log ( n ) 2 log ( n − 1 ) . (25) \xi(n)=\frac{C-s\log(n)}{2\log(n-1)}. \tag*{(25)} ξ ( n ) = 2 log ( n − 1 ) C − s log ( n ) . (25)
Whether the optimal charge is in fact flat in n n n is a testable statement: for each arm the reconstruction-optimal ξ \xi ξ is found cell by cell over the 36 cells with q ≥ 1 q\geq1 q ≥ 1 , turned into a charge by (24) , and regressed on log n \log n log n ; the charge is taken as flat when the slope is not distinguishable from zero at the 10% level. It holds for SSJ–KFS with λ \lambda λ from maximum likelihood and from contiguous blocks at K = 20 K=20 K = 20 , the two arms of Table 15 for which it is tested, and for SSJ–KFS with random folds at K = 5 K=5 K = 5 ; with random folds at K = 10 K=10 K = 10 and K = 20 K=20 K = 20 and with blocks at K = 5 K=5 K = 5 and K = 10 K=10 K = 10 the slope is positive, as it is for SSJ–MLE, and for the SSJ–ATW arms it is negative, that is, the charge these arms ask for falls with n n n faster than the plain BIC term supplies it, except with contiguous blocks at K = 5 K=5 K = 5 and K = 20 K=20 K = 20 , where it is not distinguishable from zero. Where the charge is flat, (25) tracks the observed optima closely at every sample size. Table 8 sets the rule against the observed optima for every arm whose charge is flat, and Table 7 gives it for the two SSJ–KFS arms of the table of adopted values, at the sample sizes of the design and beyond them.
Table 8. The rule of Equation (9) for the arms whose optimal charge is flat in n n n : C ^ \hat{C} C ^ is the mean charge over the 36 cells, “observed” the reconstruction-optimal ξ \xi ξ averaged over the nine cells of that sample size, and “rule” the value ( C ^ − s log n ) / { 2 log ( n − 1 ) } (\hat{C}-s\log n)/\{2\log(n-1)\} ( C ^ − s log n ) / { 2 log ( n − 1 )} , also at two sample sizes beyond the design.
Four things follow. First, the consistency floor is not violated: what the theory of Chen and Chen [1 ] constrains is the total charge per discontinuity, and the extended term supplies only what the df log n \log n log n term has not already levied, so a calibrated ξ \xi ξ well below 0.50 is consistent with a floor stated for ξ \xi ξ alone because part of the charge is paid elsewhere. On the ξ \xi ξ scale that part is largest for SSJ–ATW, whose effective constant comes close to the floor, and small for SSJ–KFS and SSJ–MLE; what the decomposition leaves is finite-sample, the floor being an asymptotic requirement for selection consistency while the calibration targets risk at the sample sizes of the design. Second, the fall of ξ \xi ξ with n n n at the baseline is not drift but the arithmetic of Equation (25) : holding the charge fixed while log n \log n log n grows requires exactly that. Third, the common charge can be read as follows: on the − 2 log L -2\log L − 2 log L scale, SSJ–ATW ends up charging each discontinuity about what the Akaike criterion charges for one free parameter, and SSJ–KFS about half of that; the difference tracks s s s , since SSJ–ATW’s degrees of freedom respond most strongly to an activated warp and more of its total charge is therefore already levied by the BIC term. Fourth, the reversal under persistence is the argument’s own premise failing rather than a contradiction of it: a fixed charge buys a fixed protection only if a longer series carries proportionally more information, and persistent errors are precisely the case in which it does not, so the charge, and with it ξ \xi ξ , has to rise with n n n instead.
In numbers. Expressed on the ξ \xi ξ scale, the implicit charge s log n / { 2 log ( n − 1 ) } s\log n/\{2\log(n-1)\} s log n / { 2 log ( n − 1 )} is 0.04 for SSJ–MLE (s = 0.07 s=0.07 s = 0.07 ), 0.07 for SSJ–KFS (s = 0.13 s=0.13 s = 0.13 ) and 0.18 for SSJ–ATW (s = 0.36 s=0.36 s = 0.36 ) at n = 100 n=100 n = 100 , so the effective constant is about 0.43 for SSJ–ATW at its reference value of 0.25, against the floor of 0.50, but only 0.22 for SSJ–KFS at 0.15 and 0.24 to 0.34 for SSJ–MLE at 0.20 to 0.30. The common charge is a reading of its own: it is about 2.1 for SSJ–KFS and, at the reference sample sizes, about 3.7 to 3.8 for SSJ–ATW; on the − 2 log L -2\log L − 2 log L scale a charge of 4 is two log-likelihood units per discontinuity, which is what the Akaike criterion charges for one free parameter, so SSJ–ATW’s calibrated penalty is close to charging each discontinuity as a single parameter and SSJ–KFS’s about half of that. And at ψ = 0.9 \psi= 0.9 ψ = 0.9 the realized lag-one autocorrelation is about 0.87, so consecutive observations share most of their information.
For practice this means that where n n n is far from the middle of the range studied here, Equation (25) should be preferred to any single constant, and that a comparison between the criterion and cross-validation which holds ξ \xi ξ fixed across sample sizes understates the criterion.
What the count rule costs in reconstruction
Where the two rules disagree, the reconstruction rule is the less demanding one, because the RMSE is nearly flat in ξ \xi ξ near its minimum; following the count rule therefore costs little in reconstruction almost everywhere, and Table 9 reports what it costs, as the increase in mean RMSE at the count-calibrated ξ \xi ξ relative to the reconstruction-calibrated one. The cost is at most about 1% in the reference case and 1.5% under serial dependence, for every estimator. It is material in one place only, a signal close to a stochastic trend with λ \lambda λ inherited from SSJ–MLE, where it reaches 9% for SSJ–KFS and 20% for SSJ–ATW; with λ \lambda λ cross-validated on contiguous blocks the same estimators pay 1% and 3%. That is the one setting in which the two objectives of the paper genuinely conflict, and it does so because at ω = 1 \omega= 1 ω = 1 the reconstruction rule wants a smaller ξ \xi ξ than in the reference case while the count rule wants a larger one; the table shows the price of resolving the conflict in favor of the count.
Table 9. The reconstruction cost of the count rule: the increase in mean RMSE at the count-calibrated ξ \xi ξ relative to the reconstruction-calibrated one, pooled over the 48 cells, in %, as a range over the arms of each estimator. A negative value means the count value does marginally better on the mean, which the minimax rule does not optimize.
Behavior by the number of true discontinuities
Table 10 shows, for seven representative arms, the mean number of discontinuities declared, the share of replications that get the number exactly right, and beneath them the median and the central half of the replication distribution, separately at q = 0 q = 0 q = 0 , 1, 3 and 5. Each arm is read at its own calibrated ξ \xi ξ from the count criterion, printed in the third column of the table.
Some considerations follow:
The declared count is close to all-or-nothing. With a single true discontinuity, more than half of the replications (i.e., between 52% and 71% of replications) declare nothing at all, and those that declare anything declare several (4.3 to 4.8 on average). The mean in the table averages those two branches and describes neither, which is why the median at q = 1 q = 1 q = 1 is zero while the central half reaches four. The shape is the same at every q q q : what the truth changes is the probability of the empty branch, which falls steadily from q = 0 q = 0 q = 0 to q = 5 q = 5 q = 5 , while the count declared on the non-empty branch stays near five throughout. The information about q q q is therefore carried almost entirely by whether anything is declared, and hardly at all by how many. The declaration rule of § 2.1.3 of the paper explains the shape: a warp is kept only if it clears both a share of the total warping and an absolute fraction of the mean sampling interval, and a fitted warping sequence that clears the absolute threshold at all usually clears it in several places at once.
Table 10. Mean number of discontinuities declared and, in round brackets, the share of replications declaring exactly the right number; beneath, in small type, the median of the same replication distribution and its central half. Each arm is read at its own calibrated ξ \xi ξ under the count rule, printed in the third column. Five settings: the baseline, two levels of the persistence and two of the stochastic-trend weight. 2,400 replications behind each entry.
Table 10 (continued).
Exact recovery rises with q q q . It is negligible at q = 1 q = 1 q = 1 and appreciable at q = 5 q = 5 q = 5 (i.e., it runs 1% to 3% at q = 1 q = 1 q = 1 against 23% to 30% at q = 5 q = 5 q = 5 ), not because the estimators count better when there is more to find but because the non-empty branch lands near five whatever the truth: it is right when the truth is five and wrong when it is one. A declared count of one or two is correspondingly rare. Reading the declared count as an estimate of q q q is therefore a mistake, and reading a declared count of zero as evidence that there is nothing to find is the same mistake in the other direction.
Persistence costs SSJ–MLE its power. By ψ = 0.6 \psi= 0.6 ψ = 0.6 SSJ–MLE declares 1.69, 2.00 and 2.17 against 1, 3 and 5, and the empty branch has taken over, at 49% of replications at q = 3 q = 3 q = 3 and 51% at q = 5 q = 5 q = 5 against 26% and 15% at the baseline. At ψ = 0.9 \psi= 0.9 ψ = 0.9 it declares 0.94, 1.80 and 2.28. The loss is not a threshold but a slide that begins at the first level of the persistence: at q = 5 q = 5 q = 5 the arm declares 4.43, 3.50, 3.02, 2.53, 2.00 and 2.03 as ψ \psi ψ runs from 0 to 0.5, and never recovers (Table 11 ). SSJ–KFS with contiguous blocks is barely touched at the strongest persistence: it declares 2.10, 3.26 and 4.38 at ψ = 0.9 \psi= 0.9 ψ = 0.9 , with the empty branch at 36%, 13% and 5%, and its exact recovery at q = 3 q = 3 q = 3 actually rises, from 16% to 36%, because the persistence damps the over-declaration that costs it accuracy under independence.
Table 11: Scenario 2, every level of the persistence. Mean number of discontinuities declared and, in brackets, the share of replications declaring exactly the right number, for seven representative arms at each true count q q q and each ψ \psi ψ . Each entry is read at that arm’s own calibrated ξ \xi ξ for that level of ψ \psi ψ , so a row is not a single penalty held fixed.
Table 11.
The false discoveries have a hump in the persistence. On null series SSJ–KFS at the maximum likelihood λ \lambda λ declares 0.28 at ψ = 0 \psi= 0 ψ = 0 , rises to 1.84 at ψ = 0.6 \psi= 0.6 ψ = 0.6 , where it is right only 68% of the time against 95% at the baseline, and falls back to 0.47 at ψ = 0.9 \psi= 0.9 ψ = 0.9 ; SSJ–MLE peaks at the same level, at 1.37, and SSJ–KFS with blocks later and lower, at 1.19 at ψ = 0.7 \psi= 0.7 ψ = 0.7 (Table 11 ). The cause is in the calibration itself: the count criterion balances over- against under-declaration, and at the levels where an arm begins to lose power on the series that contain discontinuities it answers by lowering ξ \xi ξ , from 0.25 at ψ = 0.3 \psi= 0.3 ψ = 0.3 to 0.15 at ψ = 0.6 \psi= 0.6 ψ = 0.6 for SSJ–KFS at the maximum likelihood λ \lambda λ , which is paid for on the null series. A criterion calibrated on the count alone is therefore not a criterion for specificity.
The stochastic trend costs specificity, and the fold layout buys it back. At ω = 1 \omega= 1 ω = 1 SSJ–KFS at the maximum likelihood λ \lambda λ declares discontinuities on a sizeable share of the series that have none (2.19 discontinuities and is right only 68% of the time), while the same estimator with contiguous blocks stays close to its baseline (1.00 and is right 84%); SSJ–ATW shows the same contrast (1.46 against 0.50). Where the signal wanders, how λ \lambda λ is chosen matters more for the false-discovery rate than the penalty does.
Half the departure is not half the damage. At ω = 0.5 \omega= 0.5 ω = 0.5 every arm is still close to its baseline at q = 0 q = 0 q = 0 (i.e., 84% to 96% correct against 86% to 95%), and the complete grid shows the same at ω = 0.75 \omega= 0.75 ω = 0.75 (i.e., 88% to 94% at ω = 0.75 \omega= 0.75 ω = 0.75 , Table 12 ): the specificity loss belongs entirely to the last step of the range, so a series that is partly trend-like needs no correction. Persistence is not like that: it costs SSJ–MLE its power from the first level and has taken half of it by ψ = 0.4 \psi= 0.4 ψ = 0.4 , so a practitioner who judges the departure to be moderate is safe in Scenario 3 and is not safe in Scenario 2.
Table 12: Scenario 3, every level of the stochastic-trend weight. As Table 11, against ω \omega ω .
Table 12. Scenario 3, every level of the stochastic-trend weight. As Table 11, against ω \omega ω .
Tables 11 and 12 give the whole grid rather than seven settings, so that the level at which each arm turns can be located rather than bracketed, and Figure 23 and Figure 24 draw the declared count against ξ \xi ξ rather than at one value of it, one row per true number of discontinuities, so that the opposite directions of the error at q = 1 q = 1 q = 1 and q = 5 q = 5 q = 5 , which the criteria of Section 5 average away, are visible side by side.
Figure 23. Scenario 2. Mean number of discontinuities declared against ξ \xi ξ , by the true number q q q . Color is the persistence ψ \psi ψ and line type the source of λ \lambda λ . Dashed rules mark the truth. Reading down a column shows the progression: at q = 0 q = 0 q = 0 every curve falls to zero and the only question is how soon; at q = 1 q = 1 q = 1 every curve crosses the truth from above, so the signed error has a zero for every arm; at q = 3 q = 3 q = 3 the zero survives only for some; at q = 5 q = 5 q = 5 the SSJ–MLE curve never reaches the line at any ξ \xi ξ once ψ \psi ψ is large, which is the power loss that leaves the pooled criterion without a target.
Figure 24. Scenario 3, the same against ω \omega ω . The contrast with Figure 23 is in the top row: there the curves separate by ψ \psi ψ only at small ξ \xi ξ , whereas here the ω = 1 \omega=1 ω = 1 curve stays above the rest well past ξ = 0.5 \xi=0.5 ξ = 0.5 , which is the inflation of the count on a signal that has no discontinuities at all.
Suggested calibrated values and concluding remarks
Under a departure the calibration runs over several levels of ψ \psi ψ or ω \omega ω , and the series a single constant has to serve may sit at any of them. The two rules are therefore pooled over the levels of a setting before a value is read off: the pooled rule is the minimax regret over the cells of every level at once, that is, the value of ξ \xi ξ whose worst regret over the 4 × 48 4 \times48 4 × 48 cells (RMSE) or the 4 × 36 4 \times36 4 × 36 cells (count) with ψ ≥ 0.6 \psi\ge0.6 ψ ≥ 0.6 is smallest, so that it returns one value per arm and not a range over levels. In the reference case and at ω = 1 \omega= 1 ω = 1 there is a single level and the pooled rule is the value of Tables 1 and 3. Table 13 reports the range over the arms of the calibrated values by the character of the series and by estimator, under the two pooled rules.
Three regularities are read off Table 13 .
Table 13. Calibrated values of ξ \xi ξ by the character of the series and by estimator, under minimax regret in RMSE and in the count error, each pooled over the levels of the setting. Each entry is the range over the ways of supplying λ \lambda λ ; the second row uses contiguous cross-validation folds, SSJ–MLE entering it with its own λ \lambda λ . A series with both properties was not simulated: it is bracketed by the last two rows rather than resolved by either.
First, in the reference case the two rules coincide for SSJ–KFS and SSJ–ATW and the ranges over the ways of supplying λ \lambda λ are at most two grid steps wide, so the reference calibration is settled: 0.15–0.20 for SSJ–KFS, 0.25–0.30 for SSJ–ATW, and 0.20 or 0.30 for SSJ–MLE according to the rule.
Second, under dependent errors both rules move upward with contiguous blocks (0.30–0.40 for SSJ–KFS and to 0.35–0.50 for SSJ–ATW, that is, three to five grid steps above the reference case). With random folds the reconstruction rule moves as well (to 0.25–0.30 and 0.35), while the count rule stays at its reference level, because a constant chosen on random folds has already absorbed the persistence into the fit and no penalty on the count can recover it. SSJ–MLE moves under the reconstruction rule only, to 0.30, its count rule falling to 0.15 because the estimator loses power rather than specificity as the persistence grows.
Third, under a stochastic trend the two rules separate: the reconstruction rule stays at the reference values or below, while the count rule rises to 0.20–0.45 for SSJ–KFS and 0.35–0.50 for SSJ–ATW, the upper ends of both ranges being the arms with λ \lambda λ inherited from SSJ–MLE. In every row SSJ–KFS lies below SSJ–ATW, so the constant is a property of the estimator as much as of the data.
To synthesize the interval values from Table 13 into a single value for each setting we consider a midpoint between the value calibrated according to the RMSE minimax and the count minimax (i.e., the average of the two pooled values rounded up to the grid).
Table 15 fixes, for each scenario and for each source of λ \lambda λ (i.e., from SSJ–MLE or by cross-validation at K = 5 K = 5 K = 5 , 10 and 20, with folds drawn at random or as contiguous blocks), the midpoint of the two pooled rules; the lower panel reports the two rules themselves. The number of folds is not what the value depends on: between K = 5 K = 5 K = 5 and K = 20 K = 20 K = 20 it moves by at most one grid step in every cell, and K = 10 K = 10 K = 10 lies within one step of either except with contiguous blocks under dependent errors. The paper reports K = 5 K = 5 K = 5 and K = 20 K = 20 K = 20 , with the latter being the better choice. With contiguous blocks since holding out a fifth of the series (i.e., K = 5 K = 5 K = 5 ) in one stretch over-smooths (Section 6.2).
Recall that the values are pooled over the sample sizes of the design and depend on n n n : Table 5 gives the calibration at each sample size and at every level of the departures.
Table 14. Calibrated values of ξ \xi ξ by the character of the series and by estimator, under the two syntheses of the pooled rules: the midpoint, rounded up to the grid, and the larger of the two. Each entry is the range over the ways of supplying λ \lambda λ ; the second row uses contiguous cross-validation folds, SSJ–MLE entering it with its own λ \lambda λ .
To testify the robustness of our choice, we also consider alternative synthesis criteria, such as the separate rules, and the larger of the two, which provides a more conservative penalty (Table 16 ). The two agree to within one grid step in the reference case and under dependent errors, where the rules themselves are close; they part company under a stochastic trend for the arms with λ \lambda λ inherited from SSJ–MLE, where the count rule alone asks for 0.45 and 0.50 and the reconstruction rule for 0.15, so the midpoint settles at 0.30–0.35 and the larger at the count value, and, less widely, for SSJ–ATW with contiguous blocks at K = 5 K = 5 K = 5 and 10, where the count rule asks for 0.45 and 0.40 against 0.20 for reconstruction (midpoint 0.35 and 0.30, larger 0.45 and 0.40). The midpoint is the synthesis adopted below: the two rules protect different things, the accuracy of the curve and the number of discontinuities declared, and the midpoint is the single value that gives up the least of both, since the RMSE is nearly flat around its minimum and the count regret rises slowly on the side of the larger penalty. Table 16 reports the same values under the four syntheses (i.e., each rule alone, the midpoint and the larger of the two) as a robustness check on the decision rule: the midpoint and the larger differ by at most one grid step everywhere except under a stochastic trend for the arms with λ \lambda λ from SSJ–MLE and for SSJ–ATW with contiguous blocks at K = 5 K = 5 K = 5 and 10, where the choice between them is the choice between the reconstruction of the curve and the number of discontinuities declared, at the reconstruction cost of Section 6.6.
The findings that carry over to practice are the following:
In the reference case the two minimax regret rules largely agree (Table 1), and supplying λ \lambda λ from a preliminary SSJ–MLE fit, which fixes one parameter before the budget is chosen, calls for a penalty no heavier than a cross-validated λ \lambda λ does (§ 20.1 );
Serial dependence and stochastic trends call for a heavier penalty than the reference case through different channels: persistence acts through λ \lambda λ , and the penalty responds to it when λ \lambda λ is cross-validated on contiguous blocks, and for SSJ–ATW also when λ \lambda λ is inherited from SSJ–MLE, but not with random folds (Table 2); a stochastic trend leaves reconstruction unaffected and inflates the declared count, so only the count rule moves (Table 3);
Exact recovery of the true number rises with q q q , because the declared count is close to all-or-nothing ; persistence costs SSJ–MLE its power; a stochastic trend costs specificity, which contiguous folds buy back;
The property of the extended BIC for which a smoothing constant fixed before the budget is chosen calls for a milder ξ \xi ξ (Section 1) is confirmed: SSJ–KFS with λ \lambda λ from SSJ–MLE asks for less than SSJ–MLE itself in every scenario (0.15 against 0.25 in the reference case, 0.20 against 0.25 under dependent errors and 0.30 against 0.45 under a stochastic trend, Table 15 ), and the same ordering holds at every sample size under the reconstruction rule (Table 4).
Application Supplement — Smoothing splines with some discontinuities
Matteo Pelagatti; Paolo Maranzano; Giulio Cesare Gatto Ronchieri
This document collects the material that supports the two applications of Section 4 of the paper. Section S1 reports the temperature–demand application over the whole grid of the extended-BIC constant ξ \xi ξ and compares the estimated curve with the degree-day representation of the temperature response. Section S2 gathers everything on the fuel-price application: the check of the calibration of ξ \xi ξ on the policy dates, the 2026 episode, the two excise episodes and the boundary experiment at a readable size, the application re-estimated at the three values of ξ \xi ξ that bracket the one the paper adopts, and the evidence behind the number of cross-validation folds. The calibration of ξ \xi ξ itself, with every table and figure behind it, is the subject of a separate document, the Calibration Supplement.
supplement-S-applications.pdf
Application Supplement
Smoothing splines with some discontinuities
The temperature–demand application
This section supports § 3.1 of the paper: § 24.1 reports the discontinuities declared over the grid of ξ \xi ξ , and § 24.2 compares the estimated curve with the degree-day construct.
The declared discontinuities over the grid of ξ \xi ξ
Table S1 gives, for the two algorithms of Figure 4 of the paper, the number of discontinuities for every value of ξ \xi ξ from 0.10 to 0.50. According to the calibration, the recommended values are around 0.30 − 0.35 for SSJ–MLE and around 0.25 − 0.30 for SSJ–ATW. The number of detected discontinuities is rather stable in the respective ranges and above.
Table S1. Number of temperatures declared as discontinuities by the two estimators of Figure 4 of the paper, at each value of ξ \xi ξ : warps declared according to the rule described in the main paper.
Figure S1 shows the position with respect to temperature and the strength of the warps as a function of the EBIC constant ξ \xi ξ . As already noticed, the warps’ placement stabilizes when the values of ξ \xi ξ are equal to or above the values suggested by the calibration exercises. The discontinuities detected at any value of ξ \xi ξ occur at 11.9 and 27.1 degrees Celsius, with the second being much stronger.
Figure S1. The value of the EBIC parameter ξ \xi ξ is on the x x x -axis, the value of the temperature is on the y y y -axis, the size of the point is proportional to the warp identified by each algorithm.
Comparison with degree days
It is worth setting the estimated curve against the device that most energy systems use in its place. Heating demand is conventionally summarized by degree days : one fixes a base temperature T 0 T_0 T 0 and accumulates, throughout the season, the positive part of T 0 − T ˉ d T_0 - \bar{T}_d T 0 − T ˉ d , where T ˉ d \bar{T}_d T ˉ d is the mean outdoor temperature on day d d d . The implied model of the temperature response is exactly two straight lines joined at T 0 T_0 T 0 , flat above the base and linear below it, and its single free parameter is fixed by convention rather than estimated. Italy sets the base at 20 ∘ C 20^\circ\mathrm{C} 2 0 ∘ C , and does so for a reason that makes it higher than the bases used elsewhere: the figure in DPR 412/1993 is the conventional indoor temperature, differenced directly against the outdoor daily mean, without taking into account the internal and solar gains that the lower foreign bases implicitly subtract. The same regulation partitions the country into climatic zones A–F by the resulting annual totals, and those totals govern how long the heating season may run in each municipality.
Our estimate allows us to ask what the data would have chosen. The curve fitted by SSJ–MLE (Figure 4 of the paper) attains its minimum at 18.9 ∘ C 18.9^\circ\mathrm{C} 18. 9 ∘ C , and at exactly 20 ∘ C 20^\circ\mathrm{C} 2 0 ∘ C , the convention, demand sits within 0.1 % 0.1\% 0.1% of that minimum. The reason 20 ∘ C 20^\circ\mathrm{C} 2 0 ∘ C does so well is not that the curve turns there: demand stays within 1 % 1\% 1% of its minimum throughout the interval [ 16.0 , 23.4 ] ∘ C [16.0, 23.4]^\circ\mathrm{C} [ 16.0 , 23.4 ] ∘ C , so the vertex that the degree-day construct posits is not a point the data can locate, since there is nothing sharp there to locate, and any base between roughly 16 16 16 and 23 ∘ C 23^\circ\mathrm{C} 2 3 ∘ C would describe this sample about equally well. The two arms are also asymmetric: below 14 ∘ C 14^\circ\mathrm{C} 1 4 ∘ C each additional degree of cold adds about 270 270 270 MWh, while above 24 ∘ C 24^\circ\mathrm{C} 2 4 ∘ C each additional degree of heat adds about 320 320 320 , a fifth more, and about 340 340 340 per degree between 27 27 27 and 30 ∘ C 30^\circ\mathrm{C} 3 0 ∘ C . A single base temperature cannot serve both sides, which is why the Italian regulation defines degree days for heating only and leaves cooling without a counterpart.
Where the two descriptions truly part company is at the discontinuities. The estimator places jumps at about 11.9 ∘ C 11.9^\circ\mathrm{C} 11. 9 ∘ C and, in the cooling regime, at about 27 ∘ C 27^\circ\mathrm{C} 2 7 ∘ C and 29.7 ∘ C 29.7^\circ\mathrm{C} 29. 7 ∘ C , that is, at temperatures at which demand steps up rather than merely bending upward, as air conditioning is switched on across a large number of premises at once. A degree-day representation cannot express this at any base temperature, because it is continuous by construction; the best it can do is absorb the steps into a steeper slope and spread them over the whole of the cooling range. The estimators of Section 2 of the paper do not have to choose between a kink and a step, and here they find both.
The fuel-price application
This section collects everything that supports § 3.1.1 of the paper: the check of the calibration of ξ \xi ξ on the policy dates (§ 25.1 ), the 2026 episode (§ 25.2 ), the two excise episodes at a readable resolution and the boundary experiment (§ 25.3 ), the same application re-estimated at the other values of the extended-BIC constant that the paper discusses (§ 25.4 ), and the evidence behind the number of cross-validation folds (§ 25.5 ).
Checking the calibration on the policy dates
Because the dates of the interventions are known, the fuel-price series allow the calibration of ξ \xi ξ summarized in § 2.1.3 of the paper to be checked on real data. Figure S2 sweeps ξ \xi ξ over [ 0 , 1 ] [0,1] [ 0 , 1 ] and records every week that each estimator declares as a break, with the enforcement weeks of Table 4 of the paper marked by black asterisks. Because ξ \xi ξ enters only the selection of the budget and never a fit, the smoothing constants are computed once per series and held fixed, so the figure isolates the penalty and nothing else.
Figure S2. Weeks declared as discontinuities by each estimator (columns) on each series (rows), as ξ \xi ξ runs from 0 to 1 in steps of 0.05. Each band is one date and each tile a value of ξ \xi ξ at which it is declared, colored by the size and sign of the level shift. Black asterisks mark the weeks in which the excise duty changed Table 4 of the paper: on a colored tile a hit, on white a miss.
The pattern is the one the simulations predict. At the reference-case value the estimators declare a long tail of dates that are not policy events: SSJ–ATW at the maximum likelihood λ \lambda λ reports 16 such weeks on the two pump-price series alongside 11 of the 17 true events, and with contiguous folds the tail is longer still, 20 weeks alongside 8 events. Raising ξ \xi ξ to 0.45 removes much of it while costing little of the signal: at the maximum likelihood λ \lambda λ the estimator then reports 10 spurious weeks and still recovers 9 events, so precision rises from 0.41 to 0.47, and the two events given up are small ones, the partial restoration of +0.10 euro of December 2022 on one fuel and the final +0.05 step of July 2026 on the other. Beyond 0.45 the maximum likelihood arm does not change at all and the blocked one gives up an event, so that is where the gain has been realized (§ 25.4 gives the counts).
The declared sets are nested in ξ \xi ξ : every date enters at ξ = 0 \xi= 0 ξ = 0 and, once it leaves, does not return, for all 107 dates declared anywhere in the four series, so each occupies one uninterrupted band whose length measures how much penalty it can absorb. The great majority are transient and disappear within one or two grid steps of the reference-case value; the survivors at every admissible penalty are 14 March 2022, 28 March 2022 and 9 January 2023 (the market shock, the excise cut and its restoration) on either fuel. Four of the seventeen events are recovered by no estimator at any ξ \xi ξ , and they are the same four throughout, namely the two realignment steps on both fuels, which move the duty by 1.5 and 4 cents against a weekly standard deviation of the pump price several times larger, and which are visible in the figure as rows of asterisks on white across every column. On the industrial price, from which the duty is excluded, both SSJ–ATW variants declare a discontinuity at the 2022 policy date at the smallest penalties and both still do at 0.45; the one with contiguous folds keeps it up to 0.90, so a penalty above that would be needed before neither does. Here the two objectives of the calibration differ, and the gap is the price of insisting on no false alarms rather than on an accurate curve.
Two asymmetries in the detection map are also worth recording. SSJ–ATW with a block cross-validated λ \lambda λ retains far more of its pump-price dates across the whole grid than the other three estimators of the map: that constant is the smallest of those reported, the budget it supports is correspondingly large, and the extended term has comparatively little purchase on it, so ξ \xi ξ buys less there than elsewhere—which is visible in the figure as a column of long unbroken bands, and is why the block-cross-validated variant is the one whose precision barely improves when the penalty is raised. And CSSD, which is the conservative benchmark on the pump prices with six dates, declares 15 on the industrial prices and keeps all of them, more than any SSJ estimator retains; its parsimony is a property of the series it is applied to rather than of the estimator.
The 2026 episode
The 2026 episode inverts the pattern of 2022 and illustrates what the decomposition into gross and net prices buys. The pump price barely reacts to the cut of 23 March 2026: it falls by 0.040 euro per liter for gasoline and by 0.010 for diesel against a mechanical 0.244, and no discontinuity is declared on it, while a large one appears on the industrial price, estimated at +0.157 to +0.164 for gasoline and +0.179 to +0.191 for diesel. This reading deserves caution. The two weeks preceding the cut carry a violent market surge, with the industrial price rising by about 0.06 euro per liter per week on gasoline and 0.13 on diesel, that is, of the same order as the fiscal measure. What we measure at the policy date is the one-step change of the fitted signal, which superimposes the discontinuity on one week of the prevailing drift; netting out a drift of that size would raise the implied transmission to roughly one half on gasoline and two thirds on diesel. The qualitative conclusion, that the second intervention reached the pump far less completely than the first, is robust to the correction, but the magnitude is not, and we do not press it further. Panel (b) of Figure S3 shows the episode week by week.
The two excise episodes and the boundary experiment
Figure S3 expands the two excise episodes, at a resolution at which the individual weeks can be separated and the agreement between the three estimators can be checked one date at a time.
Figure S4 and Figure S5 report the boundary experiment of § 3.1.1 of the paper, on the pump price and on the industrial price respectively. A change point that falls in the middle of a long sample is an easy target. The formulation of Section 2 of the paper places no restriction on where a jump may occur, since γ i \gamma_i γ i is defined on every one of the n − 1 n-1 n − 1 intervals; near the boundary, however, the spline has almost no data on one side, and the roughness penalty alone must arbitrate between a discontinuity and a steep continuous transition. Because the date of the March 2022 cut is known, this can be put to the test directly. We build three families of nested subsamples: one in which the first observation is moved forward until the cut becomes the second observation of the sample, one in which the last observation is moved backward until the cut becomes the last, and one repeating the second exercise on the March 2026 cut; all three procedures are then re-estimated on each of the 41 windows. Every horizontal line is one estimation, run on a different subsample; the shaded blocks are the weeks that estimation could not see, so that moving down a panel the sample is progressively cut back toward the excise-duty change, which sits at zero on the horizontal axis. The vertical axis reports h h h , the number of weekly observations still separating the change from the moving edge of the sample, from the full sample at the top to h = 1 h=1 h = 1 at the bottom. The symbols are the discontinuities found in that estimation, positioned in weeks from the excise-duty change and split into one sub-row per estimator, so that an estimator insensitive to the position of the break in the sample produces a vertical stack of symbols at zero all the way down.
Figure S4. Boundary robustness on the pump (gross) price. Panels (a) and (b) shrink the sample toward the cut of 28 March 2022 from the left and from the right respectively; panel (c) repeats the exercise on the cut of 23 March 2026.
Figure S5. Boundary robustness on the industrial (net) price. Panels (a) and (b) shrink the sample toward the cut of 28 March 2022 from the left and from the right respectively; panel (c) repeats the exercise on the cut of 23 March 2026.
Four patterns recur. A column at zero is the excise-duty change itself. The column two weeks earlier, in the 2022 panels, is the market shock of 14 March 2022: on the pump price it is a second and smaller discontinuity, on the industrial price it is the only one, since the 2022 cut was passed on almost in full and left the industrial price alone, which is why Figure S5 has its main column at − 2 -2 − 2 and nothing at 0. Isolated symbols away from zero are further genuine discontinuities that a particular window happens to contain, such as the movements in the industrial price of diesel in late August and mid-October 2022; they disappear as soon as the window no longer reaches them, which is what makes them look isolated. Finally, the short runs of adjacent symbols visible in the shortest windows are the warping spread over a few consecutive weeks rather than concentrated on one: SSJ–ATW does this near the boundary, and it is worth stressing that it still places the jump in the right neighborhood rather than displacing it.
Figure S6 reports the estimated size of the recovered discontinuity in each of those windows, which is the quantity the paper’s claim of stability rests on: on the pump price it stays at − 0.267 -0.267 − 0.267 euro per liter from the full sample down to the shortest window, so the boundary neither displaces the jump nor attenuates it.
Figure S6. Estimated size of the discontinuity at the excise-duty change, in each window of the boundary experiment. Axes as in Figure S4 and Figure S5 : the vertical axis is the number of weekly observations still separating the change from the moving edge of the sample. A horizontal series of symbols is an estimator whose measurement of the jump does not move as the sample is cut back toward it.
The application at other values of ξ \xi ξ
Section 4.2 of the paper sets ξ = 0.45 \xi= 0.45 ξ = 0.45 , for reasons given there: the two simulation experiments bracket 0.35 to 0.50 for a series that is both serially dependent and closer to a stochastic trend than to a fixed smooth curve, and 0.45 lies inside that band without being the value that maximizes detection accuracy on these data. This section re-runs the whole application at the other three values, so that the reader can see what each of them would have produced. Everything else is held fixed: the same data, the same budget grid, the same λ \lambda λ for each variant, the same seed.
Table S2 scores the outcome against the seventeen excise events on the two pump-price series, counting a run of adjacent declarations around one event as one recovered event and allowing a tolerance of one week between the legal date and the price response.
Table S2. The fuel-price application at four values of ξ \xi ξ . For each estimator and each ξ \xi ξ : the total number of discontinuities declared on the two pump-price series (all ), the number of the seventeen excise events recovered (hit ), and the number of declared weeks not within one week of any excise event (false ).
Three things are visible.
The gain is concentrated between 0.25 and 0.35, and it is large. For SSJ–ATW at the maximum likelihood λ \lambda λ the declared set falls from 32 weeks to 24 and the false detections from 16 to 10, while the events recovered fall only from 11 to 9. Precision rises from 0.41 to 0.47. The two events given up are small ones, the partial restoration of +0.10 euro of December 2022 on one fuel and the final +0.05 step of July 2026 on the other; the realignment steps of 1.5 and 4 cents, which move the pump price by less than its weekly standard deviation, are recovered at no value of ξ \xi ξ (Section S2.1).
Beyond 0.35 the penalty has nothing left to remove. That arm declares exactly the same 24 weeks at 0.35, at the 0.45 the paper adopts and at 0.50, so the reading is insensitive to the choice within the band the calibration brackets; the blocked arm, which is still declaring 31 weeks at 0.45, is the only one that moves at 0.50, and it moves by giving up an event as well as five false detections.
Two estimators do not respond at all, for opposite reasons. SSJ–MLE and CSSD declare five and six dates respectively and keep them at every ξ \xi ξ : they are already parsimonious and the penalty has nothing to remove. SSJ–ATW with a block cross-validated λ \lambda λ also barely moves, from 32 weeks to 31 at ξ = 0.45 \xi= 0.45 ξ = 0.45 , but here the cause is the opposite — the blocked criterion selects the smallest smoothing constant of the variants reported and the warping budget it supports is correspondingly large, so the extended term has comparatively little purchase and ξ \xi ξ buys less there than elsewhere. That variant is the one whose precision stays near 0.30 throughout, and it is the reason the paper leads with SSJ–ATW at the maximum likelihood λ \lambda λ .
Sensitivity of the cross-validated λ \lambda λ to the number of folds
Section 4.2 of the paper uses K = 20 K = 20 K = 20 contiguous blocks for the cross-validated variant of SSJ–ATW. On simulated data with a known amount of dependence (Scenario 2 of the Calibration Supplement) the number of folds barely matters and the layout matters a great deal: at ψ = 0.5 \psi= 0.5 ψ = 0.5 the RMSE against the true signal is 0.086 with blocks at either K = 10 K = 10 K = 10 or K = 20 K = 20 K = 20 , against 0.098 with random folds. On the fuel prices that conclusion does not transfer. The simulated design has n n n between 50 and 400 and stationary errors; these are 591 weekly prices whose level is close to a random walk, and the length of a block interacts with that. Selecting λ \lambda λ on all four series at each K K K gives, in effective degrees of freedom out of 591:
Table 17.
For the random scheme the choice is immaterial (the same λ \lambda λ is selected at K = 5 K = 5 K = 5 , 10 and 20 on every series) and the criterion runs to the floor of the search grid on both pump-price series whatever K K K is, which is the failure that Scenario 2 of the Calibration Supplement predicts and which is why the blocked variant was introduced. For the blocked scheme the choice is not immaterial at all, and each value of K K K fails somewhere: at K = 10 K = 10 K = 10 the criterion over-smooths the gasoline pump price so severely that the variant declares no discontinuities at all on the series carrying the largest fiscal shock in the sample; at K = 5 K = 5 K = 5 it declares nine on the gasoline industrial price, which is the placebo series; at K = 20 K = 20 K = 20 the criterion runs to the floor of the grid on the gasoline industrial price, although the fit that results still declares only three. K = 20 K = 20 K = 20 is adopted as the best behaved by outcome, at 4, 3, 5 and 5 declared discontinuities across the four series.
That a criterion should move by three orders of magnitude between K = 10 K = 10 K = 10 and K = 20 K = 20 K = 20 on one series is a real weakness of cross-validating the smoothing constant on data of this kind. It is also the reason the paper’s headline estimator is SSJ–ATW at the maximum likelihood λ \lambda λ , which involves no folds at all, with the cross-validated variants reported beside it rather than in place of it. The value of ξ \xi ξ plays no part in any of this: the selected λ \lambda λ is identical at ξ = 0.25 \xi= 0.25 ξ = 0.25 and ξ = 0.45 \xi= 0.45 ξ = 0.45 in all twenty-four combinations above.