betaFeedback

WorksPhysics and Astronomy

Scalable Non-Parametric Bayesian Inference of Free Energy Landscapes and Emission Functions from Observations

Abstract

Free-energy landscapes govern processes from protein conformational change to climate oscillations, but are typically observed only indirectly, through observables such as FRET intensity or optical-tweezer transmittance. Neither the landscape shape, nor the emission function mapping latent state to signal, is usually known, and distinct states may produce the same signal. We therefore infer landscape and emission function jointly and non-parametrically, using the exact non-linear Bayesian filtering likelihood, to fully use the information contained in observable trajectories that are often noisy, scarce and costly to acquire. A novel parallel-in-time algorithm makes this tractable: it is up to 27 times faster than sequential filtering on one GPU, scales near-ideally across GPUs, and, given enough resources, has runtime independent of trajectory length. We demonstrate the method on a single-protein optical tweezer recording, where distinct conformations produce the same signal, and beyond biology on the El Nino-Southern Oscillation and financial market volatility.

bayesian_filtering_arxiv_version.pdf

Many processes across the natural sciences admit as a common description an overdamped Brownian dynamics on a free energy landscape. Examples include protein conformational changes,[1] ion channel gating,[2] and the El Niño–Southern Oscillation (ENSO).[3, 4] The shape of the landscape encodes the physics of each system, with barrier heights defining transition rates and well depths defining equilibrium populations.

Yet this landscape lives in a latent space, and what is measured is not the latent state itself but a function of it, such as fluorescence intensity,[5] ionic conductance,[2] or sea-level pressure differences.[6] This mapping, the emission function, is generally non-linear and non-invertible. The task is thus to recover the free energy landscape and the emission function jointly, from observed trajectories alone and without assuming parametric forms for either. We will address this problem by Bayesian inference.

Making this inference tractable has so far required additional assumptions on the latent space or the emission function. The most common is the discrete-state hidden Markov model (HMM).[7, 8] For continuous systems, like proteins probed by single-molecule FRET,9–12 discretization coarsens the shape of the landscape. Few states collapse complex transition paths into single rate constants,13,14 whereas many states turn inference into an ill-posed inverse problem. Non-parametric HMMs15 regularize the number of states, but not the smoothness of the landscape, which is the natural prior but difficult to apply on a discrete graph of states.

Continuous diffusion models, in contrast, preserve the landscape but assume the latent coordinate is observed directly, as in single-molecule pulling,16 position-dependent diffusion models,17 and econometric diffusion estimators.18 Bayesian filtering19 lifts this assumption but is analytically solvable only for the quadratic landscape of the Kalman–Bucy filter.20 Extended and unscented Kalman filters21,22 handle non-linear dynamics but force the latent density to be Gaussian. Non-parametric landscapes have been inferred jointly with the latent state for single-molecule FRET.23–25 The non-linear dynamics and the high-dimensional representation of the latent density make such inference costly. Particle MCMC estimates the exact likelihood by Monte Carlo,26,27 whereas simulation-based inference replaces it with a neural surrogate.28,29

All of these approaches assume a fully specified or parametric emission function. Jointly inferring the emission allows analysis of systems without a closed-form emission, such as the mapping from a latent climate regime to atmospheric pressures. It also avoids imposing physical laws that are only approximate in practice, such as the 1/r61/r^{6} law of FRET efficiency,30 given linker flexibility and fluorophore orientation. Both gains come at a further computational cost.

With non-linear dynamics and joint landscape and emission inference, the filtering likelihood becomes computationally costly to evaluate. Worse, its recursion is sequential: each step conditions on the previous one, so runtime grows with trajectory length whatever the parallel resources available. Parallelizing this recursion requires restructuring it. Parallel prefix scans do so by recasting it as an associative operation,31,32 but require prediction and update to compose in closed form, which holds exactly for linear-Gaussian models or transition densities expressible as dense matrices, and only approximately otherwise, through iterated Gaussian linearization.33 A parallel-in-time (PinT) particle smoother exists,34 though it draws the particles of each time step independently, resulting in error growing faster with trajectory length than that of sequential smoothers. Parallel fixed-point solvers sidestep associativity altogether, evaluating general non-linear recursions, MCMC chains included, by Newton or Picard iteration over the whole trajectory.35–39

Under filter stability,40 filters started from different initial densities converge as observations accumulate, leaving distant time points only weakly coupled, the regime in which PinT methods excel.41,42 Yet PinT iteration has not been combined with filter stability to parallelize Bayesian filtering.

Here we infer the landscape and the emission function jointly and non-parametrically, under a prior that imposes a controlled level of smoothness. The model likelihood is computed by Bayesian filtering, which propagates the non-parametric latent density through the Fokker–Planck equation of overdamped Brownian dynamics. We remove the sequential bottleneck of this filtering with a parallel-in-time algorithm that splits the trajectory into short blocks, filters them in parallel, and iterates until it reproduces exactly the output of the sequential filter. For the model studied here, filter stability40 on its compact latent domain makes the iteration count independent of the number of observations NN. The span of a likelihood evaluation, its runtime given unlimited parallel resources, therefore falls from O(N)O(N) to O(1)O(1), at the cost of a small multiple of total work. Because the algorithm treats the filter as a black box, it requires neither the associative reformulation of scan methods nor the Jacobians of fixed-point solvers, and applies to other hidden Markov models, though not necessarily with the same span.

Results

Non-parametric hidden Brownian dynamics model

Figure 1. a INPUT; observed y(t)y(t); Joint Bayesian Inference; P(ΔG,f,x(⋅)∣y)P(\Delta G, f, x(\cdot)\mid y); OUTPUTS; latent state x(t)x(t); density; landscape G(x)G(x); posterior mean; 95% CI; ΔG\Delta G (RT); emission y=f(x)y=f(x); observable yy; latent coordinate xx. b 1 Sequential filtering; N=64N=64 steps, strictly one after another; 2 Folded into BB blocks; block 1; block 2; block 3; block 16; 3 Parallel-in-time filtering; iteration k=1k=1; iteration k=2k=2; iteration k=3k=3; Filtering 16 blocks in parallel; Converged (equivalent to sequential); Unconverged; exact π0\pi_0; trial; converged.

Image not extracted

Figure 1 (PDF p. 5). Joint inference of landscape, emission and latent state, and parallel-in-time evaluation of the likelihood. (a) Inference goal: given observable trajectories y(t)y(t) (Input panel, teal), we jointly infer the free-energy landscape G(x)G(x), the emission function ff mapping latent space coordinate xx to observable y=f(x)y=f(x), and the latent state trajectory x(t)x(t) (Outputs panel), all three in the form of a Bayesian posterior distribution. (Synthetic data shown). (b) Parallel-in-time (PinT) evaluation of the filtering likelihood. Sequential filtering (1) processes the NN steps in order. Folding the trajectory into BB blocks (2) exposes parallelism, but each block needs the final density of its predecessor. PinT filtering (3) starts the first block from the exact initial density π0\pi_0 and every other block from a trial density, filters all blocks in parallel, and passes boundary densities on at each iteration until they converge.

We model the latent process as overdamped Brownian dynamics on a free-energy landscape G(x)G(x), with xx the latent space coordinate, observed through a non-linear emission function f(x)f(x) that need not be injective (Figure 1a). Here we focus on a one-dimensional latent coordinate, obeying dXt=−DkBT∇G(Xt) dt+2D dWtdX_t=-\frac{D}{k_{\mathrm{B}}T}\nabla G(X_t)\,\mathrm{d}t+\sqrt{2D}\,\mathrm{d}W_t. Observations are drawn conditionally independently given the latent trajectory, from a user-specified distribution whose location parameter is controlled by the emission, Ytk∣Xtk∼pobs(⋅∣f(Xtk))Y_{t_k}\mid X_{t_k}\sim p_{\mathrm{obs}}(\cdot\mid f(X_{t_k})). The dynamics are thus assumed reversible and unforced, with a time-independent landscape and a position- and time-independent diffusion coefficient DD. Within these assumptions, GG and ff are inferred non-parametrically, with a prior on their wavelet coefficients expansion targeting a Besov smoothness class (Methods).43

The likelihood is evaluated by Bayesian filtering,19, 22, 44 which alternates prediction, propagating the latent density by the Fokker–Planck equation, and update, conditioning it on the new observation. The generator of the Fokker–Planck equation is projected onto a basis and diagonalized, yielding the propagator. The model likelihood is the product of the normalization factor of the update steps ∏k∫pobs(ytk∣f(x))p(x∣yt1:tk−1) dx\prod_k\int p_{\mathrm{obs}}(y_{t_k}\mid f(x))p(x\mid y_{t_1:t_{k-1}})\,\mathrm{d}x.

Non-injective emissions give rise to local maxima in the posterior, as emissions of different preimage multiplicity are separated by regions of low likelihood. Our automated inference protocol therefore samples a variety of candidate emission functions of different preimage multiplicities, takes each to their local maximum of the posterior, and selects the top candidate for posterior sampling (Methods). The method is implemented in PlanX, an open-source, GPU-accelerated Python package built on JAX,45 whose filtering model can be embedded within larger Bayesian models.

To parallelize filtering, the sequential trajectory (Figure 1b, panel 1) is split into BB blocks of LL time steps that are filtered concurrently (panels 2–3). Each block needs as its initial density the final density of its predecessor, which is not available a priori. The first block therefore starts from the exact initial density π0\pi_0, and all others from a trial density, here peqp_{\mathrm{eq}} for both. Each iteration filters all blocks in parallel, then hands the final density of each block to its successor, until these block-boundary densities stop changing (Methods). For our model, the number of iterations is set by how quickly the filter forgets an incorrect initial density, not by the trajectory length. The wall-clock time of a likelihood evaluation therefore does not grow with the number of observations, at the price of filtering each block a few times over.

Method validation

We validated the method on synthetic trajectories generated from known landscapes and emission functions (Figure 2a and Figure 2b). As a particular challenge, both emissions are non-injective, a single observation being consistent with up to two latent positions in (a) and three in (b), and both landscapes contain fine structure within their wells. The data were limited to 32 trajectories of two to ten transitions each, with observation noise narrow (σ=0.25\sigma= 0.25) compared to the emission range (−5-5 to 15), so that any ambiguity in the latent state arises from non-injectivity rather than noise. Inference followed the protocol of the application cases, without knowledge of the ground truth except for the diffusion coefficient D=1D = 1, which only sets the gauge of the latent coordinate (Methods). The posterior recovers the landscape and the emission quite accurately, including the emission degeneracies and the fine structure of the landscape, with the ground truth well within its credible intervals.

Figure 2. Inference on synthetic data and latent ambiguity resolution. (a), (b) Inferred posterior for the landscape (top) and emission function (bottom) from synthetic trajectories, shown against the ground truth (orange dashed line). The posterior mean (solid blue line) is shown alongside the interquartile range (darker blue band) and the 95% credible interval (lighter blue band) quantifying posterior uncertainty. (c) Filtered latent density for a trajectory of system (a), as a function of time (xx-axis) and latent coordinate (yy-axis). The latent density is shown as highest-density-region (HDR) level in shades of blue (darker is more probable), with the true latent trajectory superimposed (orange line). The observed value yy at each step is shown in the plot below (green line).

Recovering a landscape despite non-injective emissions requires tracking the latent state through the resulting ambiguities in localization (Figure 2c, one trajectory of system (a)). The emission of (a) is an inverted “V”, its two branches spanning similar ranges of the observable, so that a single observation cannot resolve on which side of the peak the system is. When the latent trajectory reaches the peak, around time step 745, the observations become compatible with relaxation into either well, and the filtered density accordingly splits into two modes, at −4-4 and 44, until later observations fall outside the range of one branch and resolve the ambiguity. This example shows the advantage of a non-parametric latent density over established Gaussian-latent filters, such as the extended or unscented Kalman filter,21 which must represent the latent state by a single mode. Every subsequent observation would then be evaluated against an incorrect latent, biasing the likelihood and thus the inferred landscape and emission.

Scaling and benchmark

Accurate inference of this kind requires many thousands of likelihood-and-gradient evaluations, so its practicality rests on the speed of each evaluation. We verify empirically the two properties of our PinT filter proven in Supplementary Information S1, exactness at convergence and, for our model, O(1)O(1) span, then measure its throughput against the sequential filter and on multiple GPUs.

Six-panel figure showing evaluation error distributions, throughput comparisons, GPU scaling, parallel efficiency, likelihood and gradient-evaluation scaling, and weak-scaling efficiency

Figure 7 (PDF p. 8).

We computed the model log-likelihood with both the sequential filter and PinT filter on 14 test systems, at parameters both far from and near the high-density region of the posterior (Methods). The relative error of the PinT log-likelihood, compared to sequential, is typically around 10−910^{-9}, and always well below our target of 10−610^{-6}, with no increase with trajectory length, no systematic difference between systems, and only a slight increase with GPU count (Figure 3a and Supplementary Information S3). Our algorithm is therefore numerically equivalent to the sequential filter, to within single-precision round-off.

Image not extracted

Figure 3. Correctness, speed-up and scaling of PinT filtering. (a) Cumulative distribution of the relative error of the PinT log-likelihood against the sequential filter, for 1 to 4 GPUs (colors), across 14 test systems and trajectory lengths from 400 to 168k time points (Methods). The dashed line marks the accuracy target of 10−610^{-6}. (b) Throughput, in likelihood-and-gradient evaluations per second within the Hamiltonian Monte Carlo sampler, of the sequential and PinT filters on a single GPU, with the PinT speed-up above each pair of bars. Error bars show the standard deviation across replicate runs. (c) Strong scaling: throughput as a function of GPU count, for five trajectory lengths (labeled at right), with ideal linear scaling dashed. Replicate-to-replicate variation is below 1%, so error bars are omitted. (d) Parallel efficiency, the speed-up over a single GPU divided by the GPU count, as a function of trajectory length and GPU count, with the 70, 80 and 90% iso-efficiency contours in white. (e) Weak scaling: throughput as a function of GPU count, for fixed workloads of 1k to 54k time points per GPU (vertical axis broken), with ideal weak scaling dashed. Error bars show 95% confidence intervals from the inter-replicate variance. (f) Weak-scaling efficiency E(nGPU)E(n_{\mathrm{GPU}}), the throughput with nGPUn_{\mathrm{GPU}} GPUs and an nGPUn_{\mathrm{GPU}}-fold workload relative to a single GPU, as a function of GPU count, for a range of workloads per GPU (color scale). Error bars as in (e).

Near the maximum a posteriori estimate, the iteration count is constant over close to three orders of magnitude of trajectory length, from hundreds to hundreds of thousands of time points, showing a constant span empirically (Supplementary Information S3). The iteration count depends on how quickly the filter forgets its initial condition, a property of the landscape and emission function that varies between systems. Since the iteration count, here 2 to 4, is also the factor by which total work exceeds that of the sequential filter, the speed-ups below cost a corresponding increase in total computation. Despite this, on a single GPU, PinT filtering evaluates the likelihood and gradient 9 times faster than the sequential filter at 1k time points, and 26.8 times faster at 5k (Figure 3b). This seemingly paradoxical speed-up results from better use of within-GPU parallelism. The chain of matrix-vector products in the sequential filter is a thin, latency-bound workload, whereas our algorithm uses dense matrix multiplications that use compute and memory bandwidth more efficiently.

Strong scaling, the speed-up from adding GPUs at fixed trajectory length (Figure 3c), is limited by an unparallelizable fixed cost of about 4 ms per evaluation, dominated by operator diagonalization, which accounts for more than half of single-GPU runtime below about 6k time points (Supplementary Information S3). The 4k-point trajectory accordingly plateaus beyond two GPUs, the 50k-point trajectory speeds up sublinearly up to eight, and trajectories of 200k time points and more scale linearly up to eight. Ninety percent parallel efficiency is reached at approximately 60k and 100k time points on two and three GPUs, and above 200k on eight (Figure 3d), lengths typical of single-molecule recordings. Weak scaling, the ability to maintain a constant runtime as trajectory length and GPU count grow together, is shown in Figure 3e. Performance drops mainly from one to two GPUs, at the onset of inter-GPU communication. Each block exchanges only 3 to 6 kB with its neighbors (Methods), so this cost is latency-dominated, at about 0.4 ms per evaluation. The weak-scaling efficiency stays above 0.90 for workloads above 1k time points per GPU, and approaches 0.98 above 10k (Figure 3f). A trajectory spread over enough GPUs to keep at least 10k time points on each thus scales almost ideally, for instance on two GPUs for the 24k-point optical tweezer recording analyzed below. Tens to low hundreds of likelihood-and-gradient evaluations per second are therefore readily achievable, even for long trajectories of several hundred thousand time points. These figures are deliberately conservative, as they were obtained on five-year-old GPUs with inter-GPU transfers staged through host memory (Methods).

Application examples

Next, we applied our method to three representative systems from different fields, a single-molecule optical tweezer trajectory, a century-long climate index, and a daily financial index.

The first application is a single-protein nano-aperture optical tweezer recording of bovine serum albumin (BSA),46 in which the transmission of the trapping laser reports on the conformational state of the protein. The recovered landscape has two steep-walled wells of uncertain relative depth (Figure 4a), with an inter-well relaxation time (325 ms) much slower than all others (≤20.8\leq20.8 ms; Figure 4d), justifying a two-state description. The free energy difference between the wells is 0.84 kBTk_{\mathrm{B}}T (CI95 0.17 to 1.69) and the barrier height 4.03 kBTk_{\mathrm{B}}T (CI95 3.58 to 4.68). The original non-Bayesian analysis46 reported a free energy difference near 1.5 kBTk_{\mathrm{B}}T and a barrier around 3 kBTk_{\mathrm{B}}T.

Figure 4. Landscape, emission and kinetics of a single BSA molecule in a nano-aperture optical tweezer. (a) Posterior distribution of the landscape (top, in kBTk_{\mathrm{B}}T) and the emission function (bottom, in arbitrary units equal to 1/30 volt; see Methods), showing the posterior mean (solid blue line), the interquartile range (IQR, dark band) and the 95 percent credible interval (CI95, light band). The free energy difference between the wells ΔG\Delta G and the barrier height ΔG‡\Delta G^{\ddagger} are marked by arrows. Diffusion D=5⋅10−5D = 5 \cdot10^{-5}. (b) Autocorrelation of the observable, as observed (dashed red line) and under the posterior predictive distribution (dark blue line, with IQR and CI95 as shaded bands). Lag is given in time steps (bottom axis) and in milliseconds (top axis). (c) Standardized pre-update residuals, i.e., the difference between each observed value and its one-step-ahead predictive mean, divided by the predictive standard deviation. The observation distribution is a mixture of 99 percent Gaussian and 1 percent uniform; the Gaussian component alone is drawn as a solid dark blue line and the full mixture as a dashed orange line. (d) The five slowest relaxation timescales of the inferred landscape, in milliseconds, as a box plot. The posterior median (dark blue line), IQR (box) and CI95 (vertical line) are shown with the median printed above each.

The inferred emission function is non-injective, rising at the barrier between wells, so that, as in our most challenging validation example (Figure 2a), a given transmission value is compatible with latent positions on both sides of the barrier. Unlike FRET efficiency, which follows from the donor–acceptor distance through a known law, optical transmission reports on the polarizability of the protein, a function of its conformation which is well-defined, but whose functional form is system-dependent and difficult to specify. Such non-injectivity may therefore be frequent in this type of experiment, where it could prevent interpretation by other methods assuming a monotonic emission. Posterior predictive checks reveal excess autocorrelation at 20 to 135 ms of lag (Figure 4b), pointing to a slow degree of freedom hidden by the one-dimensional projection, such as conformational substates within the wells, or to apparatus drift (Supplementary Information S6.4). The one-step-ahead residuals also show ten isolated jumps of 4 to 5 standard deviations (Figure 4c), which the observation noise model absorbs (Methods). Both of these findings suggest partial model misspecification, though the inferred landscape and emission are likely still reliable.

The second application is the El Niño–Southern Oscillation (ENSO), the dominant mode of interannual climate variability. The observable is the monthly Southern Oscillation Index (SOI) from 1866 to 2025, the normalized sea-level pressure difference between Tahiti and Darwin, which measures the atmospheric side of the ENSO. The inferred landscape is broadly harmonic and the emission linear: an Ornstein–Uhlenbeck model fitted to the posterior mean stays within the 95% credible interval throughout (Figure 5a, dashed lines). Our non-parametric approach thus identified this simpler model from the data rather than assuming it.

Landscape, emission and external validation of the ENSO latent state

Figure 5. Landscape, emission and external validation of the ENSO latent state. (a) Posterior for the landscape (top) and for the emission function (bottom), showing the posterior mean (solid blue line), the interquartile range (IQR, dark band) and the 95 percent credible interval (CI95, light band). The dashed orange line is a fitted Ornstein–Uhlenbeck model, harmonic in the landscape and linear in the emission. Diffusion D=0.35D = 0.35. (b) Smoothed latent coordinate (posterior mean, with CI95 as vertical bars) against the Oceanic Niño Index (ONI), a sea surface temperature index over the Niño 3.4 region that defines the ENSO phase (color). Dashed vertical lines mark the standard NOAA thresholds at ±0.5\pm0.5 K. (c) Autocorrelation of the observable, as observed (dashed red line) and under the posterior predictive (dark blue line, with IQR and CI95 as shaded bands).

To test whether the inferred latent coordinate tracks the ENSO state, we compare it with the Oceanic Niño Index (ONI), which was not used during inference (Figure 5b). The ONI, a sea surface temperature anomaly in the Pacific, measures the oceanic side of ENSO and defines its El Niño and La Niña phases. The smoothed latent coordinate correlates strongly with the ONI (Spearman ρs=−0.865\rho_s = -0.865) and classifies the NOAA phases with a ROC AUC of 0.93 for El Niño and 0.94 for La Niña. Inferred from atmospheric pressure alone, the latent coordinate thus captures the full ENSO state, oceanic as well as atmospheric.

The posterior predictive autocorrelation of the observable matches the observed one over most lags, including the sharp drop from lag zero to one that measures the observation noise (Figure 5c). The exception is a negative lobe between 14 and 25 months of lag, implying a period of roughly 30 to 38 months. Reversible dynamics cannot produce a negative autocorrelation at any lag, so this lobe reveals an oscillatory component outside the model class, most likely the quasi-biennial component of ENSO.47

Figure 1 (PDF p. 14).

The third system is financial market volatility, observed through the logarithm of the CBOE Volatility Index (logVIX), recorded daily from 1990 to 2026. The inferred landscape has two wells, a low- and a high-volatility regime holding 74 and 26 percent of the equilibrium population, clearly separated by a substantial barrier (Figure 6a). The emission has one rising branch on each side of the barrier, so that the same logVIX value can arise in either regime (Figure 6a). The regimes are thus distinguished by the temporal pattern of fluctuations, which the filter tracks through the latent state, rather than by the level of logVIX alone.

Image not extracted

Figure 6. Two volatility regimes in the logVIX record, recovered from a single Brownian dynamics. (a) Posterior for the landscape (top) and for the emission function (bottom), showing the posterior mean (solid line), the interquartile range (IQR, darker band) and the 95 percent credible interval (CI95, lighter band). The dashed vertical line marks the barrier top separating the low-volatility (blue) and high-volatility (red) regimes. Diffusion D=0.02D = 0.02. (b) Smoothed posterior mean of the latent coordinate over time. The dashed horizontal line marks the barrier top, and the band between (b) and (c) shows the regime occupied at each time, labeled where a period corresponds to a known episode. (c) Model surprise over time: measured as the Kullback–Leibler divergence between the predicted and the filtered latent density at each step, sharing its time axis with (b). The dashed horizontal line is the flagging threshold, set at the 95th percentile of that divergence over the whole series. Flagged dates are labeled with the corresponding market event where one is clearly identifiable. (d), (e) Landscape (top) and emission function (bottom) of the low- and high-volatility halves of (a), the landscape re-referenced to the minimum of each well, with posterior mean and CI95. Panel titles give the equilibrium weight of each regime. Dashed lines show three classical volatility models, log Ornstein–Uhlenbeck, Heston and 3/2, each fitted to the corresponding half. (f), (g) Distribution of the observable within the low- and high-volatility regime respectively, as observed (histogram) and under the posterior predictive (solid line, with CI95 as shaded band). Observations are assigned to a regime by the side of the barrier on which the smoothed latent state lies.

Remarkably, the smoothed latent trajectory shows abrupt regime switches, separated by long stretches, often years, within a single well (Figure 6b). High-volatility periods align with episodes of prolonged market stress, from the Asian, Russian and dot-com crises of the late 1990s to the 2007 subprime crisis and the COVID-19 pandemic. Brief volatility jumps lasting days are flagged by a Kullback–Leibler (KL) surprise metric, the divergence between the predicted and filtered latent densities, which is large when an observation is unlikely under the model (Figure 6c). The largest surprises, over 4 days, coincide with identifiable shocks such as the 2010 Flash Crash or “Liberation day” tariffs, though some correspond to no identifiable event. They occur in both regimes and only some accompany a regime switch, suggesting that short-lived shocks and persistent regime switches are distinct phenomena.

We fitted three classical volatility models, the log Ornstein–Uhlenbeck, Heston and 3/2 models, 48 to each well (Figure 6d,Figure 6e). All three stay close to the credible interval in the high-volatility regime, but none reproduces local features such as the shoulder in the low-volatility well, nor has any notion of regime switching. Markov-switching models49 introduce such switching as explicit jumps between separate regimes. In our model, by contrast, both regimes and the switches between them emerge from a single Brownian dynamics on the inferred landscape, which also reproduces the observed logVIX distribution within each regime, including their overlap and the heavier tail of the high-volatility regime (Figure 6f,Figure 6g).

Discussion

We have developed a Bayesian method to infer the underlying free energy landscape and emission function jointly and non-parametrically from trajectories of a single observable. We have tested the method on three representative systems from different fields such as protein dynamics, climate science and finance. We have further demonstrated that our method can evaluate the model likelihood for trajectories of hundreds of thousands of points, without approximations such as particle filters, linearization or a discretized latent state, so that the resolution of the inferred landscape and emission is limited only by the data.

Several features distinguish our method from other approaches, each with practical consequences. First, the emission is inferred rather than specified, and may be non-injective, with several latent states producing the same observed value. Boltzmann inversion, by contrast, assumes an identity emission, and Gaussian-latent filters such as the extended Kalman filter21,22 cannot represent the multimodal latent density that non-injectivity creates. In contrast, our method can be used for systems for which the emission functional form is not known a priori, such as the mapping between transmittance and protein conformation in our optical tweezer application, which moreover was shown to be non-injective. In the financial market volatility example, non-injectivity is likewise what allows two volatility regimes to emerge.

Second, the landscape is non-parametric and its shape is constrained only by a smoothness prior. This contrasts with fixed-form landscapes, such as the harmonic potential of the Ornstein–Uhlenbeck process,50 and with parametric landscapes, whose number of wells is fixed and only their depths and barriers adjusted.51 Using our method, the suitability of a simpler model is therefore a result supported by data, not an a priori assumption. Our ENSO application from climate science illustrated this, with our posterior validating the use of a simpler quadratic-linear model. In our logVIX application, in contrast, we showed that classical volatility models are severe approximations, and a two-regime model better describe the data.

Third, our model likelihood is exact and explicitly computed, rather than replaced by a surrogate as in simulation-based inference.28,29 As a consequence, when the true process is not an overdamped Brownian dynamics, i.e., when the model is misspecified, the posterior concentrates asymptotically on the overdamped Brownian dynamics closest to the data in Kullback–Leibler divergence.52 Under mild misspecification, the recovered landscape and emission are thus still an informative, albeit inexact, model of the true process, a guarantee of robustness that surrogate likelihoods do not generally share. Further, misspecification is straightforward to detect, since the observable and statistics predicted under the model can be directly compared to the data. These checks also make it possible to identify the type of misspecification, such as a hidden slow degree of freedom, external forcing, or non-Brownian jumps, all three demonstrated in our applications.

Finally, our parallel-in-time algorithm makes our likelihood computationally practical to evaluate. Already faster than sequential filtering on a single GPU, it scales well when using multiple GPUs. Without our algorithm, inference on the optical tweezer application would have been impractically slow, and our other applications would require fivefold more runtime. Unlike parallel prefix scans,31,32 the algorithm requires no closed-form composition of prediction and update. It therefore generalizes beyond our model, to hidden Markov models whose filter propagates a probability density, discrete-state models included. We conjecture that its span complexity, proven independent of trajectory length for our model, would grow only logarithmically in the general case. More broadly, our parallel-in-time algorithm reduces the need to trade off model sophistication and data volume in likelihood-based inference for such models.

The following limitations of our approach point to directions for future work. It currently supports only a single observable and a one-dimensional latent space, whereas many systems are probed through several observables simultaneously. Further observables would enter as additional factors in the observation likelihood, providing the information needed for higher-dimensional latent inference. However, the explicit basis we use to represent the latent density, and with it the computational cost, grows exponentially with the latent dimension, becoming prohibitive beyond two dimensions. Particle filters could avoid this cost, scaling instead with the number of particles, but whether or not our parallel-in-time algorithm could be adapted to them is an open question. If such adaptation were possible can, our approach would extend to models whose transitions can be simulated but not evaluated, a class currently addressed mainly by simulation-based inference.28,29

Methods

Our inference problem combines three components: a model for the latent dynamics, a model for the emission function, and Bayesian filtering to compute the likelihood of observed trajectories given both. We define each in turn, then the wavelet representation with its smoothness priors. The filtering likelihood follows, with the parallel-in-time algorithm that evaluates it, the sampling and benchmarking protocols, and the synthetic and application cases. The convergence proof of the parallel-in-time algorithm, complete parametrizations, and implementation details are given in the Supplementary Information (S1, S6 and S2, respectively).

Model definition

We consider a continuous–discrete filtering model: a continuous-time, continuous-state latent process, observed at discrete times through an emission function that is non-linear and need not be injective.

The latent process {Xt}t≥0\{X_t\}_{t \ge0} takes values in the state space X=[a,b]n⊂Rn\mathcal{X} = [a,b]^n \subset\mathbb{R}^n and satisfies the overdamped Langevin equation

dXt=−DkBT∇Gθland(Xt) dt+2D dWt,(1)\mathrm{d}X_t = -\frac{D}{k_{\mathrm{B}}T}\nabla G_{\theta_{\mathrm{land}}}(X_t)\,\mathrm{d}t + \sqrt{2D}\,\mathrm{d}W_t, \tag*{(1)}

where D>0D > 0 is the diffusion coefficient and {Wt}t≥0\{W_t\}_{t \ge0} the standard Wiener process. The equation holds for overdamped dynamics, in which momentum relaxes fast compared with the observation interval. The free energy landscape Gθland:X→RG_{\theta_{\mathrm{land}}} : \mathcal{X} \to\mathbb{R} decomposes as Gθland=Ginterior,θland+GwallG_{\theta_{\mathrm{land}}} = G_{\mathrm{interior},\theta_{\mathrm{land}}} + G_{\mathrm{wall}}, the first term parameterized by θland∈Θland\theta_{\mathrm{land}} \in\Theta_{\mathrm{land}}, the second fixed and known, rising steeply near ∂X\partial\mathcal{X} (Supplementary Information S2). Both terms are assumed continuously differentiable with bounded second derivative on X\mathcal{X}, and a formal zero-flux condition at ∂X\partial\mathcal{X} closes the domain. The stationary density is then peq(x)=Z−1e−βGθland(x)p_{\mathrm{eq}}(x) = Z^{-1}e^{-\beta G_{\theta_{\mathrm{land}}}(x)}, with β=1/kBT\beta= 1/k_{\mathrm{B}}T and ZZ the normalizing constant.

The observed process {Ytk}k=1N\{Y_{t_k}\}_{k=1}^{N} takes values in the observation space Y⊆R\mathcal{Y} \subseteq\mathbb{R}, sampled at times tk=kΔtt_k = k\Delta t for simplicity, though the method allows non-uniform observation times. Conditional on the latent state, observations are generated via the emission model

Ytk∣Xtk∼pobs ⁣(⋅∣fθem(Xtk))(2)Y_{t_k}\mid X_{t_k} \sim p_{\mathrm{obs}}\!\left(\cdot\mid f_{\theta_{\mathrm{em}}}(X_{t_k})\right) \tag*{(2)}

where fθem:X→Sf_{\theta_{\mathrm{em}}} : \mathcal{X} \to\mathcal{S} is the emission function parameterized by θem∈Θem\theta_{\mathrm{em}} \in\Theta_{\mathrm{em}}, assumed continuously differentiable with bounded second derivative. Here S\mathcal{S} is the parameter space of the observation (noise) distribution and pobs(⋅∣s)p_{\mathrm{obs}}(\cdot\mid s) is a probability density on Y\mathcal{Y} indexed by s∈Ss \in\mathcal{S}. If pobsp_{\mathrm{obs}} is Gaussian with fixed σ\sigma and variable mean, for instance, then S=R\mathcal{S} = \mathbb{R} and ss is that mean. The observation distribution is specified by the user rather than inferred, subject only to fθemf_{\theta_{\mathrm{em}}} controlling a location parameter of it. Its identifiability jointly with the landscape and emission is untested. In the application cases, we chose it empirically from the one-step-ahead residuals of a preliminary run. Given the trajectory {Xt}\{X_t\}, the observations {Ytk}\{Y_{t_k}\} are conditionally independent.

The model is identified only up to gauge freedoms of the latent coordinate: isometries such as reflections and rotations, and its scale set by the diffusion coefficient DD (Supplementary Information S4). We break the isometries with a weak prior and fix DD rather than infer it. Its value comes from a preliminary run in which DD is free, fixed within the range the posterior supports and reported alongside the posteriors (gauge-fixing procedure, Supplementary Information S4).

Representation and regularization

Recovering GG and ff from finite, noisy trajectories is an inverse problem, and unregularized solutions to such problems fit detail that the data cannot resolve. Bayesian inference mitigates this by returning a posterior rather than a point estimate, so that unsupported detail appears as uncertainty. We regularize nonetheless on two grounds. First, as an inductive prior: a projection onto a few collective coordinates averages over many degrees of freedom, so that the resulting landscape and emission are generally smooth. Second, for efficiency, as restricting the search space reduces the sampling needed to reach a given accuracy. Besov spaces Bp,psB^s_{p,p} comprise the functions of a prescribed regularity, indexed by a smoothness order ss and a sparsity type pp. We target this class by expanding GG and ff in a wavelet basis and placing the corresponding prior on the coefficients, which shrinks them geometrically with resolution level and is discretization-invariant in the sense of Lassas et al.43 We use the biorthogonal 3.3 synthesis wavelet, a compactly supported quadratic spline, continuously differentiable with bounded second derivative, so that finite expansions meet the regularity assumed above. Defaults are s=2s = 2, the lowest order that penalizes curvature, which enters the Fokker–Planck operator in the symmetrized form we diagonalize, and p=2p = 2, which makes the coefficient prior Gaussian. Both are user-settable (Supplementary Information S5). The posterior is robust to the prior scales and to the weight of the isometry-breaking prior, more sensitive to ss, and concentrates on the ground truth as the trajectory lengthens (Supplementary Information S6.6).

The wavelet basis carries the prior, but filtering is computed in a second basis. parametrize GG and ff in the wavelet basis, 24 coefficients each, and compute in a basis of Gaussian functions on a regular grid, Ng=96N_g = 96 per dimension (256 for the logVIX case), giving Nb=NgnN_b = N_g^n basis functions. A linear change of basis maps between the two. Gaussians are chosen for quadrature, the non-linear operations of filtering being evaluated at Gauss-Hermite nodes and projected back by the quadrature weights, efficiently on GPU (Supplementary Information S2). Computation is mixed precision: filtering runs in single precision, while operator assembly, diagonalization and the log-likelihood reduction run in double.

Filtering and smoothing

References

  1. [1]E. Lelarasmee, A.E. Ruehli, and A.L. Sangiovanni-Vincentelli. The Waveform Relaxation Method for Time-Domain Analysis of Large Scale Integrated Circuits. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 1(3):131–145, 1982.DOI
  2. [2]Jacques-Louis Lions, Yvon Maday, and Gabriel Turinici. Résolution d’EDP par un schéma en temps «parallèle». Comptes Rendus de l’Académie des Sciences-Series I-Mathematics, 332(7):661–668, 2001.DOI
  3. [3]François Le Gland and Nadia Oudjane. Stability and uniform approximation of nonlinear filters using the Hilbert metric and application to particle filters. The Annals of Applied Probability, 14(1):144–187, 2004.DOI
  4. [4]Jin Won Kim and Prashant G Mehta. Variance decay property for filter stability. IEEE Transactions on Automatic Control, 69(12):8140–8155, 2024.DOI
  5. [5]Yves Meyer. Wavelets and operators. Number 37. Cambridge university press, 1992.
  6. [6]Stphane Mallat. A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way. Academic Press, Inc., USA, 3rd edition, 2008.
  7. [7]Matti Lassas, Eero Saksman, and Samuli Siltanen. Discretization-invariant Bayesian inversion and Besov space priors. Inverse Problems and Imaging, 3(1):87–122, Feb 2009.
  8. [8]Matthew Peters, Tianyu Zhao, Sherin George, Viet Giang Truong, Síle Nic Chormaic, Cuifeng Ying, René A Nome, and Reuven Gordon. Energy landscape of conformational changes for a single unmodified protein. npj Biosensing, 1(1):14, 2024.DOI

Paper details

vlnc:W_mek29ubm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Claude

View statement
Looking for related papers…
Contents