WorksPhysics and Astronomy
Efficient regime boundary detection in discontinuous simulation models
Abstract
Many computational models exhibit multiple operating regimes separated by abrupt or non-smooth transitions as the input parameters vary. Such regime transitions complicate the construction of global parametric reduced-order models, often necessitating large training datasets to capture the distinct system behaviors. Accurate regime boundary detection instead enables localized modeling strategies by partitioning the parameter space into regions of qualitatively distinct behavior. In many cases, regime switching corresponds to the excursion set of a simulator output. When such a quantity is discontinuous or non-smooth, regression-based algorithms for level set estimation can struggle to correctly approximate the boundary. In this work, we show that solving a resulting binary classification problem can improve sample efficiency and we further propose a novel adaptive sampling strategy for support vector machines. Unlike uncertainty-based sampling schemes, the proposed approach uses a continuous search space and efficiently explores the currently estimated boundary in its tangential directions by gradient-based dynamic sampling. Because the decision function admits analytical gradients, this exploration incurs negligible computational overhead relative to the underlying model evaluations. Then, a space-filling subset of the generated samples is selected as the next batch to be queried. We demonstrate improved sample efficiency compared to existing active learning algorithms on analytical and numerical examples. The proposed method provides a sample-efficient framework for regime boundary detection in a broad class of discontinuous simulation-driven problems.
Introduction
Abrupt changes in the dominant system dynamics arise in a wide range of parameter-dependent computational models, ranging from fluid dynamics [1, 2] and fluid-structure interaction [3] to neurosciences [4, 5], as well as systems with friction [6, 7]. A sudden change in the dominant features of the dynamics poses challenges to the efficient construction of both intrusive and non-intrusive reduced-order models for downstream tasks such as optimization, control, and uncertainty quantification. A large class of projection-based and non-intrusive reduced-order models relies on low-dimensional representations extracted from high-fidelity solution data, with Proper Orthogonal Decomposition (POD) being one of the most widely used approaches. In such a case, the reduced basis may be constructed for each training sample in the parameter space or globally over the entire parameter domain. In the former case, standard manifold-based interpolation schemes [8] become ill-conditioned when large principal angles arise due to abrupt changes in the dynamics; in the latter case, the number of POD modes required to capture dominant dynamics across regimes may be very large, making the application of hyper-reduction techniques particularly challenging [9, 10] and more generally hindering the efficiency of the reduced-order model.
To address these limitations, when multiple regimes coexist within a parametric space, treating each regime separately enables the construction of accurate and efficient localized surrogate or reduced-order models. Beyond enabling regime-specific models, inferring the regime boundary is of standalone interest, as it can provide valuable insight into the underlying system behavior for safety assessment or reliability analysis. Central to this divide-and-conquer type of approach is a discriminant that predicts, a priori, the regime associated with a given set of input parameters, thereby partitioning the parameter space into regime-specific regions. In many real-world engineering systems, the regime transition occurs across a codimension-one surface in parameter space, corresponding to an observable or function thereof exceeding a critical threshold value. In correspondence of such a critical value, the system response switches between qualitatively distinct behaviors. For example, in aeroelasticity, combinations of flow and structural parameters for which the aerodynamic damping crosses zero define a boundary separating stable from unstable dynamics.
In practice, this separating surface is rarely known in closed form and must be inferred from experimental observations or high-fidelity simulations. When individual model evaluations are computationally expensive, exhaustive sampling of the parameter space is infeasible, making the accurate reconstruction of the regime boundary from a limited number of queries a central challenge. This motivates adaptive sampling strategies that concentrate new evaluations in regions that are most informative for identifying and refining the separating surface, rather than relying on static space-filling designs.
In this work, we consider a computational model that depends on input parameters , where is a bounded domain, and a scalar simulator output that exhibits discontinuities or non-smoothness with respect to the parameters. We are interested in efficiently approximating the boundary of the excursion set
for a prescribed threshold that determines the regime boundary. When is sufficiently regular, the boundary of the -excursion set corresponds to the -level set of . A common approach to level set estimation is then to construct a regression surrogate for , often using Gaussian processes (GPs), and exploit the associated uncertainty estimates to guide adaptive sampling. When is continuous but non-smooth (for example, when it features a kink), accurately reconstructing the response and its level set may require a large number of model evaluations. In the presence of discontinuities, the difficulty is more fundamental, since the boundary of an excursion set need not coincide with the corresponding level set of ; indeed, the latter may be empty, despite a well-defined transition between regimes. Moreover, GPs with smooth kernels tend to regularize discontinuities and sharp variations, potentially leading to over-smoothed predictions and inaccurate localization of the regime boundary.
These considerations motivate addressing regime boundary detection directly as a classification problem rather than through regression of the quantity of interest of the underlying computational model. Accordingly, in this work we formulate the identification of in (1) as a binary classification problem and approximate the separating surface using the decision function of a support vector machine (SVM).
Building on this classification formulation, we propose a novel strategy for active learning of : unlike standard margin-based sampling schemes that select informative points from a discrete candidate pool based on their proximity to the current decision boundary, we consider a continuous search space for the next points to be queried. This is achieved by using to naturally induce a potential energy proportional to , that is minimized on the -dimensional decision surface ; the currently estimated boundary can then be efficiently explored in its tangential directions using Hamiltonian Monte Carlo. While this requires gradient evaluations of the potential, it remains computationally cheap owing to the sparse representation of support vector machines and the availability of analytical gradients. Subsequently, we select a space-filling subset of the generated samples, naturally promoting underrepresented parts of the estimated boundary and improving the global approximation of the separating surface. This mitigates the sample redundancy that is typical of margin-based schemes and improves sample efficiency; we show in particular that the gains are increasingly pronounced for boundaries with higher curvature and in higher-dimensional parameter spaces.
The remainder of this manuscript is organized as follows. In Section 2 we clarify the notation, give the problem statement and review existing adaptive sampling and localized approaches for surrogate and reduced-order models. In Section 3 we first introduce active learning strategies for level set estimation using Gaussian processes; we then turn to support vector machines by motivating their use in the discontinuous model response setting and reviewing the corresponding active learning techniques. Section 4 describes the algorithmic building blocks of our proposed approach; in this section, we detail the boundary exploration mechanism and compare it with other popular choices, and we show on an analytical example that the additional computational overhead for sample generation is negligible for real-world scenarios where labeling cost dominates. In Section 5 we assess the performance of the proposed algorithm. First, we show that it often provides improved sample efficiency, compared to other active classification algorithms, on a variety of analytical examples; second, we consider increasingly complex simulation models with discontinuous response, where we show the failure mode of regression-based algorithms and that our approach generally yields superior results. Finally, in Section 6 we draw conclusions, discuss strengths and limitations, and highlight future directions.
Background
Problem statement
We consider a bounded parametric domain and assume a scalar function that can be evaluated anywhere in and returns a simulator output of an underlying computational model. We treat as a deterministic black-box quantity of interest; extensions to stochastic or noisy model evaluations and multiple model fidelities are left for future work. The quantity of interest depends non-smoothly on and may exhibit a finite number of discontinuities within the parametric domain . We further assume that there exists a critical value for which the behavior of the system changes fundamentally. That is, let us define
with ; then, the regime boundary is defined by the interface
Our goal is to construct an accurate approximation of while minimizing the number of queries to the underlying computational model represented through the simulator output .
Adaptive sampling and localized modeling
To construct an accurate approximation of the regime boundary , adaptive sampling strategies, often referred to as active learning schemes, are often preferred to static learning due to their better sample efficiency. In its broader scope, which is not limited to the regime boundary detection setting we consider in this work, active learning works by repeated construction of an approximate solution to the original problem, coupled with an oracle that, given such a solution, provides valuable information on where to query next to expand the training dataset [11, 12]. This procedure can be repeated iteratively until the computational budget is exhausted or the solutions computed in two successive active learning steps do not differ considerably. For the construction of surrogate models, [13] proposed using bootstrap-based local error estimates of a PCE surrogate to decide where to enrich the experimental design. Recent work has proposed to balance exploration and exploitation to improve active learning algorithms in the context of reliability analysis [14, 15]. In the context of discontinuous function approximation, treed Gaussian processes [16, 17] or multi-element generalized polynomial chaos (gPC) [18] perform recursive partitioning of the input space or element-wise domain decompositions, respectively. Finally, [19] maps the obtained subdomains to distinct hypercubes where polynomial chaos expansions are used for approximation. These approaches can be combined with adaptive sampling strategies exploiting for example the GP posterior uncertainty, however they inherit the poor scaling of tensor-product PCE and Gaussian processes in higher dimensions. Moreover, their computational cost increases with the number and complexity of the discontinuities, eventually rendering them computationally intractable. For reduced-order models, [20] introduced a greedy sampling strategy coupled with a cheap error estimator to query high-fidelity model evaluations where the error of a global reduced-order model is likely to be high. Similar approaches have been proposed for contact mechanics [21]. However, due to interpolation and because the error indicator is residual-based, it is not obvious why such an error indicator would query points close to the regime boundary. Recently, there have been a number of attempts to investigate subspace distances between parameter-dependent POD subspaces to capture sensitive areas of the parameter space in an active learning fashion [22, 23]. While such schemes were successfully applied to structural and CFD problems, the identification was only applied in low-dimensional parameter spaces, and the sensitivity measure was not used to guide a parameter space partitioning prior to the construction of localized reduced-order models. Attempts in this direction exist [24, 25], but the regime-awareness of the clustering is not discussed. More generally, the majority of the work on localized models relies on static one-shot clustering [26, 27, 28], either in parameter space or state space, via -means [29] or similar variance-based clustering algorithms. As highlighted and discussed in [30, 31], the computed cluster need not reflect the actual dynamics regimes. Finally, for surrogate models, [32] uses clustering and classification to detect discontinuities of a simulation output; localized Gaussian processes are then used for regression on each side of the discontinuity with a mixture of experts approach; however, this is done in a static fashion without adaptive sampling for discontinuity detection.
Therefore, this work aims to develop an adaptive sampling framework that identifies distinct regimes of the underlying computational model, thereby partitioning the parameter space into physically meaningful clusters. Within each cluster, tailored local surrogate models or reduced-order models can then be constructed. This further motivates the need to provide an accurate approximation from a limited number of model evaluations. In particular, because the quantity of interest of our computational model is discontinuous, rather than constructing a regression-based surrogate for and recovering the boundary from its -level set, we formulate the problem directly as binary classification. This avoids the need to accurately reconstruct a potentially non-smooth model response and, in the presence of discontinuities, remains applicable even when the excursion-set boundary does not coincide with the -level set of . Specifically, we construct a classifier that approximates the labeling function
The motivation for this algorithmic choice can be further supported by three considerations. First, in regime boundary identification, one is typically interested in a single threshold , or at most in a small number of nearby threshold values. In such cases, retaining an accurate global approximation of may be unnecessary, particularly when reconstructing the model response over the entire parameter domain is difficult because of discontinuities or sharp variations. Second, if several nearby thresholds are of interest, for example to assess the sensitivity of the inferred boundary to , an approximation to can provide an effective initialization for the estimation of when . Third, discontinuous function approximation remains very expensive and, unlike classification, scales badly with the dimension of the parameter space. Two-stage approaches, where regimes are firstly identified and regime-specific surrogates or reduced-order models are constructed subsequently, can therefore be more data efficient by exploiting a divide-and-conquer paradigm. This can be particularly advantageous when dealing with expensive computational models.
For the purpose of classification, in this work we employ a nonparametric approximation using support vector machines (SVM) [33, 34]. In Section 3, after a brief review on regression-based level set estimation, we detail the SVMs methodology. The adaptive sampling strategy we propose for support vector machines, which moves away from so-called uncertainty sampling schemes and instead favors tangential boundary exploration, will be detailed in § 4.
Active learning for excursion set estimation
In this section, we review active learning methodologies for excursion set estimation by firstly presenting the ones relying on function approximation, and then those relying on a resulting (and equivalent, in the continuous case) classification problem. For the latter case, we further describe the implicit representation of the regime boundary obtained by support vector machines, a popular supervised machine learning technique for classification. SVMs are particularly well suited to the present setting because they provide an explicit decision function defining the boundary and perform effectively in low-data regimes. We summarize the main properties and computational advantages of SVMs before reviewing established active learning strategies for adaptively refining their decision boundary.
Gaussian process regression
Gaussian processes (GPs) have long served as a regression tool, widely known as Kriging [35]. A GP is a collection of random variables , indexed by the -dimensional vector , such that any subset of these variables has a multivariate Gaussian distribution. A GP is therefore a non-parametric approximation that can be fully characterized by its mean function, , where denotes expectation, and its covariance function
In GP regression, a prior distribution is placed over the unknown response function and is subsequently conditioned on the available model evaluations. For simplicity, assuming a zero-mean GP with covariance kernel , the unknown response function can be approximated from measurements , , where denotes the number of training data points. This approximation is given by the regressor
where is an matrix with entries ,
and is a vector that contains the collection of measurements . The regressor (6) represents the mean of the conditional GP, with its variance given by
Gaussian process-based active learning
In the context of level set estimation, whenever is at least continuous and its -level set is well defined, Gaussian processes are used as a surrogate for the quantity of interest . Once the regressor is trained, reconstructing is straightforward and provides an approximation to . Due to the uncertainty estimates that come from evaluating the variance (7), using Kriging as a regression tool naturally gives information about the most informative regions of the parameter space. A variety of active learning paradigms have therefore been developed in this direction. Here, we summarize representative approaches, some of which are used in § 5 as benchmarks for comparison with classification-based methods in the presence of discontinuous model response.
Level Set Estimation. This algorithm, proposed in [36], defines at every active learning iteration a confidence interval of the form
for any point . In practice, (9) is computed on a dense grid of points. This confidence interval allows us to determine whether a point can be classified in the super-level or sub-level sets or whether such decision is still uncertain; therefore, it guides the sampling process towards points that are likely to be informative with respect to the desired level set. The next point to be evaluated is the one that maximizes the so-called ambiguity, defined as
where the confidence region reads as
Straddle. This algorithm, proposed in [37], is very similar in spirit to the aforementioned one, but it chooses in (9) and uses the confidence interval instead of the confidence region in (10). This leads to the acquisition function
Maximum variance. This approach is completely agnostic to the vicinity to the level set and simply acquires the candidate sample with the maximum posterior uncertainty, i.e.
The acquisition functions presented above are written for the sequential case, with new query per active learning iteration. In many applications, to exploit parallelization, it is better to produce a batch of size . To this aim, regression-based approaches use the fact that the GP posterior uncertainty in (8) does not depend on the measurements but only on the set of points. As such, to produce a batch of size , the uncertainty is updated after every addition, while keeping the posterior mean fixed until the model evaluations for the complete batch become available. For more details, we refer again to [36].
Support vector machines
Some of the GP-based acquisition strategies described in the previous subsection admit theoretical convergence guarantees under appropriate assumptions on the response and the GP model. Their effectiveness, however, depends on the ability of the regression surrogate to adequately represent in the vicinity of the target boundary. For continuous but non-smooth responses, accurate regression may require a large number of model evaluations, while in the presence of discontinuities an excursion-set boundary need not coincide with a level set of . As we discussed in Section 2, alternative approaches for regime boundary detection when is discontinuous are generally either data demanding or sensitive to the geometry of the discontinuities, which limits their applicability when dealing with computationally expensive underlying simulation models. Furthermore, they scale suboptimally with the dimensionality of the problem and may need dimensionality reduction as a preprocessing step, e.g., by projection onto a latent space through principal component analysis [38] or active subspaces [39, 40]. However, the amount of data required to reveal such a low-dimensional structure is a priori unclear. For such reasons, we focus on the related binary classification problem for approximation of the labeling function (4).
Given a dataset of measurements , with , we assign to each sample a label according to (4), with . The labeled dataset is used to train a support vector machine classifier. This is an accurate and efficient technique used to build a boundary, between the different classes of points, that approximates the true separating surface. The idea is to obtain an explicit representation of the decision function of the form
In (14), the coefficients are Lagrange multipliers associated with the training datapoints and are obtained from the solution to a dual problem [34], is a positive semidefinite Mercer kernel [41] and is an offset. Evaluation of the kernel yields the dot product between the inputs in a high-dimensional feature space; that is, for every valid kernel there exists a map such that . A linear classification problem is then solved in the higher-dimensional feature space. Therefore, (14) can equivalently be rewritten as
For a binary classification problem, the prediction of the classifier in a new query is determined by its sign: if and vise versa if . The two approximate regimes are separated by the -dimensional manifold
which acts as an approximation to the true boundary defined in (3). Importantly, for the computational cost of our proposed approach, the solution to the dual problem is generally sparse and only when the corresponding datapoint is a so-called support vector, which means that it is close to the decision boundary between the two classes and actively determines its final representation. As a consequence, we can rewrite
where SV denotes the subset of support vectors, with in most practical applications. Furthermore, is often simple enough and amenable to parsimonious differentiation. A very standard choice when no prior information about the distribution of the two classes is available is that of a Gaussian kernel, which corresponds to a lifting map to infinite dimensions. In this case and we have
whose computation can be efficiently vectorized. From expressions (17) and (18), we conclude that the computational cost of a function and gradient evaluation is , which is proportional to the number of support vectors rather than the total size of the training dataset. This makes repeated evaluations of the acquisition function tractable. Finally, the training cost of kernel support vector machines (that is, with a nontrivial lifting ) ranges between and . This is generally cheaper than the training cost of GPs, and is nonetheless negligible, in low-data regime, compared to the expensive model evaluations.
Active learning for support vector machines
Due to their well understood mathematical foundation and properties, support vector machines have been the gold standard for classification in low-data regime. Furthermore, due to the low retraining cost given new incoming data, a wide variety of active learning strategies have been proposed to refine the classification boundary. We now introduce the most popular techniques, as this will allow us to naturally set the stage for our proposed approach. To this end, we first provide some additional remarks on support vector machines.
Points for which for a given are said to be within the -margin of the classifier, while larger magnitudes of imply an increasing distance from the boundary and an increasing confidence of the classifier in its prediction. Moreover, as we discussed previously, is a positive semi-definite kernel, corresponding to a feature map such that . This allows computing distances in the feature space as
Equation (19) shows that the kernel induces a similarity distance in the input space through the feature representation. Thus, a common scale-invariant similarity measure is the cosine similarity between and , corresponding to the normalized kernel
The most common active learning techniques stem from considerations about proximity to the current separating surface and appropriate variations. In this work, we consider a pool-based setting in which, at each iteration, a surrogate boundary of the form (14) is trained on the labeled set and a batch of new points is selected from an unlabeled candidate pool ; the pool can either be fixed or regenerated at each iteration, for example using a low-discrepancy sequence in . All algorithms can either select one point at a time or be executed in batch mode, by picking the points with highest score according to the acquisition function; the latter choice is usually preferred to enable parallelism and reduce wall-clock time.
Margin/entropy sampling. In their naive form, margin sampling [42] and entropy sampling [43] start from an unlabeled pool and propose to query the points that are closest to the current separating surface. In the former case, such closeness is measured by the magnitude of as a proxy for uncertainty; that is, the acquisition rule reads
In the latter case [43], it uses as a proxy the entropy of the posterior probability that can be fitted to a support vector machine boundary via Platt scaling [44]. Therefore, in this case
where in our case but the above definition generalizes to an arbitrary number of classes. The acquisition functions (21) and (22) rely on point-wise uncertainty estimates. A well-known issue of these approaches is their mode-collapse behavior, meaning that they tend to produce sample redundancy by oversampling the same boundary region. They are therefore not incline to exploring unrepresented parts of the boundary, where refinement could greatly improve predictive accuracy. Besides, two other issues should be mentioned: (i) the performance can be very sensitive to the initial estimate of the boundary, and (ii) pure exploitation in normal direction to the boundary may lead to poor performance in the case of boundaries with multiple disconnected components.
Local refinements. This approach was presented in [45] and was used for discontinuity detection as refinement strategy after the application of polynomial annihilation [46, 47, 48] to obtain a very good initial estimate of the separating surface. It proposes a new point by solving
The candidate is accepted among the next batch to be queried if it satisfies two criteria. First, it must be sufficiently far from the current labeled set , i.e.
for some user-defined . Second, it must be close to a region where class labels disagree, which is enforced by requiring that the ball contains points from both classes, i.e.
A similar conclusion to the one drawn for margin/entropy sampling emerges here: while condition (24) ensures that training points are at least away from each other, this algorithm only improves the local approximation of the boundary unless a large is chosen.
Clustering approach. The work presented in [49] promotes diversity among the generated samples by clustering the subset of lying inside the -margin of the classifier, using a k-means algorithm [29]; the centroids of each cluster are then selected as the next batch of points to be queried. Such an algorithm has a higher computational cost compared to the previous alternatives as it requires performing clustering at every active learning iteration. Furthermore, the original work shows improvements only in the low-data regime, as the accuracy tends to plateau afterwards. As a remedy, it proposes to switch back to standard uncertainty sampling after a certain number of active learning iterations, but such a switch cannot be finetuned in advance. Finally, the effectiveness of variance-based clustering algorithms is known to deteriorate as soon as the parameter space has moderate dimension.
Incorporation of diversity. In a similar spirit, [50] incorporates diversity in the selected samples by taking as an acquisition function a convex combination of uncertainty and diversity. More precisely, the acquisition rule in this case reads as
where denotes the set of unlabeled points that lie within the classifier margin, is the set of labeled points, and the diversity measure is the cosine similarity defined in (20). Finally, is a parameter that must be tuned to balance uncertainty and diversity. Note that the case reduces to the margin sampling acquisition rule (21). The numerical experiments in [50] have shown the superior sample efficiency obtained when taking . This algorithm works well and conceptually is the closest to our proposed approach; the main disadvantages are that adjusting may still be cumbersome and that, because it still relies on a discrete search space, its effectiveness can degrade in higher dimensions.
Our proposed approach
Our proposed approach stems from ideas that have been explored for rare-event estimation and reliability analysis [51]. The fundamental paradigm is to view the current estimate of the level set as a geometric object and to prioritize its exploration with mechanisms that (i) keep samples close to the estimated boundary and (ii) allow coherent motion along it. Given a currently available training dataset at active learning iteration , and a trained SVM , we define the estimated regime boundary
and the potential energy
where the inverse-temperature parameter controls the growth of the potential energy (28) in the normal direction to the boundary . The potential energy is minimized at the estimated boundary; furthermore, since we know that four support vector machines, the margin between the two classes is maximized, defining such a potential to be symmetric with respect to is thorough. The task of generating candidate samples that are well distributed along the tangent directions to can be tackled with a variety of gradient-based methods that efficiently explore low-energy landscapes in the proximity of manifolds. As we mentioned earlier, this is motivated in our setting by the fact that computing the gradient in (28) has a negligible computational cost. In this direction, candidate samples are generated by Hamiltonian Monte Carlo (HMC) [52, 53]. We will elaborate on this choice, compared to other sampling mechanisms for the problem at hand, in Subsection 4.2. From the set of generated samples, a space-filling subset is extracted for the next batch to be queried.
The framework presents a variety of practical advantages. First, by using gradient-based dynamics, it generates a large pool of candidates distributed in a tube around ; furthermore, this pool is sought in a continuous search space, unlike margin-based schemes that are evaluating uncertainty estimates on fixed grids (or quasi-random variants thereof). Using gradient information, despite the increased computational cost, helps the adaptive sampling process to avoid function evaluations far away from in order to accurately represent it; this aspect becomes increasingly important in higher-dimensional spaces, where memory constraints become more significant. Moreover, moving away from a discrete search space for the next batch to be queried is expected to provide benefits when the target boundary is complex and has high curvature. Secondly, the subsequent space-filling down-selection discourages repeatedly querying the same local portion of the boundary, which therefore mitigates the mode-collapse behavior often observed in naive uncertainty sampling. It also provides a very natural framework to exploit batch parallelism, as different Markov chains can be evolved in parallel to contain the computational overhead. Lastly, the parameter provides a single knob to trade off global exploration versus local refinement: smaller yields broader exploration in the normal direction (that can accelerate convergence when the initial boundary estimate is crude), whereas larger concentrates samples closer to and effectively focuses the search on refining the interface location; could also be increased during active learning according to a simulated-annealing schedule [54, 55].
In our implementation, because tuning an annealing schedule for is generally challenging, we select a constant value from a pilot run carried out on a toy two dimensional example. We discuss more on this in Section 5. Furthermore, because we are interested in the restriction of the interface to the bounded parametric domain , we enforce box constraints through reflective boundaries: whenever a leapfrog step proposes a point outside , the critical coordinates are mirrored back into the domain and the corresponding momentum component is flipped, yielding a simple, stable constrained dynamics. This is a computationally cheap alternative to reparameterization, which would require modifying all function and gradient evaluations.
The proposed active learning technique is summarized in pseudocode in Algorithm 1. After starting from a coarse initial dataset , the active learning loop starts: given the current labeled dataset, EstimateSVMBoundary trains a new support vector machine. Afterwards, given a pre-defined inverse-temperature , CollectHMCSamples collects informative boundary samples using Hamiltonian Monte Carlo. Lastly, in CollectDiverseSubset, a greedy algorithm selects a diverse space-filling subset of size . Here, a standard Euclidean distance is used, but the framework naturally allows us to use more application-specific distances, such as in (20). This batch is fed to the underlying computational model for the next queries, and the procedure is repeated until the computational budget is exhausted.
Algorithm 1: Active learning for excursion set estimation with HMC-guided sampling
Data: Total budget , number of initial points , batch size , labeling function and inverse-temperature schedule .
Draw an initial sample of size with a low-discrepancy sequence
Collect initial dataset
while do
end
Output: Final dataset
Hamiltonian Monte Carlo
Hamiltonian Monte Carlo (HMC) is a Markov Chain Monte Carlo (MCMC) method designed to sample from a generic target probability distribution of the form
with a potential energy function that is finite and differentiable. To sample with HMC, one needs access to and its gradient . In our setting, is induced by the current SVM decision function at active learning iteration and takes the form , with that can be computed analytically.
The success of HMC stems from augmenting the state space with an auxiliary momentum variable, which enables long-range, coherent proposals that follow approximate iso-probability contours of the target. This mitigates the diffusive random-walk behavior typical of many Metropolis-type samplers, especially in moderate to high dimensions and for strongly correlated targets. In its vanilla version, HMC works as follows: given a (constant) symmetric and positive-definite mass matrix , it defines the Hamiltonian
where is the kinetic energy. The corresponding Hamiltonian dynamics reads as
In practice, from a given configuration , the dynamics (31) is integrated for leapfrog steps with step size , producing a proposal . The pair controls the trade-off between exploration and acceptance rate: too large a step size induces large integration error and low acceptance, while too small a step size leads to high acceptance but short effective moves. A Metropolis acceptance-rejection mechanism corrects the bias introduced by numerical integration errors and ensures that the resulting chain targets (29); that is, proposals are accepted with probability
The mass matrix can be interpreted as a linear preconditioner: choosing to match the scale and anisotropy of the target improves mixing and reduces the need for excessively small . When the geometry of the target distribution is particularly complex, mixing can be improved by taking a position-dependent mass matrix , for example, the Monge metric or the local negative Hessian . When , the resulting Hamiltonian is no longer separable and time integration of (31) requires implicit solves and a higher computational cost; while remedies have been proposed, e.g. by using hierarchical schemes [56], in our experiments we guarantee quality of the generated samples, while containing the computational overhead, by:
Rescaled inputs. Input parameters are rescaled to the unit hypercube , helping to simplify the geometry of the target distribution.
Warm-start/boundary initialization. We partially alleviate the need for long burn-in periods by initializing each chain close to the current estimate of the separating surface. This is done to avoid problematic initializations in regions where (which can happen far away from the support vectors when using a Gaussian kernel). This is obtained by solving (23) with a standard Gauss-Newton scheme from a randomly initialized point in , or by applying clustering to a lightweight grid of points within the margin.
Parallel HMC. We run a larger number of shorter chains from different initializations to improve coverage and promote diversity; this also exposes embarrassingly parallel structure, as each chain can be evolved independently.
Constant mass matrix. Due to the above, we can guarantee the quality of the boundary coverage even when using a constant mass matrix . This keeps leapfrog updates explicit and inexpensive, avoiding the implicit solves required by Riemann-manifold variants.
Boundary exploration mechanism
In this subsection, we further motivate the use of HMC compared to other alternatives for the boundary exploration mechanism. Our objective is to generate candidate samples that remain close to the current estimated boundary while efficiently spreading along it. To this end, we consider target distributions of the form
which concentrates mass where . Here we use a slight abuse of notation, since is effectively constrained to be within the parameter space . Among possible schemes to generate candidate samples distributed according to (33), we mention:
Unadjusted Langevin Algorithm (ULA). Langevin dynamics-based approaches [57, 58] propose to integrate the stochastic differential equation
e.g., with an Euler–Maruyama scheme. In (34), is a Brownian motion differential. Under a very general set of conditions on , the numerical bias introduced by discretizing (34) vanishes and for long enough time horizons the samples are distributed according to (33). Such conditions can be summarized with being strongly-convex, which is not the case for a linear combination of Gaussian kernels; even in the case of a linear SVM, whose Hessian is rank deficient, this condition is not met unless regularization is added.
Metropolis-Adjusted Langevin Algorithm (MALA)
To mitigate for the numerical bias introduced by the discretization of (34), MALA uses a Langevin step as a proposal and adds a Metropolis acceptance-rejection mechanism. The (overdamped) Langevin proposal with step size reads
which corresponds to a Gaussian proposal density . The proposal is accepted with probability
For both MALA and ULA, as for HMC, the dynamics is constrained to remain in through reflective boundaries. In our setting, we notice that the drift term vanishes as . Therefore, even with Metropolis correction, the proposal becomes dominated by Gaussian noise and the motion becomes effectively diffusive along the low-energy region, causing slow mixing. This issue is exacerbated in higher parameter space dimensions and when the estimated boundary has flat directions with vanishing gradients.
Random-walk level set exploration
In low dimensions, a computationally inexpensive alternative is to perform an explicit random walk along the level set and then generate adversarial samples in the normal direction to obtain candidate samples. A standard space-filling criterion can subsequently select a batch for query. Denoting , the orthogonal projector onto the tangent space is
A simple tangential random-walk step with step-size can then be written as
possibly followed by a projection step to enforce . In two dimensions, the tangent space is one-dimensional, so (up to orientation) the unit tangent direction is uniquely determined by ; a convenient choice is
Hence, the motion is fully deterministic except for choosing the sign . Once a boundary point is obtained, adversarial samples in the normal direction can be proposed as
where controls the offset from the currently estimated interface. When scaling this to higher dimensions, tangential exploration gets increasingly difficult, and constrained MCMC mechanisms need to be introduced to allow fast tangential mixing; this becomes basically equivalent (in terms of tangential sample spread and computational cost) to running HMC with a large value of .
Compared to the diffusive exploration of Langevin schemes and constrained random walks in higher dimensions, HMC introduces momentum, which enables long, coherent moves along low-energy regions of (33). A simple geometric picture can help to motivate the term tangential exploration. We assume that is continuously differentiable and in a neighborhood of the estimated interface , so that is locally a smooth -dimensional manifold. This geometric condition is in practice satisfied most of the time: for a Gaussian kernel SVM, vanishing gradients happen only far from the support vectors and hence from the boundary. This further motivates our initialization strategy described earlier. At a point , the normal direction is spanned by and the tangent space is
Since is proportional to , the force term acts (to first order) in the normal direction, while tangential components of the momentum are not directly penalized. Consequently, once the chain has reached the low-energy region around , leapfrog trajectories can propagate efficiently along the level sets of instead of relying on a random-walk exploration. Finally, the tangential component of the HMC dynamics is not directly affected by the choice of : this parameter only controls the spread of the samples in the normal direction to the boundary, while tangential momentum can persist along the low-energy level sets. Figure 1 shows an illustration of the generated samples for different values of on a simple two-dimensional example.
Figure 1. Illustration of the samples generated by Hamiltonian Monte Carlo for increasing values of the inverse-temperature on a simple two-dimensional example. The dashed line is the true boundary and the solid line is the SVM estimate that is used to define the potential energy. As increases, samples concentrate more closely to the boundary. Tangential momentum, leading to good boundary coverage, is not directly affected by the choice of .
Implementation details and computational cost
In this subsection, we discuss the computational cost associated with the proposed algorithm. Compared with uncertainty-based criteria, our method introduces an additional candidate-generation step based on HMC sampling. The key point is that for regime boundary detection in real-world engineering systems this overhead is very small compared to a single high-fidelity model evaluation. Furthermore, we discussed in Subsection 4.1 about some of the implementation details to contain this additional cost.
We now quantify the resulting wall-clock times on a dimension-dependent analytical example and show that, for moderate dimensions (the ones considered in this manuscript, and the ones frequently encountered in parametric computational models), the additional overhead remains small relative to typical simulation times. We consider the analytical regime boundary
which we will use again to study the performance of the proposed approach in Section 5. Here, we target the level set of (42); our focus is purely on timing as a function of dimension and the number of support vectors. For each dimension , we start from points and add batches of size until the budget is exhausted.
For candidate generation, we use HMC chains with burn-in steps, leapfrog steps, initial step-size and retained samples per chain, yielding candidate points per iteration; we then extract points using a greedy space-filling selection. We report wall-clock times averaged over 20 random seeds for each dimension. Figure 2 shows various statistics obtained: despite the increased number of support vectors, the candidate generation time per iteration remains approximately constant throughout the active learning loop, and the total sampling wall-clock time over the entire active learning run remains on the order of one minute. The reported timings are sequential; distributing chains across multiple processors can further reduce the overhead. For example, if we were to parallelize over 5 processors (consistently with the batch size ), then the total overhead would decrease to about 10 seconds. Therefore, whenever each model evaluation is in the order of tens of seconds or minutes, even in the low-data regime, this additional CPU time is negligible. In Section 5, we show improved sample efficiency on analytical and real-world simulation models, which further justifies these additional costs.
Figure 2. Dimension scaling timings: (a) growth of ; (b) serial wall-clock time growth during active learning; (c) total serial wall-clock time across dimensions. Shaded regions represent a 95% CI on the mean.
Numerical examples
We now demonstrate the performance of the proposed approach on a variety of problems: analytical level sets in low dimension, boundaries of increasing dimensions, and finally real-world examples ranging from computational biology to hyperbolic PDEs.
This section is split as follows: we start by considering analytical examples. Such cases are smooth, so they could be solved more efficiently with regression-based estimation; however, our goal is just to compare active learning strategies for classification. Specifically, we compare our proposed approach with margin and cluster sampling, that are investigated as baselines, as well as random sampling (in which case, space-filling points are selected from a low-discrepancy sequence agnostically to the zero level set location). Moreover, to isolate the effects of the different components of our proposed approach, that is the integration to sample from (29) and the space-filling downselection, we consider a margin sampling scheme with diversity-promoting downselection, as well as a MALA integration scheme with the same downstream extraction. Later in the section, we then show real-world engineering systems with discontinuous simulator outputs, where classification outperforms regression in reconstructing the regime boundary. For regression, we consider the Straddle acquisition function (12) as a competitive scheme for regression-based estimation and the maximum variance acquisition function (13) as a baseline; both are executed in batch version.
For all considered classification problems, if not specified otherwise, we use a Gaussian kernel. On the other hand, Gaussian process regression uses a Matérn kernel, which is more robust in non-smooth scenarios. In both cases, after every active learning iteration, we cross-validate the main parameters of the kernel representation using the same search space for all algorithms tested. In particular, for SVMs we finetune appearing in (18) and the parameter controlling the penalty for misclassifications; for GPs, we finetune the length scales and the parameter of the Matérn kernel. For operations with support vector machines, Gaussian processes, and -means clustering, we use the implementations provided in scikit-learn [59]; on the other hand, HMC uses an in-house implementation. Lastly, all experiments were carried on the ETH cluster Euler. Among the implementation details:
For pool-based strategies, we generate an unlabeled candidate pool of size 2000 with Latin Hypercube sampling (LHS) in and regenerate at each active learning iteration.
For the proposed algorithm, we tune on a toy two dimensional example and keep such value constant, that is we implement a constant schedule for the inverse-temperature throughout active learning iterations; more precisely, in all the examples, we use without annealing. Importantly, for in the ranges , we observed that the algorithm’s performance was essentially unchanged. The HMC setup is the same described in Subsection 4.3. The same setup, inverse-temperature, and conclusions hold for MALA.
For cluster sampling and margin with diversity in active classification, we apply -means and the space-filling greedy selection, respectively, on the set of candidate points with for the same value of used in HMC and MALA.
To compare the performance of the different algorithms, we average over 200 seeds corresponding to different initializations and report the following quantities: (i) we compute the mean score on an independent test set of size , unless specified otherwise; (ii) in the analytical examples, we compute additionally the Hausdorff distance between true and approximated boundary by evaluating its discrete counterpart on a grid with size , that is points, and normalizing it as with ; and (iii) we compute the percentage of seeds reaching a target score for a given computational budget, to draw conclusions on the robustness of the different approaches with respect to the initial design.
Analytical two-dimensional functions
We start by fixing and consider the sequence of increasingly complex level set functions
where the scaling and translation factors are selected so that the target level set of the above functions has only one connected component in . The three examples, of which the first two are taken from [45], have progressively increasing curvature. Therefore, for (43), we start with initial points and incrementally add batches of size , until points; on the other hand, for (44) and (45), from the same initialization, we add batches of size until the budget is exhausted.
We start by considering (43). In Figure 3 the adaptive sampling trend is depicted. In this case, the target boundary is simple enough so that all active classification algorithms converge to high scores before the computational budget is exhausted. The only noticeable difference stands in the gap between naive margin sampling and other algorithms in the low-data regime. The Hausdorff distance decay reveals a more interesting trend, as cluster sampling is suboptimal and we observe a clearer advantage when using algorithms performing a space-filling downselection. Among them, our proposed method has a slight advantage throughout the active learning iterations. In Figure 4, we report the percentage of seeds that reaches a certain target score for a given computational budget; this helps to compare the robustness of the algorithms to initialization from a different perspective with respect to the shaded regions in Figure 3. The space-filling algorithms (HMC, MALA, margin with diversity) are essentially equivalent. We notice furthermore that for high target scores cluster sampling plateaus.
Figure 3. Adaptive sampling results on (43). (a) Mean score; (b) mean Hausdorff distance. Shaded regions represent a 95% CI on the mean.

Figure 4. Percentage of seeds reaching different score targets on (43).
Lastly, Figure 5 shows the progression of the boundary approximation, for all the algorithms tested, on a representative seed: all space-filling algorithms produce an even distribution of samples along the estimated boundary, leading to improved accuracy (as already evidenced by the lower Hausdorff distance), while margin and cluster sampling tend to oversample the same regions, creating local clusters and promoting local overfitting.

Figure 5. Adaptive sampling on (43). From top to bottom HMC, MALA, margin, margin + diversity, cluster and random sampling. Dashed/solid lines are the true/SVM boundary, respectively. (a) . (b) . (c) . (d) .
A clearer distinction between the space-filling methods can be appreciated by increasing the curvature of the target boundary. In Figure 6 we consider (44) and observe a different trend. Margin and cluster sampling are both struggling to efficiently approximate the boundary (and are actually worse than random sampling on the Hausdorff distance, because they tend to miss entire regions of the boundary; see Figure 8), while among the space-filling algorithms, MALA plateaus and HMC outperforms margin with diversity in both score and Hausdorff distance, with a larger gap in the latter metric. We also observe that the width of the confidence intervals is smaller for the proposed algorithm, indicating its superior robustness. In Figure 7, we equally notice that the proposed algorithm is consistently more likely to achieve very high scores given a fixed computational budget. This example provides useful insights: first, with an increasing complexity of the boundary, diversity-enforcing schemes have better sample efficiency. Secondly, the score and the Hausdorff distance are complementary metrics for boundary approximation quality, i.e. two methods can have similar scores but still infer very different boundaries; as such, the Hausdorff distance, when computable, gives a more thorough assessment of the performance of the different algorithms. The last observation, related to the aforementioned point, is that for complex boundaries continuous search spaces can be more effective than discrete ones, even in low dimensions.
Figure 6. Adaptive sampling results on (44). (a) Mean score; (b) mean Hausdorff distance. Shaded regions represent a 95% CI on the mean.

Figure 7. Percentage of seeds reaching different score targets on (44).

(a) .

(b) .

(c) .

(d) .
Figure 8. Adaptive sampling on (44). From top to bottom HMC, MALA, margin, margin + diversity, cluster and random sampling. Dashed/solid lines are the true/SVM boundary, respectively.
To confirm the trend, we further increase the complexity of the boundary to be inferred by considering (45). It exhibits very high curvature, albeit while remaining a single connected component. Figure 9 shows a trend similar to that observed for (44): schemes with a space-filling downstream selection are the only ones that outperform random sampling in Hausdorff distance, and the score is again somewhat deceiving, as cluster sampling starts off with the highest score in the early data regime, but its Hausdorff distance is always suboptimal. HMC and margin with diversity are again the two methods that offer the best performance, and our proposed approach shows once again a clear improvement in Hausdorff distance throughout the entire adaptive sampling iterations and, therefore, superior boundary approximation quality. This can be equally appreciated in Figure 11 where we illustrate the progression of all algorithms on a representative seed. A final, but important observation is the following: Figure 9 shows that random sampling is the best scheme for both metrics in the early stages. This signals that if such complexity is known a priori, then the initial design could be enlarged and adaptive sampling could be started later on. The framework proposed in this work would allow for a similar approach by imposing an annealing schedule on ; this could also improve stability and mitigate the early plateauing of both HMC and MALA before they eventually recover a positive trend. However, as mentioned, picking a sound schedule is generally complex.
Figure 9. Adaptive sampling results on (45). (a) Mean score; (b) mean Hausdorff distance. Shaded regions represent a 95% CI on the mean.
Figure 11. (a) . (b) . (c) . (d) . Adaptive sampling on (45). From top to bottom HMC, MALA, margin, margin + diversity, cluster and random sampling. Dashed/solid lines are the true/SVM boundary, respectively.

Figure 10. Percentage of seeds reaching different score targets on (45).
Dimension scaling
The two-dimensional analytical examples discussed in Subsection 5.1 have shown that space-filling schemes are generally outperforming margin and cluster sampling. Therefore, in this example and throughout the rest of the results section, we only compare HMC, MALA and margin with diversity, and retain random sampling as a baseline. Our goal is now to study the scaling of the different active learning strategies with the dimensionality of the parameter space; to this aim, we consider the analytical regime boundary (42) that we have already discussed in Section 4.3, and again targets its level set.
When it is feasible computationally to get converged measurements of the Hausdorff distance (in our case, up to dimension ), we show in Figure 12 that with an increase in the dimension the gap between our proposed approach and competing methods gets larger and larger; this shows improved robustness and scaling of the concentration of samples in a tube around the boundary, but also the effect of their efficient spread tangentially. Indeed, while also MALA tries to concentrate samples in a tube around the boundary, its sample efficiency is comparable and in some cases even worse than that of the margin with diversity approach. On the other hand, in Figure 13 we show for an increasing dimension of the parameter space the percentage of seeds in which each adaptive sampling strategy reaches a certain F1 score target (0.97, 0.98, 0.99) within the prescribed computational budgets. While in lower dimensions the three approaches have similar performance, from onward margin with diversity tends to be unstable and struggles to reach high scores, while MALA is generally more stable but is largely trailing HMC. This effect becomes more pronounced as the dimension increases, highlighting the greater reliability and robustness of the proposed approach.
Figure 12. Hausdorff distance decay on (42) for different diversity-enforcing schemes in different dimensions. Shaded regions represent a 95% CI on the mean.

Figure 13. Percentage of seeds reaching different score targets on (42) for different dimensions, from top to bottom .
It is worth mentioning that these considerations hold for a fixed sample size (in this case, 2000 points); while the latter could be enlarged and this would definitely lead to better results for margin-based schemes, it wouldn’t fundamentally resolve the curse of dimensionality and is likely to lead to memory issues and increased computational costs. On the other hand, HMC is by construction more robust in higher dimensions for a given sample size, because it uses gradient-based dynamics to focus on a tube around the boundary while being excellent at traversing low-energy regions, therefore guaranteeing a more even coverage of the boundary before the space-filling downselection is performed. Furthermore, while to increase predictive accuracy with margin-based schemes we have little choice but to increase the sample size, with the proposed approach we observed that performance could be improved by simply increasing the number of chains , while keeping fixed the total number of generated candidates.
Rolling ball on a Gaussian terrain
In this example, we consider a rigid disk of radius m rolling on a one-dimensional terrain formed by two inverted Gaussian wells
with fixed centers m, m, widths m and heights m; see Figure 14a for an illustration. The ball is launched from the initial position with initial velocity along the terrain tangent and is constrained to remain on the surface; the tangential motion is governed by rolling inertia, gravity, and rolling friction with a coefficient . Two reflective walls placed at m and m have a shared restitution coefficient . Given the parametric triplet with ranges defined in Table 1, the considered discontinuous output is the final horizontal position of the ball , with a threshold value corresponding to the peak of the terrain .
| Parameter | Range |
| Reflective walls restitution coefficient (, dimensionless) | (0.5, 0.8) |
| Initial velocity (, in m/s) | (2, 4) |
| Rolling friction coefficient (, dimensionless) | (0.04, 0.08) |
Table 1. Parameters range for the rolling ball example.
(a) (a)
(b) (b)
Figure 14. Rolling ball example: (a) problem setting; (b) mean score. Shaded regions represent a 95% CI on the mean.
In this example, we start from points and add batches of size until the computational budget is exhausted. Because the final position can only take discrete values and is therefore discontinuous with respect to the parameters, we observe in Figure 14b that regression-based level set estimation has a good performance in low-data regime but then trails classification-based approaches when higher scores are targeted. In particular, Figure 15 shows that Straddle never reaches , which is likely a consequence of the fact that regression cannot accurately resolve the discontinuity front. On the other hand, classification methods are able to target high scores more efficiently. Due to the low dimensionality of the problem, for a given budget, the space-filling classification algorithms are basically equivalent, with HMC and margin leading for most target scores and MALA generally trailing (except for ).

Figure 15. Percentage of seeds reaching different score targets on the rolling ball example depicted in Figure 14a.
Genetic toggle switch
We now consider a genetic toggle switch model proposed in [60], which has been already used for parameter inference [61] and discontinuity detection [62]. This is a differential-algebraic model of a genetic circuit implemented in E. coli plasmids, for which the equations of motion read as follows:
In (47), and represent the concentration of the first and second repressor, respectively; and are the effective rates of synthesis of the repressors; and represent the cooperativity of repression of the two promoters, respectively; finally, [IPTG] is the concentration of the chemical compound that induces the switch. The parameters and describe the binding of [IPTG] to the first repressor. In this experiment, we fix [IPTG] = , , and , as well as the initial condition for time integration of (47). We consider the four parameters , which vary within a hypercube centered around the nominal configuration , with a C.o.V. of . As the parameters change, the system exhibits a bistable regime characterized by two alternative stable states, determined by the dominant repressor concentration at steady state. The quantity of interest is therefore the level set of the function , which is discontinuous in .
We start from points and enrich the training dataset with points at each iteration until we reach the training size . In Figure 16a, we observe that classification largely outperforms regression; indeed, closer inspection reveals that the latter never reaches a 0.95 score on any of the seeds. On the other hand, Figure 16b and Figure 17 show that among classification algorithms, the proposed approach is the most robust and efficient for a given computational budget and with respect to initialization, respectively. In particular, within the given budget, it is the only algorithm that reaches 0.99 score on a non-negligible percentage of seeds. On the other hand, MALA performs well but is generally less sample-efficient and margin with diversity shows deterioration at higher target scores. Contrarily to the previous example, therefore, here we observe a clearer separation between the different algorithms; this could be justified by the larger parameter space dimension.
Figure 16. Adaptive sampling results on the genetic toggle switch (47). (a) Mean score; (b) mean score, only classification methods. Shaded regions represent a 95% CI on the mean.

Figure 17. Percentage of seeds reaching different score targets on the genetic toggle switch (47).
Dam-break shallow water equation
We take as final example a two-dimensional dam-break shallow water equation [63, 64], which is a widely utilized reference for numerical schemes and model order reduction [65]. The hyperbolic equations describing the water height and the velocity are given in conservative form by
where and . In (48), we consider a Gaussian hill bottom topography
and Manning friction
Finally, is the acceleration due to gravity. The equations above are defined in the spatial domain with m and are endowed with the initial conditions
and , where is the absolute height of the water free surface. Lastly, we use outflow boundary conditions on the boundaries and and reflective boundary conditions at and .
The equations (48) are solved numerically using a cell-averaged finite-volume method. We discretize by a Cartesian grid with and finite volume cells, with a finer discretization along because we consider wet-bed simulations () and therefore a shock develops in that direction. The discretization has been chosen to get converged results in the downstream computation of quantities of interest. For interface fluxes, a local Lax-Friedrichs scheme [66] is employed. The terms and are computed analytically by differentiating (49). Finally, we use a Heun time-stepping scheme subject to a CFL condition and time-march up to s.
Regarding the parametric configuration, we fix m, m, m. The variation of the four parameters that influence the flow characteristics is summarized in Table 2.
| Parameter | Range |
| Initial left (, in m) | (8, 14) |
| Initial right (, in m) | (2, 6) |
| Bottom roughness coefficient (, dimensionless) | (0, 0.4) |
| Gaussian hill peak position (, in m) | (100, 150) |
Table 2. Parameters range for the dam-break shallow water.
Our objective is to determine the persistence of a supercritical flow region during the time interval . To do this, we compute the critical water height
The field quantity is negative if the flow is subcritical and signals that the flow is locally supercritical otherwise. To track the time persistence of supercritical flows, we can therefore compute a time-dependent quantity given by
where is the region starting from the downstream slope of the Gaussian hill and the area where the flow is most likely to be supercritical, and . For safety and design considerations, we are interested in the amount of time that such an integrand is positive, that is,
We then define (54) as our simulator output, together with a critical operational time seconds.
The setup is similar to the previous example: we start from quasi-random points and add batches of size until the computational budget is exhausted. In this example, we report the mean score computed on an independent set of uniformly drawn samples and averaged over 50 seeds. Because the transition between subcritical and supercritical flows features steep changes of the quantity of interest [54], regression-based strategies struggle to converge to acceptable accuracies, as we infer from Figure 18a. Once again, on the other hand, Figure 18b shows that HMC remains the most sample-efficient method and generally exhibits a more robust trend, whereas margin with diversity yields oscillatory scores and MALA suffers from diffusive tangential motion. The improved sample efficiency with HMC is confirmed also by looking at the percentage of seeds reaching higher targets in Figure 19.
Figure 18. Adaptive sampling results on the dam-break shallow water [48]. (a) Mean score; (b) mean score, only classification methods. Shaded regions represent a 95% CI on the mean.

Figure 19. Percentage of seeds reaching different score targets on the dam-break shallow water [48].
Conclusions
In this work, we proposed a novel methodology for regime boundary detection in discontinuous simulation models. For such instances, very often the regime switch happens across the boundary of an excursion set. We argued that since the simulator output defining such excursion set is discontinuous or non-smooth, regression-based schemes for level set estimation are inadequate and solving a related binary classification problem provides superior sample efficiency. To this aim, boundary estimation was carried out with support vector machines, and we further proposed a novel active learning strategy to mitigate for the mode-collapse behavior of uncertainty-based sampling. Our approach moves away from discrete search spaces on fixed grids and instead uses gradient-based dynamics to explore the estimated boundary tangentially. To be more precise, we derived a potential energy function from the estimated separating surface and utilized Hamiltonian Monte Carlo to foster coherent tangential motion along its boundary. Owing to the compact representation offered by support vector machines, function and gradient evaluations that need to be performed in this procedure remain computationally cheap; furthermore, we discussed how one can improve mixing, therefore allowing the additional overhead to be near-negligible compared to typical model evaluation costs for complex computational models.
In the results section, we started by showing on a series of analytical examples that the proposed approach has better sample efficiency with respect to both uncertainty-based sampling and an overly diffusive gradient-based exploration, even when both are improved by a diversity-promoting downselection. We presented examples in which this was observed in as low dimension as two (when the target boundary features high curvature), and with an increasing dimension of the parameter space, where the proposed approach can better contrast the curse of dimensionality. Subsequently, we considered a variety of increasingly complex simulation models with discontinuous outputs, and we showed that regression-based approaches are inaccurate; on the other hand, among the active classification strategies, the proposed algorithm was, in the majority of the examples, the most robust with respect to initialization and the most likely to reach a target score given a fixed computational budget.
To conclude, the limitations of the proposed approach should be mentioned. In very high dimensions, the computational overhead may become too large due to the increased number of support vectors and costly gradient evaluations. Slower mixing could also become an issue, although it could be mitigated with preconditioners coupled with hierarchical schemes that mitigate the increased computational cost. Finally, while we improve boundary coverage with gradient-based dynamics, the algorithm proposed in this work improves sample efficiency in the refinement of a connected component that was already identified with the initial static design, but in general it is not obvious why it would “discover” another connected component. While the framework allows for annealing/tempering on the inverse-temperature and this could provide benefits, we discussed that tuning such a parameter can be cumbersome unless a priori knowledge of some features of the true target boundary is available. Nevertheless, for many discontinuous engineering problems with expensive queries, the proposed algorithm remains computationally cheap and allows for accurate regime boundary detection with improved robustness and sample efficiency compared to existing approaches.
Acknowledgments
The authors acknowledge the funding of the ETH Center for Sustainable Future Mobility project “ESTRA – Efficient and Safe Train Dynamics”, within the context of ETH Mobility grants. Federico Betti acknowledges Dr. Nora Lüthen for fruitful discussions at the conceptualization stage of this work.
References
- [1]John David Crawford and Edgar Knobloch. Symmetry and symmetry-breaking bifurcations in fluid dynamics. Annual Review of Fluid Mechanics, 23(1):341–387, 1991.DOI
- [2]Henk A Dijkstra, Fred W Wubs, Andrew K Cliffe, Eusebius Doedel, Ioana F Dragomirescu, Bruno Eckhardt, Alexander Yu Gelfgat, Andrew L Hazel, Valerio Lucarini, Andy G Salinger, et al. Numerical bifurcation methods and their application to fluid dynamics: analysis beyond simulation. Communications in Computational Physics, 15(1):1–45, 2014.
- [3]Moaad Khamlich, Federico Pichi, and Gianluigi Rozza. Model order reduction for bifurcating phenomena in fluid-structure interaction problems. International Journal for Numerical Methods in Fluids, 94(10):1611–1640, 2022.DOI
- [4]John G Milton, André Longtin, Anne Beuter, Michael C Mackey, and Leon Glass. Complex dynamics and bifurcations in neurology. Journal of theoretical biology, 138(2):129–147, 1989.pubmed.ncbi.nlm.nih.gov/2691758
- [5]Eugene M Izhikevich. Dynamical systems in neuroscience. MIT press, 2007.DOI
- [6]Yong Li and ZC Feng. Bifurcation and chaos in friction-induced vibration. Communications in Nonlinear Science and Numerical Simulation, 9(6):633–647, 2004.DOI
- [7]Alexander Pavlovich Ivanov. Bifurcations in systems with friction: Basic models and methods. Regular and Chaotic Dynamics, 14(6):656–672, 2009.DOI
- [8]David Amsallem and Charbel Farhat. Interpolation method for adapting reduced-order models and application to aeroelasticity. AIAA journal, 46(7):1803–1813, 2008.DOI
- [9]Sebastian Grimberg, Charbel Farhat, Radek Tezaur, and Charbel Bou-Mosleh. Mesh sampling and weighting for the hyperreduction of nonlinear petrov–galerkin reduced-order models with local reduced-order bases. International Journal for Numerical Methods in Engineering, 122(7):1846–1874, 2021.arxiv.org/abs/2008.02891
- [10]JA Hernández, Jose Raul Bravo, and S Ares de Parga. CECM: A continuous empirical cubature method with application to the dimensional hyperreduction of parameterized finite element models. Computer Methods in Applied Mechanics and Engineering, 418:116552, 2024.DOI
- [11]Francesco Di Fiore, Michela Nardelli, and Laura Mainini. Active learning and Bayesian optimization: A Unified Perspective to Learn with a Goal. Archives of Computational Methods in Engineering, 31(5):2985–3013, 2024.DOI
- [12]Burr Settles. Active learning literature survey. 2009.
- [13]Stefano Marelli and Bruno Sudret. An active-learning algorithm that combines sparse polynomial chaos expansions and bootstrap for structural reliability analysis. Structural Safety, 75:67–74, 2018.DOI
- [14]Linxiong Hong, Huacong Li, and Jiangfeng Fu. A novel surrogate-model based active learning method for structural reliability analysis. Computer Methods in Applied Mechanics and Engineering, 394:114835, 2022.DOI
- [15]Yulin Guo, Paromita Nath, Sankaran Mahadevan, and Paul Witherell. Active learning for adaptive surrogate model improvement in high-dimensional problems. Structural and Multidisciplinary Optimization, 67(7):122, 2024.DOI
- [16]Ilias Bilionis and Nicholas Zabaras. Multi-output local Gaussian process regression: Applications to uncertainty quantification. Journal of Computational Physics, 231(17):5718–5746, 2012.
- [17]Robert B Gramacy and Herbert K H Lee. Bayesian treed gaussian process models with an application to computer modeling. Journal of the American Statistical Association, 103(483):1119–1130, 2008.arxiv.org/abs/0710.4536
- [18]Xiaoliang Wan and George Em Karniadakis. Multi-element generalized polynomial chaos for arbitrary probability measures. SIAM Journal on Scientific Computing, 28(3):901–928, 2006.DOI
- [19]Khachik Sargsyan, Cosmin Safta, Bert Debusschere, and Habib Najm. Uncertainty quantification given discontinuous model response and a limited number of model runs. SIAM Journal on Scientific Computing, 34(1):B44–B64, 2012.
- [20]Arthur Paul-Dubois-Taine and David Amsallem. An adaptive and efficient greedy procedure for the optimal training of parametric reduced-order models. International Journal for Numerical Methods in Engineering, 102(5):1262–1292, 2015.
- [21]Maciej Balajewicz, David Amsallem, and Charbel Farhat. Projection-based model reduction for contact problems. International Journal for Numerical Methods in Engineering, 106(8):644–663, 2016.arxiv.org/abs/1503.01000
- [22]Maria Cruz Varona, Boris Lohmann, et al. Automatic adaptive sampling in parametric model order reduction by matrix interpolation. In 2017 IEEE International Conference on Advanced Intelligent Mechatronics (AIM), pages 472–477. IEEE, 2017.
- [23]Harshit Kapadia, Peter Benner, and Lihong Feng. Subspace-Distance-Enabled Active Learning for Efficient Data-Driven Model Reduction of Parametric Dynamical Systems. arXiv preprint arXiv:2505.00460, 2025.arxiv.org/abs/2505.00460
- [24]Konstantinos Vlachas, Konstantinos Tatsis, Konstantinos Agathos, Adam R. Brink, Dane Quinn, and Eleni Chatzi. On the coupling of reduced order modeling with substructuring of structural systems with component nonlinearities. In Dynamic Substructures, Volume 4: Proceedings of the 39th IMAC, A Conference and Exposition on Structural Dynamics 2021, pages 35–43. Springer, 2021.DOI
- [25]Konstantinos Vlachas, Anthony Garland, D Dane Quinn, and Eleni Chatzi. Parametric reduced-order modeling for component-oriented treatment and localized nonlinear feature inclusion: K. vlachas et al. Nonlinear Dynamics, 112(5):3399–3420, 2024.DOI
- [26]David Amsallem, Matthew J Zahr, and Charbel Farhat. Nonlinear model order reduction based on local reduced-order bases. International Journal for Numerical Methods in Engineering, 92(10):891–916, 2012.DOI
- [27]Rudy Geelen and Karen Willcox. Localized non-intrusive reduced-order modelling in the operator inference framework. Philosophical Transactions of the Royal Society A, 380(2229):20210206, 2022.DOI
- [28]Margarita Chasapi. Parameterized crack modelling based on a localized non-intrusive reduced basis method. Computational Mechanics, pages 1–22, 2026.
- [29]Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9(8):1295, 2020.DOI
- [30]David Amsallem and Bernard Haasdonk. PEBL-ROM: Projection-error based local reduced-order models. Advanced Modeling and Simulation in Engineering Sciences, 3(1):6, 2016.DOI
- [31]Kyle Washabaugh, David Amsallem, Matthew Zahr, and Charbel Farhat. Nonlinear model reduction for CFD problems using local reduced-order bases. In 42nd AIAA Fluid Dynamics Conference and Exhibit, page 2686, 2012.DOI
- [32]Maliki Moustapha and Bruno Sudret. Learning non-stationary and discontinuous functions using clustering, classification and gaussian process modelling. arXiv preprint arXiv:2211.16909, 2022.arxiv.org/abs/2211.16909
- [33]Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28, 1998.
- [34]Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008.
- [35]Noel Cressie. The origins of kriging. Mathematical geology, 22(3):239–252, 1990.DOI
- [36]Alkis Gotovos. Active learning for level set estimation. 2013.DOI
- [37]Brent Bryan, Robert C Nichol, Christopher R Genovese, Jeff Schneider, Christopher J Miller, and Larry Wasserman. Active learning for identifying function threshold boundaries. Advances in neural information processing systems, 18, 2005.
- [38]Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433–459, 2010.
- [39]Paul G Constantine, Eric Dow, and Qiqi Wang. Active subspace methods in theory and practice: applications to kriging surfaces. SIAM Journal on Scientific Computing, 36(4):A1500–A1524, 2014.
- [40]Paul G Constantine. Active subspaces: Emerging ideas for dimension reduction in parameter studies. SIAM, 2015.
- [41]Ingo Steinwart and Clint Scovel. Mercer’s theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation, 35(3):363–417, 2012.DOI
- [42]Greg Schohn and David Cohn. Less is more: Active learning with support vector machines. In IcmI, volume 2, page 6, 2000.
- [43]Ran Wang and Sam Kwong. Sample selection based on maximum entropy for support vector machines. In 2010 International Conference on Machine Learning and Cybernetics, volume 3, pages 1390–1395. IEEE, 2010.DOI
- [44]John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999.
- [45]Alex Gorodetsky and Youssef Marzouk. Efficient localization of discontinuities in complex computational simulations. SIAM Journal on Scientific Computing, 36(6):A2584–A2610, 2014.arxiv.org/abs/1402.2845
- [46]Neeraj Kayal. The complexity of the annihilating polynomial. In 2009 24th Annual IEEE Conference on Computational Complexity, pages 184–193. IEEE, 2009.DOI
- [47]Cecilia Surace, Richard Archibald, and Rishu Saxena. On the use of the polynomial annihilation edge detection for locating cracks in beam-like structures. Computers & Structures, 114:72–83, 2013.DOI
- [48]Rick Archibald, Anne Gelb, and Jungho Yoon. Polynomial fitting for edge detection in irregularly sampled signals and images. SIAM journal on numerical analysis, 43(1):259–279, 2005.DOI
- [49]Zhao Xu, Kai Yu, Volker Tresp, Xiaowei Xu, and Jizhi Wang. Representative sampling for text classification using support vector machines. In European conference on information retrieval, pages 393–407. Springer, 2003.DOI
- [50]Klaus Brinker. Incorporating diversity in active learning with support vector machines. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 59–66, 2003.
- [51]Deepyaman Chakraborty, Ruben Harris, Rupert Klein, Guillermo Olicón-Méndez, Sebastian Reich, and Claudia Schillings. Affine Invariant Langevin Dynamics for rare-event sampling. arXiv preprint arXiv:2601.00107, 2025.arxiv.org/abs/2601.00107
- [52]Michael Betancourt. A conceptual introduction to Hamiltonian Monte Carlo. arXiv preprint arXiv:1701.02434, 2017.arxiv.org/abs/1701.02434
- [53]Mark Girolami and Ben Calderhead. Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 73(2):123–214, 2011.DOI
- [54]Peter JM Van Laarhoven and Emile HL Aarts. Simulated annealing. In Simulated annealing: Theory and applications, pages 7–15. Springer, 1987.
- [55]Dimitris Bertsimas and John Tsitsiklis. Simulated annealing. Statistical science, 8(1):10–15, 1993.
- [56]Miika Kailas, Matti Vihola, and Jonas Wallin. Adaptive Riemannian Manifold Hamiltonian Monte Carlo with Hierarchical Metric. arXiv preprint arXiv:2604.09832, 2026.arxiv.org/abs/2604.09832
- [57]Giovanni Bussi and Michele Parrinello. Accurate sampling using Langevin dynamics. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 75(5):056707, 2007.pubmed.ncbi.nlm.nih.gov/17677198
- [58]Tatiana Xifara, Chris Sherlock, Samuel Livingstone, Simon Byrne, and Mark Girolami. Langevin diffusions and the Metropolis-adjusted Langevin algorithm. Statistics & Probability Letters, 91:14–19, 2014.DOI
- [59]Oliver Kramer. Scikit-learn. In Machine learning for evolution strategies, pages 45–53. Springer, 2016.
- [60]Timothy S Gardner, Charles R Cantor, and James J Collins. Construction of a genetic toggle switch in Escherichia coli. Nature, 403(6767):339–342, 2000.pubmed.ncbi.nlm.nih.gov/10659857
- [61]Youssef Marzouk and Dongbin Xiu. A stochastic collocation approach to Bayesian inference in inverse problems. 2009.DOI
- [62]John D Jakeman, Richard Archibald, and Dongbin Xiu. Characterization of discontinuities in high-dimensional stochastic problems on adaptive sparse grids. Journal of Computational Physics, 230(10):3977–3997, 2011.DOI
- [63]Arno Roland Ndengna Ngatcha. High order shallow water equations: application to dam break problems. Journal of Mechanics, 40:820–842, 2024.DOI
- [64]Denys Dutykh and Dimitrios Mitsotakis. On the relevance of the dam break problem in the context of nonlinear shallow water equations. arXiv preprint arXiv:0903.3670, 2009.DOI
- [65]Md Rezwan Bin Mizan, Maxim Olshanskii, and Ilya Timofeyev. A parametric tensor ROM for the shallow water dam break problem. Computers & Fluids, page 107005, 2026.
- [66]VA Riestiana, R Setiyowati, and VY Kurniawan. Numerical solution of the one dimensional shallow water wave equations using finite difference method: Lax-Friedrichs scheme. In AIP Conference Proceedings, volume 2326, page 020022. AIP Publishing LLC, 2021.DOI










