betaFeedback

WorksPhysics and Astronomy

Machine Learning Conservation Laws of Dynamical systems

Abstract

Conservation laws are of great theoretical and practical interest. We describe a novel approach to machine learning conservation laws of finite-dimensional dynamical systems using trajectory data. It is the first such approach based on kernel methods instead of neural networks which leads to lower computational costs and requires a lower amount of training data. We propose the use of an "indeterminate" form of kernel ridge regression where the labels still have to be found by additional conditions. We use here a simple approach minimising the length of the coefficient vector to discover a single conservation law.

Introduction

Conservation laws of dynamical systems are of great theoretical and practical interest. In physics, many fundamental principles take the form of a conservation law with the conservation of energy probably the most prominent example. But also in biological and chemical models, conservation of mass and other conservation principles play a prominent role. On a more theoretical side, knowledge of a conservation law almost always provides important insight into a system. Practically, conservation laws may e.g. allow for a model reduction, if certain degrees of freedom can be expressed through others and thus be eliminated from the model equations.

Recently, there have been some efforts in discovering conservation laws of dynamical systems via machine learning, see e.g. [1, 2, 3, 4, 5, 6, 7, 8, 9] and references therein. In these works, quite different characterisations of conservation laws and quite different techniques from machine learning are used. Some approaches identify only a single conservation law, others try to find all of them. Most approaches are based on trajectory data, i. e. they do not require knowledge of the dynamical system (1) itself — they are “model-agnostic”, as the authors of [7] call it —, but [5] uses explicitly the dynamical system and no trajectory data. Common to all these works is the use of neural networks as basic technology and that only well-known textbook examples of conservation laws have been “discovered”. A conservation law is usually also found as a neural network approximating it. Most of the cited works then try in a subsequent symbolic regression step to obtain a closed-form expression for the conservation law. Judging from the presented examples, this seems to work pretty well for conservation laws which are essentially rational functions, but difficulties arise when transcendental functions with non-trivial arguments appear.

In this article, we present a novel approach to machine learning conservation laws using kernel methods, more precisely kernel ridge regression [10, 11, 12]. On the downside, this implies that we can only discover conservation laws living in the Hilbert space defined by the used kernel. Our basic tool is the inhomogeneous polynomial kernel meaning that we search mainly for polynomial conservation laws, but we will discuss how this restriction can be lifted. On the upside, kernel methods always provide explicit closed-form expressions for the discovered conservation laws so that no subsequent symbolic regression is necessary. We furthermore believe that such an approach is not only computationally much more efficient than neural networks, it also requires much less training data. As long as numerical trajectory data are used, it is usually fairly cheap to produce large amounts of data. However, in principle such approaches can also be applied to experimental data which may be much harder to get in large quantities. Of course, this raises then the question how robust the used methods are against noisy data. Kernel ridge regression includes a regularization parameter to mitigate the effects of noise, thus enhancing the reliability of the discovered conservation laws.

The paper is organized as follows: In Section II, we briefly recall the basic properties of conservation laws. Section III introduces the concept of an indeterminate regression, the key component of our method. We explain the process of discovering a single polynomial conservation law, discuss its implementation and possible validation strategies, and finally present some examples. In Section IV, we extend our analysis to the discovery of several conservation laws and non-polynomial conservation laws. Section V addresses some complexity considerations. Section VI explores further applications of our method, including its applicability to discrete dynamical systems and implicitization problems. Finally, some conclusions are given.

Conservation laws

In this work, we will be dealing with (continuous) DD-dimensional dynamical systems of the form

x˙d=fd(x)d=1,…,D(1)\dot{x}_d=f_d(\mathbf{x}) \qquad d=1,\ldots,D \tag*{(1)}

where the vector field f\mathbf{f} on the right hand side is assumed to be at least once continuously differentiable (most dynamical systems in applications are at least smooth if not even analytic). We will usually assume that (1) is explicitly given, but for our basic approach this is not necessary, as it only requires knowledge of finitely many points on finitely many trajectories.

A conservation law or a conserved quantity or a first integral is by definition a function Φ:RD→R\Phi:\mathbb{R}^{D}\to\mathbb{R} which is constant along each trajectory of the system (1). Assuming that Φ\Phi is also at least once continuously differentiable and using the Lie or orbital derivative along the vector field f\mathbf{f}, this is equivalent to Φ\Phi satisfying the linear partial differential equation

f1(x)∂Φ∂x1+⋯+fD(x)∂Φ∂xD=0.(2)f_1(\mathbf{x})\frac{\partial\Phi}{\partial x_1}+\cdots+f_D(\mathbf{x})\frac{\partial\Phi}{\partial x_D}=0. \tag*{(2)}

As we will be interested only in differentiable conservation laws, equation (2) provides a simple rigorous validation criterion for candidate functions Φ\Phi (if the vector field f\mathbf{f} is explicitly given).

Many dynamical systems in physics stem from a variational principle and their conservation laws are related to variational symmetries via Noether’s theorem [13] which often allows their explicit construction with symmetry methods. By contrast, most biological models are of a more phenomenological nature without an underlying variational principle and for them it is much harder to find conservation laws. While linear ones can often be easily constructed with linear algebra (see Remark 3 below), nonlinear ones are rarely known. In principle, methods based on (adjoint) Lie symmetries allow to construct them systematically [14], but for ordinary differential equations it is usually very hard, if not impossible, to find their Lie symmetries.

A first consequence of the above definition is that any constant function Φ\Phi defines a conservation law. Obviously, such conservation laws are not useful and they are called trivial. In the sequel, we will only be concerned with non-trivial ones. The definition of a conservation law also implies that if the system (1) admits one, it automatically admits infinitely many. Indeed, if Φ1,…,ΦL\Phi_1,\ldots,\Phi_L are some conservation laws, then any function of the form g(Φ1,…,ΦL)g(\Phi_1,\ldots,\Phi_L) for an arbitrary function gg is a conservation law, too. If one speaks about finding “all” conservation laws, one actually means finding a maximal set of functionally independent conservation laws. Differentiable functions Φ1,…,ΦL\Phi_1,\ldots,\Phi_L are functionally independent, if and only if their Jacobian ∂Φ/∂x\partial\Phi/\partial\mathbf{x} has almost everywhere maximal rank meaning that the gradient vectors ∇xΦ1,…,∇xΦL\nabla_{\mathbf{x}}\Phi_1,\ldots,\nabla_{\mathbf{x}}\Phi_L are almost everywhere linearly independent. In physics, a Hamiltonian system of dimension D=2dD=2d admitting dd functionally independent conservation laws is called completely integrable [15]. A dynamical system (1) may possess up to D−1D-1 functionally independent conservation laws; if this is the case, each trajectory is uniquely determined by the values of these laws. A Hamiltonian system with more than dd conservation laws is often called superintegrable [16].

Indeterminate Regression

Our approach is based on the literal definition of a conservation law: a function Φ\Phi which is conserved, i. e. constant, along trajectories. Like most approaches presented in the literature, we therefore use numerical trajectory data and not the dynamical system (1) itself. Given finitely many points on a trajectory, any conservation law Φ\Phi must evaluate to the same value at all these points. As several trajectories may lie on the same level set of Φ\Phi, we cannot assume that at points on different trajectories Φ\Phi must evaluate to different values. We consider this situation as an indeterminate regression problem, i. e. a regression problem where the labels are unknown at the beginning and must be determined later by additional conditions.

Discovering One Polynomial Conservation Law

We assume that we are given NN points xm,n∈RD\mathbf{x}_{m,n}\in\mathbb{R}^{D} on each of MM trajectories Tm\mathcal{T}_m of the dynamical system (1). One could easily work with a different number of points on each trajectory. However, to avoid a bias in the regression, we prefer to take always the same number of points. Thus our indices always satisfy m∈{1,…,M}m\in\{1,\ldots,M\} and n∈{1,…,N}n\in\{1,\ldots,N\} and we have a total of MNMN data points. In addition, we introduce MM yet undetermined labels ymy_m and our regression problem consists of finding a function Φ\Phi such that Φ(xm,n)=ym+ϵm,n\Phi(\mathbf{x}_{m,n})=y_m+\epsilon_{m,n} with error terms ϵm,n\epsilon_{m,n} which are minimal in a suitable sense. Note that the desired value for Φ(xm,n)\Phi(\mathbf{x}_{m,n}) does not depend on nn. This fact encodes that we search for functions which are constant along trajectories and hence the number of labels is much smaller than the total number of data points. Ha and Jeong [3] speak here of “grouped data”.

We use kernel ridge regression. Let K:RD×RD→R\mathcal{K}:\mathbb{R}^{D}\times\mathbb{R}^{D}\to\mathbb{R} be the chosen kernel. We will actually work with the polynomial kernel Kc,q(x,x′)=(x⋅x′+c)q\mathcal{K}_{c,q}(\mathbf{x},\mathbf{x}')=(\mathbf{x}\cdot\mathbf{x}'+c)^q parameterised by a non-negative real number c≥0c\geq0 (set in all our experiments to c=1c=1) and a degree q∈Nq\in\mathbb{N}. If KK is the corresponding kernel matrix of dimension MNMN defined by Kmn,mˉnˉ=K(xm,n,xmˉ,nˉ)K_{mn,\bar{m}\bar{n}}=\mathcal{K}(\mathbf{x}_{m,n},\mathbf{x}_{\bar{m},\bar{n}}), then the coefficients of the unique solution of the regression problem can be expressed in closed form as

α=(K+λ1MN)−1Y(3)\boldsymbol{\alpha} = (K + \lambda\mathbf{1}_{MN})^{-1}\mathbf{Y} \tag*{(3)}

where 1MN\mathbf{1}_{MN} denotes the MNMN-dimensional identity matrix, λ\lambda the regularisation parameter of the ridge regression and Y=1N⊗y\mathbf{Y} = \mathbf{1}_{N} \otimes\mathbf{y} the Kronecker product of an NN-dimensional vector 1N\mathbf{1}_{N} consisting only of ones and the MM-dimensional label vector y\mathbf{y}. Here and in the sequel, double indices m,nm,n are always sorted first by the value of mm and then by the value of nn. In other words, we consider first all points on the first trajectory, then all points on the second trajectory and so on. The solution itself is given by the linear combination

Φ(x)=∑m=1M∑n=1Nαm,nφm,n(x).(4)\Phi(\mathbf{x}) = \sum_{m=1}^{M} \sum_{n=1}^{N} \alpha_{m,n}\varphi_{m,n}(\mathbf{x}). \tag*{(4)}

of the base functions φm,n(x)=Kc,q(x,xm,n)\varphi_{m,n}(\mathbf{x}) = \mathcal{K}_{c,q}(\mathbf{x},\mathbf{x}_{m,n}).

In (3), it is inconvenient to work with the label vector Y\mathbf{Y}, as so many of its entries are identical. Consider the matrix K^∈RMN×M\widehat{K} \in\mathbb{R}^{MN \times M} defined as

K^=(K+λ1MN)−1⋅(1M⊗1N).(5)\widehat{K} = (K + \lambda\mathbf{1}_{MN})^{-1} \cdot(\mathbf{1}_{M} \otimes\mathbf{1}_{N}). \tag*{(5)}

Its first column is the sum of the first NN columns of (K+λ1MN)−1(K + \lambda\mathbf{1}_{MN})^{-1}, its second column the sum of the next NN columns of the inverse and so on. This allows us to express (3) more compactly as α=K^y\boldsymbol{\alpha} = \widehat{K}\mathbf{y}. Defining the vector valued function k(x)\mathbf{k}(\mathbf{x}) by

kmˉ(x)=∑m=1M∑n=1NK^mn,mˉK(x,xm,n),(6)k_{\bar{m}}(\mathbf{x}) = \sum_{m=1}^{M} \sum_{n=1}^{N} \widehat{K}_{mn,\bar{m}} K(\mathbf{x},\mathbf{x}_{m,n}), \tag*{(6)}

we can finally write the solution (4) in a form emphasising the linear dependency on the yet unknown labels y\mathbf{y}:

Φ(x)=∑mˉ=1Mymˉkmˉ(x).(7)\Phi(\mathbf{x}) = \sum_{\bar{m}=1}^{M} y_{\bar{m}} k_{\bar{m}}(\mathbf{x}). \tag*{(7)}

There remains the problem of determining labels y\mathbf{y} such that (7) possibly defines a conservation law. A simple approach – in line with the general philosophy of a ridge regression – proceeds as follows: we set one label, say y1y_1, equal to 1 (to avoid the trivial solution y=0\mathbf{y} = 0) and determine the remaining labels in y\mathbf{y} by the condition that the Euclidean (or L2L^2) norm of α=K^y\boldsymbol{\alpha} = \widehat{K}\mathbf{y} should be minimal. Obviously, this condition leads to a quadratic minimisation problem for the labels which can be easily solved in closed form. With thus determined labels, we have finally obtained a candidate Φ\Phi for a conservation law expressed either in the form (4) or (7).

Practical Realisation

Since we are working with “grouped” data, one has to choose two parameters for determining the size of used data set: the number MM of trajectories and the number NN of points on each trajectory. The total number of points is thus MNMN. As generically each trajectory provides information about a further level set of the conserved quantity, we generally prefer larger values of MM and smaller values of NN. For a reasonable sampling of the phase space, one should probably take N≥DN \ge D with DD the phase space dimension. Furthermore, the parameters MM and NN are chosen such that both the hold-out and the training sets in a stratified five-fold cross-validation can be filled with the same number of points (thus we typically choose values which are multiples of 25).

As the points on each trajectory should be about equally spaced to avoid a sampling bias, we do not integrate the dynamical system (1) in the given form, but normalise the vector field f\mathbf{f}. Thus we use as right hand side the field f/∥f∥\mathbf{f}/\lVert\mathbf{f}\rVert. This modification does not change the trajectories, but only their time parametrisation. If we now take on each trajectory points at times separated by a fixed time interval Δt\Delta t, they are automatically equally spaced with respect to the arc length. The initial points for the different trajectories are randomly picked inside a rectangular cuboid and we integrate from each initial point both forward and backward. The size of the cuboid must be chosen such that the obtained labels do not cluster too closely around 1.

Once the data has been produced, the regularisation parameter λ\lambda and the labels y\mathbf{y} must be determined. We consider them all as hyperparameters and compute them simultaneously. For λ\lambda we use a grid search (first on a logarithmic grid for getting the right order of magnitude and then on a linear grid for refinement) and for y\mathbf{y} a 5-fold cross-validation. We first set aside as hold-out set a stratified random sample of 20% of the generated data points; stratified means here that we randomly choose on each trajectory 20% of the points on it.

The remaining 80% of the data points are randomly divided in five disjoint subsets which are again stratified, i.e. each set contains the same number of points from each trajectory. This yields five different splits where always four of the subsets are used as training set and the remaining subset as test set. For each value of λ\lambda on our grid, we determine the optimal labels yλ\mathbf{y}_{\lambda} by the condition that the mean value of the L2L^2 norms of the five coefficient vectors αλ(i)\boldsymbol{\alpha}_{\lambda}^{(i)} obtained for the five different training sets is minimal. Since this choice is identical to the loss function for the individual regressions, there is no trade-off between λ\lambda and αλ(i)\boldsymbol{\alpha}_{\lambda}^{(i)}.

Each vector αλ(i)\boldsymbol{\alpha}_{\lambda}^{(i)} determines a function Φ(i)\Phi^{(i)} which we evaluate on the test points of the iith split. The result for each point is compared with the entry of the label vector yλ\mathbf{y}_{\lambda} corresponding to the trajectory on which the test point lies. We then compute the root mean squared error over all test points of all splits and choose that value for λ\lambda which yields the minimal error (together with the corresponding label vector yλ\mathbf{y}_{\lambda}).

For the chosen values of λ\lambda and yλ\mathbf{y}_{\lambda}, we determine the final coefficient vector αλ\boldsymbol{\alpha}_{\lambda} using all data points outside the hold-out set which then defines our final candidate conservation law Φλ\Phi_\lambda. The function Φλ\Phi_\lambda is then evaluated at all points in the hold-out set and the results are again compared with the labels for the corresponding trajectories. We call the root mean squared error over all these tests ϵgen\epsilon_{\mathrm{gen}} and consider it as a measure for the generalisation error of our final model. Figure 1 depicts the complete procedure.

Flowchart of the indeterminate regression

Figure 1. Flowchart of the indeterminate regression.

Entering the computed vector αλ\boldsymbol{\alpha}_\lambda into (4) yields our candidate conservation law Φ\Phi. Assuming that we are using the polynomial kernel Kc,qK_{c,q}, this candidate Φ\Phi is a polynomial – but written as a linear combination of the base functions φm,n(x)\varphi_{m,n}(\mathbf{x}). As these functions are dense polynomials and not convenient for a human reader, we explicitly write Φ\Phi as an expanded polynomial in DD variables of degree at most qq

Φ(x)=∑0≤∣μ∣≤qaμxμ(8)\Phi(\mathbf{x})=\sum_{0\leq\lvert\boldsymbol{\mu}\rvert\leq q}a_{\boldsymbol{\mu}}\mathbf{x}^{\boldsymbol{\mu}} \tag*{(8)}

where μ=(μ1,…,μD)\boldsymbol{\mu}=(\mu_1,\ldots,\mu_D) denotes an exponent vector and the aμa_{\boldsymbol{\mu}} are numerical coefficients determined from α\boldsymbol{\alpha}. For a normalisation, we divide all coefficients by the one with the largest absolute value, but continue to call them aμa_{\boldsymbol{\mu}}. Typically, most of the obtained coefficients aμa_{\boldsymbol{\mu}} are close to zero. As the largest coefficient is 1 after the normalisation, we consider all coefficients with an absolute value several orders of magnitude smaller as numerical artifacts and set them exactly zero. In our experience, a reasonable threshold is often 10−510^{-5} or 10−610^{-6}. As a “beautification”, one may also think about rounding all coefficients to the corresponding number of digits, as this often helps to make coefficients exactly equal which differ only by a very small amount. But we will present here in the next section a more rigorous alternative.

Validation and Refinement

The above outlined procedure will always produce some candidate function Φ\Phi, but Φ\Phi does not necessarily define a conservation law. In fact, the studied dynamical system (1) might not possess any conservation law at all – at least not within the Hilbert space defined by the used kernel. Thus we need a validation procedure for accepting or rejecting a candidate.

There are two fairly immediate possibilities for a purely numerical validation. One verifies directly that Φ\Phi is constant along each of the computed trajectories which can be done independently of the dynamical system (1) or one uses the equivalent characterisation of conservation laws via the partial differential equation (2) which, however, requires explicit knowledge of the vector field f\mathbf{f}.

In the first approach, we compute for each trajectory the mean value Φ‾m\overline{\Phi}_m of the candidate Φ\Phi evaluated at the data points on this trajectory, i.e.

Φ‾m=1N∑n=1NΦ(xm,n)(9)\overline{\Phi}_m=\frac{1}{N}\sum_{n=1}^{N}\Phi(\mathbf{x}_{m,n}) \tag*{(9)}

and then consider the root mean squared relative deviation of Φ\Phi from these mean values

Δ1(Φ)=1MN∑m=1M∑n=1N(Φ(xm,n)−Φ‾m)2Φ‾m2(10)\Delta_1(\Phi)=\sqrt{\frac{1}{MN}\sum_{m=1}^{M}\sum_{n=1}^{N}\frac{\left(\Phi(\mathbf{x}_{m,n})-\overline{\Phi}_m\right)^2}{\overline{\Phi}_m^2}} \tag*{(10)}

We accept Φ\Phi as a conservation law, if Δ1(Φ)<ϵ1\Delta_1(\Phi)<\epsilon_1 for a user determined validation threshold ϵ1>0\epsilon_1>0.

Kernel methods provide us automatically with a candidate Φ\Phi in a symbolic form without any need for a symbolic regression. Furthermore, it is easy to compute its derivatives ∂Φ/∂x\partial\Phi/\partial\mathbf{x}. Thus, a direct symbolic validation via the partial differential equation (2) seems possible. However, as the coefficients in the linear combinations (4) or (7), respectively, have been determined numerically, we cannot expect that (2) is satisfied exactly. Thus we resort instead again to a purely numerical approach by evaluating the left hand side of (2) at all data points xm,n\mathbf{x}_{m,n} and considering the root mean squared error

Δ2(Φ)=1MN∑m,n(∑d=1Dfd(xm,n)∂Φ∂xd(xm,n))2(11)\Delta_2(\Phi)=\sqrt{\frac{1}{MN}\sum_{m,n}\left(\sum_{d=1}^{D}f_d(\mathbf{x}_{m,n})\frac{\partial\Phi}{\partial x_d}(\mathbf{x}_{m,n})\right)^2} \tag*{(11)}

We accept Φ\Phi as a conservation law, if Δ2(Φ)<ϵ2\Delta_2(\Phi)<\epsilon_2 for some prescribed threshold ϵ2\epsilon_2. One should note that while it is natural to use the given data points for computing Δ2(Φ)\Delta_2(\Phi), one could in principle use any set of test points that reasonably samples some open subset of the state space. It is not even necessary that the points are grouped, i. e. come from some trajectories.

From a mathematical point of view, the approaches are equivalent and in experiments one indeed observes a close correlation between them. Δ1(Φ)\Delta_1(\Phi) is easier to compute and as a relative variation easier to interpret which in turn makes it easier to choose the threshold ϵ1\epsilon_1. In our experiments, we therefore preferred to work with Δ1(Φ)\Delta_1(\Phi).

If the vector field f\mathbf{f} is explicitly known, one can use the symbolic evaluation of the partial differential equation (2) for the refinement of candidates which have passed the numerical validation. Our starting point is the representation (8) of Φ\Phi as an expanded polynomial – after all very small coefficients have been set to zero. Let

Φˉ(x,δ)=∑0≤∣μ∣≤qaμ≠0(aμ+δμ)xμ(12)\bar{\Phi}(\mathbf{x},\boldsymbol{\delta}) = \sum_{\substack{0\leq|\boldsymbol{\mu}|\leq q\\a_{\boldsymbol{\mu}}\ne0}}(a_{\boldsymbol{\mu}}+\delta_{\boldsymbol{\mu}})\mathbf{x}^{\boldsymbol{\mu}} \tag*{(12)}

be a perturbed form of this representation with yet undetermined perturbations δμ\delta_{\boldsymbol{\mu}}. As typically many coefficients aμa_{\boldsymbol{\mu}} in (8) will be very small, we can expect that the number of these perturbations will be much smaller than the dimension nD,q=(D+qq)n_{D,q}=\binom{D+q}{q} of the vector space of polynomials of degree at most qq in DD variables.

We assume now furthermore that our vector field f\mathbf{f} is polynomial, too, with entries of degree at most qfq_f. Entering (12) into the left hand side of the partial differential equation (2) and expanding, we obtain a polynomial

∑d=1Dfd(x)∂Φˉ(x,δ)∂xd=∑0≤∣ν∣<q+qfbν(δ)xν(13)\sum_{d=1}^{D}f_d(\mathbf{x})\frac{\partial\bar{\Phi}(\mathbf{x},\boldsymbol{\delta})}{\partial x_d}=\sum_{0\leq|\boldsymbol{\nu}|<q+q_f}b_{\boldsymbol{\nu}}(\boldsymbol{\delta})\mathbf{x}^{\boldsymbol{\nu}} \tag*{(13)}

where the coefficients bν(δ)b_{\boldsymbol{\nu}}(\boldsymbol{\delta}) depend linearly on the perturbations δμ\delta_{\boldsymbol{\mu}}, since (12) is linear in them and (2) is a linear differential equation. Any perturbation δ\boldsymbol{\delta} with Δ2(Φˉ(x,δ))<ϵ2\Delta_2(\bar{\Phi}(\mathbf{x},\boldsymbol{\delta}))<\epsilon_2 is consistent with the used data (obviously, here using Δ2\Delta_2 is more natural than taking Δ1\Delta_1).

We compute the least-squares solution δ∗\boldsymbol{\delta}^* of the linear system bν(δ)=0b_{\boldsymbol{\nu}}(\boldsymbol{\delta})=0 for 0≤∣ν∣<q+qf0\leq|\boldsymbol{\nu}|<q+q_f. Note that despite its outer appearance this is an inhomogeneous linear system, as the coefficients bν(δ)b_{\boldsymbol{\nu}}(\boldsymbol{\delta}) usually contain constant terms. Generally, we can expect this system to be overdetermined, as the number of coefficients bνb_{\boldsymbol{\nu}} is larger than the number of perturbations δμ\delta_{\boldsymbol{\mu}} for qf>1q_f>1 (i. e. for a nonlinear dynamical system). However, depending on the exact form of f\mathbf{f} and Φˉ\bar{\Phi}, it may happen in exceptional cases that the system is underdetermined. Then the least-squares solution is not unique and it is natural to take the unique one of minimal norm. After possibly rounding coefficients to a prescribed number of digits, our final conservation law is then given by Φˉ(x,δ∗)\bar{\Phi}(\mathbf{x},\boldsymbol{\delta}^*).

Examples

Most dynamical systems appearing in physics or biology depend on parameters and these also show up in their conservation laws. We will therefore always consider the parameters as further state variables with trivial dynamics. Obviously, this approach increases the dimension DD of the dynamical system – in biological systems often significantly. Furthermore, one generally has now to work with a kernel of higher degree qq to accommodate for the dependency on the parameters.

Example 1. A classical textbook example of a system with a conservation law is the Hénon-Heiles system from astronomy:

q˙1=p1,q˙2=p2,\dot{q}_1=p_1,\qquad\dot{q}_2=p_2,
p˙1=−q1−2q1q2,p˙2=−q2−q12+q22.(14)\dot{p}_1=-q_1-2q_1q_2,\qquad\dot{p}_2=-q_2-q_1^2+q_2^2. \tag*{(14)}

It is a Hamiltonian system and thus has as a conserved quantity the total energy given by

E=12(p12+p22+q12+q22)+q12q2−13q23.(15)E=\frac{1}{2}\left(p_1^2+p_2^2+q_1^2+q_2^2\right)+q_1^2q_2-\frac{1}{3}q_2^3. \tag*{(15)}

The trajectories are rather different for different values of EE. For E<1/6E<1/6, all trajectories are bounded and regular; for E>1/6E>1/6 most trajectories show a chaotic behaviour.

We conducted some experiments on the Hénon-Heiles system using different initial data sets, some composed entirely of points on chaotic trajectories and some using only points on regular trajectories. Some results are shown in Figure 2. We were particularly interested in estimating how many data points are necessary to discover the conservation law using a cubic polynomial kernel. It turned out that with 200 data points no correct polynomial was learned, whereas with 300 data points good results were achieved. These numbers should be contrasted with the dimension n4,3=35n_{4,3}=35 of the space of polynomials of degree 3 in four variables: we needed roughly eight times as many data points as coefficients were to be determined. A further increase of the number of data points did not lead to notable improvements in the validation. Our refinement procedure produced here the exact expression for the energy eliminating all numerical errors in the coefficients.

Hénon-Heiles model

Figure 2. Hénon-Heiles model: (2a) shows the projection to the q1q_1-q2q_2-plane of a regular trajectory at energy E=0.07E = 0.07. Figures (2b) and (2c) depict the correlation coefficients for 300 and 200 data points, respectively. The relative variations along some random regular and chaotic trajectories are shown in (2d) and (2e), respectively.

While the basic results were very similar for data sets consisting of regular or chaotic trajectories, respectively, one could see some small differences in a closer analysis. It seems that for a fixed total number of data points it is better in the chaotic case to use a smaller number of trajectories with more points on each of them, whereas in the regular case more trajectories with a lower number of points work better. A possible explanation could be that a chaotic trajectory has a higher fractal dimension. On the one hand, it thus contains more information about the corresponding level set of the energy, but on the other hand more data points are necessary to extract this information.

Example 2. As a “scalable” example, we consider (generalised) Fermi–Pasta–Ulam–Tsingou (FPUT) lattices [17] described by second-order systems of the form

x¨ℓ=f(xℓ+1−xℓ)−f(xℓ−xℓ−1).(16)\ddot{x}_{\ell} = f(x_{\ell+1} - x_{\ell}) - f(x_{\ell} - x_{\ell-1}). \tag*{(16)}

For simplicity, we restrict to periodic lattices, i. e. we assume that xℓ+L=xℓx_{\ell+L} = x_{\ell} for some given L∈NL \in\mathbb{N}. It then suffices to consider only ℓ=1,…,L\ell= 1,\ldots,L and rewriting (16) as a first-order system yields a system of dimension D=2LD = 2L. The obtained system is Hamiltonian so that it possesses a conservation law:

H=12∑ℓ=1Lx˙ℓ2+∑ℓ=1LF(xℓ+1−xℓ)(17)H = \frac{1}{2}\sum_{\ell=1}^{L}\dot{x}_{\ell}^{2} + \sum_{\ell=1}^{L}F(x_{\ell+1} - x_{\ell}) \tag*{(17)}

where FF is an antiderivative of ff. One usually assumes that FF can be represented by a power series and the original FPUT α\alpha-model is obtained for the simplest non-trivial choice F(x)=12kx2+13αx3F(x) = \frac{1}{2}kx^{2} + \frac{1}{3}\alpha x^{3}. In the β\beta-model, one adds one further term 14βx4\frac{1}{4}\beta x^{4}.

We used the α\alpha-model treating the parameter kk and α\alpha as further state variables with trivial dynamics. Hence, our systems were of dimension 2L+22L + 2 and we had to use a polynomial kernel of degree 4. In figure (3), we show the relative variation along some random trajectories for L=2L = 2 and L=5L = 5. While the results are significantly worse for the larger system, they are still fairly good. In the case L=5L = 5, a set of 200 data points was not sufficient for discovering the conservation law, whereas 500 data points yielded good results. If this is contrasted with the dimension n12,4=1820n_{12,4} = 1820 of the space of polynomials in 12 variables and maximal degree 4, then one sees that comparatively very few data points allow for the discovery.

Extensions

Discovering More Conservation Laws

A dynamical system might possess several (functionally independent) conservation laws. A simple way to discover these consists of an iteration of the approach presented in Section III A following an idea already proposed by Wetzel et al. [2]. It is, however, only applicable, if it is possible to produce data points for arbitrary initial conditions, and thus generally cannot be used with experimental data. Assume that the validation has confirmed the discovery of a first conservation law Φ1\Phi_1. Then we generate new trajectory data such that all trajectories lie on a level set of Φ1\Phi_1, i.e. we choose for the trajectories initial data x0(i)\mathbf{x}_0^{(i)} for i=1,…,Mi = 1, \ldots, M such that Φ1(x0(1))=⋯=Φ1(x0(M))\Phi_1(\mathbf{x}_0^{(1)}) = \cdots= \Phi_1(\mathbf{x}_0^{(M)}). Applying our indeterminate regression to this new data, we will obtain a new candidate Φ2(x)\Phi_2(\mathbf{x}). As our ansatz ensures that Φ2\Phi_2 cannot be constant on all data points, Φ2\Phi_2 must be functionally independent of Φ1\Phi_1.

Relative variation of the discovered conservation law along random trajectories for FPUT lattices

Figure 3. Relative variation of the discovered conservation law along random trajectories for FPUT lattices with L=2L = 2 (left) and L=5L = 5 (right) vertices.

If validation confirms Φ2\Phi_2 as a conservation law, we produce again new trajectory data such that all trajectories lie on common level set of Φ1\Phi_1 and Φ2\Phi_2. By the same reasoning as above, the new candidate Φ3\Phi_3 must be functionally independent of the already found conservation laws Φ1\Phi_1 and Φ2\Phi_2. We continue in this manner, until the validation rejects the last candidate.

Note that this approach does not need any a priori assumption on the actual number of functionally independent conservation laws. It automatically stops, when a complete set has been found within the considered Hilbert space. Furthermore, this approach allows for an easy integration of a priori known conservation laws: one simply uses initial data such that all known conservation laws are constant on all initial points.

Remark 3. Linear conservation laws are easy to compute directly for almost any dynamical systems so that there is no need to resort to machine learning for them. Assume that (1) can be brought into the “pseudolinear” form x˙=Ng(x)\dot{\mathbf{x}} = N\mathbf{g}(\mathbf{x}) with a constant matrix N∈RD×BN \in\mathbb{R}^{D \times B} and a BB-dimensional vector g\mathbf{g} of “building blocks”. Such a structure appears e.g. naturally in chemical reaction networks where NN is the stoichiometric matrix and g\mathbf{g} the vector of reaction rates [18, 19]. More generally, any polynomial vector field is of this form with the vector g\mathbf{g} consisting of all terms appearing in the vector field. One can show with elementary linear algebra that for any vector v∈ker⁡NT\mathbf{v} \in\ker N^{T} the linear function Φ(x)=v⋅x\Phi(\mathbf{x}) = \mathbf{v} \cdot\mathbf{x} defines a conservation law and that all linear conservation laws are of this form – see e.g. [20]. Thus, one can precompute a complete set of independent linear conservation laws and then use machine learning only for discovering additional nonlinear conservation laws functionally independent of the linear ones.

For the generation of the new initial data in each iteration of the above outline procedure, we use a simple approach. Assume that we have already found the conservation laws Φ1,…,Φk\Phi_1, \ldots, \Phi_k. We randomly choose a first initial point x0(1)\mathbf{x}_0^{(1)}; the values ci=Φi(x0(1))c_i = \Phi_i(\mathbf{x}_0^{(1)}) then define the level set on which we choose further points. We randomly pick values (xˉk+1,…,xˉn)(\bar{x}_{k+1}, \ldots, \bar{x}_n) and compute one solution of the system Φi(x1,…,xk,xˉk+1,…,xˉn)=ci\Phi_i(x_1, \ldots, x_k, \bar{x}_{k+1}, \ldots, \bar{x}_n) = c_i. If the (numerical) solution of this system makes problems, we simply discard our random pick and choose a new one. We iterate this procedure, until we have obtained the necessary number of initial points.

Note that it is only for notational convenience that we consider the first kk coordinates xix_i as unknowns here. Any choice of kk coordinates is possible here and in practise one should make the choice in dependence of the actual form of the conservation laws Φ1,…,Φk\Phi_1, \ldots, \Phi_k. If for example a conservation law is linear in some variable xix_i, then of course we simply solve it for this variable and ignore it in the system to be solved numerically.

Example 4. The following three-dimensional Lotka–Volterra system [21 Ex. 3]

x˙1=x1(x2−x3),x˙2=x2(x3−x1),x˙3=x3(x1−x2)(18)\dot{x}_1 = x_1(x_2 - x_3), \qquad\dot{x}_2 = x_2(x_3 - x_1), \qquad\dot{x}_3 = x_3(x_1 - x_2) \tag*{(18)}

has two polynomial conservation laws of degree 1 and 3, respectively, namely

Φ1=x1+x2+x3,Φ2=x1x2x3.(19)\Phi_1 = x_1 + x_2 + x_3,\qquad\Phi_2 = x_1x_2x_3. \tag*{(19)}

For a linear conservation law, the level sets are hyperplanes so that it is trivial to generate initial data on a randomly chosen but fixed level set. When searching for the cubic conservation law, we have here D=d=3D=d=3 and thus nD,d=20n_{D,d}=20. In our experiments, 50 data point were not sufficient to discover Φ2\Phi_2; we needed 100. Figure 4 shows again the relative conservation error along random trajectories.

Relative variation of the discovered cubic conservation law along random trajectories of the 3D Lotka–Volterra system

Figure 4. Relative variation of the discovered cubic conservation law along random trajectories of the 3D Lotka–Volterra system

Example 5. The non-dissipative Lorenz model is described by the three-dimensional system

x˙=σy,y˙=−xz+rx,z˙=xy(20)\dot{x}=\sigma y,\qquad\dot{y}=-xz+rx,\qquad\dot{z}=xy \tag*{(20)}

with two parameters σ,r\sigma,r. In contrast to the better known chaotic Lorenz model, it possesses two conservation laws:

Φ1=12x2−σz,Φ2=rz−12y2−12z2.(21)\Phi_1=\frac{1}{2}x^2-\sigma z,\qquad\Phi_2=rz-\frac{1}{2}y^2-\frac{1}{2}z^2. \tag*{(21)}

A first iteration of our indeterminate regression procedure discovered after refinement the exact conservation law Φ1\Phi_1. Since Φ1\Phi_1 is linear in zz and one considers for σ\sigma only positive values, we produced new initial data with x,y,σ,rx,y,\sigma,r chosen randomly and zz determined by Φ1\Phi_1. Due to the parameters, we have here D=5D=5 and d=2d=2 and thus nD,d=21n_{D,d}=21. Already with 40 data points, Φ2\Phi_2 is discovered exactly, if one truncates the coefficients to six digits (with less data points, it is difficult to perform a cross validation). The relative conservation error of the learned Φ2\Phi_2 is shown in Figure 5.

Relative conservation error of $\Phi_2$ along random trajectories for non-dissipative Lorenz system

Figure 5. Relative conservation error of Φ2\Phi_2 along random trajectories for non-dissipative Lorenz system

More General Conservation Laws

If H\mathcal{H} is the reproducing kernel Hilbert space uniquely associated with the chosen kernel K\mathcal{K}, then our approach searches for conservation laws only in the finite-dimensional subspace of H\mathcal{H} of all functions of the form (7). In the case of an inhomogeneous polynomial kernel of degree dd, we can thus only find conservation laws which are polynomials in x\mathbf{x} of maximal degree dd.

If one has a priori knowledge which other, say transcendental or rational, functions may appear in a conservation law, it is rather easy to adapt our approach such that also polynomials in these functions are discovered. Assume we suspect that the functions η1(x),…,ηL(x)\eta_1(\mathbf{x}),\ldots,\eta_L(\mathbf{x}) will occur (in [6], this is called the “theorist setup” with known “basis functions”). In our basic approach, the feature vector for the kernel ridge regression coincides with the coordinate vector x\mathbf{x} of the dynamical system. Now, we simply extend the feature vector by the expressions η1(x),…,ηL(x)\eta_1(\mathbf{x}),\ldots,\eta_L(\mathbf{x}), before we start the kernel ridge regression. Obviously, this will increase the dimension of the feature vector from DD to D+LD+L and thus the computational costs. But this approach allows us to find any conservation law that is polynomial in the variables x\mathbf{x} and the chosen transcendental functions η(x)\boldsymbol{\eta}(\mathbf{x}) (here we have a slight difference to [6] where it is assumed that a linear basis of the considered function space is known).

Example 6. It is well-known that many Lotka–Volterra systems admit conservation laws containing logarithmic terms [21] (this is also very common in chemical reaction networks, as the entropy is a logarithmic quantity [18]). This is easily accommodated by extending the feature vector: we add for each coordinate xdx_d its logarithm ln⁡xd\ln x_d, i.e. we use (x,ln⁡x)(\mathbf{x},\ln\mathbf{x}) as feature vector for the regression. For general dynamical systems, one might worry what happens for negative xdx_d. But most biological or chemical models are positive and for them only the positive orthant x∈R>0D\mathbf{x}\in\mathbb{R}_{>0}^D is relevant as state space.

As a concrete example, we consider the following four-dimensional Lotka–Volterra system

x˙1=x1(3−x2−x3−x4),x˙2=x2(x1−x3),\dot{x}_1=x_1(3-x_2-x_3-x_4),\qquad\dot{x}_2=x_2(x_1-x_3),
x˙3=x3(−1+x1+x2−x4),x˙4=x4(−2+x1+x3),(22)\dot{x}_3=x_3(-1+x_1+x_2-x_4),\qquad\dot{x}_4=x_4(-2+x_1+x_3), \tag*{(22)}

belonging to the class considered in [21 Ex. 7]. It possesses the logarithmic conservation law

Φ=∑d=14(xd−ln⁡xd).(23)\Phi= \sum_{d=1}^{4}(x_d-\ln x_d). \tag*{(23)}

As the conservation law Φ\Phi is linear in the variables x\mathbf{x} and ln⁡x\ln\mathbf{x}, we can work with d=1d=1 and D=8D=8 and thus have a search space of dimension n8,1=9n_{8,1}=9. Nevertheless, we needed 100 data points to discover Φ\Phi, 50 points were not sufficient.

Relative conservation error along random trajectories for 4D Lotka–Volterra system

Figure 6. Relative conservation error along random trajectories for 4D Lotka–Volterra system

Example 7. The discrete sine-Gordon equation in the form

x¨ℓ=k(xℓ+1−2xℓ+xℓ−1)−gsin⁡xℓ(24)\ddot{x}_{\ell}=k(x_{\ell+1}-2x_{\ell}+x_{\ell-1})-g\sin x_{\ell} \tag*{(24)}

describes a chain of pendula coupled by torsion springs [22 pp. 42–44], which among other applications is sometimes used as a simple model to describe the mechanics of DNA [23]. For simplicity, we restrict again to periodic chains and set xℓ+L=xℓx_{\ell+L}=x_{\ell} for some given chain length L∈NL \in\mathbb{N}. Rewriting (24) as a first-order system yields a system of dimension 2L2L which is Hamiltonian with a conservation law

H=12∑ℓ=1Lx˙ℓ2+∑ℓ=1L[k2(xℓ+1−xℓ)2+g(1−cos⁡xℓ)].(25)H=\frac{1}{2}\sum_{\ell=1}^{L}\dot{x}_{\ell}^{2}+\sum_{\ell=1}^{L}\left[\frac{k}{2}(x_{\ell+1}-x_{\ell})^{2}+g(1-\cos x_{\ell})\right]. \tag*{(25)}

Pretending that we did not know HH, we extended the feature vector by both sin⁡xℓ\sin x_{\ell} and cos⁡xℓ\cos x_{\ell} and thus obtained a vector of dimension 4L4L. Since we also treated the two parameters kk, gg as dynamical variables, we have here D=4L+2D=4L+2 and d=3d=3. We worked with a very small chain with L=3L=3, so that nD,d=680n_{D,d}=680. With 480 data points we could not discover HH, but with 600. Figure 7 shows the relative conservation error along random trajectories. Again, our refinement procedure yielded (after a truncation of the coefficients to six digits) the exact conservation law.

Relative conservation error along random trajectories for discrete sine-Gordon equation (L = 3)

Figure 7. Relative conservation error along random trajectories for discrete sine-Gordon equation (L=3L=3)

Some Complexity Considerations

The core computation in our approach is the kernel ridge regression requiring mainly the inversion of the matrix K+λ1K+\lambda\mathbf{1} (because of our indeterminate regression, we indeed need the inverse and not only the solution of a linear system). The dimension of this matrix equals the number MNM_N of used data points. It is a priori independent of the dimension DD of the feature vector and of the degree qq of the used polynomial kernel, although our experiments show that typically MNM_N has to be roughly of magnitude O(nD,q)O(n_{D,q}) where again nD,qn_{D,q} is the dimension of the space of polynomials in DD variables and of degree up to qq. This implies in particular that modifications of our basic algorithm – like the “promotion” of parameters to dynamical variables or the inclusion of additional functions into the feature vector x\mathbf{x} which all increase DD and may require higher values of qq – have only a moderate effect on the complexity of this step. The matrix inversion requires O((MN)3)O((M_N)^3) operations. The fact that our matrix is symmetric and positive definite leads to better constants in this estimate, but does not change the cubic complexity. The (system) dimension DD is relevant, when the data is produced by numerical integration. One may estimate the costs for this step to be roughly O(DMN)O(DM_N) which is, however, negligible compared to the costs of the matrix inversion.

As soon as the final coefficient vector α\boldsymbol{\alpha} has been computed and the candidate conservation law Φ\Phi defined by (4) has been validated, we are in principle done and have found a conservation law. Compared with approaches where conservation laws are represented by neural networks, our result is much more explicit, as it is a closed form expression (with numerical coefficients). However, as the conservation law is given as a linear combination of the base functions φm,n\varphi_{m,n}, it does not arise in a form convenient for any subsequent analysis.

One should therefore transform the conservation law Φ\Phi obtained in the form (4) into the usual representation (8) of a polynomial. One may consider this rewriting as the analogue in our method to the symbolic regression step used in approaches based on neural networks to obtain a closed form expression for the discovered conservation law. If one does the expansion straightforwardly in a symbolic computation, then each of the base functions φm,n(x)=(xTxm,n+c)q\varphi_{m,n}(\mathbf{x}) = (\mathbf{x}^{T}\mathbf{x}_{m,n} + c)^q expands into a dense polynomial of degree qq in DD variables and we must take a linear combination of MNMN such polynomials. Any such polynomial contains nD,qn_{D,q} terms. Indeed, we noticed in experiments that – already for moderate values of DD and qq – such a symbolic computation leads to a bottleneck.

However, it is straightforward to compute directly the coefficients of the representation (8) in a purely numerical manner via a multinomial expansion. We first introduce the following notations. Let μ^=(μ0,μ1,…,μD)=(μ0,μ)\hat{\mu} = (\mu_0,\mu_1,\ldots,\mu_D) = (\mu_0,\mu) be an exponent vector with D+1D+1 entries and μ\mu the vector with DD entries obtained by removing μ0\mu_0. We can then write the base functions in the form

φm,n(x)=∑∣μ^∣=q(qμ^)cμ0xm,nμ(26)\varphi_{m,n}(\mathbf{x}) = \sum_{|\hat{\mu}|=q} \binom{q}{\hat{\mu}} c^{\mu_0}\mathbf{x}_{m,n}^{\mu} \tag*{(26)}

with (qμ^)\binom{q}{\hat{\mu}} a multinomial coefficient and our conservation law expands to

Φ(x)=∑m=1M∑n=1N∑∣μ^∣=qαm,n(qμ^)cμ0xm,nμxμ.(27)\Phi(\mathbf{x}) = \sum_{m=1}^{M}\sum_{n=1}^{N}\sum_{|\hat{\mu}|=q} \alpha_{m,n}\binom{q}{\hat{\mu}}c^{\mu_0}\mathbf{x}_{m,n}^{\mu}\mathbf{x}^{\mu}. \tag*{(27)}

By simply collecting the terms with the same power product, we obtain for the coefficients aμa_{\mu} in (8) the formula

aμ=cμ0(qμ^)∑m=1M∑n=1Nαm,nxm,nμ,(28)a_{\mu} = c^{\mu_0}\binom{q}{\hat{\mu}}\sum_{m=1}^{M}\sum_{n=1}^{N}\alpha_{m,n}\mathbf{x}_{m,n}^{\mu}, \tag*{(28)}

where μ0=q−∣μ∣\mu_0 = q - |\mu|. The numerical evaluation of this sum is straightforward. The total cost of the rewriting is then O(nD,qMN)O(n_{D,q}MN) and neglectable against the cost of the kernel ridge regression.

One may wonder why we bother with machine learning techniques, if we search only for polynomial conservation laws – in particular, in view of our refinement method. One could instead directly make a simple ansatz for a general polynomial of degree qq in DD variables with indeterminate coefficients, enter the ansatz into the partial differential equation (2) and then solve the arising linear system of dimension for its nD,qn_{D,q} coefficients.

For an exact computation, one must then assume that the right hand side of the dynamical system (1) consists at most of rational functions over the rational numbers Q\mathbb{Q}, as only then entering a polynomial ansatz yields after clearing denominators a linear system over Q\mathbb{Q}. The usual complexity estimates for nullspace computations assumes that any arithmetical operation can be performed in constant time. While this is true for floating point operations, it is not true for computations over the rationals Q\mathbb{Q}. A naive implementation of Gaussian implementation will even have an exponential bit complexity, as the number of digits will double in each elimination step. With more advanced methods like modular arithmetics, one can again obtain algorithms with a cubic complexity. However, the constants are much worse than in a floating point computation and depend on the size of the results. As a rule of thumb, one can treat on a single processor machine linear systems over Q\mathbb{Q} up to a size of about a few thousand indeterminates within reasonable computation times.

If one replaces the exact nullspace computation of the linear system by a numerical one, one has a complexity of O(nD,q3)O(n_{D,q}^{3}). Assuming that MNMN and nD,qn_{D,q} are of the same order of magnitude (which is roughly the case in all examples that we studied), we find the same complexity as in our approach. A key difference is that in our approach it is not necessary to know the dynamical system (1) explicitly; it suffices, if enough trajectory data are available. If (1) is explicitly given, then our refinement step takes into account only those power products which according to the regression analysis actually appear in the conservation law. Thus generally in our refinement step one has to solve a much smaller linear system and its costs are again neglectable.

In principle, one can also work with an ansatz containing more general functions than polynomials. The “theorist” approach of [6] is an example how this can be realized purely numerically leading to a linear regression problem. For exact computations, their approach is not suitable, as it would lead to a linear system over a function field instead of the rationals Q\mathbb{Q}. For doing exact linear algebra computations over such a field, one must be able to decide whether or not an expression is zero. Richardson’s theorem (see [24 Sect. 5]) asserts that this problem becomes rapidly undecidable, if one adds transcendental functions like exp⁡x\exp x or sin⁡x\sin x.

However, with a suitable setup we can always reduce to linear algebra over the rationals. We first choose a (finite-dimensional) Q\mathbb{Q}-linear function space V1V_1 which represents the search space in which we look for conservation laws. Then we have to construct a second (finite-dimensional) Q\mathbb{Q}-linear function space V2V_2 which must contain for each function Φ∈V1\Phi\in V_1 and for each variable xdx_d the product (∂Φ/∂xd)fd(\partial\Phi/\partial x_d)f_d. Let {g1,…,gR}\{g_1,\ldots,g_R\} be a linear basis of V1V_1 and {h1,…,hS}\{h_1,\ldots,h_S\} one of V2V_2. Making the ansatz Φ=∑r=1Rβrgr\Phi= \sum_{r=1}^{R}\beta_r g_r with yet undetermined coefficients βr∈Q\beta_r \in\mathbb{Q} and entering it into the partial differential equation (2), we obtain a condition of the form ∑s=1Scs(β)hs=0\sum_{s=1}^{S}c_s(\boldsymbol{\beta})h_s = 0 where the coefficients csc_s are linear in the unknowns β\boldsymbol{\beta}. The ansatz defines an exact conservation law, if and only if cs(β)=0c_s(\boldsymbol{\beta}) = 0 for s=1,…,Ss = 1,\ldots,S. This condition defines a linear system over Q\mathbb{Q} for the coefficients β\boldsymbol{\beta}. Compared with the purely polynomial case, the dimensions RR and SS are rapidly growing here for more complicated situations, so that exact computations will often be unfeasible.

Further Applications

The basic idea of our approach is fairly independent of the notion of a conservation law of a dynamical system. It is concerned with predicting functions from knowing points in their level sets (the “grouped data” of Ha and Jeong [3]). But such questions appear also in other situations. We briefly discuss here two applications. The first one is very close to what we have done so far: we consider discrete dynamical systems which can be handled by our approach without any real changes. The second one is quite different: the implicitisation of curves or higher-dimensional surfaces which is important in algebraic geometry, geometric modeling and computer vision.

Discrete Dynamical Systems

Our approach can also be applied to discrete dynamical systems, i. e. systems where the time tt is a discrete variable. We assume t∈Zt \in\mathbb{Z} and consider an autonomous first-order system of the form

xt+1=f(xt).(29)\mathbf{x}_{t+1}=\mathbf{f}(\mathbf{x}_t). \tag*{(29)}

A conservation law of it is a function Φ:RD→R\Phi:\mathbb{R}^{D}\to\mathbb{R} which remains constant along solutions of (29), i. e. which satisfies Φ(xt+1)=Φ(xt)\Phi(\mathbf{x}_{t+1})=\Phi(\mathbf{x}_t) (see e. g. [25]).

Thus we are in exactly the same situation, as in the continuous case: we know points on level sets of Φ\Phi and can apply the approach developed above. One may even say that the discrete case is slightly easier to handle, as evaluation of (29) along a trajectory does not require the use of numerical approximations as in the case of differential equations. On the other hand, it is more difficult to provide a more or less uniform sampling of the state space, as one has no control about the distance between consecutive points.

Remark 8. In the differential case, it was not really necessary to assume that we treat a first-order system. Most numerical methods are geared towards such systems, but e. g. there also exist methods for second-order systems. For our approach this is irrelevant; we only need the trajectory data (this is different for the approach by Liu et al. [5] using the partial differential equation (2) which assumes a first-order system). In the discrete case, the above definition of a conservation law is valid only for first-order systems. For a system of order QQ, conservation laws are functions defined on RQD\mathbb{R}^{QD}, as they depend not only on xt\mathbf{x}_t, but also on the points xt+1,…,xt+Q−1\mathbf{x}_{t+1},\ldots,\mathbf{x}_{t+Q-1}, i. e. on a whole segment of the trajectory [25]. In principle, our approach can be adapted to such a situation, but it is probably easier to rewrite the given dynamical system as a first-order one.

Example 9. We consider the second-order difference equation

xt+2=tt+1xt+1xt+1(30)x_{t+2}=\frac{t}{t+1}x_t+\frac{1}{x_{t+1}} \tag*{(30)}

appearing in [25]. One readily verifies that

Φ(t,xt,xt+1)=xtxt+1−12t(t+1)(31)\Phi(t,x_t,x_{t+1})=x_tx_{t+1}-\frac{1}{2}t(t+1) \tag*{(31)}

is constant along solutions and thus defines a polynomial conservation law. For applying our approach to this equation, we first rewrite it as a first-order system by introducing yt=xt+1y_t=x_{t+1} and then render it autonomous by introducing zt=tz_t=t. This yields the rational system

xt+1=yt,yt+1=xtztzt+1+1yt,zt+1=zt+1(32)x_{t+1}=y_t,\qquad y_{t+1}=\frac{x_tz_t}{z_t+1}+\frac{1}{y_t},\qquad z_{t+1}=z_t+1 \tag*{(32)}

with a polynomial conservation law of degree 3

Φ(xt,yt,zt)=xtytzt−12zt(zt+1).(33)\Phi(x_t,y_t,z_t)=x_ty_tz_t-\frac{1}{2}z_t(z_t+1). \tag*{(33)}

Applying our approach to (32) and rounding the obtained coefficients to four digits, we discover the exact conservation law (33).

Remark 10. As the approach of Arora et al. [7] is based on finding via discrete gradients a particularly well adapted discretisation of the given continuous dynamical system, it cannot be extended to discrete dynamical systems. Similarly, the approach of Kaiser et al. [1] uses the Koopman theory of continuous dynamical systems and cannot be extended. The neural deflation approach of Zhu et al. [9] only applies to Hamiltonian systems and requires a Poisson structure. While there exist discrete analogues of these concepts, it is unclear whether an extension is possible. The approach of Liu et al. [5] is based on learning solutions of the partial differential equation (2). For discrete dynamical systems, one can derive a discrete analog to this partial differential equation where partial derivatives are replaced by shift operators [25]. It is unclear how solutions of this difference equation can be learned, but our refinement procedure could probably be adapted to this partial difference equation. By contrast, the approaches of Ha and Jeong [3] via a noise-variance loss and Wetzel et al. [2] via Siamese neural networks, respectively, can also be straightforwardly extended to the discrete case.

Implicitisation of Curves and Surfaces

In geometry, two different types of representations of objects like curves and surfaces (of dimension 2 or higher) in some affine space RD\mathbb{R}^{D} are typically used. In an explicit representation, a KK-dimensional object is given via a parametrisation xd=φd(t1,…,tK)\mathbf{x}_d=\varphi_d(t_1,\ldots,t_K) with parameters (t1,…,tK)∈P⊂RK(t_1,\ldots,t_K)\in P\subset\mathbb{R}^{K} from some real parameter set. In an implicit representation, the object is described as the zero set of some functions Φj:RD→R\Phi_j:\mathbb{R}^{D}\to\mathbb{R}. Each representation is preferable for certain tasks. For example, an explicit representation allows us to produce easily points on the object, whereas an implicit representation is better for checking whether a given point lies on the object. It is therefore of importance to be able to switch between these two types of representations and implicitisation is the problem of going from a given parametrisation to an implicit representation. An extensive discussion in the context of geometric modeling can be found in [26].

In algebraic geometry, the given parametrisation φd\varphi_d is typically rational and one looks for polynomials Φj\Phi_j, i. e. it is assumed that the represented object is a variety. Then implicitisation can be formulated as an elimination problem and techniques like resultants or Gröbner bases are available for its solution (see [26] and references therein). However, these techniques are expensive and often lead to polynomials of high degree making subsequent computations costly and numerically unstable (which is a problem for applications in CAGD).

Consequently, numerous methods for approximate implicitisation have been developed (see the references in the recent work [27]) and our approach is related to a class of methods called discrete approximate implicitisation. The authors of [28] propose to use an autoencoder for fitting polynomial equations, i. e. a neural network. Like for conservation laws, we consider our approach via kernel methods as much cheaper.

While many of the algebraic methods can – at least in principle – handle arbitrary dimensions K<DK < D, most of the approximate approaches concentrate on the case K=D−1K = D-1 for D=2,3D = 2, 3, i. e. on planar curves and surfaces in 3D, as these are the dominant situations in CAGD and geometric modelling. This restriction means that the implicit representation consists of a single polynomial. For simplicity, we also consider here only this case.

Our discovery of conservation laws of ordinary differential equations from trajectory data corresponds to the implicitisation of whole families of curves: we are looking for an implicit representation valid not only for a single curve, but for many curves simultaneously. This means that we search for functions Φ\boldsymbol{\Phi} such that the kkth curve is described by the equations Φ(x)=c(k)\boldsymbol{\Phi}(\boldsymbol{x}) = \boldsymbol{c}^{(k)} for some constant vector c(k)\boldsymbol{c}^{(k)}. The extension of our approach to families of two or higher dimensional surfaces is straightforward: one only needs a way to generate sufficiently many data points on the surfaces that do not lie on lower-dimensional subsets. While for a single curve or surface always an implicitisation exists (though not necessarily one with polynomials as desired in algebraic geometry), this represents a special property for families.

In the case of a single curve or surface, one needs in addition to data points on the curve or surface one point off the curve or surface; this point should not lie too close. Then we take as labels for all points on the curve or surface 0 and for the one additional point 1.

Example 11. A classical planar algebraic curve is the trifolium shown in Figure 8. A trigonometric parametrisation of it is given by

x=4sin⁡(t)4−3sin⁡(t)2,y=−sin⁡(t)cos⁡(t)(4sin⁡(t)2−3)(34)\begin{aligned} x &= 4\sin(t)^4 - 3\sin(t)^2, \\ y &= -\sin(t)\cos(t)(4\sin(t)^2 - 3) \tag*{(34)} \end{aligned}
Regular trifolium

Figure 8. Regular trifolium

with 0≤t≤π0 \le t \le\pi. As an implicit description of it, one may use the polynomial equation of degree 4

(x2+y2)2=x(x2−3y2).(35)(x^2+y^2)^2 = x(x^2-3y^2). \tag*{(35)}

A point outside of the curve is for example (−1,0)(-1,0). By employing our kernel method, we discover – after rounding the coefficients to five digits – the exact solution x4−x3+2x2y2+3xy2+y4=(x2+y2)2−x(x2−3y2)x^4-x^3+2x^2y^2+3xy^2+y^4=(x^2+y^2)^2-x(x^2-3y^2).

Example 12. Whitney’s umbrella is the algebraic surface shown in Figure 9 – note that the “handle”, i. e. the zz-axis, is part of the surface. It is implicitly defined by the polynomial equation of degree 3

x2z=y2.(36)x^2z = y^2. \tag*{(36)}

A polynomial parametrisation of the two-dimensional part of the surface is given by

x=u,y=uv,z=v2.(37)x = u,\qquad y = uv,\qquad z = v^2. \tag*{(37)}
Whitney’s umbrella

Figure 9. Whitney’s umbrella

Points on the zz-axis are not really necessary for the implicitisation; it suffices to consider only points on the two-dimensional part of the surface. A point obviously off Whitney’s umbrella is (1,1,−1)(1,1,-1). By employing a polynomial kernel of degree 3, we discover after rounding the coefficients to 7 digits the exact implicit description.

Conclusions

In this work, we proposed an approach to discover conservation laws of dynamical systems which is based on a kernel method, namely an “indeterminate” ridge regression, instead of some variants of neural networks. We believe that this is computationally more efficient and in particular requires much less data which is important, if the data comes from experiments and not from numerical simulations. Indeed, in all our examples a few dozens to a few hundred data points were sufficient (600 points was the maximum we used). By contrast, in [2, 7, 9], the authors report that for problems of similar sizes they trained their neural networks with 50, 000 to 200, 000 points; only the 2, 000 data points used in [3] are somewhat close to our values. The neural networks typically consist of about 200 to 2, 000 neurons in the hidden layers and the training requires between 1, 000 and 50, 000 epochs. The computational costs of such trainings are probably several orders of magnitudes larger than the matrix inversion needed in a kernel ridge regression, even if one takes into account that the determination of the labels in our indeterminate regression and the tuning of the regularisation parameter require repeated inversions. The comparatively low number of data points ensures that the costs for the inversions remain moderate. We also emphasize that our approach yields immediately a symbolic representation of a discovered conservation law, whereas methods based on neural networks still need a subsequent symbolic regression step which usually also comes with considerable computational costs.

The underlying idea of our approach is more general than the problem of discovering conservation laws: it is about learning functions from information about their level sets and thus should be adaptable to other problems, too. We presented here only a basic version of our approach and applied it to a number of popular example systems. There are many questions that call for a deeper analysis. An obvious one is the robustness with respect to noisy data which is again very important, if one thinks of using experimental data. One could hope that up to a certain noise level the regularization parameter of the kernel ridge regression may compensate the noise. But only experiments will tell whether this is really the case. Our current approach to discover multiple conservation laws requires to produce new data in each step adapted to the already found conserved quantities. Thus it cannot be applied with experimental data. In principle, it is not difficult to formulate optimization problems that iteratively discover functionally independent conservation laws using always the same data, e. g. following ideas presented in [5] or [9]. However, these problems are no longer simple ridge regressions where one has a closed-form expression for the exact solution, but must be solved numerically. Thus one has to study more closely the properties of these problems and available solution methods.

Finally, we want to mention that our approach should also be able to discover approximate conservation laws. These could be for example quantities that are almost conserved on shorter time scales, but show some evolution on longer time scales or quantities that remain conserved only after an initial transient phase. While in physics many models are constructed in such a way to possess exact symmetries and thus related exact conservation laws, we expect that in the phenomenological models that prevail in biology such approximate conservation laws are much more frequent than exact ones. In our approach, one can study such phenomena by a suitable selection of the used data. Thus one could e. g. consider only trajectory points up to a certain maximal time or conversely only points at times larger than a certain minimal time. By playing with the time threshold, one may even glean information about the corresponding time scales.

Data Availability

A repository with the Python code and the data used for the examples presented in this article is publicly available on Zenodo at the DOI https://doi.org/10.5281/zenodo.11279856.

Acknowledgments

The work of MAM and WMS was performed within the Research Training Group Biological Clocks on Multiple Time Scales (GRK 2749) at Kassel University funded by the German Research Foundation (DFG).

References

  1. [1]E. Kaiser, J. Kutz, and S. Brunton, Discovering conservation laws from data for control, in 2018 IEEE Conference on Decision and Control (IEEE, 2018) pp. 6415–6421.arxiv.org/abs/1811.00961
  2. [2]S. Wetzler, R. Melko, J. Scott, M. Panju, and V. Ganesh, Discovering symmetry invariants and conserved quantities by interpreting Siamese neural networks, Phys. Rev. 2, 033499 (2020).
  3. [3]S. Ha and H. Jeong, Discovering conservation laws from trajectories via machine learning, Preprint arXiv:2102.04008 (2021).arxiv.org/abs/2102.04008
  4. [4]Z. Liu and M. Tegmark, Machine learning conservation laws from trajectories, Phys. Rev. Lett. 126, 180604 (2021).DOI
  5. [5]Z. Liu, V. Madhavan, and M. Tegmark, Machine learning conservation laws from differential equations, Phys. Rev. E 106, 045307 (2022).DOI
  6. [6]Z. Liu, P. Sturm, S. Bharadwaj, S. Silva, and M. Tegmark, Interpretable conservation laws as sparse invariants, Phys. Rev. E 109, L023301 (2024).DOI
  7. [7]S. Arora, A. Bihlo, R. Brecht, and P. Holba, Model-agnostic machine learning of conservation laws from data, Preprint arXiv:2301.07503 (2023).arxiv.org/abs/2301.07503
  8. [8]P. Lu, R. Dangosvki, and Soljačić, Discovering conservation laws using optimal transport and manifold learning, Nature Comm. 14, 4744 (2023).DOI
  9. [9]W. Zhu, H. Zhang, and P. Kevrekidis, Machine learning of independent conservation laws through neural deflation, Phys. Rev. E 108, L022301 (2023).DOI
  10. [10]S. Kung, Kernel Methods and Machine Learning (Cambridge University Press, Cambridge, 2014).DOI
  11. [11]B. Schölkopf and A. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (MIT Press, Boston, 2018).
  12. [12]J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis (Cambridge University Press, New York, 2004).DOI
  13. [13]P. Olver, Applications of Lie Groups to Differential Equations, Graduate Texts in Mathematics 107 (Springer-Verlag, New York, 1986).
  14. [14]G. Bluman and S. Anco, Symmetry and Integration Methods for Differential Equations, Applied Mathematical Sciences 154 (Springer-Verlag, New York, 2002).
  15. [15]X. Zhang, Integrability of Dynamical Systems: Algebra and Analysis, Developments in Mathematics (Springer Nature, Singapore, 2018).DOI
  16. [16]W. Miller Jr, S. Post, and P. Winternitz, Classical and quantum superintegrability with applications, J. Phys. A: Math. Theor. 46, 423001 (2013).DOI
  17. [17]E. Fermi, J. Pasta, and S. Ulam, Studies of non linear problems, Los Alamos Report LA-1940 (1955).
  18. [18]D. Angeli, A tutorial on chemical reaction network dynamics, Eur. J. Control 15, 398 (2009).DOI
  19. [19]M. Feinberg, Foundations of Chemical Reaction Network Theory, Applied Mathematical Sciences 202 (Springer Nature, Cham, 2019).DOI
  20. [20]H. Errami, M. Eiswirth, D. Grigoriev, W. Seiler, T. Sturm, and A. Weber, Detection of Hopf bifurcations in chemical reaction networks using convex coordinates, J. Comput. Phys. 291, 279 (2015).DOI
  21. [21]R. Schimming, Conservation laws for Lotka–Volterra models, Math. Meth. Appl. Sci. 26, 1517 (2003).DOI
  22. [22]T. Dauxois and M. Peyrard, Physics of Solitons (Cambridge University Press, New York, 2006).DOI
  23. [23]S. Englander, N. Kallenbach, A. Heeger, J. A. Krumhansl, and S. Litwint, Nature of the open state in long polynucleotide double helices: Possibility of soliton excitations, Proc. Natl. Acad. Sci. 77, 7222 (1980).
  24. [24]B. Buchberger and R. Loos, Algebraic simplification, in Computer Algebra: Symbolic and Algebraic Computation, Computing Supplementum 4, edited by B. Buchberger, G. Collins, R. Loos, and R. Albrecht (Springer-Verlag, Wien, 1982) pp. 11–43.
  25. [25]P. Hydon, Symmetries and first integrals of ordinary difference equations, Proc. Roy. Soc. A 456, 2835 (2000).DOI
  26. [26]C. Hoffmann, Geometric and Solid Modeling: An Introduction (Morgan Kaufmann, San Mateo, 1989).
  27. [27]M. Guo, Y. Gao, and Z. Pan, Adaptive approximate implicitization of planar parametric curves via weak gradient constraints, Preprint arXiv:2302.11767 (2023).arxiv.org/abs/2302.11767
  28. [28]G. Wang, W. Li, L. Zhang, L. Sun, P. Chen, L. Yu, and X. Ning, Encoder-X: Solving unknown coefficients automatically in polynomial fitting by using an autoencoder, IEEE Trans. Neural Netw. Learn. Sys. 33, 3264 (2022).DOI

Paper details

vlnc:W_fjfydz5p
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents