betaFeedback

WorksComputer Science

Func-JEPA: Learning a Sobolev Space of Functions as Latent Vectors

Abstract

Pretraining an encoder once and reusing it across tasks is a widespread approach in vision and language. However, neural symbolic mathematics still trains a dedicated sequence-to-sequence model for each task, so that no representation of a function is shared. In this work, we propose the Function Joint-Embedding Predictive Architecture (Func-JEPA, abbreviated f-JEPA): an encoder of mathematical functions, pretrained on their symbolic form, whose latent space imitates the Sobolev space H^1 and in which differentiation is a learned latent operator. The norm of a latent is computed exactly from the function, whereas its direction is learned to match Sobolev angles. The frozen encoder is then reused by lightweight modules for decoding, composition, integration, and regression from samples. The representation and four downstream tasks are evaluated within and outside the pretraining distribution. f-JEPA recovers exact formulas from noisy samples more often than the compared symbolic-regression methods, whereas computer algebra and task-specific models remain stronger on standard integration and differential-equation benchmarks.

Code: https://github.com/amorarc/Func-JEPA

Model: https://huggingface.co/amorarc/func-jepa

Introduction

Nowadays, encoders pretrained once on large corpora and reused for many downstream tasks are the standard approach in language and vision [2, 18]. Neural symbolic mathematics has not followed this path: integration, the solution of ordinary differential equations (ODEs), and symbolic regression are posed as translation problems, each solved by a dedicated sequence-to-sequence transformer trained on its own synthetic corpus [7, 12, 16, 23, 28]. Whilst these models achieve remarkable accuracy, the representation of a function learned for one task is not reused by the others.

A reusable encoder of functions should reflect what a function is in analysis: an element of a vector space with an inner product, on which differentiation, integration, and composition act as operators. Joint-embedding predictive architectures (JEPA) [2, 6, 13, 29] are well-suited to this purpose, since they predict the representation of a target from that of a context: the context is a function ff, the target is its derivative f′f', and the predictor is the derivative operator. However, a predictive loss alone is minimised by collapsed representations, which current methods avoid with architectural asymmetries [2, 20] or with regularisers on the distribution of the embeddings [4, 5]. For functions, a stronger anchor is available: the inner product of two functions can be computed exactly from their values, so the target geometry of the latent space is defined by the data, a property that existing embeddings of mathematical expressions do not exploit [1, 34].

This work demonstrates that a transformer encoder can be pretrained, from symbolic text alone, into a latent space that is close to an isometric embedding of the Sobolev space H1H^1 and in which the derivative is a learned operator. This leads us to propose the Function Joint-Embedding Predictive Architecture (Func-JEPA, abbreviated f-JEPA throughout this paper): a novel pretrained encoder whose latent vectors factorise into an exactly computed H1H^1 norm and a learned unit direction, trained with an angle-only Sobolev geometry, a linearity objective, and a JEPA derivative objective. The derivative predictor is factored into a gain and a direction, which makes it exactly homogeneous. From this, a set of lightweight modules that reuse the frozen encoder is proposed: a grammar-constrained decoder from latents to formulas, a bank of composition operators, an integrator, and an encoder of sampled values.

In order to evaluate f-JEPA, a series of experiments are conducted at three levels: the pretrained representation, compared with released encoders of symbolic expressions; four downstream tasks on the frozen encoder, namely symbolic integration, symbolic regression, and the solution and discovery of ODEs; and the transfer to out-of-distribution data and standard benchmarks. In every task the learned components are compared with a network-free control that differs only in the learned part and with released models and established tools [14, 16, 17, 23, 28], and every reported solution is verified with SymPy [36] or numerically. As a concrete example, the encoder maps the text of f=4sin⁡7x−3cos⁡7xf = 4\sin7x - 3\cos7x to a latent whose norm is exactly ∥f∥H1\lVert f\rVert_{H^1}, the operator DD maps it to a prediction of the latent of f′f', and the decoder turns this prediction into 21sin⁡7x+28cos⁡7x21\sin7x + 28\cos7x, the exact derivative, without any symbolic differentiation (Figure 4).

Figure 4. True functions (black) against formulas decoded from latents (orange): (a, b) ff from zfz_f, (c, d) f′f' from D(zf)\mathcal{D}(z_f), (e) f′′f'' from D(D(zf))\mathcal{D}(\mathcal{D}(z_f)), and (f) the antiderivative from I(zf)\mathcal{I}(z_f). The examples are taken in a fixed random order, and more are shown in Appendix § 11

Related Work

Representations of Mathematical Expressions EqNet [1] and later sequence-to-sequence and contrastive encoders [19, 52] learn embeddings in which equivalent expressions are close, using pairs of equivalent rewritings as supervision. SNIP [34] aligns a symbolic and a numeric encoder with a contrastive objective, and Valentino et al. [48] learn mathematical operations, including differentiation, as transformations between expression embeddings evaluated by retrieval. In f-JEPA, the target of the geometry is instead a Sobolev inner product computed from function values, so that norms, angles, and distances have an analytical meaning.

Joint-Embedding Predictive Architectures Methods in this family pretrain encoders by predicting representations rather than inputs [2, 6, 29], and VL-JEPA [13] employs a lightweight text decoder only at inference, which is the template followed by our decoder. Collapse is avoided by stop-gradient and teacher-student asymmetries [20], by variance and covariance penalties [5], or by matching an isotropic Gaussian distribution [4], and BRo-JEPA [22] imposes the circular structure of modular arithmetic on the predictor. In f-JEPA, the context-target pair is a function and its derivative, and collapse is prevented mainly by a data-defined Gram target, with only a weak variance floor.

Transformers for Symbolic Mathematics Lample and Charton [28] showed that sequence-to-sequence transformers trained on synthetic prefix expressions can integrate functions and solve ODEs, and later work studied linear algebra [12]. Such models are brittle outside their training distribution [51], which has motivated search-based alternatives in which a transformer selects rules applied by a symbolic engine [46]. We adopt their prefix notation, but every task shares a single pretrained encoder and is solved by operators and searches in its latent space, followed by exact verification.

Neural Symbolic Regression Transformer-based methods encode a set of points and decode a prefix expression [7, 23, 40, 49], usually followed by constant refinement. Search-based methods include genetic programming [14], recursive decomposition [45], reinforcement learning [37, 38], and large language models proposing equation skeletons [42], many of which are compared in SRBench [10]. In our case, the value encoder is trained to reach the latent of the frozen symbolic encoder, so that symbolic regression shares its decoder with the other tasks.

Operator Learning and Equation Discovery Neural operators learn maps between function spaces from sampled values [27, 31, 32], function encoders represent functions as coefficients on a learned basis [21], and Koopman-based methods search for coordinates in which a nonlinear operator acts linearly [26, 33, 41]. Equation discovery is commonly addressed by sparse regression on libraries of candidate terms [8, 35], and more recently with transformers that output symbolic ODEs [11, 16]. f-JEPA differs in that its input is symbolic and its outputs can be decoded back to exact formulas, and, unlike Sobolev training [15], it employs the Sobolev inner product as the geometry of the latent space rather than as a regression target.

Proposed Method

Function-space geometry

f-JEPA is built in two stages: an encoder of functions is pretrained once from symbolic text (Sections 3.1 to 3.4), and is then frozen and reused by lightweight modules (Section 3.5), as summarised in Figure 1. The objective of the pretraining is that the latent space imitates a Sobolev space of functions, so that the operations of analysis become operations on vectors. For this reason, every structural property of the function space is enforced on the latent space, either by construction or by a training objective:

Modules of f-JEPA and the data flow between them

Figure 1. Modules of f-JEPA and the data flow between them. Solid borders mark the modules pretrained jointly and dashed borders mark the modules trained afterwards on the frozen encoder

  • Norm: ∥zf∥=∥f∥H1\lVert z_f\rVert= \lVert f\rVert_{H^1}, computed exactly by quadrature (Section 3.2).

  • Inner product: zfTzg≈⟨f,g⟩H1z_f^{\mathsf{T}}z_g \approx\langle f,g\rangle_{H^1}, since latent angles are fitted to Sobolev angles (geometry objective).

  • Linearity: zaf+bg≈azf+bzgz_{af+bg} \approx a z_f + b z_g, including zcf≈czfz_{cf} \approx c z_f (linearity objective).

  • Differentiation: D(zf)≈zf′D(z_f) \approx z_{f'}, with an operator DD that is exactly homogeneous by construction (Section 3.3) and learned with the JEPA objective.

For univariate real functions on an interval [a,b][a,b], we consider the inner product and its induced norm

⟨f,g⟩[a,b]=∫abf(x)g(x) dx,∥f∥[a,b]=⟨f,f⟩[a,b].(1)\langle f,g\rangle_{[a,b]} = \int_a^b f(x)g(x)\,\mathrm{d}x,\qquad\lVert f\rVert_{[a,b]} = \sqrt{\langle f,f\rangle_{[a,b]}}. \tag*{(1)}

The encoder employs the normalised version of Equation (1) on Ω=[a,b]\Omega=[a,b] and its Sobolev extension:

⟨f,g⟩Ω=1b−a⟨f,g⟩[a,b],⟨f,g⟩H1(Ω)=⟨f,g⟩Ω+λ2⟨f′,g′⟩Ω,(2)\langle f,g\rangle_{\Omega} = \frac{1}{b-a}\langle f,g\rangle_{[a,b]},\qquad\langle f,g\rangle_{H^1(\Omega)} = \langle f,g\rangle_{\Omega}+\lambda^2\langle f',g'\rangle_{\Omega}, \tag*{(2)}

with induced norm ∥f∥H1(Ω)=⟨f,f⟩H1(Ω)1/2\lVert f\rVert_{H^1(\Omega)}=\langle f,f\rangle_{H^1(\Omega)}^{1/2}, where λ=0.5\lambda=0.5 and the integrals are evaluated with the N=64N=64-node Gauss–Legendre rule, using the values of ff and of its symbolic derivative f′f' at the nodes. The H1H^1 geometry is preferred over L2L^2 because differentiation is unbounded on L2L^2, whereas ∥f′∥Ω≤λ−1∥f∥H1(Ω)\lVert f'\rVert_{\Omega}\leq\lambda^{-1}\lVert f\rVert_{H^1(\Omega)}, which keeps the ratio between the norms of a function and of its derivative well conditioned.

A function encoder on Ω\Omega is a map EΩE_{\Omega} from the symbolic form of ff to a latent vector zf∈Rdz_f\in\mathbb{R}^d whose Euclidean inner product reproduces Equation (2), zfTzg≈⟨f,g⟩H1(Ω)z_f^{\mathsf{T}}z_g\approx\langle f,g\rangle_{H^1(\Omega)}. Such a representation exists: the coefficients of ff on an H1H^1-orthonormal polynomial basis define a linear map whose inner products reproduce Equation (2) up to an explicit quadrature and truncation error (Proposition 2 in Appendix B.2). The encoder therefore has to learn an approximately isometric and linear embedding from symbols alone. Two encoders are trained with the same architecture, objective, and schedule: E[−π,π]E_{[-\pi,\pi]} on [−π,π][-\pi,\pi], with which all results of Section 4 are obtained, and E[0,5]E_{[0,5]} on [0,5][0,5], evaluated in Appendix C.1.

The pretraining corpus of E[−π,π]E_{[-\pi,\pi]} consists of 2.0M synthetic functions drawn from 17 parameterised families, such as polynomials, Fourier sums, Gaussians, rational functions, and solutions of linear ODEs with integer coefficients, complemented with short random expression trees. Every row stores the prefix tokens of ff and of its derivative f′f' together with their values at the quadrature nodes, and rewritings of the same function and coefficient sweeps are never split across partitions. Further details can be found in Appendix § 7.§ 1.

Encoder with computed magnitude

The encoder EΩE_{\Omega} is a BERT-style transformer [18] with 10 layers, hidden size 512 and about 33M parameters, which reads the prefix tokens of ff. Its [CLS] state is projected to vf∈Rd\mathbf{v}_f \in\mathbb{R}^{d} with d=256d = 256, and the latent vector is defined as

zf=n(f)uf,uf=vf∥vf∥,n(f)=∥f∥H1(Ω),(3)\mathbf{z}_f = n(f)\mathbf{u}_f,\qquad\mathbf{u}_f = \frac{\mathbf{v}_f}{\lVert\mathbf{v}_f\rVert},\qquad n(f) = \lVert f\rVert_{H^1(\Omega)}, \tag*{(3)}

where n(f)n(f) is computed from the values of ff and f′f' at the quadrature nodes rather than learned. With Equation (3), zfTzg=n(f)n(g)ufTug\mathbf{z}_f^{\mathsf T}\mathbf{z}_g = n(f)n(g)\mathbf{u}_f^{\mathsf T}\mathbf{u}_g, so the norms are exact by construction and the only learned quantity is the angle between two latents. Proposition 3 in Appendix § 8.§ 2 makes this precise: the error ∣zfTzg−⟨f,g⟩H1(Ω)∣\lvert\mathbf{z}_f^{\mathsf T}\mathbf{z}_g-\langle f,g\rangle_{H^1(\Omega)}\rvert is at most n(f)n(g)n(f)n(g) times the angular error of the encoder, plus a quadrature error that decays geometrically with NN for analytic functions. Moreover, the H1H^1 norms of the training functions span more than four orders of magnitude, and with a computed norm homogeneity reduces to the invariance of the direction, ucf=sign⁡(c)uf\mathbf{u}_{cf}=\operatorname{sign}(c)\mathbf{u}_f.

Closeness in latent space. Since latent distances approximate Sobolev distances (Corollary 1 in Appendix § 8.§ 2), a prediction z^\hat{\mathbf{z}} is compared with its target z\mathbf{z} through the relative error δ(z^,z)=∥z^−z∥/∥z∥\delta(\hat{\mathbf{z}},\mathbf{z})=\lVert\hat{\mathbf{z}}-\mathbf{z}\rVert/\lVert\mathbf{z}\rVert, which, unlike the cosine, penalises a wrong norm. A prediction is considered correct under the ε\varepsilon-criterion when δ(z^,z)≤ε\delta(\hat{\mathbf{z}},\mathbf{z})\leq\varepsilon. The threshold is calibrated on the validation split: the relative distance from a function to its nearest distinct function has a median of 0.128 and a 5th percentile of 0.0102, and we set ε=0.0051\varepsilon=0.0051, half of this percentile, so that the ε\varepsilon-balls of two functions do not overlap for 95% of the functions (Appendix § 9.§ 2).

Factored derivative operator

The derivative operator D:Rd→Rd\mathcal{D}:\mathbb{R}^{d}\to\mathbb{R}^{d} predicts zf′\mathbf{z}_{f'} from zf\mathbf{z}_f. Writing u^=z/∥z∥\hat{\mathbf{u}}=\mathbf{z}/\lVert\mathbf{z}\rVert, it is factored into a gain and a direction:

D(z)=∥z∥γ(u^)δ(u^),γ(u^)=ek(u^)+k(−u^),δ(u^)=Au^+h(u^)−h(−u^)∥Au^+h(u^)−h(−u^)∥,(4)\mathcal{D}(\mathbf{z})=\lVert\mathbf{z}\rVert\gamma(\hat{\mathbf{u}})\delta(\hat{\mathbf{u}}),\qquad\gamma(\hat{\mathbf{u}})=e^{k(\hat{\mathbf{u}})+k(-\hat{\mathbf{u}})},\qquad\delta(\hat{\mathbf{u}})=\frac{\mathbf{A}\hat{\mathbf{u}}+h(\hat{\mathbf{u}})-h(-\hat{\mathbf{u}})}{\lVert\mathbf{A}\hat{\mathbf{u}}+h(\hat{\mathbf{u}})-h(-\hat{\mathbf{u}})\rVert}, \tag*{(4)}

where A∈Rd×d\mathbf{A}\in\mathbb{R}^{d\times d} is initialised to the identity, and hh and kk are multilayer perceptrons. The gain γ\gamma is even and positive and learns the ratio n(f′)/n(f)n(f')/n(f), whereas the direction δ\delta is odd and of unit norm.

Proposition 1. For every z≠0\mathbf{z}\ne0 and every c∈R∖{0}c\in\mathbb{R}\setminus\{0\}, the operator in Equation (4) satisfies D(cz)=cD(z)\mathcal{D}(c\mathbf{z})=c\mathcal{D}(\mathbf{z}).

The proof can be found in Appendix § 8.§ 1. Thus, homogeneity is exact by construction, and powers D2\mathcal{D}^{2} and polynomials p(D)p(\mathcal{D}) are available by composition without being trained. It is worth noting that D\mathcal{D} is not additive by construction: additivity is not enforced on the operator and is only learned indirectly, through the linearity of the latent.

Training objective

Figure 2 summarises how the objective is built. Let ui\mathbf{u}_i be the directions of the BB functions of a batch, U∈RB×d\mathbf{U}\in\mathbb{R}^{B\times d} the matrix that stacks them, and Cij=⟨fi,fj⟩H1(Ω)/(n(fi)n(fj))C_{ij}=\langle f_i,f_j\rangle_{H^1(\Omega)}/(n(f_i)n(f_j)) the target cosines. The geometry objective matches latent angles to Sobolev angles and adds a centring term whose maximum is reached by any rank-one latent:

Construction of the training objective

Figure 2. Construction of the training objective. The tokens of ff, of its derivative f′f', and of synthetic combinations hh are encoded by the same encoder, and latent norms and target cosines are computed exactly from the function values

Lgeo=1B2∑i,j(uiTuj−Cij)2+βc(1−ρ2(HUUTH,HCH)),(5)\mathcal{L}_{\mathrm{geo}}=\frac{1}{B^2}\sum_{i,j}\left(\mathbf{u}_i^{\mathsf T}\mathbf{u}_j-C_{ij}\right)^2+\beta_c\left(1-\rho^2\left(\mathbf{H}\mathbf{U}\mathbf{U}^{\mathsf T}\mathbf{H},\mathbf{H}\mathbf{C}\mathbf{H}\right)\right), \tag*{(5)}

where H\mathbf{H} is the centring matrix, ρ\rho is the Frobenius cosine and βc=1.5\beta_c = 1.5. The first term is the mean squared angular error that bounds the inner-product error in Proposition 3. Since a freshly initialised transformer maps all inputs to almost parallel directions, a variance floor Lvar=1d∑jmax⁡(0,τ−std⁡(d uj))\mathcal{L}_{\mathrm{var}} = \frac{1}{d}\sum_j \max(0,\tau-\operatorname{std}(\sqrt{d}\,u_j)) with τ=0.5\tau= 0.5 is added, in the spirit of VICReg [5].

The linearity objective is applied to synthetic combinations h=af+bgh = af + bg and to scaled copies h=cfh = cf, built directly in prefix notation with exact values:

Llin=∥zh−(azf+bzg)∥212(∥zh∥2+∥azf+bzg∥2)(6)\mathcal{L}_{\mathrm{lin}} = \frac{\lVert z_h-(az_f+bz_g)\rVert^2}{\frac{1}{2}\left(\lVert z_h\rVert^2+\lVert az_f+bz_g\rVert^2\right)} \tag*{(6)}

which is a relative squared error.

The JEPA objective predicts the latent of the derivative, and is split into a direction term and a log-gain term:

LJEPA=(1−cos⁡(D(zf),zˉf′))+βg(log⁡(∥D(zf)∥+ϵ∥zf∥)−log⁡(∥zˉf′∥+ϵ∥zf∥))2(7)\mathcal{L}_{\mathrm{JEPA}} = \left(1-\cos\left(\mathcal{D}(z_f),\bar{z}_{f'}\right)\right)+\beta_g\left(\log\left(\lVert\mathcal{D}(z_f)\rVert+\epsilon\lVert z_f\rVert\right)-\log\left(\lVert\bar{z}_{f'}\rVert+\epsilon\lVert z_f\rVert\right)\right)^2 \tag*{(7)}

where the target zˉf′\bar{z}_{f'} is the latent of f′f' with a stop-gradient, βg=1\beta_g = 1, and ϵ=0.05\epsilon= 0.05 handles constant functions, for which zf′=0z_{f'} = 0. The split is motivated by the following observation: for a prediction of norm ss at angle θ\theta from the target tt, the squared error is minimised at s=∥t∥cos⁡θs = \lVert t\rVert\cos\theta, so a mean squared error shrinks the predicted norm whenever the direction is uncertain. With Equation (7), the gain is trained independently of the direction.

The full objective is L=Lgeo+Lvar+r(t)(Llin+LJEPA)+0.05Lv\mathcal{L} = \mathcal{L}_{\mathrm{geo}}+\mathcal{L}_{\mathrm{var}}+r(t)\left(\mathcal{L}_{\mathrm{lin}}+\mathcal{L}_{\mathrm{JEPA}}\right)+0.05\mathcal{L}_v, where Lv=(log⁡(∥v∥/d))2\mathcal{L}_v = \left(\log(\lVert v\rVert/\sqrt{d})\right)^2 only keeps the raw projection well scaled for bfloat16 precision. The ramp r(t)r(t) is 0 during the first 1000 steps, so that the geometry forms before the predictive terms act on the latent, and grows linearly to 1 over the next 1000 steps. Only ff and f′f' are used during pretraining: second derivatives, compositions, and ODEs are never seen, and serve as zero-shot tests.

Modules on the frozen encoder

Once pretrained, the encoder and D\mathcal{D} are frozen, and every downstream use of f-JEPA is built from lightweight modules trained on top of them. Every module outputs a latent as a unit direction times a magnitude, with the magnitude computed whenever possible and otherwise predicted as a log-gain. Since the learned operators are accurate only up to a few percent (§ 4.1), the modules are employed to propose candidates in latent space, which are then combined, refitted, and verified with exact numerical or symbolic steps. This principle is shared by all the applications of § 4, whose procedures are detailed in Appendix § 7.7.

Decoder A 6-layer transformer decoder (17.7M parameters) maps (u,log⁡n)(\mathbf{u}, \log n) to prefix tokens under a grammar mask, so that every output parses. Beam candidates are re-ranked by re-encoding them (round trip), and constants can be refined in latent space.

Composition operators For each outer function hh, a network Mh(zg)≈zh∘g\mathcal{M}_h(\mathbf{z}_g) \approx\mathbf{z}_{h\circ g} predicts a direction and a log-gain. Seven operators are trained (sin⁡g\sin g, cos⁡g\cos g, log⁡g\log g, g\sqrt{g}, g2g^2, g3g^3, and ag2+bg+cag^2 + bg + c), and the scaling laws of their targets, such as (cg)2=c2g2(cg)^2 = c^2g^2, are satisfied by design for positive scalars (Proposition 4 in Appendix § 8.§ 3).

Integrator I(z)=∥z∥ekI(u^)δI(u^)\mathcal{I}(\mathbf{z}) = \lVert\mathbf{z}\rVert e^{k_{\mathcal{I}}(\hat{\mathbf{u}})} \delta_{\mathcal{I}}(\hat{\mathbf{u}}) predicts the latent of the zero-mean antiderivative, and is trained with an additional cycle loss through the frozen D\mathcal{D}.

Value encoder V\mathcal{V} maps 64 samples of ff to a unit direction, while the magnitude is computed from a spectral derivative of the samples, so that formulas can be recovered from data with the same decoder. Further details can be found in Appendix § 7.§ 3.

Experiments

The experiments follow the two stages of f-JEPA: the pretrained representation is evaluated first (§ 4.1), then the modules and the four downstream tasks built on the frozen encoder (§ 4.2 to § 4.6), and finally the transfer beyond the pretraining distribution (§ 4.7). The encoder E[−π,π]E_{[-\pi,\pi]} was pretrained once for 40k steps with batches of 384 functions on a single 8 GB laptop GPU (6.6 hours), and it is never fine-tuned afterwards. Unless stated otherwise, results are reported on a held-out test set of 12,146 functions that is disjoint from the training data of every component, and retrieval is measured on a fixed subset of 4,000 of them. Only 9.2% of the test functions lie within a relative H1H^1 distance of 10−310^{-3} of a training function, and excluding them leaves the main results unchanged (derivative retrieval 0.835 instead of 0.820). All models were trained once with a single seed. Further details can be found in Appendix § 7, and time, memory, and model sizes are compared in Appendix § 10.

Latent geometry and differentiation

We first evaluate the pretrained representation itself, before any module is trained on it. Table 1 is organised in five blocks: whether latent angles and inner products are those of H1H^1 (geometry), whether the latent of a combination is the combination of the latents (linearity), whether D\mathcal{D} maps the latent of a function to that of its derivative (derivative), whether relations that were never trained hold (zero-shot), and whether the latent space organises functions in an interpretable way (structure).

Metricf-JEPA
GeometryCorrelation of cos⁡(zf,zg)\cos(z_f,z_g) with cos⁡H1(f,g)\cos_{H^1}(f,g) — 0.993
Correlation of zfTzgz_f^\mathsf{T}z_g with ⟨f,g⟩H1\langle f,g\rangle_{H^1} — 0.998
Mean absolute cosine error — 0.017
LinearityScale error zcfz_{cf} vs. czfcz_f (c = 2 / 10 / −1-1) — 0.012 / 0.014 / 0.050
Combinations zaf+bgz_{af+bg} vs. azf+bzgaz_f+bz_g — 0.071
Random sums zf+gz_{f+g} vs. zf+zgz_f+z_g — 0.207
DerivativeRetrieval of f′f', top-1 / top-5 (3,956 candidates) — 0.820 / 0.960
Relative error / cosine of D(zf)\mathcal{D}(z_f) vs. zf′z_{f'} — 0.070 / 0.998
Gain ratio ∥D(zf)∥/∥zf′∥\lVert\mathcal{D}(z_f)\rVert/\lVert z_{f'}\rVert — 1.004
Best linear map, top-1 / top-5 — 0.460 / 0.680
Zero-shotD(D(zf))\mathcal{D}(\mathcal{D}(z_f)) vs. zf′′z_{f''}, error / top-1 (3,640 candidates) — 0.267 / 0.458
D(zf′)\mathcal{D}(z_{f'}) vs. zf′′z_{f''}, error / top-1 — 0.232 / 0.501
Eigenfunctions Dz=az (eax)\mathcal{D}z=az\ (e^{ax}) / D2z=−k2z (sin⁡kx)\mathcal{D}^2z=-k^2z\ (\sin kx) — 0.122 / 0.174
ODE annihilation ∥p(D)z∥\lVert p(\mathcal{D})z\rVert relative to a shuffled ODE (chance 1) — 0.209
StructureCoefficient-sweep ordering (Spearman; chance 0.57) — 0.828
Distance between equivalent rewritings (relative; random pairs 1) — 0.071
Linear probe: family accuracy / values R2\mathbb{R}^2 — 0.742 / 0.998

Table 1. Latent space properties of the encoder E[−π,π]E_{[-\pi,\pi]} on held-out functions. Errors are median relative errors, and every row is defined in Appendix § 7.§ 6.

As observed in Table 1, the latent space is close to an isometric embedding of H1H^1: latent cosines and dot products match their Sobolev counterparts with correlations of 0.993 and 0.998, and the norm is exact by construction. The latent is homogeneous to 1.2% for positive scalars and to 5.0% for negative ones, and linear combinations of the training type are reproduced to 7.1%, whereas sums of two unrelated functions, which are longer and further from the training distribution, are reproduced only to 20.7%. The encoder E[0,5]E_{[0,5]}, trained from scratch on [0,5][0,5], reaches a comparable geometry, linearity, and derivative retrieval on the test split of its own interval, although its zero-shot operator algebra is weaker (Appendix § 9.§ 1).

The learned derivative retrieves the true derivative among 3,956 candidates for 82.0% of the functions (96.0% within the top five), and its gain is calibrated, with a median ratio of 1.004 between predicted and true norms. Notably, the best linear map fitted to D\mathcal{D} retrieves only 46.0%, so differentiation is not a linear map of this latent space. Relations that were never trained hold approximately: applying D\mathcal{D} twice retrieves the true f′′f'' for 45.8% of the functions, the eigenfunction relations of eaxe^{ax} and sin⁡kx\sin kx hold to 12% and 17%, and the residual p(D)zp(\mathcal{D})z of solutions of linear ODEs is five times smaller than for a shuffled ODE. Without supervision, the latent also orders the members of coefficient sweeps (Spearman 0.83 against 0.57 by chance) and places equivalent rewritings of a function at 7% of the distance between random pairs.

However, D\mathcal{D} is accurate only up to a noise floor. Its median relative error is 0.070 and its smallest error on the validation split is 0.013, above the threshold ε=0.0051\varepsilon= 0.0051 of § 3.2, so no prediction of any learned operator satisfies the ε\varepsilon-criterion (Table 14 in Appendix § 9.§ 2). Retrieval succeeds nevertheless because the target only has to be the nearest candidate, and the median gap between distinct functions is 0.128. Therefore, the latent operators are accurate enough to propose candidates but not to identify a function by themselves, which is why every application below verifies their proposals with an exact step.

PredictionTop-1Rel. error< 0.05< 0.1
D(zf)D(z_f) vs. zf′z_{f'}0.8200.0700.3170.671
D(D(zf))D(D(z_f)) vs. zf′′z_{f''}0.4580.2670.0750.264
D(zf′)D(z_{f'}) vs. zf′′z_{f''}0.5010.2320.1780.339
Integrator I(zf)I(z_f)0.6700.1760.0020.172
Msin⁡\mathcal{M}_{\sin}0.6900.2460.0000.042
Mcos⁡\mathcal{M}_{\cos}0.6470.2750.0000.008
Mlog⁡\mathcal{M}_{\log}0.5840.1100.1460.446
M⋅\mathcal{M}_{\sqrt{\cdot}}0.5680.1790.0070.107
Mg2\mathcal{M}_{g^2}0.9220.1680.0010.135
Mg3\mathcal{M}_{g^3}0.9190.1490.0060.213
Mag2+bg+c\mathcal{M}_{ag^2+bg+c}0.7960.2750.0000.025

Table 14. Closeness of the latents predicted by the learned operators to their targets: top-1 retrieval accuracy, median relative error ∥z^−z∥/∥z∥\lVert\hat{z} - z\rVert/\lVert z\rVert, and shares with a relative error below 0.05 and 0.1. No prediction reaches the calibrated ε=0.0051\varepsilon= 0.0051

Table 2 compares the latent space of f-JEPA with the representations of two released encoders of symbolic expressions: SNIP [34] and the encoder of the integrator of Lample and Charton [28]. We report the property probes of SNIP [34] with the same protocol for every representation, standard measures of how an embedding preserves the geometry of the functions, including trustworthiness [50], and a derivative probe (Appendix § 7.§ 4). In order not to favour f-JEPA, whose training target is the H1H^1 cosine, the first geometry measure employs the plain cosine of the function values on a uniform grid.

RepresentationDim.Value cos.Dist. rankkNN@10Trust.NCRUpward.Avg. yyOsc.Top-5Align.
SNIP, symbolic encoder [34]5120.002-0.0900.0800.5700.6800.7160.8050.0840.3380.936
Lample and Charton encoder [28]10240.0120.1330.0870.6010.3190.2170.2100.0440.9790.908
f-JEPA2560.9461.0000.9000.9940.2350.3890.5830.3900.9290.997

Table 2. Comparison of function representations computed from the symbolic text on the same 4,000 test functions. The property probes follow SNIP [34] (normalised error, chance 1), and the derivative probe is fitted on the same 20k pairs for every representation.

Only f-JEPA reflects the geometry of the functions: its cosines follow those of the function values (correlation 0.946, against 0.012 and 0.002), and 90% of its 10 nearest neighbours are also H1H^1 neighbours, against fewer than 9% for the other two encoders. However, the encoder of Lample and Charton, with 2.4 times as many parameters and a latent of dimension 1,024, predicts three of the four SNIP properties more accurately and supports a more accurate derivative retrieval (top-5 0.979 against 0.929), whereas f-JEPA predicts the non-convexity ratio best and has the best aligned derivative predictions. A likely reason is the geometry itself, which places functions of the same family close to each other: 78% of the retrieval errors of D\mathcal{D} return a different member of the correct family. Thus, the representation of f-JEPA is not the most informative one for every property, but it is the only one in which norms, angles, and linear combinations have an analytical meaning.

Figure 3 shows the global organisation of the latent space, in which the unit directions are projected by a small autoencoder and the exact norm is kept as the radius. All multiples of a function lie on one ray, oscillatory functions occupy different angular sectors from exponential, power, and rational functions, and the test set covers the same sectors as the training set. It is worth noting that the projection is lossy, with a median angular error of 33∘33^\circ (Appendix § 9.8).

Figure 3. Latent space of (a) 120k training functions and (b) the 20.4k functions of the test split of the training corpus. The angle is the code of the unit direction ufu_f given by an autoencoder with a circular bottleneck, the radius is the rank of log⁡∥f∥H1\log\lVert f\rVert_{H_1}, and the colour encodes the elementary functions of each expression

Decoding and composition

From here on the encoder and D\mathcal{D} are frozen. The decoder recovers the exact function from zfz_f for 52.6% of the test functions with beam search and round-trip re-ranking, and for 67.0% after latent constant refinement, while the shape of the function (constants refitted) is correct for 84.0%. Decoding the predicted latent D(zf)\mathcal{D}(z_f) yields a derivative obtained entirely through the latent space: its shape is correct for 74.8% of the functions, but its constants are exact only for 6.8%, since D(zf)\mathcal{D}(z_f) inherits the noise floor of D\mathcal{D}. Accuracy also decreases with expression length, from 77% for 10 to 19 tokens to 10% for 30 to 39 tokens (Appendix § 9.§ 4). Figure 4 illustrates both regimes: when decoding fails, the formula usually keeps the structure of the true one with a wrong constant, such as 189 instead of 171 in the predicted derivative of a logistic function.

Table 3 shows that composition is learnable on the frozen encoder, but not with a linear map: ridge regression retrieves at most 19.6% of the targets, whereas Mh\mathcal{M}_h attains between 56.8% and 92.2% top-1 accuracy. Remarkably, the quadratic ag2+bg+cag^2+bg+c is retrieved with 44.4% accuracy without any training on it, by combining the learned square operator with the linearity of the latent. The predicted magnitudes also contribute to retrieval, as shown by the lower accuracy of the direction alone.

Methodsin⁡g\sin gcos⁡g\cos glog⁡g\log gg\sqrt{g}g2g^2g3g^3ag2+bg+cag^2+bg+c
Identity zgz_g0.0990.0340.0170.0230.0160.0380.005
Ridge regression0.0320.0210.0410.0190.0960.1960.118
Linear part of Mh\mathcal{M}_h only0.0950.0000.0320.2010.1490.6820.000
Zero-shot aMsq(zg)+bzg+cz1a\mathcal{M}_{\mathrm{sq}}(z_g)+bz_g+cz_1––––––0.444
Mh\mathcal{M}_h, direction only0.5780.5080.4810.2530.7030.6750.660
Mh\mathcal{M}_h0.6900.6470.5840.5680.9220.9190.796
Mh\mathcal{M}_h, top-50.9220.8940.8900.9560.9900.9830.885

Table 3. Accuracy comparison (top-1) of composition operators on the frozen encoder, retrieving zh∘gz_{h\circ g} among about 3k unseen inner functions gg. The last row gives the top-5 accuracy of Mh\mathcal{M}_h

A natural question is whether the learned operators add anything over decoding the input and applying computer algebra. Table 16 in Appendix § 9.§ 4 compares each learned operator with a symbolic path that employs the same frozen decoder and encoder but replaces the learned operator by the exact one, with both routes scored by retrieval among the same candidates. The symbolic path has the higher top-1 accuracy on every operator, since it is almost always correct when the decoder reconstructs the input exactly. However, on the 24% to 45% of the inputs that the decoder does not reconstruct, the learned operators are more accurate in top-5 for the derivative and for 6 of the 7 composition operators, the exception being the quadratic. Hence, the learned operators do not replace exact symbolic calculus on formulas that can be recovered, but they complement it, and they are the only option when no formula is available, as in the tasks below.

Table 16: Accuracy comparison (top-1 / top-5 retrieval) of the learned operators with a symbolic path that decodes the input latent, applies the exact operator with SymPy, and re-encodes the result. Input not decoded: inputs that the decoder does not reconstruct exactly

OperatorAll inputs: LearnedAll inputs: Symbolic pathInput not decoded: LearnedInput not decoded: Symbolic path
d/dx (D)d/dx\ (D)0.820 / 0.9600.901 / 0.9630.802 / 0.9530.783 / 0.917
sin⁡g\sin g0.675 / 0.9240.850 / 0.9090.585 / 0.8660.543 / 0.723
cos⁡g\cos g0.640 / 0.8890.849 / 0.9050.528 / 0.7920.502 / 0.686
log⁡g\log g0.591 / 0.8930.773 / 0.8860.509 / 0.8310.498 / 0.747
g\sqrt{g}0.573 / 0.9580.836 / 0.9280.509 / 0.9490.637 / 0.841
g2g^20.923 / 0.9900.932 / 0.9710.854 / 0.9710.752 / 0.894
g3g^30.923 / 0.9880.938 / 0.9810.868 / 0.9880.745 / 0.922
ag2+bg+cag^2+bg+c0.801 / 0.8970.942 / 0.9630.688 / 0.8220.784 / 0.862

Table 16. Accuracy comparison (top-1 / top-5 retrieval) of the learned operators with a symbolic path that decodes the input latent, applies the exact operator with SymPy, and re-encodes the result. Input not decoded: inputs that the decoder does not reconstruct exactly

Symbolic integration

The four downstream tasks reuse the same frozen encoder, decoder, and operators, and each adds at most one lightweight module. The integrator I\mathcal{I} retrieves the antiderivative among 11,196 candidates with a top-1 accuracy of 67.0% (91.9% top-5), against 42.7% for a search that inverts D\mathcal{D} without training. The solver of Algorithm 1 builds on it: latent proposals are decoded, refitted, and combined, and an answer is accepted only when SymPy verifies F′=fF' = f. Symbolic preprocessing is limited to splitting sums, reducing trigonometric powers, and substituting inner functions, whereas every integration step is performed by the learned integrator and the decoder.

As observed in Table 4, the solver verifies an antiderivative for 91.7% of the test integrands and 66.7% of the hard integrals, which require integration by parts or substitution, against 99.3% and 100% for SymPy. The pretrained integrator of Lample and Charton [28] solves 68.7% of our test integrands, whose coefficients often exceed the small integers of its training expressions, but 90.0% of the hard integrals. The unverified answers remain close approximations: the returned antiderivative reproduces the definite integral on [−π,π][-\pi,\pi] within 1% for 99.3% of the test integrands and for all the hard ones. For integrands without an elementary antiderivative, such as e−x2e^{-x^2} or esin⁡xe^{\sin x}, the derivative of a five-term approximation built from the same basis is within 1% of the integrand for 7 of 12 functions (Table 24 in Appendix § 9).

MethodSolve rate TestSolve rate HardDefinite integral TestDefinite integral Hard
Latent shortlists, decoding (beam 16) and constant refitting0.7730.1670.9570.900
+ exact combination of decoded pieces and pool functions0.7870.2000.9770.900
+ splitting sums into terms0.8130.2000.9770.900
+ residual peeling in latent space0.8170.2000.9830.900
+ latent tabular integration by parts0.8270.4330.9830.967
+ substitution and trigonometric reduction0.9170.6670.9931.000
Lample and Charton, FWD+BWD+IBP, beam 10 [28]0.6870.900––
SymPy integrate (reference, not a learned method)0.9931.000––

Table 4. Symbolic integration on 300 test integrands and 30 hard elementary integrals, with cumulative levels of our solver. Solve rate: antiderivative verified exactly by SymPy; definite integral: the returned antiderivative, verified or not, reproduces ∫−ππf dx\int_{-\pi}^{\pi} f\,dx within 1%.

Integrandk=1k = 1k=3k = 3k=5k = 5
e−x2e^{-x^{2}}7.5e-21.8e-22.9e-3
sin⁡(x2)\sin\left(x^{2}\right)6.7e-12.9e-11.1e-1
cos⁡(x2)\cos\left(x^{2}\right)3.9e-14.9e-21.9e-2
esin⁡(x)e^{\sin\left(x\right)}1.4e-17.0e-31.2e-3
sin⁡(x)x2+1\frac{\sin\left(x\right)}{x^{2}+1}1.1e-14.3e-21.2e-2
e−x2cos⁡(x)e^{-x^{2}}\cos\left(x\right)1.2e-11.7e-26.4e-3
x4+1\sqrt{x^{4}+1}4.2e-25.8e-31.8e-3
ecos⁡(x)e^{\cos\left(x\right)}9.0e-21.3e-21.6e-3
log⁡(sin⁡(x)+2)\log\left(\sin\left(x\right)+2\right)9.9e-21.7e-25.6e-3
ex25e^{\frac{x^{2}}{5}}2.6e-21.3e-21.1e-2
x2e−x2x^{2}e^{-x^{2}}5.3e-11.1e-14.6e-2
cos⁡(x)x2+1\frac{\cos\left(x\right)}{x^{2}+1}7.4e-28.4e-33.1e-3

Table 24. Integrands without elementary antiderivative: relative error of the derivative F′F' of the kk-term symbolic approximation with respect to the integrand ff.

Symbolic regression

Formulas are recovered from 64 samples with the value encoder, the decoder, constant refitting, and a selection by χ2\chi^2 plus description length (Algorithm 2). On 300 test functions, 80.0% of the formulas are recovered exactly from clean samples, and 63.0% and 44.7% at relative noise levels of 10−310^{-3} and 10−210^{-2}, against 87.0% when the true latent is decoded instead of the output of the value encoder (Table 17 in Appendix § 9.§ 5). Table 5 compares f-JEPA with six external methods that receive the same samples of a fixed subset of 50 test functions (Appendix § 7.§ 5).

MethodExact / within threshold σ=0\sigma= 0Exact / within threshold 10−310^{-3}Exact / within threshold 10−210^{-2}σ=0\sigma= 0 NMSE <10−3< 10^{-3}Extrap.NodesRel. err. 10−210^{-2}Classic (of 24) σ=0\sigma= 0Classic (of 24) 10−210^{-2}
E2E0.00 / 0.020.00 / 0.000.00 / 0.000.126.2e-01201.7e-010 / 00 / 0
NeSymReS0.06 / 0.140.00 / 0.120.00 / 0.080.228.0e-01122.7e-013 / 40 / 2
SNIP0.04 / 0.580.00 / 0.300.00 / 0.100.744.0e-04226.5e-030 / 100 / 2
MMSR0.00 / 0.120.00 / 0.080.00 / 0.020.284.1e-01153.2e-013 / 51 / 3
PySR0.06 / 0.740.00 / 0.580.00 / 0.180.944.0e-07225.2e-039 / 191 / 4
Operon0.06 / 0.960.00 / 0.260.00 / 0.000.962.3e-09237.5e-033 / 200 / 0
f-JEPA0.76 / 0.960.76 / 0.920.62 / 0.740.98<10−20< 10^{-20}122.8e-088 / 108 / 9

Table 5. Symbolic regression from 64 samples on the same 50 test functions for every method, and on 24 classic univariate benchmarks. Entries with a slash give exact recovery / recovery within the function-space threshold, and the columns are defined in Appendix § 7.§ 6.

Methodσ=0\sigma= 0σ=10−3\sigma= 10^{-3}σ=10−2\sigma= 10^{-2}
Nearest function in the training pool0.1100.1100.110
Value encoder + decoding (beam 8) + constant refitting0.6400.4700.210
+ beam 16 and refitting of every candidate0.7300.6300.447
+ exact combination with polynomial atoms0.7630.6300.447
+ residual peeling0.8000.6300.447
True latent zfz_f instead of V\mathcal{V} (upper bound)0.870––

Table 17. Exact symbolic recovery (share of functions) of the cumulative levels of our pipeline from 64 samples on 300 test functions, at relative noise levels σ\sigma. All levels use the χ2\chi^{2} and description-length selection

f-JEPA is the only method that recovers the exact formula for most functions, and the only one that keeps doing so under noise (76%, 76%, and 62%, against at most 6% for every other method). This advantage partly reflects the criterion and the data: the test functions belong to the pretraining families, whose rational constants the decoder emits, whereas the other methods return floating-point constants. When near-exact formulas are also counted, through a threshold of 0.0037 on the relative H1H^1 error, Operon ties f-JEPA on clean samples (0.96 each), and on the 24 classic benchmarks Operon and PySR are within the threshold of 20 and 19 targets against 10 for f-JEPA. Under noise, in contrast, the advantage of f-JEPA remains under both criteria (0.92 and 0.74 against at most 0.58 and 0.18), since the constants of the other methods fit the noise: their median relative error at σ=10−2\sigma= 10^{-2} is between 5.2⋅10−35.2 \cdot10^{-3} and 0.32, against 2.8⋅10−82.8 \cdot10^{-8} for f-JEPA. Moreover, f-JEPA is the fastest method, with 0.44 seconds per function against 1.6 to 93 seconds (Appendix § 10). At 1% noise, its typical failure is a single constant snapped to a wrong fraction, as in Figure 5b.

Figure 5. True functions (black) against recovered formulas (orange): (a, b) symbolic regression at σ=10−2\sigma= 10^{-2}, (c, d) five-term approximations of antiderivatives without elementary closed form, and (e, f) four-atom formulas for initial-value problems without closed form. More examples are shown in Appendix E

Solving ordinary differential equations

For a linear ODE p(ddx)y=0p\left(\frac{d}{dx}\right)y = 0, the latent residual ∥p(D)z∥\lVert p(D)z\rVert ranks a pool of functions, but true solutions have median relative residuals between 0.009 and 0.08, and gradient descent on zz below this level does not improve the ranking (Table 20 in Appendix § 9). Following the principle of § 3.5, the latent residual is employed to shortlist the top 1% of a pool of 2M functions, and exact steps on the stored values (re-ranking, sparse combinations, and refitting of constants) followed by SymPy verification produce the solutions (Algorithm 3). This yields 20 verified solutions among the top 20 for all 9 linear ODEs tested and for 4 nonlinear ones, including solutions absent from the pool (Table 21). Since SymPy solves such textbook ODEs directly, this experiment validates the principle rather than a use case.

ODESolutionsResidualp@20p@20 (GD)ExactNew
4y+y′′=04y+y^{\prime\prime}=0470.0810.850.7080
5y+2y′+y′′=05y+2y^{\prime}+y^{\prime\prime}=050.0690.250.25179
2y−3y′+y′′=02y-3y^{\prime}+y^{\prime\prime}=0390.0090.400.4563
y+y′′=0y+y^{\prime\prime}=0200.0490.800.5570
−y+y′=0-y+y^{\prime}=050.0130.200.2510

Table 20. Linear ODEs on the held-out pool (11,196 functions): number of solutions in the pool, their median latent residual, precision at 20 of the latent ranking and of the re-ranking after gradient descent in latent space, and SymPy-verified solutions found by exact sparse combinations of the 16 best-ranked functions (total, and absent from the pool)

ODESolutionsLatent p@20Recall@1%Verified top 20New
y′=yy^{\prime}=y2440.000.8920/202
y′=−2yy^{\prime}=-2y2950.050.9620/201
y′′=−4yy^{\prime\prime}=-4y7,5221.000.9120/202
y′′=−yy^{\prime\prime}=-y10,5671.000.9920/200
y′′=yy^{\prime\prime}=y7,4180.000.8620/202
y′′=−2y′−5yy^{\prime\prime}=-2y^{\prime}-5y2,8350.900.9620/203
y′′=3y′−2yy^{\prime\prime}=3y^{\prime}-2y2,5740.100.7720/207
y′′=4y′−4yy^{\prime\prime}=4y^{\prime}-4y6290.000.6420/201
y′′′=−y′y^{\prime\prime\prime}=-y^{\prime}19,2781.000.9720/200
y′=y2y^{\prime}=y^280.001.002/22
y′=−y2y^{\prime}=-y^290.100.331/32
y′=y−y2y^{\prime}=y-y^2580.751.0020/204
y′=2y−2y2y^{\prime}=2y-2y^2450.001.0020/200
y′=y+y2y^{\prime}=y+y^2500.901.0020/201
y′=y−y3y^{\prime}=y-y^370.001.001/10
y′=y3y^{\prime}=y^360.000.000/01
y′=yy^{\prime}=\sqrt{y}400.601.0020/204
y′′=2y3y^{\prime\prime}=2y^3110.000.402/51
y′′=6y2y^{\prime\prime}=6y^270.001.001/10

Table 21. Textbook ODEs on the 2M-function pool: number of solutions in the pool, precision at 20 of the latent ranking alone, recall of the latent shortlist (top 1%), verified solutions among the final top 20 (of the maximum possible), and verified solutions absent from the pool

We therefore consider 12 initial-value problems for which SymPy’s dsolve fails, such as the pendulum, Duffing oscillators, and a Painlevé-I type equation. A network-free spectral Newton method provides the solution values, and the task is to compress them into a formula with kk atoms, measured by the relative deviation from a high-precision numerical solution (Algorithm 4). Every learned source of atoms (latent nearest neighbours, latent ODE shortlist, latent residual peeling, and atoms created by the composition operators) is compared with a network-free control that differs only in the learned part.

As observed in Table 6, with four atoms the best learned method outperforms the best control on 11 of 12 ODEs (median ratio 1.43), and atoms created by the composition operators improve over latent atoms alone on 10 of 12. With three atoms, however, learned and network-free methods are equivalent (5 wins, 1 tie, and 6 losses). All learned formulas are one to two orders of magnitude more accurate than Padé approximants [3] with the same number of coefficients. The per-equation deviations are listed in Table 22 in Appendix C. Figure 5e,Figure 5f illustrates the two extremes: the four-atom formula is indistinguishable from the solution of the soft Duffing oscillator, whereas for the Painlevé-I type equation every four-atom formula deviates by about 5%.

k=3k = 3k=4k = 4k=7k = 7
Best learned vs. best network-free control: wins / ties / losses5 / 1 / 611 / 0 / 19 / 1 / 2
Median ratio (control / learned)0.961.431.40
Composed atoms vs. latent atoms only: wins / ties / losses7 / 5 / 010 / 1 / 110 / 2 / 0
Median ratio (latent / composed)1.151.552.00
Best learned vs. Padé approximant with k+1k + 1 coefficients: wins12 / 1212 / 1212 / 12
Median error ratio (learned / Padé)0.0370.0110.004

Table 6. Performance comparison on 12 initial-value problems that SymPy cannot solve in closed form, with kk atoms. Ratios are medians over the ODEs of the relative deviations from the numerical solution.

ODEControlLatent atoms+ composedBest learnedPadé
y′=sin⁡y+xy' = \sin y + x1.4e-24.0e-21.7e-29.5e-31.0e0
y′=cos⁡y−sin⁡xy' = \cos y - \sin x8.4e-32.6e-25.0e-35.0e-31.3e-1
y′=−y2+cos⁡xy' = -y^2 + \cos x3.3e-24.3e-24.1e-22.4e-23.5e-1
y′=sin⁡y+cos⁡xy' = \sin y + \cos x1.1e-23.4e-21.5e-27.7e-36.5e-1
y′=y+xy' = \sqrt{y} + x2.8e-33.8e-32.4e-32.4e-35.9e-3
y′=cos⁡y+xy' = \cos y + x1.2e-21.7e-27.8e-35.4e-33.9e-1
y′′=−sin⁡yy'' = -\sin y5.3e-44.8e-46.4e-44.4e-44.3e-1
y′′=y2+xy'' = y^2 + x5.1e-25.3e-23.8e-23.8e-25.6e-1
y′′=−y+y3/6y'' = -y + y^3/67.9e-41.3e-48.7e-58.7e-55.6e-1
y′′=sin⁡y+cos⁡xy'' = \sin y + \cos x3.4e-44.2e-43.6e-42.0e-44.6e-1
y′′=−y3y'' = -y^31.9e-32.2e-32.3e-32.2e-32.8e-1
y′′=−y−y3y'' = -y - y^33.9e-36.3e-32.7e-32.7e-32.8e-1

Table 22. Relative deviation from the numerical solution of four-atom formulas for the 12 initial-value problems without closed form: best network-free control, latent atoms only, latent and composed atoms, best learned method, and Padé approximant with five coefficients

Discovering ordinary differential equations

Table 7 reports the discovery of the governing equation from 64 noisy samples, with the same term search, coefficient snapping, and verification for every route (Algorithm 5). The strong-form and weak-form routes estimate the derivatives from the samples and employ no network, whereas the learned route maps the samples to a formula with the value encoder and the decoder, and then differentiates the formula exactly.

MethodLinear ODEs, σ=0\sigma= 0Linear ODEs, 10−310^{-3}Linear ODEs, 10−210^{-2}Logistic-type ODEs, σ=0\sigma= 0Logistic-type ODEs, 10−310^{-3}Logistic-type ODEs, 10−210^{-2}
Strong form (SINDy-like, no network)0.5200.2000.0730.8530.5130.200
Weak form (no network)0.6730.2330.0530.4730.3800.127
Samples →\to formula →\to exact derivatives0.4870.4400.4400.6330.6470.600
SINDy, PySINDy [17]†0.0730.0000.0000.2000.0600.013
Weak SINDy, PySINDy [35]†0.1670.0470.0130.5800.4330.080
ODEFormer, pretrained [16]†0.0470.0200.0200.0330.0400.027

Table 7. Accuracy comparison (valid@1) on ODE discovery from 64 noisy samples (150 functions per set). †External methods receive the true order of the equation and generous coefficient snapping (Appendix § 7.§ 4).

The learned route is almost insensitive to noise: at 1% noise it is the most accurate method on both families, 6.0 and 3.0 times more accurate than the best network-free route. On clean samples, however, the network-free routes are more accurate (0.673 and 0.853 against 0.487 and 0.633), since spectral derivatives of clean samples already identify most equations. The external methods, which receive the true order of the equation, identify at most 16.7% of the linear ODEs, whose states require estimated higher derivatives; on logistic-type equations weak SINDy reaches 0.580 on clean samples but drops to 0.080 at 1% noise.

Transfer beyond the pretraining distribution

In order to measure how far the frozen stack transfers, it is evaluated on out-of-distribution expressions and on standard benchmarks, restricted to the items that f-JEPA can represent (univariate, without poles on [−π,π][-\pi,\pi], and within our vocabulary; Appendix § 7.§ 5). Every method is evaluated on the same items.

Out-of-distribution expressions The out-of-distribution integrands are derivatives of random non-polynomial expression trees with 6 to 8 operators, whereas the training trees have at most 5. The solver verifies an antiderivative for 44.3% of them, against 71.8% for SymPy, and only 17.4% before the substitution level (Table 26 in Appendix § 9), which confirms the brittleness of learned integrators outside their training distribution [51]. Notably, 8 of the solved integrals are not solved by SymPy within 10 seconds (Table 25). However, these integrands are generated like the backward training data of the integrator of Lample and Charton, which solves 99.3% of them, so this benchmark is out of distribution only for f-JEPA. For symbolic regression on 100 such trees, f-JEPA recovers 10% of the formulas exactly from clean samples, against 17% for PySR, and 7% at 1% noise, where PySR recovers none (Table 19 in Appendix § 9.§ 5).

Methodσ=0\sigma= 0 Exactσ=0\sigma= 0 <10−3<10^{-3}σ=0\sigma= 0 <10−1<10^{-1}σ=10−2\sigma= 10^{-2} Exactσ=10−2\sigma= 10^{-2} <10−3<10^{-3}σ=10−2\sigma= 10^{-2} <10−1<10^{-1}
E2E transformer [23]0.0000.0100.4500.0000.0000.440
PySR [14]0.1700.6800.8900.0000.0100.880
f-JEPA0.1000.2500.6100.0700.1000.640

Table 19. Out-of-distribution symbolic regression from 64 samples on 100 random non-polynomial expression trees with 6 to 8 operators. Shares of formulas recovered exactly and with a relative error below 10−310^{-3} and below 10−110^{-1}

Integrand ffAntiderivative FF found by f-JEPA
5ex2sin⁡(5x)sin⁡(cos⁡(5x))+ex2cos⁡(cos⁡(5x))25e^{\frac{x}{2}}\sin(5x)\sin(\cos(5x))+\frac{e^{\frac{x}{2}}\cos(\cos(5x))}{2}ex2cos⁡(cos⁡(5x))e^{\frac{x}{2}}\cos(\cos(5x))
−15(cos⁡(cos⁡(cos⁡(5x)))+5)2sin⁡(5x)sin⁡(cos⁡(5x))sin⁡(cos⁡(cos⁡(5x)))-15\left(\cos(\cos(\cos(5x)))+5\right)^{2}\sin(5x)\sin(\cos(5x))\sin(\cos(\cos(5x)))(cos⁡(cos⁡(cos⁡(5x)))+5)3\left(\cos(\cos(\cos(5x)))+5\right)^{3}
(tan⁡2(cos⁡(x)+3)+1)sin⁡(x)+1)ex−tan⁡(cos⁡(x)+3)\left(\tan^{2}(\cos(x)+3)+1\right)\sin(x)+1)e^{x-\tan(\cos(x)+3)}exe−tan⁡(cos⁡(x)+3)e^{x}e^{-\tan(\cos(x)+3)}
12+4x(6x−2x)2+1\frac{12+\frac{4}{x}}{(6x-\frac{2}{x})^{2}+1}atan⁡2(6x−2x)\operatorname{atan}^{2}\left(6x-\frac{2}{x}\right)
(3x2+15)cos⁡(x3+15x)cos⁡(sin⁡(x3+15x))\left(3x^{2}+15\right)\cos(x^{3}+15x)\cos(\sin(x^{3}+15x))sin⁡(sin⁡(x3+15x))\sin\left(\sin\left(x^{3}+15x\right)\right)
−3x2sin⁡(x3)cos⁡(x)−sin⁡(x)cos⁡(x3)-3x^{2}\sin(x^{3})\cos(x)-\sin(x)\cos(x^{3})cos⁡(x)cos⁡(x3)\cos(x)\cos(x^{3})
(−3x2−3x(2x+1)−3x+12)ecos⁡(3x(x2+x−4))sin⁡(3x(x2+x−4))\left(-3x^{2}-3x(2x+1)-3x+12\right)e^{\cos\left(3x\left(x^{2}+x-4\right)\right)}\sin\left(3x\left(x^{2}+x-4\right)\right)ecos⁡(3x(x2+x−4))e^{\cos\left(3x\left(x^{2}+x-4\right)\right)}
(2π5+15)sin⁡(sin⁡(π(x+1)5))sin⁡(cos⁡(sin⁡(π(x+1)5)))cos⁡(π(x+1)5)\left(\frac{2\pi}{5}+\frac{1}{5}\right)\sin\left(\sin\left(\frac{\pi(x+1)}{5}\right)\right)\sin\left(\cos\left(\sin\left(\frac{\pi(x+1)}{5}\right)\right)\right)\cos\left(\frac{\pi(x+1)}{5}\right)cos⁡(cos⁡(sin⁡(π(x+1)5)))\cos\left(\cos\left(\sin\left(\frac{\pi(x+1)}{5}\right)\right)\right)

Table 25. Out-of-distribution integrals solved by the learned levels of f-JEPA (verified by SymPy) for which SymPy integrate returns no antiderivative within 10 seconds.

MethodSolve rateDefinite integralNot solved by SymPy
Latent shortlists, decoding and constant refitting0.1010.5440
+ exact combination, term splitting and residual peeling0.1210.6640
+ latent tabular integration by parts0.1740.6583
+ substitution and trigonometric reduction0.4430.7118
Lample and Charton, FWD+BWD+IBP, beam 10 [28]0.993–42
SymPy integrate (reference, 10 s limit)0.718––

Table 26. Out-of-distribution symbolic integration on 149 integrands of random expression trees with 6 to 8 operators, with cumulative levels of our solver. Columns as in Table 4, plus the number of solved integrands that SymPy does not solve

Standard integration benchmarks On the test sets of Lample and Charton, the MIT Integration Bee, and the Rubi suite [39] (Table 8), f-JEPA is an approximate integrator rather than an exact one. Between 36.5%36.5\% and 76.0%76.0\% of its antiderivatives have a derivative within a relative error of 10−210^{-2} of the integrand, but only between 16.5%16.5\% and 35.9%35.9\% are verified as exact, against 97.5%97.5\% to 99.5%99.5\% for the integrator of Lample and Charton on its own test sets and 56.5%56.5\% and 28.0%28.0\% on the other two. Thus, the integration results of § 4.3 do not transfer to the distributions on which sequence-to-sequence integrators are trained.

MethodFWDBWDIBPMIT BeeRubi [39]
Approximate antiderivatives (only f-JEPA produces them)
f-JEPA, relative error of F′F' below 10−210^{-2}0.7000.5300.7600.5000.365
f-JEPA, within ε\varepsilon of the true antiderivative (eH1≤εe_{H_1} \leq\varepsilon)0.6600.4850.7350.5110.375
Exact antiderivatives verified by SymPy
f-JEPA0.2500.2400.1950.3590.165
Lample and Charton [28], rerun on these items0.9750.9850.9950.5650.280

Table 8. Symbolic integration on standard benchmarks (200 representable items per benchmark, 92 for MIT Bee). Top: shares of the approximate antiderivatives of f-JEPA that are close to the true one; bottom: shares verified exactly by SymPy

Standard ODE benchmarks The ODE test sets of Lample and Charton are posed as initial-value problems, and f-JEPA only receives samples of their numerical solution (Table 9). A solution is counted as verified when its numeric residual is below 10−610^{-6} and it meets the initial conditions, because a symbolic proof rejects 29.5%29.5\% and 18.5%18.5\% of the reference solutions themselves. The pretrained solvers of Lample and Charton verify 94.5%94.5\% and 91.5%91.5\% of the problems and SymPy 62%62\% and 32%32\%, whereas f-JEPA verifies 18.5%18.5\% and 16%16\%, and 25%25\% and 23%23\% after fine-tuning the decoder and the value encoder on expressions of this distribution and adding an ODE-aware search. The gap originates in the pretrained encoder: these solutions combine products and compositions that are absent from the pretraining families, and decoding their true latent recovers only 7.5%7.5\% and 4.5%4.5\% of them exactly, against 87%87\% for our test functions (Appendix § 9.7).

MethodODE1 VerifiedODE1 Res. <10−2< 10^{-2}ODE1 Res. <10−1< 10^{-1}ODE2 VerifiedODE2 Res. <10−2< 10^{-2}ODE2 Res. <10−1< 10^{-1}
f-JEPA (numerical IVP → samples-to-formula)0.1850.2400.3050.1600.2200.280
f-JEPA + fine-tuned decoder + ODE-aware search0.2500.2950.3650.2300.2800.350
Lample and Charton, rerun on these IVPs0.9450.9600.9650.9150.9500.950
SymPy dsolve (10 s)0.6200.6200.6200.3200.3350.335

Table 9. Solving ODEs on the Lample and Charton test sets, posed as initial-value problems (200 each). Verified: numeric residual below 10−610^{-6} and initial conditions met within 10−610^{-6}

On ODEBench [16] (Table 27 in Appendix § 9), the learned discovery route reconstructs 80%80\%, 87%87\%, and 89%89\% of the trajectories at the three noise levels. It is more accurate than ODEFormer under noise and ties with weak SINDy at 1%1\% noise, but every SINDy variant is more accurate on clean samples. In contrast, f-JEPA recovers the exact equation more often than every other method (15%15\% of the trajectories on clean samples and 7%7\% under noise, against at most 7%7\% and 4%4\%), although these shares are low in absolute terms.

MethodR2>0.9R^2 > 0.9, σ=0\sigma= 0R2>0.9R^2 > 0.9, 10−310^{-3}R2>0.9R^2 > 0.9, 10−210^{-2}R2>0.99R^2 > 0.99, σ=0\sigma= 0R2>0.99R^2 > 0.99, 10−310^{-3}R2>0.99R^2 > 0.99, 10−210^{-2}Exact equation, σ=0\sigma= 0Exact equation, 10−310^{-3}Exact equation, 10−210^{-2}Extrapolation NMSE, σ=0\sigma= 0Extrapolation NMSE, 10−210^{-2}
f-JEPA: samples → formula → exact derivatives0.800.870.890.670.780.670.150.070.072.4e-031.9e-03
SINDy (PySINDy)0.960.930.480.800.670.200.070.020.001.0e-052.0e-01
Weak SINDy (PySINDy)0.930.960.890.830.870.700.070.040.005.8e-065.7e-03
Estimated derivatives + PySR0.980.540.350.980.520.170.000.000.002.2e-081.1e+01
ODEFormer (pretrained)0.850.850.830.670.670.610.000.000.002.7e-032.3e-03

Table 27. Discovery of the 23 one-dimensional ODEBench systems [16] (two initial conditions each) from 64 noisy samples. R2R^2 is measured on the trajectory integrated from the discovered equation, and an equation is exact when it has the true terms with every coefficient within 1%

Limitations

One limitation of our work is that the latent operators are accurate only up to a noise floor of a few percent: no prediction satisfies the calibrated ε\varepsilon-criterion, linear relations between latents hold up to about 7%7\%, and the learned derivative is not a linear map of the latent space. Any optimisation in latent space below this level fits the errors of the model rather than the problem, which is why the latent space only proposes candidates and exact numerical or symbolic steps combine and verify them. Where a closed form exists, computer algebra remains more reliable, and when the decoder recovers the input formula, applying the exact operator to it is more accurate than the learned operator. (Table 16). The transfer beyond the pretraining families is partial: on standard integration and ODE benchmarks f-JEPA verifies at most 35.9% and 25% of the items, far behind sequence-to-sequence models trained on those distributions, and closing this gap would require pretraining the encoder on such expressions (§ 4.7). Its advantage in exact symbolic regression is measured on functions of the pretraining families and narrows on clean samples when near-exact formulas are counted. Moreover, the study is restricted to univariate functions on a fixed interval, to a 37-token vocabulary without special functions, and to synthetic data, and the decoder rarely reproduces expressions longer than 30 tokens exactly. Finally, all models were trained once on a single 8 GB laptop GPU, so the variability across training runs and the scaling behaviour are not measured, and Proposition 3 controls the inner-product error only through the angular error of the encoder, which is measured rather than bounded.

Conclusions

In this work, we examined whether a single transformer encoder, pretrained once from symbolic text, can learn a vector space of functions on which calculus acts through latent operators, and whether this representation can be reused across tasks. The proposed f-JEPA computes the magnitude of each latent exactly as the H1H^1 norm and learns only its direction, matching Sobolev angles, enforcing linearity, and predicting the derivative with a factored and exactly homogeneous operator. The resulting latent space closely reproduces Sobolev angles and inner products, and its operators are accurate enough to propose solutions that exact steps then verify.

Without any fine-tuning of the encoder, the same representation supports symbolic integration, symbolic regression, and the solution and discovery of ODEs, with its clearest gains on noisy data, whereas computer algebra and task-specific models remain stronger on standard benchmarks. To the best of our knowledge, this is the first joint-embedding predictive architecture trained on the symbolic form of functions in which the predictor is the derivative operator and collapse is prevented by a data-defined Sobolev geometry.

For future work, we would like to extend the approach to multivariate functions with commuting partial derivative operators, to reduce the noise floor of the derivative with structured operators, and to pretrain the encoder on broader distributions of expressions.

Experimental details

Data generation

Functions are generated with SymPy [36] on Ω=[−π,π]\Omega= [-\pi, \pi] and written in prefix notation [28] with a 37-token vocabulary:

  • special tokens;

  • the binary operators add sub mul div pow;

  • the unary operators neg inv sqrt pow2 pow3 exp log sin cos tan atan;

  • the symbols x E pi, the sign tokens INT+ INT-, the ten digits and a fraction separator.

A rational coefficient is a single contiguous literal (INT+ 3 / 1 0 for 3/10), and numeric coefficients come first in products, so that functions that differ in one coefficient differ in one literal. For instance, e−x2cos⁡3xe^{-x^2}\cos3x is written mul exp mul INT- 1 pow2 x cos mul INT+ 3 x.

The families are listed in Table 10. Coefficients are integers between 1 and 10 (50%), integers between 11 and 100 (30%) or fractions p/qp/q with q∈{2,3,4,5,10}q \in\{2, 3, 4, 5, 10\} (20%), with random signs. Frequencies are at most 16 and exponential rates at most 2, so that 64 quadrature nodes resolve every function. The ODE family contains solutions of ∑kcky(k)=0\sum_k c_k y^{(k)} = 0 with a monic integer characteristic polynomial of order 1 to 4, built from integer real roots with multiplicity and complex pairs a±bia \pm bi with a∈{−1,0,1}a \in\{-1, 0, 1\} and b≤10b \le10; every row satisfies its ODE exactly. The family of polynomials of a component contains ag3+bg2+cg+dag^3 + bg^2 + cg + d for a single component g∈{x,sin⁡kx,cos⁡kx,erx,log⁡(x2+1),arctan⁡x,1/(x2+1)}g \in\{x, \sin kx, \cos kx, e^{rx}, \log(x^2+1), \arctan x, 1/(x^2+1)\}. Moreover, constants, linear and quadratic functions are guaranteed minimum shares of 1%, 4% and 6%, and the length distribution decreases from 10 tokens on (mean length 19.9 tokens).

FamilyShareFamilyShareFamilyShare
linear ODE solutions19.3Chebyshev series5.7logistic2.7
polynomials of a component13.1exponential4.7damped oscillation2.7
polynomial7.5Gaussian4.2xe−bx2x e^{-bx^2} type2.5
Fourier sum6.4Lorentzian3.5xnekxx^n e^{kx}2.4
polynomial ×\times trigonometric5.9arctan3.5random expression trees4.2
harmonic5.3rational3.5
log⁡(1+bx2)\log(1 + bx^2)2.8

Table 10. Composition of the 2.0M training functions by family (%)

A function is accepted under the following conditions:

  • it is finite on a 1024-point grid and bounded by 10510^5 in absolute value;

  • its norm agrees between 64 and 128 quadrature nodes (relative tolerance 10−310^{-3});

  • its L2L^2 norm lies in [0.05,2000][0.05, 2000];

  • it has at most 126 tokens.

The derivative f′f' is stored in its shortest equivalent form. The test set contains 12,146 functions generated with the same families and filtered so that none of them appears in the training data. The corpus also has its own validation and test splits (20.4k functions each): the former is used for calibration and model selection, and the latter only in Figure 3. The composition operators are trained on 4.17M pairs (g,h∘g)(g, h \circ g), whose inner functions gg are drawn from the training split.

Encoder training

The hyperparameters are listed in Table 11. The networks hh and kk in Equation (4) have hidden sizes 1024 and 256, respectively, and their gradients are clipped separately from those of the encoder. Batches are encoded in four chunks of similar length, since random batches contain about 70% padding; this makes training 4.9 times faster.

HyperparameterValue
ArchitectureBERT, 10 layers, hidden size 512, 8 heads, feed-forward 2176, d=256d = 256
Parameters33M (encoder), 1.6M (DD)
Initialisationtruncated normal (std 0.02), residual outputs scaled by 1/2L1/\sqrt{2L}
OptimizerAdamW, learning rate 3⋅10−43 \cdot10^{-4} (10−310^{-3} for DD), 700 warm-up steps, cosine
Steps / batch size40,000 / 384
Batch composition15% equivalence groups, 25% sweeps, plus 15% synthetic rows (half combinations, half scaled copies)
Combination coefficients±{1,2,3,5,10,1/2}\pm\{1, 2, 3, 5, 10, 1/2\}
λ\lambda, βc\beta_c, τ\tau, βg\beta_g, ϵ\epsilon0.5, 1.5, 0.5, 1, 0.05
Ramp of Llin\mathcal{L}_{\mathrm{lin}} and LJEPA\mathcal{L}_{\mathrm{JEPA}}0 until step 1000, linear to 1 at step 2000
Precisionbfloat16 autocast, gradient checkpointing, length-bucketed batches

Table 11. Hyperparameters of the encoder

Modules on the frozen encoder

The decoder has 6 layers, model size 384, 6 heads and 8 memory tokens (17.7M parameters). It is trained for 105k steps with batches of 32k tokens and a warm-up, stable and decay schedule, with dropout 0.05 and label smoothing 0.02. Its inputs are randomly rotated (cosine at least 0.99) for 40% of the rows, and 30% of the derivative rows use the predicted latent D(zf)D(z_f) as input. The composition bank contains seven networks, trained jointly with disjoint parameters and per-network gradient clipping for 40k steps; their construction is described in Appendix § 7.§ 3. The integrator and the value encoder share the same MLP size, and are trained for 60k and 20k steps, respectively. The value encoder receives the normalised samples and their spectral derivative (Legendre degree up to 40), noise with σ≤0.02\sigma\le0.02 is added to half of the training batches, and it is fine-tuned with 0.5 times the cross-entropy of the frozen decoder.

Construction of the composition operators Each composition operator Mh\mathcal{M}_h, for h∈{sin⁡,cos⁡,log⁡,⋅,(⋅)2,(⋅)3}h \in\{\sin,\cos,\log,\sqrt{\cdot},(\cdot)^2,(\cdot)^3\} and for the quadratic ag2+bg+cag^2 + bg + c, is a separate network on the frozen latent zgz_g, with ng=∥zg∥n_g = \lVert z_g \rVert and u^=zg/ng\hat{\mathbf{u}} = z_g/n_g. Its output is built in the same factored form as the latent and as DD:

Mh(zg)=eκh(x)+phlog⁡ng δh(x)+ah(log⁡ng) z1,δh(x)=Whu^+rh(x)∥Whu^+rh(x)∥,(8)\mathcal{M}_h(z_g) = e^{\kappa_h(\mathbf{x})+p_h\log n_g}\,\delta_h(\mathbf{x}) + a_h(\log n_g)\,\mathbf{z}_1,\qquad \delta_h(\mathbf{x}) = \frac{\mathbf{W}_h\hat{\mathbf{u}}+\mathbf{r}_h(\mathbf{x})}{\lVert\mathbf{W}_h\hat{\mathbf{u}}+\mathbf{r}_h(\mathbf{x})\rVert}, \tag*{(8)}

where z1\mathbf{z}_1 is the latent of the constant function 1, Wh∈Rd×d\mathbf{W}_h \in\mathbb{R}^{d \times d} is a linear skip initialised to the identity, and (rh,κh)(\mathbf{r}_h,\kappa_h) are the d+1d+1 outputs of a residual MLP ThT_h (input layer to width 768, three pre-LayerNorm residual blocks with GELU, expansion factor 2 and dropout 0.1, and a final LayerNorm). The input x\mathbf{x} of ThT_h and the constants (ph,ah)(p_h,a_h) depend on the operator:

  • g2g^2, g3g^3 and g\sqrt{g}: x=u^\mathbf{x} = \hat{\mathbf{u}}, ah=0a_h = 0 and ph=2p_h = 2, 33 and 1/21/2, respectively;

  • log⁡g\log g: x=u^\mathbf{x} = \hat{\mathbf{u}}, ph=0p_h = 0 and ah=1a_h = 1;

  • sin⁡g\sin g and cos⁡g\cos g: x=(u^,ℓ)\mathbf{x} = (\hat{\mathbf{u}}, \ell), where ℓ\ell is the standardised log⁡ng\log n_g, ph=1p_h = 1 and ah=0a_h = 0;

  • ag2+bg+cag^2+bg+c: Mquad(zg)=b+∥b∥rquad(u^,ℓ,13(a,b,c))M_{\mathrm{quad}}(z_g) = \mathbf{b} + \lVert\mathbf{b}\rVert r_{\mathrm{quad}}\left(\hat{\mathbf{u}}, \ell, \frac{1}{3}(a,b,c)\right), with the base b=aMsq(zg)+bzg+cz1\mathbf{b} = aM_{\mathrm{sq}}(z_g)+bz_g+cz_1, in which MsqM_{\mathrm{sq}} is evaluated without gradient.

The scaling laws that this construction satisfies by design are stated in Proposition 4 (Appendix B.3). Each design choice reproduces a property of the target, so that the network only has to learn what the property does not determine:

  • Equivariance under positive scaling. Since (cg)p=cpgp(cg)^p = c^p g^p, cg=c1/2g\sqrt{cg} = c^{1/2}\sqrt{g} and log⁡(cg)=log⁡g+log⁡c\log(cg) = \log g + \log c, and since the latent has an exact norm and is linear up to the scale error of the encoder (Table 1), the targets satisfy the laws of Proposition 4 up to that error. The architecture satisfies them exactly, so these networks learn a function of the direction u^\hat{\mathbf{u}} alone, and the logarithm learns its value at unit norm, with training target zlog⁡g−(log⁡ng)z1z_{\log g} - (\log n_g)z_1.

  • No scaling law for sin⁡\sin and cos⁡\cos. Since sin⁡(cg)\sin(cg) is not determined by sin⁡g\sin g, these networks receive log⁡ng\log n_g as an input and predict the gain freely.

  • Sign changes. For c<0c < 0 the laws require the parities (−g)2=g2(-g)^2 = g^2, (−g)3=−g3(-g)^3 = -g^3, sin⁡(−g)=−sin⁡g\sin(-g) = -\sin g and cos⁡(−g)=cos⁡g\cos(-g) = \cos g, which Equation (8) does not enforce because κh\kappa_h and δh\delta_h are not symmetrised as in D\mathcal{D} (Proposition 1). They are taught by sign augmentation: for a random half of each batch, zgz_g is replaced by −zg-z_g and the target by −zh∘g-z_{h\circ g} for the odd operators (sin⁡\sin, g3g^3) and by zh∘gz_{h\circ g} for the even ones (cos⁡\cos, g2g^2), and b\mathbf{b} by −b-\mathbf{b} for the quadratic; log⁡\log and ⋅\sqrt{\cdot}, whose inner functions are nonnegative on Ω\Omega in the data, are not augmented. The integrator is augmented in the same way, since integration is odd.

  • Linearity and closure under composition. The quadratic is not learned from scratch: by the linearity of the latent, zag2+bg+c≈azg2+bzg+cz1z_{ag^2+bg+c} \approx a z_{g^2} + b z_g + c z_1, so it is the composition of the learned square operator with exact latent linear algebra plus a residual scaled by the norm of the base. The base alone retrieves 44.4% of the targets without any training on the quadratic (Table 3).

  • Factored loss. As for D\mathcal{D} (§ 3.4), the direction and the log-gain are trained by separate terms, (1−cos⁡(δh,t^))+(κh−log⁡∥t∥+phlog⁡ng)2+0.1LNCE(δh,t^)(1-\cos(\delta_h,\hat{\mathbf{t}})) + (\kappa_h-\log\lVert\mathbf{t}\rVert+p_h\log n_g)^2 + 0.1L_{\mathrm{NCE}}(\delta_h,\hat{\mathbf{t}}) with temperature 0.05, where t=zh∘g−ah(log⁡ng)z1\mathbf{t}=z_{h\circ g}-a_h(\log n_g)z_1, which avoids the shrinkage of the predicted norm under an uncertain direction; the quadratic uses the relative squared error of its full prediction plus the same contrastive term.

  • Identity initialisation. With Wh=IW_h = \mathbf{I} and a residual head initialised at about 1% of the norm of u^\hat{\mathbf{u}}, every direction starts at the direction of the inner function, and the linear skip carries the part of the map that is linear in the latent (Table 3, linear part only).

External baselines

All symbolic-regression baselines receive exactly the 64 samples (and noise draws) used by f-JEPA and are scored with the same criteria; floating-point constants are snapped to simple fractions before the symbolic check, as f-JEPA does with its own constants. PySR [14] (version 2.7) uses the binary operators ++, −-, ×\times, ÷\div and the unary operators sin⁡\sin, cos⁡\cos, exp⁡\exp, log⁡\log, ⋅\sqrt{\cdot}, square, cube and arctan, a maximum expression size of 30, a budget of 20 seconds per function, deterministic serial search and its default model selection. The end-to-end transformer [23] is the publicly released pretrained model with the settings of its official example (200 input points, rescaling, refinement of 100 candidate trees by BFGS). Since its checkpoint is a pickled Python object, it was loaded with a restricted unpickler that resolves only the 42 classes and functions found at a static scan of the file, and two incompatibilities of the official code with NumPy 2 were fixed (an unused import and the alias np.infty).

The integrator of Lample and Charton [28] is the released FWD+BWD+IBP model, run with the official code and inference settings of its example notebook (input ∂xy(x)−f(x)\partial_x y(x)-f(x) in its prefix notation, beam 10, at most 2000 output tokens); an integrand is solved if any hypothesis HH satisfies H′=fH' = f with our SymPy check. The symbolic path of Table 16 decodes the input latent (greedy and beam 8, round-trip re-ranking), applies sympy.diff or the composition at token level, writes the derivative in the canonical form of the data (shortest of the raw, together and factor_terms forms), and re-encodes it with the exact norm.

For ODE discovery, ODEFormer [16] is the released pretrained model with the beam size 50 and temperature 0.1 of its official example, and SINDy [8] and weak SINDy [35] are the implementations of PySINDy 2.1 [17, 24], with the library {1,si,si2,si3}\{1, s_i, s_i^2, s_i^3\} over the state components sis_i, the sequentially thresholded least-squares optimiser, and the model chosen among 7 thresholds by the same parsimony-weighted residual as our search. These methods identify first-order systems of observed states, so for a linear ODE of order kk they receive the state (y,y′,…,y(k−1))(y,y',\ldots,y^{(k-1)}), with derivatives estimated by the finite differences of PySINDy (smoothed for noisy data) and with the true order kk, which our routes have to find. Moreover, their coefficients are snapped generously: an equation is valid if snapping with any of the denominator bounds of our search makes it exact. Both choices favour the baselines.

For the representation comparison of Table 2, SNIP [34] is the released SNIP-1d-normalized model, of which the symbolic encoder is used, with expressions written in the format of its own generator (binary sums and products, unary powers, all constants in its floating-point encoding). As a check of this input format, its symbolic and numeric embeddings retrieve each other with 42% top-1 accuracy among 200 functions of its own generator, but only 3% among 200 of our test functions, with any input range. The Lample and Charton representation is the mean of the final encoder states for the integration input. The probes of Table 2 are fitted on the same 20k training pairs for every representation: ridge regression with the regularisation chosen on a held-out 10%, and a 2-layer MLP with 1,024 hidden units trained for 3k Adam steps. The derivative probe takes [e/∥e∥,log⁡∥e∥][e/\lVert e\rVert,\log\lVert e\rVert] as input and predicts a unit direction (cosine loss) and the log-ratio log⁡(∥e(f′)∥/∥e(f)∥)\log(\lVert e(f')\rVert/\lVert e(f)\rVert), with the prediction ∥e(f)∥eekd\lVert e(f)\rVert e^{e_k} d, so that it is not biased towards small norms.

E-Gen [52] releases code and data but no trained model, so it is not included. The checkpoints of the Lample and Charton model and of SNIP contain only tensors (and, for SNIP, a path object), so they were loaded with weights_only=True, whereas the ODEFormer checkpoint is a pickled Python object and was loaded with a restricted unpickler that resolves only the 32 globals found by a static scan. The official code of ODEFormer and SNIP required the same two NumPy 2 fixes as above.

Standard benchmarks

The standard benchmarks of § 4.7 are the test splits of Lample and Charton [28] (integration FWD, BWD and IBP; ODE1 and ODE2), the indefinite integrals of the Rubi test suite [39] and of the MIT Integration Bee as collected in the SymPy integration test suites [43], and the 23 one-dimensional systems of ODEbench [16]. Each item is kept only if it can be represented by f-JEPA: a univariate expression in xx (hyperbolic and inverse functions are written exactly with exp, log and arctan, and decimal constants as rationals), at most 126 tokens of our vocabulary, real and finite on [−π,π][-\pi,\pi] without poles (no step between neighbouring points of a 20,001-point grid exceeds 5% of the range), and with a norm in the training range of the encoder. Table 12 gives the number of items kept and the main reasons for rejection; the benchmarks are scanned in a seed-0 random order, and the first 200 kept items are used.

BenchmarkKeptScannedMain rejection reasons (counts)
L&C integration FWD10002764not finite/real on [-pi, pi] (1096); outside norm range (283); operator outside vocabulary (212)
L&C integration BWD10008203not finite/real on [-pi, pi] (3463); too long (1278); operator outside vocabulary (1038)
L&C integration IBP10002098not finite/real on [-pi, pi] (580); outside norm range (283); operator outside vocabulary (143)
MIT Integration Bee92322not finite/real on [-pi, pi] (163); outside norm range (35); pole / discontinuity in [-pi, pi] (18)
Rubi (indefinite)100058104not univariate in x (51687); not finite/real on [-pi, pi] (3570); outside norm range (923)
L&C ODE1 (as IVPs)5306523not finite/real on [-pi, pi] (3317); operator outside vocabulary (964); pole / discontinuity in [-pi, pi] (517)
L&C ODE2 (as IVPs)3834346not finite/real on [-pi, pi] (2147); ill-conditioned IVP on [-pi, pi] (617); outside norm range (428)

Table 12. Coverage of the standard benchmarks: representable items kept, items scanned, and the three main reasons for rejection

The ODE test items of Lample and Charton ask for the general solution of an equation F(x,y,y′,y′′)=0F(x,y,y',y'')=0. They are converted to initial-value problems by fixing the constants of the reference solution to the first values in {1,2,−1,12,3}\{1, 2, -1, \frac{1}{2}, 3\} (pairs for ODE2) for which the particular solution is representable, and by imposing y(x0)y(x_0) (and y′(x0)y'(x_0)) at the first point x0∈{0,12,1,−12}x_0 \in\{0, \frac{1}{2}, 1, -\frac{1}{2}\} where the equation can be solved for the highest derivative; for implicit equations, the IVP also selects the branch taken by the reference solution at x0x_0. Since f-JEPA works from samples, we keep only the IVPs whose numerical solution (DOP853, relative tolerance 10−1210^{-12}) reproduces the reference solution on [−π,π][-\pi,\pi] to 10−610^{-6}, which removes equations that are singular inside the interval. The pretrained solvers of Lample and Charton receive the original equation and return a general solution, whose constants are then solved from the initial conditions with SymPy; SymPy dsolve receives the initial conditions directly.

For symbolic regression, NeSymReS [7] is the released 100M model with the inference configuration of its repository (beam search and BFGS constant fitting), SNIP [34] is the released encoder-decoder model with latent space optimisation by Grey Wolf optimisation (population 50, 80 iterations, as in its example), MMSR [30] is the released model with the settings of its benchmark script (run in a separate TensorFlow environment; the released model is bivariate, so the second input is a dummy variable drawn uniformly from [−5,5][-5,5]), and Operon [9] uses our operator set and a budget of 20 seconds per function, as PySR. Every method is given at most 600 seconds per function, after which the function counts as unsolved. On ODEBench, every method searches first-order autonomous equations y′=g(y)y'=g(y), the form of all one-dimensional ODEBench systems: the learned route of f-JEPA and SINDy use the library {1,y,y2,y3}\{1,y,y^2,y^3\} with floating-point coefficients (the constants of ODEBench are decimals), ODEFormer [16] receives the samples with their original times, and the derivative-estimation baseline fits PySR to y′=g(y)y'=g(y) on smoothed finite differences. The checkpoints of NeSymReS, SNIP and the ODE solvers of Lample and Charton contain only tensors (and, for NeSymReS, a checkpoint-callback object) and were loaded with weights_only=True after a static scan; the server that hosts the Lample and Charton files had an expired TLS certificate at the time of download, so their SHA-256 hashes are recorded and the files were only loaded in this restricted mode. The official code of these baselines required only compatibility fixes for NumPy 2, Python 3.11 and PyTorch 2 (deprecated aliases, a mutable dataclass default and byte masks), which do not change their computations. As context, Lample and Charton report accuracies of 95.6% to 99.6% on their complete integration test sets and general-solution accuracies of 97.2% (ODE1) and 88.2% (ODE2) with beam 10; their protocols differ from ours, so these numbers are not included in the tables. The reference solutions pass the numeric verification of Table 9 for 97.5% (ODE1) and 96% (ODE2) of the IVPs, where the residual is the maximum of ∣F(x,y,y′,y′′)∣|F(x,y,y',y'')| over 256 points of [−π,π][-\pi,\pi] relative to the size of the terms. ODEFormer reports its integration accuracies only sporadically and aggregated over all dimensions, so they are not reproduced. For the integration benchmarks of Table 8, the answer of f-JEPA combines, per integrand, the solver of Algorithm 1 with a second latent proposal obtained by inverting the derivative operator, z∗=arg⁡min⁡z∥D(z)−zf∥2/∥zf∥2z^*=\arg\min_z\lVert D(z)-z_f\rVert^2/\lVert z_f\rVert^2, followed by two rounds of refinement on the residual f−F^′f-\hat{F}'; the answer with the smaller residual ∥F^′−f∥/∥f∥\lVert\hat{F}'-f\rVert/\lVert f\rVert is kept, which does not use the true antiderivative, and these settings were chosen on 150 validation integrands. The second proposal alone verifies far fewer exact antiderivatives than the solver of Algorithm 1, since it lacks the substitution level, so the combination only improves the approximate rows of the table. The timeout of the ODEFormer simplifier schedules an alarm signal that it never cancels; in the first evaluation this pending alarm shortened the 60-second limit of the next search of the learned route of f-JEPA to about one second, so that 67 of its 138 searches were interrupted. The harness now cancels any pending alarm after every ODEFormer call, and the learned route was re-evaluated on the same noisy samples; the affected evaluation gave 37%, 30%, and 65% instead of the 80%, 87%, and 89% of Table 27. Three inference-only variants of the learned route (falling back to the next decoded candidate, refitting the coefficients of the decoded support in the weak form, and a wider beam) were compared on 40 synthetic validation systems, and none was more accurate than the original route, which is therefore kept.

Evaluation metrics

Scale errors are median values of ∥zcf−czf∥/∥czf∥\lVert z_{cf}-cz_f\rVert/\lVert cz_f\rVert with inputs mult ∩\cap f. Linearity compares zaf+bgz_{af+bg} with azf+bzgaz_f+bz_g for training-type combinations, and additivity compares zf+gz_{f+g} with zf+zgz_f+z_g for random pairs. Derivative retrieval finds the nearest neighbour of D(zf)\mathcal{D}(z_f) among all distinct test derivatives. ODE annihilation is the residual ∥p(D)z∥\lVert p(\mathcal{D})z\rVert of ODE-family functions relative to that obtained with a shuffled ODE (1 is chance level), and ODE precision at 20 ranks the pool by r(z)r(z). Decoding is exact when the relative L2L^2 error of the decoded expression is below 10−610^{-6}, and the shape is correct when the error is below 10−310^{-3} after refitting all constants to the true values. For ODE discovery, valid@1 is the fraction of functions whose top-ranked equation is satisfied exactly by the true function.

The rows of Table 1 are defined as follows.

  • Geometry: Pearson correlation, over random pairs of test functions, between the cosine (inner product) of the two latents and the cosine (inner product) of the two functions in H1H^1, and mean absolute difference between the two cosines; an exact isometry would give 1, 1, and 0.

  • Linearity: the scale error above, the same error for combinations af+bgaf+bg of two functions of the same family (the type seen in pretraining), and for sums f+gf+g of two unrelated test functions.

  • Derivative: top-1 and top-5 retrieval of f′f' among 3,956 candidates by nearest neighbour of D(zf)\mathcal{D}(z_f); relative error and cosine between D(zf)\mathcal{D}(z_f) and zf′z_{f'}; ratio of their norms; and retrieval with the best linear map fitted to D\mathcal{D} by ridge regression on 60k training pairs.

  • Zero-shot: second derivatives obtained by applying D\mathcal{D} twice, or once to zf′z_{f'}, retrieved among 3,640 candidates; relative errors of the eigenfunction relations Dz=az\mathcal{D}z=az for eaxe^{ax} and D2z=−k2z\mathcal{D}^2z=-k^2z for sin⁡kx\sin kx; and ODE annihilation.

  • Structure: Spearman correlation between the order of a swept coefficient and the order of the latents along the sweep; distance between the latents of two rewritings of the same function, relative to random pairs; and accuracy of a linear probe for the family of a function and R2R^2 of a linear probe for its sampled values.

In the symbolic-regression comparison of Table 5, a formula is exact when its clean relative error is below 10−610^{-6} and SymPy proves its equality with the target, and it is within the threshold when its relative H1H^1 error with respect to the clean target is at most 0.0037, the function-space analogue of ε\varepsilon (Appendix C.2), which also counts near-exact formulas with floating-point constants. The remaining columns are, at σ=0\sigma=0, the share of functions with a held-out NMSE below 10−310^{-3}, the median NMSE on [−1.25π,1.25π]∖[−π,π][-1.25\pi,1.25\pi]\setminus[-\pi,\pi] (Extrap.; 35 of the 50 formulas of f-JEPA coincide with the target), and the median number of nodes of the formula, and, at σ=10−2\sigma=10^{-2}, the median relative error with respect to the clean function. The same function-space threshold is used for the approximate antiderivatives of Table 8, with the constant fixed by F(−π)=0F(-\pi)=0. In Tables 4 and 26, the definite-integral column counts the integrands for which ∣F(π)−F(−π)−∫−ππf dx∣<10−2∫−ππ∣f∣ dx|F(\pi)-F(-\pi)-\int_{-\pi}^{\pi}f\,dx|<10^{-2}\int_{-\pi}^{\pi}|f|\,dx. In Table 16, an input is not decoded when the relative L2L^2 error of its decoded formula is above 10−610^{-6}, and the evaluation uses 4,000 functions for the derivative and 1,000 inner functions per composition operator.

Inference procedures

This section specifies the procedures used on the frozen encoder in Section 4. In them, r(z)=∥Dkz−∑jajTj(z)∥∥Dkz∥+∑j∣aj∣∥Tj(z)∥r(z)=\frac{\lVert\mathcal{D}^kz-\sum_j a_j\mathcal{T}_j(z)\rVert}{\lVert\mathcal{D}^kz\rVert+\sum_j\lvert a_j\rvert\lVert\mathcal{T}_j(z)\rVert} is the scale-free latent residual of an ODE y(k)=∑jajTj(y)y^{(k)}=\sum_j a_jT_j(y), and σ^\hat{\sigma} is the noise level estimated from the flat floor of the Legendre spectrum of the samples. Every procedure ends with an exact check, either a SymPy proof or a high-precision numerical comparison, and no procedure uses the ground truth to select its output.

Theoretical results

Proof of Proposition 1

Consider z≠0z\ne0 and c≠0c\ne0, and let u^=z/∥z∥\hat{\mathbf{u}}=z/\lVert z\rVert and s=sign⁡(c)s=\operatorname{sign}(c). Then ∥cz∥=∣c∣∥z∥\lVert cz\rVert=\lvert c\rvert\lVert z\rVert and cz/∥cz∥=su^cz/\lVert cz\rVert=s\hat{\mathbf{u}}. Since γ(u^)=eκ(u^)+κ(−u^)\gamma(\hat{\mathbf{u}})=e^{\kappa(\hat{\mathbf{u}})+\kappa(-\hat{\mathbf{u}})} is symmetric in u^\hat{\mathbf{u}} and −u^-\hat{\mathbf{u}}, we have γ(su^)=γ(u^)\gamma(s\hat{\mathbf{u}})=\gamma(\hat{\mathbf{u}}). Since δ\delta is linear and h(u^)−h(−u^)h(\hat{\mathbf{u}})-h(-\hat{\mathbf{u}}) is odd, the numerator of δ\delta is odd, and so is its normalisation, δ(su^)=s δ(u^)\delta(s\hat{\mathbf{u}})=s\,\delta(\hat{\mathbf{u}}). Therefore

D(cz)=∣c∣∥z∥γ(u^) sδ(u^)=c∥z∥γ(u^)δ(u^)=cD(z),\mathcal{D}(cz)=\lvert c\rvert\lVert z\rVert\gamma(\hat{\mathbf{u}})\,s\delta(\hat{\mathbf{u}})=c\lVert z\rVert\gamma(\hat{\mathbf{u}})\delta(\hat{\mathbf{u}})=c\mathcal{D}(z),

where ∣c∣s=c\lvert c\rvert s=c.

Input: integrand ff (tokens and values at the 64 nodes). Output: a SymPy-verified antiderivative FF, or none.

  1. zf←EΩ(f)z_f \leftarrow E_{\Omega}(f), z^F←I(zf)\hat{z}_F \leftarrow I(z_f); shortlist the 16 training functions gg nearest to z^F\hat{z}_F and the 16 with D(zg)D(z_g) nearest to zfz_f.

  2. Decode z^F\hat{z}_F with beam 16 and refit the constants of every candidate so that F′=fF' = f on the nodes.

  3. Exact combination: orthogonal matching pursuit (at most 6 atoms) over decoded pieces, shortlisted functions and {x,x2,x3}\{x,x^2,x^3\} for F′=fF' = f; coefficients snapped to fractions.

  4. Term splitting: integrate each term of a sum separately and add the results.

  5. Residual peeling (2 rounds): encode the residual f−∑jajgj′f-\sum_j a_jg'_j, integrate it with II and repeat steps 2–3.

  6. Integration by parts: for each product u hu\,h in ff, build the chain H1,H2,H3H_1,H_2,H_3 of antiderivatives of hh with II and the decoder (constants dropped) and add the atoms u(i)Hi+1u^{(i)}H_{i+1} to a small basis before the full one.

  7. Substitution and reduction: reduce powers and products of trigonometric functions to sums; for every inner function φ\varphi of ff, rewrite f/φ′f/\varphi' in U=φ(x)U=\varphi(x), integrate it with steps 1–6 and substitute back.

  8. Return the first candidate FF for which SymPy proves F′−f=0F'-f=0 within the time limit.

Algorithm 1: Symbolic integration on the frozen stack (levels of Table 4)

Input: values yy at the 64 nodes. Output: a formula.

  1. Estimate σ^\hat{\sigma}; compute n^\hat{n} as the H1H^1 norm of yy with a spectral derivative (low-pass to degree 40 if σ^>0\hat{\sigma}>0); z^←n^ V(y)\hat{z}\leftarrow\hat{n}\,\mathcal{V}(y).

  2. Decode z^\hat{z} with beam 16; refit the constants of every candidate to yy by weighted least squares and snap them to fractions with denominator at most 12.

  3. Exact combination of the candidates with the atoms {1,x,x2}\{1,x,x^2\}.

  4. Residual peeling (2 rounds): encode the residual values with V\mathcal{V}, decode them and add the results to the basis.

  5. Return the candidate minimising n^2σ^2err⁡2+4\frac{\hat{n}}{2\hat{\sigma}^2}\operatorname{err}^2+4 bits per operator + the bits of its constants (for σ^=0\hat{\sigma}=0, the shortest exact candidate).

Algorithm 2: Symbolic regression from samples

For c>0c>0 only the first step is needed, since then s=1s=1; this is the argument behind the scaling laws of the integrator and of the composition operators (Proposition 4), whose gains and directions are not symmetrised and which therefore satisfy their laws exactly only for positive scalars.

Why a finite latent vector can reproduce the Sobolev inner product

This section answers two questions in plain terms. First, can a vector of finite dimension behave exactly like a function in the Sobolev space, with dot products equal to Sobolev inner products and linear combinations equal to linear combinations of functions? Proposition 2 shows that it can: the coefficients of a function on a Sobolev-orthonormal polynomial basis form such a vector, up to a truncation error and a quadrature error that both vanish quickly for smooth functions, so the target of the pretraining is attainable. Second, how far can the learned encoder be from this ideal? Proposition 3 and Corollary 1 show that, because the norm of a latent is computed exactly, the error of every latent dot product and latent distance comes from only two sources: the angle error of the network, which training reduces, and the error of the 64-node quadrature, which is negligible for the functions used here. Hence, any remaining error of the latent geometry is the angular error of the encoder, which is measured in Appendix § 9.§ 3.

Let Ω=[a,b]\Omega=[a,b], let I(F)=1b−a∫abF(x) dxI(F)=\frac{1}{b-a}\int_a^b F(x)\,\mathrm{d}x, and let QN(F)=∑i=1NwiF(xi)Q_N(F)=\sum_{i=1}^{N}w_iF(x_i) be the NN-node Gauss–Legendre rule mapped to Ω\Omega with weights normalised so ∑iwi=1\sum_iw_i=1, which is the rule used by the encoder (N=64N=64). QNQ_N is exact for polynomials of degree at most 2N−12N-1. Let t(x)=(2x−a−b)/(b−a)t(x)=(2x-a-b)/(b-a) map Ω\Omega onto [−1,1][-1,1], and let EρE_\rho denote the open Bernstein ellipse with parameter ρ>1\rho>1 in the variable tt. We use the following classical bound [44 Theorem 19.3], divided by 2 because of the normalised measure.

Input: ODE y(k)=∑jajTj(y)y^{(k)}=\sum_j a_jT_j(y), pool of 2M2M functions with latents and stored values. Output: verified solutions.

  1. For every pool function compute r(z)r(z), with Tj(z)∈{z,Dz,D2z,Msq(z),Mcube(z),Msqrt(z),z1}T_j(z)\in\{z,Dz,D^2z,\mathcal{M}_{\mathrm{sq}}(z),\mathcal{M}_{\mathrm{cube}}(z),\mathcal{M}_{\mathrm{sqrt}}(z),z_1\}, and keep the 1%1\% with the lowest residual.

  2. Re-rank the shortlist by the exact residual of the ODE on the stored values of ff, f′f' and f′′f''.

  3. Linear ODEs: for all subsets of at most 3 functions among the 16 best and {1,x}\{1,x\}, compute the exact null vector of the residual matrix and snap it to fractions.

  4. Refit the constants of the 50 best skeletons so that the ODE holds (scale-free residual below 10−910^{-9}, every term active).

  5. Return the candidates for which SymPy proves that the ODE holds.

Algorithm 3: Solving ODEs with closed-form solutions

Input: ODE, initial conditions, solution values yNy_N from a spectral Newton method. Output: formulas with k=1,…,7k=1,\ldots,7 atoms.

  1. Build a dictionary of atoms. Learned: the 200 nearest latents to V(yN)\mathcal{V}(y_N), the 200 best functions by the latent ODE residual, decoded composition outputs Mh(zg)\mathcal{M}_h(z_g) of the 30 best, and latent residual peeling (V\mathcal{V} of the current residual, 16 neighbours and 8 decoded beams). Controls: the 200 nearest functions in value space, 200 random pool functions, and value-space residual peeling.

  2. Greedily add the atom that most decreases 10∥y−yN∥2+ODE residual+initial-condition penalty10\lVert y-y_N\rVert^2+\text{ODE residual}+\text{initial-condition penalty}, refit all coefficients by nonlinear least squares, and refine the set by swaps.

  3. Report the relative deviation of each formula from a high-precision numerical solution (not used in steps 1–2).

Algorithm 4: Compact formulas for initial-value problems without closed form

Quadrature bound If F∘t−1F\circ t^{-1} is analytic in EρE_\rho and bounded there by MM, then

∣QN(F)−I(F)∣≤3215Mρ−2Nρ2−1.(9)\lvert Q_N(F)-I(F)\rvert\le\frac{32}{15}\frac{M\rho^{-2N}}{\rho^2-1}. \tag*{(9)}

The bound decays geometrically with the number of nodes; for functions that are only mm times differentiable the decay is algebraic in NN instead. The data filter of Appendix A.1 additionally checks, for every stored function, that its norm agrees between 64 and 128 nodes to a relative tolerance of 10−310^{-3}.

Reference embedding Let ϕ0,…,ϕK−1\phi_0,\ldots,\phi_{K-1} be the basis of polynomials of degree less than KK that is orthonormal in ⟨⋅,⋅⟩H1(Ω)\langle\cdot,\cdot\rangle_{H^1(\Omega)} (Gram–Schmidt applied to 1,x,…,xK−11,x,\ldots,x^{K-1}), let ΠK\Pi_K be the orthogonal projection onto their span, TK(f)=∥f−ΠKf∥H1(Ω)T_K(f)=\lVert f-\Pi_Kf\rVert_{H^1(\Omega)} the truncation error, ck(f)=⟨f,ϕk⟩H1(Ω)c_k(f)=\langle f,\phi_k\rangle_{H^1(\Omega)} the exact coefficients, and

c^k(f)=QN(fϕk+λ2f′ϕk′),LK(f)=(c^0(f),…,c^K−1(f))∈RK,(10)\hat{c}_k(f)=Q_N(f\phi_k+\lambda^2f'\phi_k'),\qquad L_K(f)=(\hat{c}_0(f),\ldots,\hat{c}_{K-1}(f))\in\mathbb{R}^K, \tag*{(10)}

so that each coordinate of LK(f)L_K(f) is the discrete quadrature of ⟨f,ϕk⟩H1(Ω)\langle f,\phi_k\rangle_{H^1(\Omega)}, and LKL_K is linear in ff.

Proposition 2. For f,g∈H1(Ω)f,g\in H^1(\Omega) whose values and derivatives are defined at the nodes, let e(f)=∥LK(f)−c(f)∥e(f)=\lVert L_K(f)-c(f)\rVert be the quadrature error of the coefficients. Then

∣LK(f)TLK(g)−⟨f,g⟩H1(Ω)∣≤TK(f)TK(g)+e(f)∥LK(g)∥+∥f∥H1(Ω)e(g).(11)\lvert L_K(f)^\mathsf{T}L_K(g)-\langle f,g\rangle_{H^1(\Omega)}\rvert\le T_K(f)T_K(g)+e(f)\lVert L_K(g)\rVert+\lVert f\rVert_{H^1(\Omega)}e(g). \tag*{(11)}

Moreover, TK(f)→0T_K(f)\to0 as K→∞K\to\infty; e(f)=0e(f)=0 if ff is a polynomial of degree pp with p+K≤2Np+K\le2N; and if ff and f′f' are analytic in EρE_\rho with ∣fϕk+λ2f′ϕk′∣≤Mk(f)\lvert f\phi_k+\lambda^2f'\phi_k'\rvert\le M_k(f) there, then

e(f)≤3215ρ−2Nρ2−1(∑k<KMk(f)2)1/2.e(f)\le\frac{32}{15}\frac{\rho^{-2N}}{\rho^2-1}\left(\sum_{k<K}M_k(f)^2\right)^{1/2}.

Proof. Since f−ΠKff-\Pi_Kf and g−ΠKgg-\Pi_Kg are orthogonal to the span of the ϕk\phi_k, the cross terms vanish in

⟨f,g⟩H1=⟨ΠKf,ΠKg⟩H1+⟨f−ΠKf,g−ΠKg⟩H1,\langle f,g\rangle_{H^1}=\langle\Pi_Kf,\Pi_Kg\rangle_{H^1}+\langle f-\Pi_Kf,g-\Pi_Kg\rangle_{H^1},

and by orthonormality ⟨ΠKf,ΠKg⟩H1=c(f)Tc(g)\langle\Pi_Kf,\Pi_Kg\rangle_{H^1}=c(f)^\mathsf{T}c(g), which is the Parseval identity on the span. The Cauchy–Schwarz inequality gives ∣c(f)Tc(g)−⟨f,g⟩H1∣≤TK(f)TK(g)\lvert c(f)^\mathsf{T}c(g)-\langle f,g\rangle_{H^1}\rvert\le T_K(f)T_K(g). Next, $L_K(f)^\mathsf{T}L_K(g)-

Input: 64 noisy samples. Output: an equation y(k)=∑jajTjy^{(k)} = \sum_j a_j T_j, T∈{y,…,y(k−1),y2,y3,1}T \in\{y,\ldots,y^{(k-1)},y^2,y^3,1\}.

  1. Build the library of the route: spectral derivatives of the samples (strong form), weak-form projections on 40 test functions, or exact derivatives of the formula returned by Algorithm 2 (samples →\to formula).

  2. Fit all supports of at most 4 terms by least squares, rank them by relative residual, and snap each coefficient to the fraction with the smallest denominator that keeps the residual within 5%.

  3. Return the top-ranked equation, on which valid@1 is scored.

Algorithm 5: ODE discovery from noisy samples (one route)

c(f)⊤c(g)=(LK(f)−c(f))⊤LK(g)+c(f)⊤(LK(g)−c(g))\mathbf{c}(f)^\top\mathbf{c}(g) = (L_K(f)-\mathbf{c}(f))^\top L_K(g)+\mathbf{c}(f)^\top(L_K(g)-\mathbf{c}(g)), whose absolute value is at most e(f)∥LK(g)∥+∥c(f)∥e(g)e(f)\lVert L_K(g)\rVert+\lVert\mathbf{c}(f)\rVert e(g), and ∥c(f)∥≤∥f∥H1\lVert\mathbf{c}(f)\rVert\le\lVert f\rVert_{H^1} by the Bessel inequality. The triangle inequality gives (11). Since polynomials are dense in H1(Ω)H^1(\Omega) on a bounded interval, TK(f)→0T_K(f)\to0. If ff is a polynomial of degree pp, the integrands fϕkf\phi_k and f′ϕkf'\phi_k have degree at most p+K−1≤2N−1p+K-1\le2N-1, so QNQ_N is exact and e(f)=0e(f)=0. Finally, applying (9) to each coordinate and summing the squares gives the analytic bound. □

Since LKL_K is linear and reproduces the inner product up to (11), it has all the properties sought for the latent space, and for K≤dK\le d it can be embedded in Rd\mathbb{R}^d by padding with zeros.

Learned encoder The encoder does not compute LKL_K: its direction is produced by a transformer from the symbolic text. Its latent (Equation 3) uses the quadrature norm n^(f)=QN(f2+λ2f′2)1/2\hat n(f)=Q_N(f^2+\lambda^2 f'^2)^{1/2}, computed from the values of ff and of its symbolic derivative f′f' at the nodes, so that no numerical differentiation is involved. Let G^(f,g)=QN(fg+λ2f′g′)\hat G(f,g)=Q_N(fg+\lambda^2 f'g') be the quadrature Gram entry, C^(f,g)=G^(f,g)/(n^(f)n^(g))\hat C(f,g)=\hat G(f,g)/(\hat n(f)\hat n(g)) the target cosine of Equation 5, and α(f,g)=∣uf⊤ug−C^(f,g)∣\alpha(f,g)=|\mathbf{u}_f^\top\mathbf{u}_g-\hat C(f,g)| the angular error of the encoder.

Proposition 3. For the latent of (3),

∣zf⊤zg−⟨f,g⟩H1(Ω)∣≤n^(f)n^(g)α(f,g)+∣G^(f,g)−⟨f,g⟩H1(Ω)∣,(12)\left|\mathbf{z}_f^\top\mathbf{z}_g-\langle f,g\rangle_{H^1(\Omega)}\right|\le\hat n(f)\hat n(g)\alpha(f,g)+\left|\hat G(f,g)-\langle f,g\rangle_{H^1(\Omega)}\right|, \tag*{(12)}

and if fg+λ2f′g′fg+\lambda^2f'g' is analytic in EρE_\rho and bounded there by MfgM_{fg}, the last term is at most 3215Mfgρ−2N/(ρ2−1)\frac{32}{15}M_{fg}\rho^{-2N}/(\rho^2-1).

Proof By (3), zf⊤zg=n^(f)n^(g)uf⊤ug=n^(f)n^(g)(uf⊤ug−C^(f,g))+G^(f,g)\mathbf{z}_f^\top\mathbf{z}_g=\hat n(f)\hat n(g)\mathbf{u}_f^\top\mathbf{u}_g=\hat n(f)\hat n(g)(\mathbf{u}_f^\top\mathbf{u}_g-\hat C(f,g))+\hat G(f,g), since n^(f)n^(g)C^(f,g)=G^(f,g)\hat n(f)\hat n(g)\hat C(f,g)=\hat G(f,g) by definition. Subtracting ⟨f,g⟩H1\langle f,g\rangle_{H^1} and applying the triangle inequality gives (12), and (9) bounds the last term. □

Corollary 1. With q(f,g)=∣G^(f,g)−⟨f,g⟩H1(Ω)∣q(f,g)=|\hat G(f,g)-\langle f,g\rangle_{H^1(\Omega)}|,

∣∥zf−zg∥2−∥f−g∥H1(Ω)2∣≤2n^(f)n^(g)α(f,g)+q(f,f)+q(g,g)+2q(f,g).\left|\|\mathbf{z}_f-\mathbf{z}_g\|^2-\|f-g\|_{H^1(\Omega)}^2\right|\le2\hat n(f)\hat n(g)\alpha(f,g)+q(f,f)+q(g,g)+2q(f,g).

Proof Expand ∥zf−zg∥2=n^(f)2+n^(g)2−2zf⊤zg\|\mathbf{z}_f-\mathbf{z}_g\|^2=\hat n(f)^2+\hat n(g)^2-2\mathbf{z}_f^\top\mathbf{z}_g and ∥f−g∥H12=∥f∥H12+∥g∥H12−2⟨f,g⟩H1\|f-g\|_{H^1}^2=\|f\|_{H^1}^2+\|g\|_{H^1}^2-2\langle f,g\rangle_{H^1}, use n^(f)2=G^(f,f)\hat n(f)^2=\hat G(f,f), and apply Proposition 3 to the cross term. □

Thus, the only term in Equation 12 that depends on the network is the angular error α\alpha, whose square, averaged over the pairs of a batch, is the first term of the geometry objective (Equation 5).

Scaling laws of the composition operators and of the integrator

The composition operators and the integrator, whose construction is given in Appendix A.3, satisfy the scaling laws of their targets exactly for positive scalars.

Proposition 4. For every z≠0z\ne0 and every c>0c>0, Msq(cz)=c2Msq(z)\mathcal{M}_{\mathrm{sq}}(cz)=c^2\mathcal{M}_{\mathrm{sq}}(z), Mcube(cz)=c3Mcube(z)\mathcal{M}_{\mathrm{cube}}(cz)=c^3\mathcal{M}_{\mathrm{cube}}(z), Msqrt(cz)=c1/2Msqrt(z)\mathcal{M}_{\mathrm{sqrt}}(cz)=c^{1/2}\mathcal{M}_{\mathrm{sqrt}}(z), Mlog⁡(cz)=Mlog⁡(z)+(log⁡c)z1\mathcal{M}_{\log}(cz)=\mathcal{M}_{\log}(z)+(\log c)z_1, and the integrator satisfies I(cz)=cI(z)\mathcal{I}(cz)=c\mathcal{I}(z).

Proof For c>0c>0, ∥cz∥=c∥z∥\|cz\|=c\|z\| and cz/∥cz∥=u^cz/\|cz\|=\hat{\mathbf{u}}. For these operators x=u^\mathbf{x}=\hat{\mathbf{u}}, so κh\kappa_h and δh\delta_h are unchanged, and (8) gives eκh+phlog⁡(cn)δh=cpheκh+phlog⁡nδhe^{\kappa_h+p_h\log(c^n)}\delta_h=c^{p_h}e^{\kappa_h+p_h\log n}\delta_h, while the affine term of the logarithm becomes (log⁡c+log⁡n)z1(\log c+\log n)z_1. The integrator I(z)=∥z∥eκI(u^)δI\mathcal{I}(z)=\|z\|e^{\kappa_{\mathcal{I}}(\hat{\mathbf{u}})}\delta_{\mathcal{I}} is the case p=1p=1. □

Additional experimental results

A second encoder trained on [0,5][0,5]

In order to check that learning a Sobolev space does not depend on the interval, a second encoder E[0,5]E_{[0,5]} is trained once, from scratch, with the configuration of E[−π,π]E_{[-\pi,\pi]} (Table 11) on 2.0M functions generated on [0,5][0,5], and its checkpoint is selected on its validation split at step 38,000 of 40,000. Table 13 compares both encoders, each on the held-out test split of its own interval; since the two test sets contain different functions, the differences mix the effect of the encoder with that of the data. E[0,5]E_{[0,5]} matches E[−π,π]E_{[-\pi,\pi]} in geometry, scaling, linearity and derivative retrieval, and it is even slightly better on several of these metrics (cosine correlation 0.996 against 0.991, scale error at c=−1c=-1 of 0.037 against 0.054, error on random sums 0.141 against 0.216), so the approach does not depend on the interval [−π,π][-\pi,\pi]. The values of E[−π,π]E_{[-\pi,\pi]} in Table 13 are computed on a smaller sample than those of Table 1, hence the small differences between the two tables. It is weaker in the zero-shot operator algebra: the annihilation residual of ODEs is 2.5 times larger, the chain error of D2D^2 is 32% larger and the effective rank of the directions is lower. The family probe is also lower, which we attribute to the data rather than to the encoder, since 20% of the functions generated on [0,5][0,5] are numerically close to linear, against 11% on [−π,π][-\pi,\pi]. A shorter interval was discarded for the same reason: on [0,1][0,1], 44% of the generated functions are close to linear and 5.4% of random pairs have ∣cos⁡∣>0.99|\cos|>0.99, against 2.4% for the training data of E[0,5]E_{[0,5]} and 0.6% for that of E[−π,π]E_{[-\pi,\pi]}.

MetricE[−π,π]E_{[-\pi,\pi]}E[0,5]E_{[0,5]}
H1H^1 cosine correlation0.9910.996
H1H^1 inner-product correlation0.9991.000
effective rank of uu54.745.4
scale error, c=10c=100.0150.012
scale error, c=−1c=-10.0540.037
linearity error (combinations)0.0700.070
error on random sums0.2160.141
derivative retrieval, top-1 (4,000)0.8200.825
derivative retrieval, top-5 (4,000)0.9600.956
relative error of D\mathcal{D} (median)0.0700.076
gain ratio of D\mathcal{D}1.0040.991
D2D^2 chain error0.2400.316
D(D(zf))\mathcal{D}(\mathcal{D}(z_f)), top-10.4580.398
ODE annihilation0.2080.530
eigenfunction error, trigonometric0.1740.258
sweep ordering (Spearman)0.8280.868
linear probe, family accuracy0.7420.445
calibrated εrel\varepsilon_{\mathrm{rel}} (own validation split)0.00510.0038

Table 13. E[−π,π]E_{[-\pi,\pi]} and E[0,5]E_{[0,5]}, each on the test split of its own interval (1,024 functions for the geometry, scaling and ODE rows, 4,000 for retrieval). Errors are relative

Figure 11 shows the latent of E[0,5]E_{[0,5]} next to that of E[−π,π]E_{[-\pi,\pi]}, with the projection described in Appendix C.8, trained in the same configuration on 120k training directions of E[0,5]E_{[0,5]}: no angle term and radius equal to the rank of log⁡∥f∥H1\log\lVert f\rVert_{H^1}. On the test split, the mean of 1−cos⁡(uf,u^f)1-\cos(\mathbf{u}_f,\hat{\mathbf{u}}_f) is 0.182 for k=2k=2 and 0.080 for k=3k=3 (median angular errors of 20∘20^\circ and 10∘10^\circ), lower than for E[−π,π]E_{[-\pi,\pi]} because the directions of E[0,5]E_{[0,5]} have a lower effective rank (Table 13), and the validation family accuracy is 0.367 and 0.448, against 0.431 and 0.518 for E[−π,π]E_{[-\pi,\pi]}.

Calibration of the ε\varepsilon-criterion

Figure 6 shows the data behind the threshold of § 3.2, computed on the 19,765 distinct functions of the validation split. The relative latent gap to the nearest distinct function and the corresponding H1H^1 gap have almost the same distribution (medians 0.128 and 0.121), and the latent and H1H^1 nearest neighbours coincide for 77% of the functions. The relative error of D\mathcal{D} lies between the 5th percentile and the median of these gaps, which is why retrieval succeeds while the ε\varepsilon-ball is never reached. The absolute root mean square gaps span 2.5 orders of magnitude between their 5th and 95th percentiles, because the norms of the functions do, which rules out a single absolute threshold. Finally, latent distances reproduce H1H^1 distances on random pairs (Pearson 0.99997), but only to a median relative error of 7% on nearest-neighbour pairs, so that very close functions are not resolved as accurately as distant ones. When the criterion is applied to a formula rather than to a latent, the relative H1H^1 error of the formula is compared with the threshold obtained by the same rule from the function-space gaps, 0.0037.

Table 14 re-scores the predictions of every learned operator with this criterion. No prediction reaches ε\varepsilon: the smallest relative error of DD on the validation split is 0.013, whereas 67.1% of its predictions are within 0.1 of their target, against a top-1 accuracy of 82.0%.

Figure 6. Calibration of the ε\varepsilon-criterion on the validation split: (a) relative gaps to the nearest distinct function and relative error of DD, (b) the same quantities as absolute root mean square distances, and (c) latent against H1H^1 distance

Numerical check of the inner-product bound

Figure 7 measures both terms of Equation (12) for E[−π,π]E_{[-\pi,\pi]} on all pairs of 2,000 test functions, against a reference inner product computed with 512 nodes. The quadrature error decays geometrically with the number of nodes and is negligible from N=64N = 64 (median 6⋅10−156 \cdot10^{-15} relative to ∥f∥H1∥g∥H1\lVert f\rVert_{H^1}\lVert g\rVert_{H^1}), whereas the error of the encoder stays at its angular floor (median 0.0083) for every N≥32N \ge32, so that the error of the latent inner product is the angular error of the network. Since the encoder with another latent dimension was trained, panel (b) uses the projection of the latents of E[−π,π]E_{[-\pi,\pi]} on their top kk principal directions as a proxy: the error reaches the floor of the encoder at k≈64k \approx64, whereas a polynomial truncation of degree below kk, as in Proposition 2, and the optimal rank-kk approximation of the Gram matrix reach machine precision at the same dimension, so the 256 dimensions of the latent space are not the limiting factor.

Figure 7. Median relative error of the latent inner product on all pairs of 2,000 test functions: (a) against the number NN of quadrature nodes, and (b) against the dimension kk of a truncated representation

The propositions bound the inner-product error by α\alpha, but they do not bound α\alpha itself, which depends on training and is measured empirically: its mean over held-out pairs is 0.017 (Table 1).

Decoder and module predictions

Figure 8 and Table 15 detail the decoder. Errors are dominated by constants and by long expressions, whereas the function shape is correct in most cases. Second derivatives decoded from D(D(zf))D(D(z_f)) have the correct shape for 39.9% of the functions, although they are never seen by the encoder or the decoder.

Figure 8. Decoder accuracy on the test set: (a) exact reconstruction and correct shape for ff from zfz_f, f′f' from zf′z_{f'} and ff from the predicted latent D(zf)D(z_f), and (b) exact reconstruction of ff by expression length

Table 15: Examples of test functions and their decoding from zfz_f (beam search with round-trip re-ranking); the last column gives the relative L2L^2 error when the decoding is not exact

True functionDecoded functionError
4xe9x243\frac{4xe^{\frac{9x^2}{4}}}{3}4xe9x243\frac{4xe^{\frac{9x^2}{4}}}{3}✓
−21e−11x25cos⁡(9x)-21e^{-\frac{11x^2}{5}}\cos(9x)−21e−11x25cos⁡(9x)-21e^{-\frac{11x^2}{5}}\cos(9x)✓
3e−13x210sin⁡(11x)3e^{-\frac{13x^2}{10}}\sin(11x)3e−13x210sin⁡(11x)3e^{-\frac{13x^2}{10}}\sin(11x)✓
sin⁡3(x+5)\sin^3(x+5)sin⁡(sin⁡(x+5))\sin(\sin(x+5))4.0e-01
−10e−17(x−12)22-10e^{-\frac{17(x-\frac{1}{2})^2}{2}}−10e−17(x−12)22-10e^{-\frac{17(x-\frac{1}{2})^2}{2}}✓
\frac{16x^2e^{-\frac{9x^2}{4}}}{\frac{13x}{5}+83\atan(4x)}\frac{16x^2e^{-\frac{9x^2}{4}}}{3x+83\atan(4x)}6.4e-03
−70sin⁡(6x)+33sin⁡(10x)+6sin⁡(12x)+3cos⁡(5x)-70\sin(6x)+33\sin(10x)+6\sin(12x)+3\cos(5x)−66sin⁡(6x)+34sin⁡(10x)+3cos⁡(3x)-66\sin(6x)+34\sin(10x)+3\cos(3x)1.1e-01
(−7ex−20sin⁡(3x))e−x(-7e^x-20\sin(3x))e^{-x}−7−20e−xsin⁡(3x)−e−xcos⁡(3x)2-7-20e^{-x}\sin(3x)-\frac{e^{-x}\cos(3x)}{2}2.8e-02
32cos⁡(6x)+832\cos(6x)+832cos⁡(6x)+832\cos(6x)+8✓

Table 15. Examples of test functions and their decoding from zfz_f (beam search with round-trip re-ranking); the last column gives the relative L2L^2 error when the decoding is not exact

Figure 9 compares predicted and true quantities for every module. The integrator predicts the magnitude of the antiderivative with a log-correlation of 0.995 (median absolute log error 0.021), the composition operators follow the true gains over five orders of magnitude, and the median cosine between predicted and true latents is 0.998 for DD, 0.985 for II, between 0.964 (cos gg) and 0.995 (log gg) for the composition operators, and 0.994 for VV with and without noise.

Figure 9. Predicted against true quantities on the test sets: (a) gain of the integrator, (b) gain of the composition operators, and (c) distribution of 1−cos⁡1-\cos between predicted and true latents for every module

Table 16 gives the comparison of the learned operators with the symbolic path discussed in § 4.2.

Symbolic regression: pipeline levels and additional comparisons

Table 17 reports the exact recovery of each level of Algorithm 2 on 300 test functions. The value encoder is robust to noise, with a median cosine similarity of 0.994 to the true latent, and under noise the additional search levels do not change the selected formula, since candidates with more atoms do not pass the description-length criterion. Excluding the test functions that lie within a relative H1H^{1} distance of 10−310^{-3} of a training function gives 0.80, 0.63, and 0.43 instead of 0.80, 0.63, and 0.45.

Table 18 is an earlier comparison on a fixed subset of 100 test functions with PySR [14] and the end-to-end transformer [23], with three accuracy criteria. PySR fits the samples accurately but recovers the exact formula only for 3% of the clean functions, and the end-to-end transformer, designed for numerical accuracy rather than exact constants, is within 10% of the target for 37% to 44% of the functions. The exact rates of f-JEPA differ between the subsets of 300, 100, and 50 functions (Table 17, Table 18, and Table 5) because of their size; on a rerun of the noisy pipeline, the rate at σ=10−2\sigma= 10^{-2} on the 300 functions changed from 0.447 to 0.473, which gives the order of the run-to-run variation.

Methodσ=0\sigma= 0 Exactσ=0\sigma= 0 <10−3<10^{-3}σ=0\sigma= 0 <10−1<10^{-1}σ=10−3\sigma= 10^{-3} Exactσ=10−3\sigma= 10^{-3} <10−3<10^{-3}σ=10−3\sigma= 10^{-3} <10−1<10^{-1}σ=10−2\sigma= 10^{-2} Exactσ=10−2\sigma= 10^{-2} <10−3<10^{-3}σ=10−2\sigma= 10^{-2} <10−1<10^{-1}
E2E transformer [23]0.000.000.390.000.000.440.000.000.37
PySR [14]0.030.630.980.000.660.980.000.020.94
f-JEPA0.770.930.970.670.860.980.550.570.99
True latent zfz_f instead of V\mathcal{V} (upper bound)0.840.930.97––––––

Table 18. Symbolic regression from 64 samples on a fixed subset of 100 test functions: share of formulas recovered exactly and with a relative error below 10−310^{-3} and below 10−110^{-1}. The last row decodes the true latent of the formula, which does not depend on the samples

Table 19 gives the out-of-distribution comparison of § 4.7, on random trees generated like the integrands of Table 26 but disjoint from them, without poles on [−π,π][-\pi,\pi]. PySR, whose genetic search is not tied to a training distribution, is the strongest method on clean samples, but at 1% noise its formulas fit the noise. On the 24 classic univariate benchmarks Nguyen [47], Keijzer [25], and Livermore [37], the 7 that require a positive domain being composed with an affine map to [−π,π][-\pi,\pi], PySR recovers 9 formulas exactly on clean samples but a single one at 1% noise, whereas f-JEPA recovers 8 at both noise levels.

Per-equation and per-integrand results

Table 20 to Table 24 report the per-equation and per-integrand results that are summarised in § 4. For instance, the best approximation of ∫esin⁡x dx\int e^{\sin x}\,dx is 1.266 x−1.266sin⁡(cos⁡x)1.266\,x - 1.266\sin(\cos x) plus harmonics, where 1.266=I0(1)1.266 = I_0(1) is the mean value of esin⁡xe^{\sin x}.

Figure 10 complements Table 6: the deviation of the learned formulas decreases steadily with the number of atoms and stays one to three orders of magnitude below that of the Padé approximants for every k≥2k \ge2. In median, the learned and network-free dictionaries are close for most kk, with the clearest advantage of the learned atoms at k=4k = 4 and k=7k = 7; at k=4k = 4 the learned formula is more accurate than the best network-free control on 11 of the 12 equations, and more accurate than Padé on all of them.

Figure 10. Initial-value problems without closed form: (a) median deviation from the numerical solution against the number of atoms kk, and (b) deviation of the best control and of the Padé approximant divided by that of the best learned method for each ODE at k=4k = 4

As an independent check that does not rely on the numerical reference solution, Table 23 reports for the same four-atom formulas (stored with four significant digits) the scale-free ODE residual max⁡x∣y(k)−∑jajTj∣/(max⁡x∣y∣+∑j∣aj∣max⁡x∣Tj∣)\max_x \lvert y^{(k)} - \sum_j a_j T_j\rvert/(\max_x \lvert y\rvert+ \sum_j \lvert a_j\rvert\max_x \lvert T_j\rvert) over 256 points of [−π,π][-\pi,\pi] and the error in the initial conditions at x0=0x_0 = 0. All formulas satisfy the initial conditions to about 10−410^{-4}, and none of them solves the equations exactly, as expected for compact approximations; latent peeling and composed atoms have the smallest residuals (3.9% and 4.3%), against 7.2% for the best network-free control.

MethodODEsMedian residualMedian IC error
Random atoms (control)121.5e-011.0e-03
Value-space neighbours (control)127.2e-022.0e-04
Value-space peeling (control)121.0e-013.2e-04
Latent neighbours121.1e-012.4e-04
Latent ODE shortlist111.0e-012.9e-04
Latent peeling123.9e-029.4e-05
Latent shortlist + atoms111.0e-012.9e-04
+ composed atoms124.3e-022.6e-04

Table 23. Residual and initial-condition check of the four-atom formulas for the 12 initial-value problems without closed form (medians over the ODEs; formulas with four significant digits).

Table 25 lists the out-of-distribution integrands of Table 26 for which the learned levels of our solver return an antiderivative verified by SymPy, whereas SymPy’s own integrate returns none within 10 seconds. In every case the antiderivative is a nested composition, which the learned integrator reaches after the substitution or by-parts step has reduced the integrand to a function close to the training families.

Table 26 and Table 27 give the complete results of the out-of-distribution integration and of the ODEBench discovery discussed in § 4.7. On ODEBench, the stricter threshold R2>0.99R^{2} > 0.99 gives the same ranking of the methods as R2>0.9R^{2} > 0.9.

Why few ODE solutions of Lample and Charton are verified

Table 28 analyses the 400 initial-value problems of Table 9 from their reference solutions and the formulas returned by the samples-to-formula route of f-JEPA (original decoder, no ODE-aware search) (‘numerically exact’: relative error below 10−610^{-6} with respect to the reference solution on the 64 nodes, but not proven exact by SymPy). The reference solutions have the length of our training expressions and use only operators of our vocabulary, so the failures are not due to length or to unknown operators. Two causes account for them. First, a quarter to a third of the references contain irrational constants coming from the integration constants of the generator of Lample and Charton (for instance x4sin⁡9+1x^4\sin9+1 or −4x+arctan⁡x−4+cos⁡4-4x+\arctan x-4+\cos4); f-JEPA recovers about 40% of these formulas with a rational approximation of the constant (107171260049x4+1\frac{107171}{260049}x^4+1, relative error 3⋅10−103\cdot10^{-10}), which no exact test can accept. Second, for references with rational constants, about three quarters of the formulas have the wrong shape, because these solutions combine products and compositions of elementary functions that are not among the parameterised families of the pretraining data. The last row of Table 28 shows that the limitation lies in the pretrained encoder and decoder rather than in the value encoder: when the latent is computed from the symbolic form of the reference solution and given to the same solver, only 7.5% and 4.5% of the solutions are recovered exactly (34% and 32% within a relative error of 10−210^{-2}), against 87% for the functions of our test set. Both causes are addressed by the two additions of Table 9: the decoder and the value encoder are fine-tuned on expressions of the Lample and Charton training sets, with the encoder and D\mathcal{D} frozen, and the selection of a single formula is replaced by an ODE-aware search. The search keeps the whole candidate pool of the samples-to-formula solver, refits the coefficients of every candidate on the numerical solution, identifies them as exact constants by integer-relation detection (PSLQ) against e±ke^{\pm k}, sin⁡k\sin k, cos⁡k\cos k, log⁡k\log k, k\sqrt{k}, and solves the initial conditions exactly for one or two coefficients, and ranks the resulting formulas by their residual in the equation. Under the stricter symbolic proof, the two additions raise the verified shares from 5% and 5.5% to 16.5% and 18%. Ranking by the latent residual under D\mathcal{D} instead of the numeric residual was slightly worse in a preliminary run on 40 IVPs per set, since the noise floor of D\mathcal{D} cannot separate the exact formula from near-exact ones. The ODE-aware search identifies the irrational constants exactly (for instance it returns x4sin⁡9+1x^4\sin9+1 and x6+(2−e3)x+1x^6+(2-e^3)x+1), and fine-tuning the decoder on expressions of the Lample and Charton training sets raises the exact recovery from the true latent on the first 40 IVPs of each set from 15% to 25% (ODE1) and from 5% to 8% (ODE2). On ODE1, the remaining failures of the final route are formulas of the wrong shape (53.5% of the IVPs) or close to the reference without being exact (21.5%), and none of them solves the equation with initial conditions of another solution, so the IVP formulation itself is not a source of error.

ODE1ODE2
Median prefix length of the reference solution (pretraining: 18)1919
Reference beyond the 99th percentile of pretraining lengths0.0250.000
Reference with irrational constants0.3550.245
f-JEPA verified0.0000.020
f-JEPA numerically exact, not provable (rel. error <10−6< 10^{-6})0.3940.408
Reference with rational constants only0.6450.755
f-JEPA verified0.0780.066
f-JEPA wrong shape (rel. error ≥10−2\ge10^{-2})0.7360.748
True latent zf\mathbf{z}_f instead of V\mathcal{V}: exact recovery0.0750.045

Table 28. Diagnosis of the IVPs of Table 9 (200 per test set): reference solutions and outcome of f-JEPA; shares within each group of references

Projections of the latent space

The projection of Figure 3 and Figure 11 is built as follows. An autoencoder with layers 256→256→128→k256 \to256 \to128 \to k and k→128→256→256k \to128 \to256 \to256 maps the unit directions uf\mathbf{u}_f to codes cf\mathbf{c}_f normalised to the unit sphere Sk−1\mathbb{S}^{k-1}, and is trained for 60 epochs on the 120k training directions with the loss 1−cos⁡(uf,u^f)1-\cos(\mathbf{u}_f,\hat{\mathbf{u}}_f). The scale is then reintroduced as the radius, cf r(n(f))c_f\,r(n(f)), and the display was chosen on the validation split by the 10-nearest-neighbour accuracy of the function family in the displayed coordinates, among the radii r=log⁡(1+n)r=\log(1+n) and r=0.25+0.75 rank⁡(log⁡n)r=0.25+0.75\,\operatorname{rank}(\log n) and an additional loss term (ciTcj−uiTuj)2(\mathbf{c}_i^{\mathsf{T}}\mathbf{c}_j-\mathbf{u}_i^{\mathsf{T}}\mathbf{u}_j)^2 that preserves the H1H^1 angles. The angle term raises the correlation between code and latent cosines from 0.50 to 0.62 for k=2k=2 and from 0.61 to 0.73 for k=3k=3, but it does not improve the family accuracy, so it is not used. On the test split, the mean of 1−cos⁡(uf,u^f)1-\cos(\mathbf{u}_f,\hat{\mathbf{u}}_f) is 0.283 for k=2k=2 and 0.115 for k=3(medianangularerrorsof33∘k=3 (median angular errors of 33^\circ and 16∘16^\circ), and the validation family accuracy is 0.431 and 0.518, against 0.630 for the UMAP projection of the same functions, which preserves local neighbourhoods better but discards the norms and has no explicit map.

Figure 11. Latents of E[−π,π]E_{[-\pi,\pi]} (left) and E[0,5]E_{[0,5]} (right) projected in two (top) and three (bottom) dimensions, each for (a) 120k training functions and (b) 20.4k test functions of its own interval. The construction and colouring are those of Figure 3, which is the top left panel

Time and memory consumption

Table 29 reports the training cost of each component of f-JEPA; peak GPU memory during encoder training was 2.7 GB with gradient checkpointing. All components together have 120M parameters (encoder and DD 35.1M, decoder 17.7M, composition bank 52.9M, integrator 7.6M and value encoder 7.4M), and every one of them is trained on a single 8 GB laptop GPU.

ComponentParametersStepsTime
Encoder and DD35.1M40k6.6 h (0.6 s/step)
Decoder17.7M105k3.5 h (0.11 s/step)
Composition bank (7 operators)52.9M40k1 h
Integrator7.6M60k13 min (0.013 s/step)
Value encoder (fine-tuning)7.4M20k39 min

Table 29. Training cost of the components of f-JEPA on a single RTX 4070 Laptop GPU (8 GB)

Table 30 compares the inference cost and the size of f-JEPA with those of the other methods, measured in our own runs on the same machine (16 CPU cores and the same GPU). For f-JEPA, the parameters are those of the modules that the task uses: the value encoder and the decoder for symbolic regression and for the tasks that start from samples (ODE solving and discovery), and the encoder with DD, the integrator and the decoder for integration. On every task, f-JEPA has fewer parameters than every pretrained neural network it is compared with: 25.1M against 26.4M (NeSymReS) to 166.7M (SNIP) for symbolic regression, 60.4M against 185.0M for integration and for ODE solving with the models of Lample and Charton, and 25.1M against 60.6M for ODE discovery with ODEFormer. Even all the modules of f-JEPA together (120M) are smaller than a single integrator of Lample and Charton. For symbolic regression, f-JEPA is also the fastest method: a forward pass of the value encoder, a beam search of the decoder and a few least-squares refits take 0.44 seconds per function, whereas the search-based methods use their whole budget (PySR and Operon, 20 seconds), SNIP refines its latent by an evolutionary search (23 seconds) and MMSR needs 93 seconds; the pretrained transformers E2E and NeSymReS take 2.0 and 1.6 seconds. For integration and ODE solving, in contrast, f-JEPA is slower than the sequence-to-sequence models, since it explores several routes and verifies every candidate with SymPy: 3.4 seconds per integrand against 0.15 to 0.91 seconds for Lample and Charton, and 1.4 seconds per IVP against 0.29 seconds. In terms of memory, the integration solver peaks at 4.1 GB of GPU memory, most of which holds the latents of the 2M-function pool used for the shortlists.

TaskMethodSeconds per itemParameters
Symbolic regressionf-JEPA (value encoder and decoder)0.4425.1M
Symbolic regressionNeSymReS [7]1.626.4M
Symbolic regressionE2E transformer [23]2.093.5M
Symbolic regressionOperon [9] (20 s budget)20.0–
Symbolic regressionPySR [14] (20 s budget)20.2–
Symbolic regressionSNIP [34]23.1166.7M
Symbolic regressionMMSR [30]92.790.0M
Integrationf-JEPA (encoder, DD, integrator, decoder)3.460.4M
IntegrationLample and Charton [28]0.15–0.91185.0M
IntegrationSymPy integrate0.03–0.43–
ODE solving (IVPs)f-JEPA (value encoder and decoder)1.4325.1M
ODE solving (IVPs)Lample and Charton [28]0.29185.0M
ODE solving (IVPs)SymPy dsolve4.34–
ODE discoveryf-JEPA (value encoder and decoder)3.2125.1M
ODE discoveryODEFormer [16]0.5460.6M
ODE discoverySINDy (PySINDy)0.03–
ODE discoveryWeak SINDy (PySINDy)0.35–
ODE discoveryEstimated derivatives + PySR20.3–

Table 30. Inference time per item (median wall-clock seconds on the same machine) and number of parameters of the networks used by each method. For f-JEPA on symbolic regression and integration, the time is the mean over the 300 test items

Galleries of predictions

The following figures show unselected examples (the first test functions in a fixed random order, or every problem of a benchmark), except Figure 13, whose examples are selected as described in its caption; the examples already shown in Figure 4 and Figure 5 are not repeated. In Figure 4, the panels show the first exact (a) and the first inexact (b) reconstruction of ff, the first exact (c) and an inexact (d) decoding of D(zf)\mathcal{D}(z_f) with the correct shape, the first decoding of D(D(zf))\mathcal{D}(\mathcal{D}(z_f)) with the correct shape (e), and the first inexact decoding of I(zf)\mathcal{I}(z_f) (f), with antiderivatives shown with zero mean; in Figure 5, panels (a) and (b) are the first exact and the first inexact recovery. The decoded formula is the top candidate of beam search without re-ranking or constant refitting, so these examples include the failure modes of the latent operators and of the decoder. Below each panel, the true formula is written in black and the produced formula in orange (floating-point coefficients rounded to three significant digits; very long sums are truncated).

Figure 13. Composition outputs Mh(zg)M_h(z_g) decoded to formulas, two per operator in the order sin, cos, log, x\sqrt{\phantom{x}}, square, cube, and quadratic. These are the decodings with the smallest error among the first 150 random test pairs, and are not representative of the average decoding

In this unselected sample (Figure 12) no derivative is decoded exactly, in agreement with the 6.8% exact rate of § 4.2: 6 of the 11 decoded derivatives have the correct family and frequencies with relative errors below 0.25, whereas the other 5 miss a phase, a term or a scale, mostly for oscillatory functions with large coefficients.

Derivatives predicted in latent space and decoded for eleven test functions, showing true and decoded curves with relative errors

Figure 12. Derivatives predicted in latent space and decoded, f′f' from D(zf)\mathcal{D}(z_f), for the first test functions in a fixed random order, without the one shown in Figure 4 (titles give ff)

Decoding the outputs of the composition operators is much harder than retrieving them, since the decoder is not trained on composed latents and the composed expressions are long; nevertheless, Figure 13 shows that for every operator some outputs are decoded into formulas that follow the true function closely, two of them exactly (sin⁡(sin⁡(4x+1))\sin(\sin(4x + 1)) and cos⁡3(4x+2)\cos^{3}(4x + 2)), and that the typical residual error is a constant or a frequency slightly off, whereas the same latents retrieve the correct function for 57% to 92% of the pairs (Table 3).

Figure 14 to Figure 16 complete the picture for the applications: at 1% noise, symbolic regression errors are small constant errors, the five-term approximations of non-elementary antiderivatives deviate by at most 3.1% from the antiderivative obtained by quadrature (Table 24 reports the larger error of their derivative with respect to the integrand), and four-atom formulas track the numerical solutions of all twelve initial-value problems, with the Painlevé-I type equation of Figure 5 as the only case with a visible deviation.

Figure 14. Symbolic regression from 64 samples at σ=10−2\sigma= 10^{-2} for the first 12 test functions in a fixed random order that are not shown in Figure 5 (titles give the true function)

Ten initial-value problems without closed-form solution

Figure 16. The ten initial-value problems without closed-form solution that are not shown in Figure 5: four-atom formula of the best learned method against the numerical solution

Figure 15. The integrands without elementary antiderivative of Table 24 that are not shown in Figure 5: five-term symbolic approximation against cumulative quadrature (relative L2L^2 error of the antiderivative, both curves with zero mean). Panels: ∫cos⁡(x2) dx\int\cos(x^2)\,\mathrm{d}x, relative error 0.0079; formula: 0.583arctan⁡(x)−0.0456sin⁡(22x)+0.536sin⁡(x)/x+0.0248xcos⁡(x)−0.00278xcos⁡(x2)0.583\arctan(x)-0.0456\sin(2\sqrt{2}x)+0.536\sin(x)/x+0.0248x\cos(x)-0.00278x\cos(x^2). ∫e−sin⁡(x) dx\int e^{-\sin(x)}\,\mathrm{d}x, relative error 0.00016; formula: 1.27+1.27x+0.00137sin⁡(4x)−0.0348cos⁡(3x)−1.27sin⁡(cos⁡(x))−0.136sin⁡(2x)+0.296cos⁡(13/8)cos⁡(x)1.27+1.27x+0.00137\sin(4x)-0.0348\cos(3x)-1.27\sin(\cos(x))-0.136\sin(2x)+0.296\cos(13/8)\cos(x). ∫sin⁡(x)/(x2+1) dx\int\sin(x)/(x^2+1)\,\mathrm{d}x, relative error 0.0031; formula: −0.00704cos⁡(4x)+0.000229x3sin⁡(5x)−0.66e−x2/2cos⁡(x/2)−0.0067x2sin⁡(cos⁡(x))…-0.00704\cos(4x)+0.000229x^3\sin(5x)-0.66e^{-x^2/2}\cos(x/2)-0.0067x^2\sin(\cos(x))\ldots. ∫e−x2cos⁡(x) dx\int e^{-x^2}\cos(x)\,\mathrm{d}x, relative error 0.00086; formula: 0.00206sin⁡(5x)+0.486arctan⁡(x)+(0.486x)/(x2+1)+0.0217xcos⁡(11x/10−0.001)+0.143x3e−x2cos⁡(11x/10−0.001)…0.00206\sin(5x)+0.486\arctan(x)+(0.486x)/(x^2+1)+0.0217x\cos(11x/10-0.001)+0.143x^3e^{-x^2}\cos(11x/10-0.001)\ldots. ∫x4+1 dx\int\sqrt{x^4+1}\,\mathrm{d}x, relative error 0.00029; formula: −5.08+0.301x3+0.269cos⁡(13/7)+0.678x+0.015sin⁡(3x)+0.00823sin⁡(4x)−0.015sin⁡(x)+0.269sin⁡(18/7)sin⁡(17x/9)…-5.08+0.301x^3+0.269\cos(13/7)+0.678x+0.015\sin(3x)+0.00823\sin(4x)-0.015\sin(x)+0.269\sin(18/7)\sin(17x/9)\ldots. ∫ecos⁡(x) dx\int e^{\cos(x)}\,\mathrm{d}x, relative error 0.00016; formula: 3.02arctan⁡(x)+0.000378x−0.00508excos⁡(x)sin⁡(4x)−0.00051x2sin⁡(3x)−0.0881xexcos⁡(x)cos⁡(4x/3)3.02\arctan(x)+0.000378x-0.00508e^x\cos(x)\sin(4x)-0.00051x^2\sin(3x)-0.0881xe^x\cos(x)\cos(4x/3). ∫log⁡(sin⁡(x)+2) dx\int\log(\sin(x)+2)\,\mathrm{d}x, relative error 0.0022; formula: 0.109arctan⁡(x+8/5)+0.583x+0.0442sin⁡(2x)+0.00615cos⁡(3x)−0.583cos⁡(x)−0.00378xsin⁡(x)0.109\arctan(x+8/5)+0.583x+0.0442\sin(2x)+0.00615\cos(3x)-0.583\cos(x)-0.00378x\sin(x). ∫ex2 dx\int e^{x^2}\,\mathrm{d}x, relative error 0.0067; formula: 0.0102x5+1.01x+0.0327x3+0.00322sin⁡(6x)−0.0041sin⁡(5x)+…0.0102x^5+1.01x+0.0327x^3+0.00322\sin(6x)-0.0041\sin(5x)+\ldots. ∫x2e−x2 dx\int x^2e^{-x^2}\,\mathrm{d}x, relative error 0.0085; formula: 0.442sin⁡(6x/11)+0.0111cos⁡(3x+2)+0.00506xsin⁡(x)+0.111e−x2sin⁡(2x+3)…0.442\sin(6x/11)+0.0111\cos(3x+2)+0.00506x\sin(x)+0.111e^{-x^2}\sin(2x+3)\ldots. ∫cos⁡(x)/(x2+1) dx\int\cos(x)/(x^2+1)\,\mathrm{d}x, relative error 0.0003; formula: 0.000912sin⁡(4x)+0.00047sin⁡(6x)−0.00581sin⁡(3x)+(2.02x)/(x2+2)+0.0213x2e−x2/10sin⁡(10x/17)0.000912\sin(4x)+0.00047\sin(6x)-0.00581\sin(3x)+(2.02x)/(x^2+2)+0.0213x^2e^{-x^2/10}\sin(10x/17).

References

  1. [1]Miltiadis Allamanis, Pankajan Chanthirasegaram, Pushmeet Kohli, and Charles Sutton. Learning continuous semantic representations of symbolic expressions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 80–88, 2017.arxiv.org/abs/1611.01423
  2. [2]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629, 2023.arxiv.org/abs/2301.08243
  3. [3]George A. Baker and Peter Graves-Morris. Padé Approximants. Cambridge University Press, 2 edition, 1996.
  4. [4]Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025.arxiv.org/abs/2511.08544
  5. [5]Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2105.04906
  6. [6]Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2404.08471
  7. [7]Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascondolo. Neural symbolic regression that scales. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 936–945, 2021.arxiv.org/abs/2106.06427
  8. [8]Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016.DOI
  9. [9]Bogdan Burlacu, Gabriel Kronberger, and Michael Kommenda. Operon C++: An efficient genetic programming framework for symbolic regression. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pages 1562–1570, ACM, 2020. doi: 10.1145/3377929.3398099.DOI
  10. [10]William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabrício Olivetti de França, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason H. Moore. Contemporary symbolic regression methods and their relative performance. In Advances in Neural Information Processing Systems, Track on Datasets and Benchmarks, 2021.arxiv.org/abs/2107.14351
  11. [11]Yang Chang, Kuang-Da Wang, Ping-Chun Hsieh, Cheng-Kuan Lin, and Wen-Chih Peng. DDOT: A derivative-directed dual-decoder ordinary differential equation transformer for dynamic system modeling. arXiv preprint arXiv:2506.18522, 2025.arxiv.org/abs/2506.18522
  12. [12]François Charton. Linear algebra with transformers. Transactions on Machine Learning Research, 2022.arxiv.org/abs/2112.01898
  13. [13]Delong Chen, Mustafa Shukor, Theo Moutakanni, Willy Chung, Jade Yu, Tejaswi Kasarla, Allen Bolourchi, Yann LeCun, and Pascale Fung. VL-JEPA: Joint embedding predictive architecture for vision-language. In International Conference on Learning Representations, 2026. arXiv:2512.10942.arxiv.org/abs/2512.10942
  14. [14]Miles Cranmer. Interpretable machine learning for science with PySR and SymbolicRegression.jl. arXiv preprint arXiv:2305.01582, 2023.arxiv.org/abs/2305.01582
  15. [15]Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Świszcz, and Razvan Pascanu. Sobolev training for neural networks. In Advances in Neural Information Processing Systems, volume 30, pages 4278–4287, 2017.arxiv.org/abs/1706.04859
  16. [16]Stéphane d’Ascoli, Sören Becker, Philippe Schwaller, Alexander Mathis, and Niki Kilbertus. ODEFormer: Symbolic regression of dynamical systems with transformers. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.05573
  17. [17]Brian M. de Silva, Kathleen Champion, Markus Quade, Jean-Christophe Loiseau, J. Nathan Kutz, and Steven L. Brunton. PySINDy: A Python package for the sparse identification of nonlinear dynamical systems from data. Journal of Open Source Software, 5(49):2104, 2020. doi: 10.21105/joss.02104.DOI
  18. [18]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, 2019.arxiv.org/abs/1810.04805
  19. [19]Neeraj Gangwar and Nickvash Kani. Semantic representations of mathematical expressions in a continuous vector space. Transactions on Machine Learning Research, 2023.arxiv.org/abs/2211.08142
  20. [20]Jean-Bastien Grill, Florian Strub, Florent Alché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2006.07733
  21. [21]Tyler Ingebrand, Adam J. Thorpe, and Ufuk Topcu. Zero-shot transfer of neural ODEs. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2405.08954
  22. [22]Divyansh Jha, Yuanfang Xie, Varan Mehra, and Brennen Yu. BRo-JEPA: Learning modular arithmetic in latent space. arXiv preprint arXiv:2606.01372, 2026.arxiv.org/abs/2606.01372
  23. [23]Pierre-Alexandre Kamienny, Stéphane d’Ascoli, Guillaume Lample, and François Charton. End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems, volume 35, 2022.
  24. [24]Alan A. Kaptanoglu, Brian M. de Silva, Urban Fasel, Kadierdan Kaheman, Andy J. Goldschmidt, Jared Callaham, Charles B. Delahunt, Zachary G. Nicolaou, Kathleen Champion, Jean-Christophe Loiseau, J. Nathan Kutz, and Steven L. Brunton. PySINDy: A comprehensive Python package for robust sparse system identification. Journal of Open Source Software, 7(69):3994, 2022. doi: 10.21105/joss.03994.DOI
  25. [25]Maarten Keijzer. Improving symbolic regression with interval arithmetic and linear scaling. In Genetic Programming: 6th European Conference, EuroGP 2003, volume 2610 of Lecture Notes in Computer Science, pages 70–82, 2003.DOI
  26. [26]Bernard O. Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences, 17(5):315–318, 1931.pubmed.ncbi.nlm.nih.gov/16577368
  27. [27]Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizadehnesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89):1–97, 2023.
  28. [28]Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations, 2020.arxiv.org/abs/1912.01412
  29. [29]Yann LeCun. A path towards autonomous machine intelligence. OpenReview preprint, 2022.
  30. [30]Yanjie Li, Jingyi Liu, Min Wu, Lina Yu, Weijun Li, Xin Ning, Wenqiang Li, Meilan Hao, Yusong Deng, and Shu Wei. MMSR: Symbolic regression is a multi-modal information fusion task. Information Fusion, 114:102681, 2025. doi: 10.1016/j.inffus.2024.102681.DOI
  31. [31]Zongyi Li, Nikola Kovachki, Kamyar Azizadehnesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
  32. [32]Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3:218–229, 2021.
  33. [33]Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9:4950, 2018.DOI
  34. [34]Kazem Meidani, Parshin Shojaee, Chandan K. Reddy, and Amir Barati Farimani. SNIP: Bridging mathematical symbolic and numeric realms with unified pre-training. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.02227
  35. [35]Daniel A. Messenger and David M. Bortz. Weak SINDy: Galerkin-based data-driven model selection. Multiscale Modeling & Simulation, 19(3):1474–1497, 2021.DOI
  36. [36]Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, et al. SymPy: Symbolic computing in Python. PeerJ Computer Science, 3:e103, 2017.
  37. [37]T. Nathan Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P. Santiago, Daniel Faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In Advances in Neural Information Processing Systems, volume 34, 2021.
  38. [38]Brenden K. Petersen, Mikel Landajuela Lar​​ma, T. Nathan Mundhenk, Claudio P. Santiago, Soo K. Kim, and Joanne T. Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations, 2021.
  39. [39]Albert Rich, Patrick Scheibe, and Nasser Abbasi. Rule-based integration: An extensive system of symbolic integration rules. Journal of Open Source Software, 3(32):1073, 2018. doi: 10.21105/joss.01073.DOI
  40. [40]Paul Saegert and Ullrich Köthe. Breaking the simplification bottleneck in amortized neural symbolic regression. arXiv preprint arXiv:2602.08885, 2026.arxiv.org/abs/2602.08885
  41. [41]Peter J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics, 656:5–28, 2010.
  42. [42]Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K. Reddy. LLM-SR: Scientific equation discovery via programming with large language models. In International Conference on Learning Representations, 2025.arxiv.org/abs/2404.18400
  43. [43]SymPy Development Team. integration-test-suites: A corpus of symbolic integration problems. https://github.com/sympy/integration-test-suites, 2026. Commit 871364f.
  44. [44]Lloyd N. Trefethen. Approximation Theory and Approximation Practice. SIAM, Philadelphia, 2013.
  45. [45]Silviu-Marian Udrescu and Max Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaba2631, 2020.DOI
  46. [46]Mert Ünsal, Timon Gehr, and Martin Vechev. AlphaIntegrator: Transformer action search for symbolic integration proofs. arXiv preprint arXiv:2410.02666, 2024.arxiv.org/abs/2410.02666
  47. [47]Nguyen Quang Uy, Nguyen Xuan Hoai, Michael O’Neill, R. I. McKay, and Edgar Galván-López. Semantically-based crossover in genetic programming: Application to real-valued symbolic regression. Genetic Programming and Evolvable Machines, 12(2):91–119, 2011.DOI
  48. [48]Marco Valentino, Jordan Meadows, Lan Zhang, and André Freitas. Multi-operational mathematical derivations in latent space. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1446–1458, 2024.
  49. [49]Martin Vastl, Jonáš Kulhánek, Jiří Kubalík, Erik Derner, and Robert Babuška. SymFormer: End-to-end symbolic regression using transformer-based architecture. IEEE Access, 12:37840–37849, 2024.arxiv.org/abs/2205.15764
  50. [50]Jarkko Venna and Samuel Kaski. Neighborhood preservation in nonlinear projection methods: An experimental study. In Artificial Neural Networks – ICANN 2001, volume 2130 of Lecture Notes in Computer Science, pages 485–491. Springer, 2001.DOI
  51. [51]Sean Welleck, Peter West, Jize Cao, and Yejin Choi. Symbolic brittleness in sequence models: On systematic generalization in symbolic mathematics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8629–8637, 2022.arxiv.org/abs/2109.13986
  52. [52]Hongbo Zheng, Suyuan Wang, Neeraj Gangwar, and Nickvash Kani. E-Gen: Leveraging E-graphs to improve continuous representations of symbolic expressions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 11772–11788, 2025.arxiv.org/abs/2501.14951

Paper details

vlnc:W_ek9uvsd4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents