WorksComputer Science
Func-JEPA: Learning a Sobolev Space of Functions as Latent Vectors
Abstract
Pretraining an encoder once and reusing it across tasks is a widespread approach in vision and language. However, neural symbolic mathematics still trains a dedicated sequence-to-sequence model for each task, so that no representation of a function is shared. In this work, we propose the Function Joint-Embedding Predictive Architecture (Func-JEPA, abbreviated f-JEPA): an encoder of mathematical functions, pretrained on their symbolic form, whose latent space imitates the Sobolev space H^1 and in which differentiation is a learned latent operator. The norm of a latent is computed exactly from the function, whereas its direction is learned to match Sobolev angles. The frozen encoder is then reused by lightweight modules for decoding, composition, integration, and regression from samples. The representation and four downstream tasks are evaluated within and outside the pretraining distribution. f-JEPA recovers exact formulas from noisy samples more often than the compared symbolic-regression methods, whereas computer algebra and task-specific models remain stronger on standard integration and differential-equation benchmarks.
Code: https://github.com/amorarc/Func-JEPA
Model: https://huggingface.co/amorarc/func-jepa
Introduction
Nowadays, encoders pretrained once on large corpora and reused for many downstream tasks are the standard approach in language and vision [2, 18]. Neural symbolic mathematics has not followed this path: integration, the solution of ordinary differential equations (ODEs), and symbolic regression are posed as translation problems, each solved by a dedicated sequence-to-sequence transformer trained on its own synthetic corpus [7, 12, 16, 23, 28]. Whilst these models achieve remarkable accuracy, the representation of a function learned for one task is not reused by the others.
A reusable encoder of functions should reflect what a function is in analysis: an element of a vector space with an inner product, on which differentiation, integration, and composition act as operators. Joint-embedding predictive architectures (JEPA) [2, 6, 13, 29] are well-suited to this purpose, since they predict the representation of a target from that of a context: the context is a function , the target is its derivative , and the predictor is the derivative operator. However, a predictive loss alone is minimised by collapsed representations, which current methods avoid with architectural asymmetries [2, 20] or with regularisers on the distribution of the embeddings [4, 5]. For functions, a stronger anchor is available: the inner product of two functions can be computed exactly from their values, so the target geometry of the latent space is defined by the data, a property that existing embeddings of mathematical expressions do not exploit [1, 34].
This work demonstrates that a transformer encoder can be pretrained, from symbolic text alone, into a latent space that is close to an isometric embedding of the Sobolev space and in which the derivative is a learned operator. This leads us to propose the Function Joint-Embedding Predictive Architecture (Func-JEPA, abbreviated f-JEPA throughout this paper): a novel pretrained encoder whose latent vectors factorise into an exactly computed norm and a learned unit direction, trained with an angle-only Sobolev geometry, a linearity objective, and a JEPA derivative objective. The derivative predictor is factored into a gain and a direction, which makes it exactly homogeneous. From this, a set of lightweight modules that reuse the frozen encoder is proposed: a grammar-constrained decoder from latents to formulas, a bank of composition operators, an integrator, and an encoder of sampled values.
In order to evaluate f-JEPA, a series of experiments are conducted at three levels: the pretrained representation, compared with released encoders of symbolic expressions; four downstream tasks on the frozen encoder, namely symbolic integration, symbolic regression, and the solution and discovery of ODEs; and the transfer to out-of-distribution data and standard benchmarks. In every task the learned components are compared with a network-free control that differs only in the learned part and with released models and established tools [14, 16, 17, 23, 28], and every reported solution is verified with SymPy [36] or numerically. As a concrete example, the encoder maps the text of to a latent whose norm is exactly , the operator maps it to a prediction of the latent of , and the decoder turns this prediction into , the exact derivative, without any symbolic differentiation (Figure 4).
Figure 4. True functions (black) against formulas decoded from latents (orange): (a, b) from , (c, d) from , (e) from , and (f) the antiderivative from . The examples are taken in a fixed random order, and more are shown in Appendix § 11
Related Work
Representations of Mathematical Expressions EqNet [1] and later sequence-to-sequence and contrastive encoders [19, 52] learn embeddings in which equivalent expressions are close, using pairs of equivalent rewritings as supervision. SNIP [34] aligns a symbolic and a numeric encoder with a contrastive objective, and Valentino et al. [48] learn mathematical operations, including differentiation, as transformations between expression embeddings evaluated by retrieval. In f-JEPA, the target of the geometry is instead a Sobolev inner product computed from function values, so that norms, angles, and distances have an analytical meaning.
Joint-Embedding Predictive Architectures Methods in this family pretrain encoders by predicting representations rather than inputs [2, 6, 29], and VL-JEPA [13] employs a lightweight text decoder only at inference, which is the template followed by our decoder. Collapse is avoided by stop-gradient and teacher-student asymmetries [20], by variance and covariance penalties [5], or by matching an isotropic Gaussian distribution [4], and BRo-JEPA [22] imposes the circular structure of modular arithmetic on the predictor. In f-JEPA, the context-target pair is a function and its derivative, and collapse is prevented mainly by a data-defined Gram target, with only a weak variance floor.
Transformers for Symbolic Mathematics Lample and Charton [28] showed that sequence-to-sequence transformers trained on synthetic prefix expressions can integrate functions and solve ODEs, and later work studied linear algebra [12]. Such models are brittle outside their training distribution [51], which has motivated search-based alternatives in which a transformer selects rules applied by a symbolic engine [46]. We adopt their prefix notation, but every task shares a single pretrained encoder and is solved by operators and searches in its latent space, followed by exact verification.
Neural Symbolic Regression Transformer-based methods encode a set of points and decode a prefix expression [7, 23, 40, 49], usually followed by constant refinement. Search-based methods include genetic programming [14], recursive decomposition [45], reinforcement learning [37, 38], and large language models proposing equation skeletons [42], many of which are compared in SRBench [10]. In our case, the value encoder is trained to reach the latent of the frozen symbolic encoder, so that symbolic regression shares its decoder with the other tasks.
Operator Learning and Equation Discovery Neural operators learn maps between function spaces from sampled values [27, 31, 32], function encoders represent functions as coefficients on a learned basis [21], and Koopman-based methods search for coordinates in which a nonlinear operator acts linearly [26, 33, 41]. Equation discovery is commonly addressed by sparse regression on libraries of candidate terms [8, 35], and more recently with transformers that output symbolic ODEs [11, 16]. f-JEPA differs in that its input is symbolic and its outputs can be decoded back to exact formulas, and, unlike Sobolev training [15], it employs the Sobolev inner product as the geometry of the latent space rather than as a regression target.
Proposed Method
Function-space geometry
f-JEPA is built in two stages: an encoder of functions is pretrained once from symbolic text (Sections 3.1 to 3.4), and is then frozen and reused by lightweight modules (Section 3.5), as summarised in Figure 1. The objective of the pretraining is that the latent space imitates a Sobolev space of functions, so that the operations of analysis become operations on vectors. For this reason, every structural property of the function space is enforced on the latent space, either by construction or by a training objective:

Figure 1. Modules of f-JEPA and the data flow between them. Solid borders mark the modules pretrained jointly and dashed borders mark the modules trained afterwards on the frozen encoder
Norm: , computed exactly by quadrature (Section 3.2).
Inner product: , since latent angles are fitted to Sobolev angles (geometry objective).
Linearity: , including (linearity objective).
Differentiation: , with an operator that is exactly homogeneous by construction (Section 3.3) and learned with the JEPA objective.
For univariate real functions on an interval , we consider the inner product and its induced norm
The encoder employs the normalised version of Equation (1) on and its Sobolev extension:
with induced norm , where and the integrals are evaluated with the -node Gauss–Legendre rule, using the values of and of its symbolic derivative at the nodes. The geometry is preferred over because differentiation is unbounded on , whereas , which keeps the ratio between the norms of a function and of its derivative well conditioned.
A function encoder on is a map from the symbolic form of to a latent vector whose Euclidean inner product reproduces Equation (2), . Such a representation exists: the coefficients of on an -orthonormal polynomial basis define a linear map whose inner products reproduce Equation (2) up to an explicit quadrature and truncation error (Proposition 2 in Appendix B.2). The encoder therefore has to learn an approximately isometric and linear embedding from symbols alone. Two encoders are trained with the same architecture, objective, and schedule: on , with which all results of Section 4 are obtained, and on , evaluated in Appendix C.1.
The pretraining corpus of consists of 2.0M synthetic functions drawn from 17 parameterised families, such as polynomials, Fourier sums, Gaussians, rational functions, and solutions of linear ODEs with integer coefficients, complemented with short random expression trees. Every row stores the prefix tokens of and of its derivative together with their values at the quadrature nodes, and rewritings of the same function and coefficient sweeps are never split across partitions. Further details can be found in Appendix § 7.§ 1.
Encoder with computed magnitude
The encoder is a BERT-style transformer [18] with 10 layers, hidden size 512 and about 33M parameters, which reads the prefix tokens of . Its [CLS] state is projected to with , and the latent vector is defined as
where is computed from the values of and at the quadrature nodes rather than learned. With Equation (3), , so the norms are exact by construction and the only learned quantity is the angle between two latents. Proposition 3 in Appendix § 8.§ 2 makes this precise: the error is at most times the angular error of the encoder, plus a quadrature error that decays geometrically with for analytic functions. Moreover, the norms of the training functions span more than four orders of magnitude, and with a computed norm homogeneity reduces to the invariance of the direction, .
Closeness in latent space. Since latent distances approximate Sobolev distances (Corollary 1 in Appendix § 8.§ 2), a prediction is compared with its target through the relative error , which, unlike the cosine, penalises a wrong norm. A prediction is considered correct under the -criterion when . The threshold is calibrated on the validation split: the relative distance from a function to its nearest distinct function has a median of 0.128 and a 5th percentile of 0.0102, and we set , half of this percentile, so that the -balls of two functions do not overlap for 95% of the functions (Appendix § 9.§ 2).
Factored derivative operator
The derivative operator predicts from . Writing , it is factored into a gain and a direction:
where is initialised to the identity, and and are multilayer perceptrons. The gain is even and positive and learns the ratio , whereas the direction is odd and of unit norm.
Proposition 1. For every and every , the operator in Equation (4) satisfies .
The proof can be found in Appendix § 8.§ 1. Thus, homogeneity is exact by construction, and powers and polynomials are available by composition without being trained. It is worth noting that is not additive by construction: additivity is not enforced on the operator and is only learned indirectly, through the linearity of the latent.
Training objective
Figure 2 summarises how the objective is built. Let be the directions of the functions of a batch, the matrix that stacks them, and the target cosines. The geometry objective matches latent angles to Sobolev angles and adds a centring term whose maximum is reached by any rank-one latent:

Figure 2. Construction of the training objective. The tokens of , of its derivative , and of synthetic combinations are encoded by the same encoder, and latent norms and target cosines are computed exactly from the function values
where is the centring matrix, is the Frobenius cosine and . The first term is the mean squared angular error that bounds the inner-product error in Proposition 3. Since a freshly initialised transformer maps all inputs to almost parallel directions, a variance floor with is added, in the spirit of VICReg [5].
The linearity objective is applied to synthetic combinations and to scaled copies , built directly in prefix notation with exact values:
which is a relative squared error.
The JEPA objective predicts the latent of the derivative, and is split into a direction term and a log-gain term:
where the target is the latent of with a stop-gradient, , and handles constant functions, for which . The split is motivated by the following observation: for a prediction of norm at angle from the target , the squared error is minimised at , so a mean squared error shrinks the predicted norm whenever the direction is uncertain. With Equation (7), the gain is trained independently of the direction.
The full objective is , where only keeps the raw projection well scaled for bfloat16 precision. The ramp is 0 during the first 1000 steps, so that the geometry forms before the predictive terms act on the latent, and grows linearly to 1 over the next 1000 steps. Only and are used during pretraining: second derivatives, compositions, and ODEs are never seen, and serve as zero-shot tests.
Modules on the frozen encoder
Once pretrained, the encoder and are frozen, and every downstream use of f-JEPA is built from lightweight modules trained on top of them. Every module outputs a latent as a unit direction times a magnitude, with the magnitude computed whenever possible and otherwise predicted as a log-gain. Since the learned operators are accurate only up to a few percent (§ 4.1), the modules are employed to propose candidates in latent space, which are then combined, refitted, and verified with exact numerical or symbolic steps. This principle is shared by all the applications of § 4, whose procedures are detailed in Appendix § 7.7.
Decoder A 6-layer transformer decoder (17.7M parameters) maps to prefix tokens under a grammar mask, so that every output parses. Beam candidates are re-ranked by re-encoding them (round trip), and constants can be refined in latent space.
Composition operators For each outer function , a network predicts a direction and a log-gain. Seven operators are trained (, , , , , , and ), and the scaling laws of their targets, such as , are satisfied by design for positive scalars (Proposition 4 in Appendix § 8.§ 3).
Integrator predicts the latent of the zero-mean antiderivative, and is trained with an additional cycle loss through the frozen .
Value encoder maps 64 samples of to a unit direction, while the magnitude is computed from a spectral derivative of the samples, so that formulas can be recovered from data with the same decoder. Further details can be found in Appendix § 7.§ 3.
Experiments
The experiments follow the two stages of f-JEPA: the pretrained representation is evaluated first (§ 4.1), then the modules and the four downstream tasks built on the frozen encoder (§ 4.2 to § 4.6), and finally the transfer beyond the pretraining distribution (§ 4.7). The encoder was pretrained once for 40k steps with batches of 384 functions on a single 8 GB laptop GPU (6.6 hours), and it is never fine-tuned afterwards. Unless stated otherwise, results are reported on a held-out test set of 12,146 functions that is disjoint from the training data of every component, and retrieval is measured on a fixed subset of 4,000 of them. Only 9.2% of the test functions lie within a relative distance of of a training function, and excluding them leaves the main results unchanged (derivative retrieval 0.835 instead of 0.820). All models were trained once with a single seed. Further details can be found in Appendix § 7, and time, memory, and model sizes are compared in Appendix § 10.
Latent geometry and differentiation
We first evaluate the pretrained representation itself, before any module is trained on it. Table 1 is organised in five blocks: whether latent angles and inner products are those of (geometry), whether the latent of a combination is the combination of the latents (linearity), whether maps the latent of a function to that of its derivative (derivative), whether relations that were never trained hold (zero-shot), and whether the latent space organises functions in an interpretable way (structure).
| Metric | f-JEPA |
| Geometry | Correlation of with — 0.993 |
| Correlation of with — 0.998 | |
| Mean absolute cosine error — 0.017 | |
| Linearity | Scale error vs. (c = 2 / 10 / ) — 0.012 / 0.014 / 0.050 |
| Combinations vs. — 0.071 | |
| Random sums vs. — 0.207 | |
| Derivative | Retrieval of , top-1 / top-5 (3,956 candidates) — 0.820 / 0.960 |
| Relative error / cosine of vs. — 0.070 / 0.998 | |
| Gain ratio — 1.004 | |
| Best linear map, top-1 / top-5 — 0.460 / 0.680 | |
| Zero-shot | vs. , error / top-1 (3,640 candidates) — 0.267 / 0.458 |
| vs. , error / top-1 — 0.232 / 0.501 | |
| Eigenfunctions / — 0.122 / 0.174 | |
| ODE annihilation relative to a shuffled ODE (chance 1) — 0.209 | |
| Structure | Coefficient-sweep ordering (Spearman; chance 0.57) — 0.828 |
| Distance between equivalent rewritings (relative; random pairs 1) — 0.071 | |
| Linear probe: family accuracy / values — 0.742 / 0.998 |
Table 1. Latent space properties of the encoder on held-out functions. Errors are median relative errors, and every row is defined in Appendix § 7.§ 6.
As observed in Table 1, the latent space is close to an isometric embedding of : latent cosines and dot products match their Sobolev counterparts with correlations of 0.993 and 0.998, and the norm is exact by construction. The latent is homogeneous to 1.2% for positive scalars and to 5.0% for negative ones, and linear combinations of the training type are reproduced to 7.1%, whereas sums of two unrelated functions, which are longer and further from the training distribution, are reproduced only to 20.7%. The encoder , trained from scratch on , reaches a comparable geometry, linearity, and derivative retrieval on the test split of its own interval, although its zero-shot operator algebra is weaker (Appendix § 9.§ 1).
The learned derivative retrieves the true derivative among 3,956 candidates for 82.0% of the functions (96.0% within the top five), and its gain is calibrated, with a median ratio of 1.004 between predicted and true norms. Notably, the best linear map fitted to retrieves only 46.0%, so differentiation is not a linear map of this latent space. Relations that were never trained hold approximately: applying twice retrieves the true for 45.8% of the functions, the eigenfunction relations of and hold to 12% and 17%, and the residual of solutions of linear ODEs is five times smaller than for a shuffled ODE. Without supervision, the latent also orders the members of coefficient sweeps (Spearman 0.83 against 0.57 by chance) and places equivalent rewritings of a function at 7% of the distance between random pairs.
However, is accurate only up to a noise floor. Its median relative error is 0.070 and its smallest error on the validation split is 0.013, above the threshold of § 3.2, so no prediction of any learned operator satisfies the -criterion (Table 14 in Appendix § 9.§ 2). Retrieval succeeds nevertheless because the target only has to be the nearest candidate, and the median gap between distinct functions is 0.128. Therefore, the latent operators are accurate enough to propose candidates but not to identify a function by themselves, which is why every application below verifies their proposals with an exact step.
| Prediction | Top-1 | Rel. error | < 0.05 | < 0.1 |
| vs. | 0.820 | 0.070 | 0.317 | 0.671 |
| vs. | 0.458 | 0.267 | 0.075 | 0.264 |
| vs. | 0.501 | 0.232 | 0.178 | 0.339 |
| Integrator | 0.670 | 0.176 | 0.002 | 0.172 |
| 0.690 | 0.246 | 0.000 | 0.042 | |
| 0.647 | 0.275 | 0.000 | 0.008 | |
| 0.584 | 0.110 | 0.146 | 0.446 | |
| 0.568 | 0.179 | 0.007 | 0.107 | |
| 0.922 | 0.168 | 0.001 | 0.135 | |
| 0.919 | 0.149 | 0.006 | 0.213 | |
| 0.796 | 0.275 | 0.000 | 0.025 |
Table 14. Closeness of the latents predicted by the learned operators to their targets: top-1 retrieval accuracy, median relative error , and shares with a relative error below 0.05 and 0.1. No prediction reaches the calibrated
Table 2 compares the latent space of f-JEPA with the representations of two released encoders of symbolic expressions: SNIP [34] and the encoder of the integrator of Lample and Charton [28]. We report the property probes of SNIP [34] with the same protocol for every representation, standard measures of how an embedding preserves the geometry of the functions, including trustworthiness [50], and a derivative probe (Appendix § 7.§ 4). In order not to favour f-JEPA, whose training target is the cosine, the first geometry measure employs the plain cosine of the function values on a uniform grid.
| Representation | Dim. | Value cos. | Dist. rank | kNN@10 | Trust. | NCR | Upward. | Avg. | Osc. | Top-5 | Align. |
| SNIP, symbolic encoder [34] | 512 | 0.002 | -0.090 | 0.080 | 0.570 | 0.680 | 0.716 | 0.805 | 0.084 | 0.338 | 0.936 |
| Lample and Charton encoder [28] | 1024 | 0.012 | 0.133 | 0.087 | 0.601 | 0.319 | 0.217 | 0.210 | 0.044 | 0.979 | 0.908 |
| f-JEPA | 256 | 0.946 | 1.000 | 0.900 | 0.994 | 0.235 | 0.389 | 0.583 | 0.390 | 0.929 | 0.997 |
Table 2. Comparison of function representations computed from the symbolic text on the same 4,000 test functions. The property probes follow SNIP [34] (normalised error, chance 1), and the derivative probe is fitted on the same 20k pairs for every representation.
Only f-JEPA reflects the geometry of the functions: its cosines follow those of the function values (correlation 0.946, against 0.012 and 0.002), and 90% of its 10 nearest neighbours are also neighbours, against fewer than 9% for the other two encoders. However, the encoder of Lample and Charton, with 2.4 times as many parameters and a latent of dimension 1,024, predicts three of the four SNIP properties more accurately and supports a more accurate derivative retrieval (top-5 0.979 against 0.929), whereas f-JEPA predicts the non-convexity ratio best and has the best aligned derivative predictions. A likely reason is the geometry itself, which places functions of the same family close to each other: 78% of the retrieval errors of return a different member of the correct family. Thus, the representation of f-JEPA is not the most informative one for every property, but it is the only one in which norms, angles, and linear combinations have an analytical meaning.
Figure 3 shows the global organisation of the latent space, in which the unit directions are projected by a small autoencoder and the exact norm is kept as the radius. All multiples of a function lie on one ray, oscillatory functions occupy different angular sectors from exponential, power, and rational functions, and the test set covers the same sectors as the training set. It is worth noting that the projection is lossy, with a median angular error of (Appendix § 9.8).
Figure 3. Latent space of (a) 120k training functions and (b) the 20.4k functions of the test split of the training corpus. The angle is the code of the unit direction given by an autoencoder with a circular bottleneck, the radius is the rank of , and the colour encodes the elementary functions of each expression
Decoding and composition
From here on the encoder and are frozen. The decoder recovers the exact function from for 52.6% of the test functions with beam search and round-trip re-ranking, and for 67.0% after latent constant refinement, while the shape of the function (constants refitted) is correct for 84.0%. Decoding the predicted latent yields a derivative obtained entirely through the latent space: its shape is correct for 74.8% of the functions, but its constants are exact only for 6.8%, since inherits the noise floor of . Accuracy also decreases with expression length, from 77% for 10 to 19 tokens to 10% for 30 to 39 tokens (Appendix § 9.§ 4). Figure 4 illustrates both regimes: when decoding fails, the formula usually keeps the structure of the true one with a wrong constant, such as 189 instead of 171 in the predicted derivative of a logistic function.
Table 3 shows that composition is learnable on the frozen encoder, but not with a linear map: ridge regression retrieves at most 19.6% of the targets, whereas attains between 56.8% and 92.2% top-1 accuracy. Remarkably, the quadratic is retrieved with 44.4% accuracy without any training on it, by combining the learned square operator with the linearity of the latent. The predicted magnitudes also contribute to retrieval, as shown by the lower accuracy of the direction alone.
| Method | |||||||
| Identity | 0.099 | 0.034 | 0.017 | 0.023 | 0.016 | 0.038 | 0.005 |
| Ridge regression | 0.032 | 0.021 | 0.041 | 0.019 | 0.096 | 0.196 | 0.118 |
| Linear part of only | 0.095 | 0.000 | 0.032 | 0.201 | 0.149 | 0.682 | 0.000 |
| Zero-shot | – | – | – | – | – | – | 0.444 |
| , direction only | 0.578 | 0.508 | 0.481 | 0.253 | 0.703 | 0.675 | 0.660 |
| 0.690 | 0.647 | 0.584 | 0.568 | 0.922 | 0.919 | 0.796 | |
| , top-5 | 0.922 | 0.894 | 0.890 | 0.956 | 0.990 | 0.983 | 0.885 |
Table 3. Accuracy comparison (top-1) of composition operators on the frozen encoder, retrieving among about 3k unseen inner functions . The last row gives the top-5 accuracy of
A natural question is whether the learned operators add anything over decoding the input and applying computer algebra. Table 16 in Appendix § 9.§ 4 compares each learned operator with a symbolic path that employs the same frozen decoder and encoder but replaces the learned operator by the exact one, with both routes scored by retrieval among the same candidates. The symbolic path has the higher top-1 accuracy on every operator, since it is almost always correct when the decoder reconstructs the input exactly. However, on the 24% to 45% of the inputs that the decoder does not reconstruct, the learned operators are more accurate in top-5 for the derivative and for 6 of the 7 composition operators, the exception being the quadratic. Hence, the learned operators do not replace exact symbolic calculus on formulas that can be recovered, but they complement it, and they are the only option when no formula is available, as in the tasks below.
Table 16: Accuracy comparison (top-1 / top-5 retrieval) of the learned operators with a symbolic path that decodes the input latent, applies the exact operator with SymPy, and re-encodes the result. Input not decoded: inputs that the decoder does not reconstruct exactly
| Operator | All inputs: Learned | All inputs: Symbolic path | Input not decoded: Learned | Input not decoded: Symbolic path |
| 0.820 / 0.960 | 0.901 / 0.963 | 0.802 / 0.953 | 0.783 / 0.917 | |
| 0.675 / 0.924 | 0.850 / 0.909 | 0.585 / 0.866 | 0.543 / 0.723 | |
| 0.640 / 0.889 | 0.849 / 0.905 | 0.528 / 0.792 | 0.502 / 0.686 | |
| 0.591 / 0.893 | 0.773 / 0.886 | 0.509 / 0.831 | 0.498 / 0.747 | |
| 0.573 / 0.958 | 0.836 / 0.928 | 0.509 / 0.949 | 0.637 / 0.841 | |
| 0.923 / 0.990 | 0.932 / 0.971 | 0.854 / 0.971 | 0.752 / 0.894 | |
| 0.923 / 0.988 | 0.938 / 0.981 | 0.868 / 0.988 | 0.745 / 0.922 | |
| 0.801 / 0.897 | 0.942 / 0.963 | 0.688 / 0.822 | 0.784 / 0.862 |
Table 16. Accuracy comparison (top-1 / top-5 retrieval) of the learned operators with a symbolic path that decodes the input latent, applies the exact operator with SymPy, and re-encodes the result. Input not decoded: inputs that the decoder does not reconstruct exactly
Symbolic integration
The four downstream tasks reuse the same frozen encoder, decoder, and operators, and each adds at most one lightweight module. The integrator retrieves the antiderivative among 11,196 candidates with a top-1 accuracy of 67.0% (91.9% top-5), against 42.7% for a search that inverts without training. The solver of Algorithm 1 builds on it: latent proposals are decoded, refitted, and combined, and an answer is accepted only when SymPy verifies . Symbolic preprocessing is limited to splitting sums, reducing trigonometric powers, and substituting inner functions, whereas every integration step is performed by the learned integrator and the decoder.
As observed in Table 4, the solver verifies an antiderivative for 91.7% of the test integrands and 66.7% of the hard integrals, which require integration by parts or substitution, against 99.3% and 100% for SymPy. The pretrained integrator of Lample and Charton [28] solves 68.7% of our test integrands, whose coefficients often exceed the small integers of its training expressions, but 90.0% of the hard integrals. The unverified answers remain close approximations: the returned antiderivative reproduces the definite integral on within 1% for 99.3% of the test integrands and for all the hard ones. For integrands without an elementary antiderivative, such as or , the derivative of a five-term approximation built from the same basis is within 1% of the integrand for 7 of 12 functions (Table 24 in Appendix § 9).
| Method | Solve rate Test | Solve rate Hard | Definite integral Test | Definite integral Hard |
| Latent shortlists, decoding (beam 16) and constant refitting | 0.773 | 0.167 | 0.957 | 0.900 |
| + exact combination of decoded pieces and pool functions | 0.787 | 0.200 | 0.977 | 0.900 |
| + splitting sums into terms | 0.813 | 0.200 | 0.977 | 0.900 |
| + residual peeling in latent space | 0.817 | 0.200 | 0.983 | 0.900 |
| + latent tabular integration by parts | 0.827 | 0.433 | 0.983 | 0.967 |
| + substitution and trigonometric reduction | 0.917 | 0.667 | 0.993 | 1.000 |
| Lample and Charton, FWD+BWD+IBP, beam 10 [28] | 0.687 | 0.900 | – | – |
| SymPy integrate (reference, not a learned method) | 0.993 | 1.000 | – | – |
Table 4. Symbolic integration on 300 test integrands and 30 hard elementary integrals, with cumulative levels of our solver. Solve rate: antiderivative verified exactly by SymPy; definite integral: the returned antiderivative, verified or not, reproduces within 1%.
| Integrand | |||
| 7.5e-2 | 1.8e-2 | 2.9e-3 | |
| 6.7e-1 | 2.9e-1 | 1.1e-1 | |
| 3.9e-1 | 4.9e-2 | 1.9e-2 | |
| 1.4e-1 | 7.0e-3 | 1.2e-3 | |
| 1.1e-1 | 4.3e-2 | 1.2e-2 | |
| 1.2e-1 | 1.7e-2 | 6.4e-3 | |
| 4.2e-2 | 5.8e-3 | 1.8e-3 | |
| 9.0e-2 | 1.3e-2 | 1.6e-3 | |
| 9.9e-2 | 1.7e-2 | 5.6e-3 | |
| 2.6e-2 | 1.3e-2 | 1.1e-2 | |
| 5.3e-1 | 1.1e-1 | 4.6e-2 | |
| 7.4e-2 | 8.4e-3 | 3.1e-3 |
Table 24. Integrands without elementary antiderivative: relative error of the derivative of the -term symbolic approximation with respect to the integrand .
Symbolic regression
Formulas are recovered from 64 samples with the value encoder, the decoder, constant refitting, and a selection by plus description length (Algorithm 2). On 300 test functions, 80.0% of the formulas are recovered exactly from clean samples, and 63.0% and 44.7% at relative noise levels of and , against 87.0% when the true latent is decoded instead of the output of the value encoder (Table 17 in Appendix § 9.§ 5). Table 5 compares f-JEPA with six external methods that receive the same samples of a fixed subset of 50 test functions (Appendix § 7.§ 5).
| Method | Exact / within threshold | Exact / within threshold | Exact / within threshold | NMSE | Extrap. | Nodes | Rel. err. | Classic (of 24) | Classic (of 24) |
| E2E | 0.00 / 0.02 | 0.00 / 0.00 | 0.00 / 0.00 | 0.12 | 6.2e-01 | 20 | 1.7e-01 | 0 / 0 | 0 / 0 |
| NeSymReS | 0.06 / 0.14 | 0.00 / 0.12 | 0.00 / 0.08 | 0.22 | 8.0e-01 | 12 | 2.7e-01 | 3 / 4 | 0 / 2 |
| SNIP | 0.04 / 0.58 | 0.00 / 0.30 | 0.00 / 0.10 | 0.74 | 4.0e-04 | 22 | 6.5e-03 | 0 / 10 | 0 / 2 |
| MMSR | 0.00 / 0.12 | 0.00 / 0.08 | 0.00 / 0.02 | 0.28 | 4.1e-01 | 15 | 3.2e-01 | 3 / 5 | 1 / 3 |
| PySR | 0.06 / 0.74 | 0.00 / 0.58 | 0.00 / 0.18 | 0.94 | 4.0e-07 | 22 | 5.2e-03 | 9 / 19 | 1 / 4 |
| Operon | 0.06 / 0.96 | 0.00 / 0.26 | 0.00 / 0.00 | 0.96 | 2.3e-09 | 23 | 7.5e-03 | 3 / 20 | 0 / 0 |
| f-JEPA | 0.76 / 0.96 | 0.76 / 0.92 | 0.62 / 0.74 | 0.98 | 12 | 2.8e-08 | 8 / 10 | 8 / 9 |
Table 5. Symbolic regression from 64 samples on the same 50 test functions for every method, and on 24 classic univariate benchmarks. Entries with a slash give exact recovery / recovery within the function-space threshold, and the columns are defined in Appendix § 7.§ 6.
| Method | |||
| Nearest function in the training pool | 0.110 | 0.110 | 0.110 |
| Value encoder + decoding (beam 8) + constant refitting | 0.640 | 0.470 | 0.210 |
| + beam 16 and refitting of every candidate | 0.730 | 0.630 | 0.447 |
| + exact combination with polynomial atoms | 0.763 | 0.630 | 0.447 |
| + residual peeling | 0.800 | 0.630 | 0.447 |
| True latent instead of (upper bound) | 0.870 | – | – |
Table 17. Exact symbolic recovery (share of functions) of the cumulative levels of our pipeline from 64 samples on 300 test functions, at relative noise levels . All levels use the and description-length selection
f-JEPA is the only method that recovers the exact formula for most functions, and the only one that keeps doing so under noise (76%, 76%, and 62%, against at most 6% for every other method). This advantage partly reflects the criterion and the data: the test functions belong to the pretraining families, whose rational constants the decoder emits, whereas the other methods return floating-point constants. When near-exact formulas are also counted, through a threshold of 0.0037 on the relative error, Operon ties f-JEPA on clean samples (0.96 each), and on the 24 classic benchmarks Operon and PySR are within the threshold of 20 and 19 targets against 10 for f-JEPA. Under noise, in contrast, the advantage of f-JEPA remains under both criteria (0.92 and 0.74 against at most 0.58 and 0.18), since the constants of the other methods fit the noise: their median relative error at is between and 0.32, against for f-JEPA. Moreover, f-JEPA is the fastest method, with 0.44 seconds per function against 1.6 to 93 seconds (Appendix § 10). At 1% noise, its typical failure is a single constant snapped to a wrong fraction, as in Figure 5b.
Figure 5. True functions (black) against recovered formulas (orange): (a, b) symbolic regression at , (c, d) five-term approximations of antiderivatives without elementary closed form, and (e, f) four-atom formulas for initial-value problems without closed form. More examples are shown in Appendix E
Solving ordinary differential equations
For a linear ODE , the latent residual ranks a pool of functions, but true solutions have median relative residuals between 0.009 and 0.08, and gradient descent on below this level does not improve the ranking (Table 20 in Appendix § 9). Following the principle of § 3.5, the latent residual is employed to shortlist the top 1% of a pool of 2M functions, and exact steps on the stored values (re-ranking, sparse combinations, and refitting of constants) followed by SymPy verification produce the solutions (Algorithm 3). This yields 20 verified solutions among the top 20 for all 9 linear ODEs tested and for 4 nonlinear ones, including solutions absent from the pool (Table 21). Since SymPy solves such textbook ODEs directly, this experiment validates the principle rather than a use case.
| ODE | Solutions | Residual | p@20 | p@20 (GD) | Exact | New |
| 47 | 0.081 | 0.85 | 0.70 | 8 | 0 | |
| 5 | 0.069 | 0.25 | 0.25 | 17 | 9 | |
| 39 | 0.009 | 0.40 | 0.45 | 6 | 3 | |
| 20 | 0.049 | 0.80 | 0.55 | 7 | 0 | |
| 5 | 0.013 | 0.20 | 0.25 | 1 | 0 |
Table 20. Linear ODEs on the held-out pool (11,196 functions): number of solutions in the pool, their median latent residual, precision at 20 of the latent ranking and of the re-ranking after gradient descent in latent space, and SymPy-verified solutions found by exact sparse combinations of the 16 best-ranked functions (total, and absent from the pool)
| ODE | Solutions | Latent p@20 | Recall@1% | Verified top 20 | New |
| 244 | 0.00 | 0.89 | 20/20 | 2 | |
| 295 | 0.05 | 0.96 | 20/20 | 1 | |
| 7,522 | 1.00 | 0.91 | 20/20 | 2 | |
| 10,567 | 1.00 | 0.99 | 20/20 | 0 | |
| 7,418 | 0.00 | 0.86 | 20/20 | 2 | |
| 2,835 | 0.90 | 0.96 | 20/20 | 3 | |
| 2,574 | 0.10 | 0.77 | 20/20 | 7 | |
| 629 | 0.00 | 0.64 | 20/20 | 1 | |
| 19,278 | 1.00 | 0.97 | 20/20 | 0 | |
| 8 | 0.00 | 1.00 | 2/2 | 2 | |
| 9 | 0.10 | 0.33 | 1/3 | 2 | |
| 58 | 0.75 | 1.00 | 20/20 | 4 | |
| 45 | 0.00 | 1.00 | 20/20 | 0 | |
| 50 | 0.90 | 1.00 | 20/20 | 1 | |
| 7 | 0.00 | 1.00 | 1/1 | 0 | |
| 6 | 0.00 | 0.00 | 0/0 | 1 | |
| 40 | 0.60 | 1.00 | 20/20 | 4 | |
| 11 | 0.00 | 0.40 | 2/5 | 1 | |
| 7 | 0.00 | 1.00 | 1/1 | 0 |
Table 21. Textbook ODEs on the 2M-function pool: number of solutions in the pool, precision at 20 of the latent ranking alone, recall of the latent shortlist (top 1%), verified solutions among the final top 20 (of the maximum possible), and verified solutions absent from the pool
We therefore consider 12 initial-value problems for which SymPy’s dsolve fails, such as the pendulum, Duffing oscillators, and a Painlevé-I type equation. A network-free spectral Newton method provides the solution values, and the task is to compress them into a formula with atoms, measured by the relative deviation from a high-precision numerical solution (Algorithm 4). Every learned source of atoms (latent nearest neighbours, latent ODE shortlist, latent residual peeling, and atoms created by the composition operators) is compared with a network-free control that differs only in the learned part.
As observed in Table 6, with four atoms the best learned method outperforms the best control on 11 of 12 ODEs (median ratio 1.43), and atoms created by the composition operators improve over latent atoms alone on 10 of 12. With three atoms, however, learned and network-free methods are equivalent (5 wins, 1 tie, and 6 losses). All learned formulas are one to two orders of magnitude more accurate than Padé approximants [3] with the same number of coefficients. The per-equation deviations are listed in Table 22 in Appendix C. Figure 5e,Figure 5f illustrates the two extremes: the four-atom formula is indistinguishable from the solution of the soft Duffing oscillator, whereas for the Painlevé-I type equation every four-atom formula deviates by about 5%.
| Best learned vs. best network-free control: wins / ties / losses | 5 / 1 / 6 | 11 / 0 / 1 | 9 / 1 / 2 |
| Median ratio (control / learned) | 0.96 | 1.43 | 1.40 |
| Composed atoms vs. latent atoms only: wins / ties / losses | 7 / 5 / 0 | 10 / 1 / 1 | 10 / 2 / 0 |
| Median ratio (latent / composed) | 1.15 | 1.55 | 2.00 |
| Best learned vs. Padé approximant with coefficients: wins | 12 / 12 | 12 / 12 | 12 / 12 |
| Median error ratio (learned / Padé) | 0.037 | 0.011 | 0.004 |
Table 6. Performance comparison on 12 initial-value problems that SymPy cannot solve in closed form, with atoms. Ratios are medians over the ODEs of the relative deviations from the numerical solution.
| ODE | Control | Latent atoms | + composed | Best learned | Padé |
| 1.4e-2 | 4.0e-2 | 1.7e-2 | 9.5e-3 | 1.0e0 | |
| 8.4e-3 | 2.6e-2 | 5.0e-3 | 5.0e-3 | 1.3e-1 | |
| 3.3e-2 | 4.3e-2 | 4.1e-2 | 2.4e-2 | 3.5e-1 | |
| 1.1e-2 | 3.4e-2 | 1.5e-2 | 7.7e-3 | 6.5e-1 | |
| 2.8e-3 | 3.8e-3 | 2.4e-3 | 2.4e-3 | 5.9e-3 | |
| 1.2e-2 | 1.7e-2 | 7.8e-3 | 5.4e-3 | 3.9e-1 | |
| 5.3e-4 | 4.8e-4 | 6.4e-4 | 4.4e-4 | 4.3e-1 | |
| 5.1e-2 | 5.3e-2 | 3.8e-2 | 3.8e-2 | 5.6e-1 | |
| 7.9e-4 | 1.3e-4 | 8.7e-5 | 8.7e-5 | 5.6e-1 | |
| 3.4e-4 | 4.2e-4 | 3.6e-4 | 2.0e-4 | 4.6e-1 | |
| 1.9e-3 | 2.2e-3 | 2.3e-3 | 2.2e-3 | 2.8e-1 | |
| 3.9e-3 | 6.3e-3 | 2.7e-3 | 2.7e-3 | 2.8e-1 |
Table 22. Relative deviation from the numerical solution of four-atom formulas for the 12 initial-value problems without closed form: best network-free control, latent atoms only, latent and composed atoms, best learned method, and Padé approximant with five coefficients
Discovering ordinary differential equations
Table 7 reports the discovery of the governing equation from 64 noisy samples, with the same term search, coefficient snapping, and verification for every route (Algorithm 5). The strong-form and weak-form routes estimate the derivatives from the samples and employ no network, whereas the learned route maps the samples to a formula with the value encoder and the decoder, and then differentiates the formula exactly.
| Method | Linear ODEs, | Linear ODEs, | Linear ODEs, | Logistic-type ODEs, | Logistic-type ODEs, | Logistic-type ODEs, |
| Strong form (SINDy-like, no network) | 0.520 | 0.200 | 0.073 | 0.853 | 0.513 | 0.200 |
| Weak form (no network) | 0.673 | 0.233 | 0.053 | 0.473 | 0.380 | 0.127 |
| Samples formula exact derivatives | 0.487 | 0.440 | 0.440 | 0.633 | 0.647 | 0.600 |
| SINDy, PySINDy [17]† | 0.073 | 0.000 | 0.000 | 0.200 | 0.060 | 0.013 |
| Weak SINDy, PySINDy [35]† | 0.167 | 0.047 | 0.013 | 0.580 | 0.433 | 0.080 |
| ODEFormer, pretrained [16]† | 0.047 | 0.020 | 0.020 | 0.033 | 0.040 | 0.027 |
Table 7. Accuracy comparison (valid@1) on ODE discovery from 64 noisy samples (150 functions per set). †External methods receive the true order of the equation and generous coefficient snapping (Appendix § 7.§ 4).
The learned route is almost insensitive to noise: at 1% noise it is the most accurate method on both families, 6.0 and 3.0 times more accurate than the best network-free route. On clean samples, however, the network-free routes are more accurate (0.673 and 0.853 against 0.487 and 0.633), since spectral derivatives of clean samples already identify most equations. The external methods, which receive the true order of the equation, identify at most 16.7% of the linear ODEs, whose states require estimated higher derivatives; on logistic-type equations weak SINDy reaches 0.580 on clean samples but drops to 0.080 at 1% noise.
Transfer beyond the pretraining distribution
In order to measure how far the frozen stack transfers, it is evaluated on out-of-distribution expressions and on standard benchmarks, restricted to the items that f-JEPA can represent (univariate, without poles on , and within our vocabulary; Appendix § 7.§ 5). Every method is evaluated on the same items.
Out-of-distribution expressions The out-of-distribution integrands are derivatives of random non-polynomial expression trees with 6 to 8 operators, whereas the training trees have at most 5. The solver verifies an antiderivative for 44.3% of them, against 71.8% for SymPy, and only 17.4% before the substitution level (Table 26 in Appendix § 9), which confirms the brittleness of learned integrators outside their training distribution [51]. Notably, 8 of the solved integrals are not solved by SymPy within 10 seconds (Table 25). However, these integrands are generated like the backward training data of the integrator of Lample and Charton, which solves 99.3% of them, so this benchmark is out of distribution only for f-JEPA. For symbolic regression on 100 such trees, f-JEPA recovers 10% of the formulas exactly from clean samples, against 17% for PySR, and 7% at 1% noise, where PySR recovers none (Table 19 in Appendix § 9.§ 5).
Table 19. Out-of-distribution symbolic regression from 64 samples on 100 random non-polynomial expression trees with 6 to 8 operators. Shares of formulas recovered exactly and with a relative error below and below
| Integrand | Antiderivative found by f-JEPA |
Table 25. Out-of-distribution integrals solved by the learned levels of f-JEPA (verified by SymPy) for which SymPy integrate returns no antiderivative within 10 seconds.
| Method | Solve rate | Definite integral | Not solved by SymPy |
| Latent shortlists, decoding and constant refitting | 0.101 | 0.544 | 0 |
| + exact combination, term splitting and residual peeling | 0.121 | 0.664 | 0 |
| + latent tabular integration by parts | 0.174 | 0.658 | 3 |
| + substitution and trigonometric reduction | 0.443 | 0.711 | 8 |
| Lample and Charton, FWD+BWD+IBP, beam 10 [28] | 0.993 | – | 42 |
| SymPy integrate (reference, 10 s limit) | 0.718 | – | – |
Table 26. Out-of-distribution symbolic integration on 149 integrands of random expression trees with 6 to 8 operators, with cumulative levels of our solver. Columns as in Table 4, plus the number of solved integrands that SymPy does not solve
Standard integration benchmarks On the test sets of Lample and Charton, the MIT Integration Bee, and the Rubi suite [39] (Table 8), f-JEPA is an approximate integrator rather than an exact one. Between and of its antiderivatives have a derivative within a relative error of of the integrand, but only between and are verified as exact, against to for the integrator of Lample and Charton on its own test sets and and on the other two. Thus, the integration results of § 4.3 do not transfer to the distributions on which sequence-to-sequence integrators are trained.
| Method | FWD | BWD | IBP | MIT Bee | Rubi [39] |
| Approximate antiderivatives (only f-JEPA produces them) | |||||
| f-JEPA, relative error of below | 0.700 | 0.530 | 0.760 | 0.500 | 0.365 |
| f-JEPA, within of the true antiderivative () | 0.660 | 0.485 | 0.735 | 0.511 | 0.375 |
| Exact antiderivatives verified by SymPy | |||||
| f-JEPA | 0.250 | 0.240 | 0.195 | 0.359 | 0.165 |
| Lample and Charton [28], rerun on these items | 0.975 | 0.985 | 0.995 | 0.565 | 0.280 |
Table 8. Symbolic integration on standard benchmarks (200 representable items per benchmark, 92 for MIT Bee). Top: shares of the approximate antiderivatives of f-JEPA that are close to the true one; bottom: shares verified exactly by SymPy
Standard ODE benchmarks The ODE test sets of Lample and Charton are posed as initial-value problems, and f-JEPA only receives samples of their numerical solution (Table 9). A solution is counted as verified when its numeric residual is below and it meets the initial conditions, because a symbolic proof rejects and of the reference solutions themselves. The pretrained solvers of Lample and Charton verify and of the problems and SymPy and , whereas f-JEPA verifies and , and and after fine-tuning the decoder and the value encoder on expressions of this distribution and adding an ODE-aware search. The gap originates in the pretrained encoder: these solutions combine products and compositions that are absent from the pretraining families, and decoding their true latent recovers only and of them exactly, against for our test functions (Appendix § 9.7).
| Method | ODE1 Verified | ODE1 Res. | ODE1 Res. | ODE2 Verified | ODE2 Res. | ODE2 Res. |
| f-JEPA (numerical IVP → samples-to-formula) | 0.185 | 0.240 | 0.305 | 0.160 | 0.220 | 0.280 |
| f-JEPA + fine-tuned decoder + ODE-aware search | 0.250 | 0.295 | 0.365 | 0.230 | 0.280 | 0.350 |
| Lample and Charton, rerun on these IVPs | 0.945 | 0.960 | 0.965 | 0.915 | 0.950 | 0.950 |
| SymPy dsolve (10 s) | 0.620 | 0.620 | 0.620 | 0.320 | 0.335 | 0.335 |
Table 9. Solving ODEs on the Lample and Charton test sets, posed as initial-value problems (200 each). Verified: numeric residual below and initial conditions met within
On ODEBench [16] (Table 27 in Appendix § 9), the learned discovery route reconstructs , , and of the trajectories at the three noise levels. It is more accurate than ODEFormer under noise and ties with weak SINDy at noise, but every SINDy variant is more accurate on clean samples. In contrast, f-JEPA recovers the exact equation more often than every other method ( of the trajectories on clean samples and under noise, against at most and ), although these shares are low in absolute terms.
| Method | , | , | , | , | , | , | Exact equation, | Exact equation, | Exact equation, | Extrapolation NMSE, | Extrapolation NMSE, |
| f-JEPA: samples → formula → exact derivatives | 0.80 | 0.87 | 0.89 | 0.67 | 0.78 | 0.67 | 0.15 | 0.07 | 0.07 | 2.4e-03 | 1.9e-03 |
| SINDy (PySINDy) | 0.96 | 0.93 | 0.48 | 0.80 | 0.67 | 0.20 | 0.07 | 0.02 | 0.00 | 1.0e-05 | 2.0e-01 |
| Weak SINDy (PySINDy) | 0.93 | 0.96 | 0.89 | 0.83 | 0.87 | 0.70 | 0.07 | 0.04 | 0.00 | 5.8e-06 | 5.7e-03 |
| Estimated derivatives + PySR | 0.98 | 0.54 | 0.35 | 0.98 | 0.52 | 0.17 | 0.00 | 0.00 | 0.00 | 2.2e-08 | 1.1e+01 |
| ODEFormer (pretrained) | 0.85 | 0.85 | 0.83 | 0.67 | 0.67 | 0.61 | 0.00 | 0.00 | 0.00 | 2.7e-03 | 2.3e-03 |
Table 27. Discovery of the 23 one-dimensional ODEBench systems [16] (two initial conditions each) from 64 noisy samples. is measured on the trajectory integrated from the discovered equation, and an equation is exact when it has the true terms with every coefficient within 1%
Limitations
One limitation of our work is that the latent operators are accurate only up to a noise floor of a few percent: no prediction satisfies the calibrated -criterion, linear relations between latents hold up to about , and the learned derivative is not a linear map of the latent space. Any optimisation in latent space below this level fits the errors of the model rather than the problem, which is why the latent space only proposes candidates and exact numerical or symbolic steps combine and verify them. Where a closed form exists, computer algebra remains more reliable, and when the decoder recovers the input formula, applying the exact operator to it is more accurate than the learned operator. (Table 16). The transfer beyond the pretraining families is partial: on standard integration and ODE benchmarks f-JEPA verifies at most 35.9% and 25% of the items, far behind sequence-to-sequence models trained on those distributions, and closing this gap would require pretraining the encoder on such expressions (§ 4.7). Its advantage in exact symbolic regression is measured on functions of the pretraining families and narrows on clean samples when near-exact formulas are counted. Moreover, the study is restricted to univariate functions on a fixed interval, to a 37-token vocabulary without special functions, and to synthetic data, and the decoder rarely reproduces expressions longer than 30 tokens exactly. Finally, all models were trained once on a single 8 GB laptop GPU, so the variability across training runs and the scaling behaviour are not measured, and Proposition 3 controls the inner-product error only through the angular error of the encoder, which is measured rather than bounded.
Conclusions
In this work, we examined whether a single transformer encoder, pretrained once from symbolic text, can learn a vector space of functions on which calculus acts through latent operators, and whether this representation can be reused across tasks. The proposed f-JEPA computes the magnitude of each latent exactly as the norm and learns only its direction, matching Sobolev angles, enforcing linearity, and predicting the derivative with a factored and exactly homogeneous operator. The resulting latent space closely reproduces Sobolev angles and inner products, and its operators are accurate enough to propose solutions that exact steps then verify.
Without any fine-tuning of the encoder, the same representation supports symbolic integration, symbolic regression, and the solution and discovery of ODEs, with its clearest gains on noisy data, whereas computer algebra and task-specific models remain stronger on standard benchmarks. To the best of our knowledge, this is the first joint-embedding predictive architecture trained on the symbolic form of functions in which the predictor is the derivative operator and collapse is prevented by a data-defined Sobolev geometry.
For future work, we would like to extend the approach to multivariate functions with commuting partial derivative operators, to reduce the noise floor of the derivative with structured operators, and to pretrain the encoder on broader distributions of expressions.
Experimental details
Data generation
Functions are generated with SymPy [36] on and written in prefix notation [28] with a 37-token vocabulary:
special tokens;
the binary operators
add sub mul div pow;the unary operators
neg inv sqrt pow2 pow3 exp log sin cos tan atan;the symbols
x E pi, the sign tokensINT+ INT-, the ten digits and a fraction separator.
A rational coefficient is a single contiguous literal (INT+ 3 / 1 0 for 3/10), and numeric coefficients come first in products, so that functions that differ in one coefficient differ in one literal. For instance, is written mul exp mul INT- 1 pow2 x cos mul INT+ 3 x.
The families are listed in Table 10. Coefficients are integers between 1 and 10 (50%), integers between 11 and 100 (30%) or fractions with (20%), with random signs. Frequencies are at most 16 and exponential rates at most 2, so that 64 quadrature nodes resolve every function. The ODE family contains solutions of with a monic integer characteristic polynomial of order 1 to 4, built from integer real roots with multiplicity and complex pairs with and ; every row satisfies its ODE exactly. The family of polynomials of a component contains for a single component . Moreover, constants, linear and quadratic functions are guaranteed minimum shares of 1%, 4% and 6%, and the length distribution decreases from 10 tokens on (mean length 19.9 tokens).
| Family | Share | Family | Share | Family | Share |
| linear ODE solutions | 19.3 | Chebyshev series | 5.7 | logistic | 2.7 |
| polynomials of a component | 13.1 | exponential | 4.7 | damped oscillation | 2.7 |
| polynomial | 7.5 | Gaussian | 4.2 | type | 2.5 |
| Fourier sum | 6.4 | Lorentzian | 3.5 | 2.4 | |
| polynomial trigonometric | 5.9 | arctan | 3.5 | random expression trees | 4.2 |
| harmonic | 5.3 | rational | 3.5 | ||
| 2.8 |
Table 10. Composition of the 2.0M training functions by family (%)
A function is accepted under the following conditions:
it is finite on a 1024-point grid and bounded by in absolute value;
its norm agrees between 64 and 128 quadrature nodes (relative tolerance );
its norm lies in ;
it has at most 126 tokens.
The derivative is stored in its shortest equivalent form. The test set contains 12,146 functions generated with the same families and filtered so that none of them appears in the training data. The corpus also has its own validation and test splits (20.4k functions each): the former is used for calibration and model selection, and the latter only in Figure 3. The composition operators are trained on 4.17M pairs , whose inner functions are drawn from the training split.
Encoder training
The hyperparameters are listed in Table 11. The networks and in Equation (4) have hidden sizes 1024 and 256, respectively, and their gradients are clipped separately from those of the encoder. Batches are encoded in four chunks of similar length, since random batches contain about 70% padding; this makes training 4.9 times faster.
| Hyperparameter | Value |
| Architecture | BERT, 10 layers, hidden size 512, 8 heads, feed-forward 2176, |
| Parameters | 33M (encoder), 1.6M () |
| Initialisation | truncated normal (std 0.02), residual outputs scaled by |
| Optimizer | AdamW, learning rate ( for ), 700 warm-up steps, cosine |
| Steps / batch size | 40,000 / 384 |
| Batch composition | 15% equivalence groups, 25% sweeps, plus 15% synthetic rows (half combinations, half scaled copies) |
| Combination coefficients | |
| , , , , | 0.5, 1.5, 0.5, 1, 0.05 |
| Ramp of and | 0 until step 1000, linear to 1 at step 2000 |
| Precision | bfloat16 autocast, gradient checkpointing, length-bucketed batches |
Table 11. Hyperparameters of the encoder
Modules on the frozen encoder
The decoder has 6 layers, model size 384, 6 heads and 8 memory tokens (17.7M parameters). It is trained for 105k steps with batches of 32k tokens and a warm-up, stable and decay schedule, with dropout 0.05 and label smoothing 0.02. Its inputs are randomly rotated (cosine at least 0.99) for 40% of the rows, and 30% of the derivative rows use the predicted latent as input. The composition bank contains seven networks, trained jointly with disjoint parameters and per-network gradient clipping for 40k steps; their construction is described in Appendix § 7.§ 3. The integrator and the value encoder share the same MLP size, and are trained for 60k and 20k steps, respectively. The value encoder receives the normalised samples and their spectral derivative (Legendre degree up to 40), noise with is added to half of the training batches, and it is fine-tuned with 0.5 times the cross-entropy of the frozen decoder.
Construction of the composition operators Each composition operator , for and for the quadratic , is a separate network on the frozen latent , with and . Its output is built in the same factored form as the latent and as :
where is the latent of the constant function 1, is a linear skip initialised to the identity, and are the outputs of a residual MLP (input layer to width 768, three pre-LayerNorm residual blocks with GELU, expansion factor 2 and dropout 0.1, and a final LayerNorm). The input of and the constants depend on the operator:
, and : , and , and , respectively;
: , and ;
and : , where is the standardised , and ;
: , with the base , in which is evaluated without gradient.
The scaling laws that this construction satisfies by design are stated in Proposition 4 (Appendix B.3). Each design choice reproduces a property of the target, so that the network only has to learn what the property does not determine:
Equivariance under positive scaling. Since , and , and since the latent has an exact norm and is linear up to the scale error of the encoder (Table 1), the targets satisfy the laws of Proposition 4 up to that error. The architecture satisfies them exactly, so these networks learn a function of the direction alone, and the logarithm learns its value at unit norm, with training target .
No scaling law for and . Since is not determined by , these networks receive as an input and predict the gain freely.
Sign changes. For the laws require the parities , , and , which Equation (8) does not enforce because and are not symmetrised as in (Proposition 1). They are taught by sign augmentation: for a random half of each batch, is replaced by and the target by for the odd operators (, ) and by for the even ones (, ), and by for the quadratic; and , whose inner functions are nonnegative on in the data, are not augmented. The integrator is augmented in the same way, since integration is odd.
Linearity and closure under composition. The quadratic is not learned from scratch: by the linearity of the latent, , so it is the composition of the learned square operator with exact latent linear algebra plus a residual scaled by the norm of the base. The base alone retrieves 44.4% of the targets without any training on the quadratic (Table 3).
Factored loss. As for (§ 3.4), the direction and the log-gain are trained by separate terms, with temperature 0.05, where , which avoids the shrinkage of the predicted norm under an uncertain direction; the quadratic uses the relative squared error of its full prediction plus the same contrastive term.
Identity initialisation. With and a residual head initialised at about 1% of the norm of , every direction starts at the direction of the inner function, and the linear skip carries the part of the map that is linear in the latent (Table 3, linear part only).
External baselines
All symbolic-regression baselines receive exactly the 64 samples (and noise draws) used by f-JEPA and are scored with the same criteria; floating-point constants are snapped to simple fractions before the symbolic check, as f-JEPA does with its own constants. PySR [14] (version 2.7) uses the binary operators , , , and the unary operators , , , , , square, cube and arctan, a maximum expression size of 30, a budget of 20 seconds per function, deterministic serial search and its default model selection. The end-to-end transformer [23] is the publicly released pretrained model with the settings of its official example (200 input points, rescaling, refinement of 100 candidate trees by BFGS). Since its checkpoint is a pickled Python object, it was loaded with a restricted unpickler that resolves only the 42 classes and functions found at a static scan of the file, and two incompatibilities of the official code with NumPy 2 were fixed (an unused import and the alias np.infty).
The integrator of Lample and Charton [28] is the released FWD+BWD+IBP model, run with the official code and inference settings of its example notebook (input in its prefix notation, beam 10, at most 2000 output tokens); an integrand is solved if any hypothesis satisfies with our SymPy check. The symbolic path of Table 16 decodes the input latent (greedy and beam 8, round-trip re-ranking), applies sympy.diff or the composition at token level, writes the derivative in the canonical form of the data (shortest of the raw, together and factor_terms forms), and re-encodes it with the exact norm.
For ODE discovery, ODEFormer [16] is the released pretrained model with the beam size 50 and temperature 0.1 of its official example, and SINDy [8] and weak SINDy [35] are the implementations of PySINDy 2.1 [17, 24], with the library over the state components , the sequentially thresholded least-squares optimiser, and the model chosen among 7 thresholds by the same parsimony-weighted residual as our search. These methods identify first-order systems of observed states, so for a linear ODE of order they receive the state , with derivatives estimated by the finite differences of PySINDy (smoothed for noisy data) and with the true order , which our routes have to find. Moreover, their coefficients are snapped generously: an equation is valid if snapping with any of the denominator bounds of our search makes it exact. Both choices favour the baselines.
For the representation comparison of Table 2, SNIP [34] is the released SNIP-1d-normalized model, of which the symbolic encoder is used, with expressions written in the format of its own generator (binary sums and products, unary powers, all constants in its floating-point encoding). As a check of this input format, its symbolic and numeric embeddings retrieve each other with 42% top-1 accuracy among 200 functions of its own generator, but only 3% among 200 of our test functions, with any input range. The Lample and Charton representation is the mean of the final encoder states for the integration input. The probes of Table 2 are fitted on the same 20k training pairs for every representation: ridge regression with the regularisation chosen on a held-out 10%, and a 2-layer MLP with 1,024 hidden units trained for 3k Adam steps. The derivative probe takes as input and predicts a unit direction (cosine loss) and the log-ratio , with the prediction , so that it is not biased towards small norms.
E-Gen [52] releases code and data but no trained model, so it is not included. The checkpoints of the Lample and Charton model and of SNIP contain only tensors (and, for SNIP, a path object), so they were loaded with weights_only=True, whereas the ODEFormer checkpoint is a pickled Python object and was loaded with a restricted unpickler that resolves only the 32 globals found by a static scan. The official code of ODEFormer and SNIP required the same two NumPy 2 fixes as above.
Standard benchmarks
The standard benchmarks of § 4.7 are the test splits of Lample and Charton [28] (integration FWD, BWD and IBP; ODE1 and ODE2), the indefinite integrals of the Rubi test suite [39] and of the MIT Integration Bee as collected in the SymPy integration test suites [43], and the 23 one-dimensional systems of ODEbench [16]. Each item is kept only if it can be represented by f-JEPA: a univariate expression in (hyperbolic and inverse functions are written exactly with exp, log and arctan, and decimal constants as rationals), at most 126 tokens of our vocabulary, real and finite on without poles (no step between neighbouring points of a 20,001-point grid exceeds 5% of the range), and with a norm in the training range of the encoder. Table 12 gives the number of items kept and the main reasons for rejection; the benchmarks are scanned in a seed-0 random order, and the first 200 kept items are used.
| Benchmark | Kept | Scanned | Main rejection reasons (counts) |
| L&C integration FWD | 1000 | 2764 | not finite/real on [-pi, pi] (1096); outside norm range (283); operator outside vocabulary (212) |
| L&C integration BWD | 1000 | 8203 | not finite/real on [-pi, pi] (3463); too long (1278); operator outside vocabulary (1038) |
| L&C integration IBP | 1000 | 2098 | not finite/real on [-pi, pi] (580); outside norm range (283); operator outside vocabulary (143) |
| MIT Integration Bee | 92 | 322 | not finite/real on [-pi, pi] (163); outside norm range (35); pole / discontinuity in [-pi, pi] (18) |
| Rubi (indefinite) | 1000 | 58104 | not univariate in x (51687); not finite/real on [-pi, pi] (3570); outside norm range (923) |
| L&C ODE1 (as IVPs) | 530 | 6523 | not finite/real on [-pi, pi] (3317); operator outside vocabulary (964); pole / discontinuity in [-pi, pi] (517) |
| L&C ODE2 (as IVPs) | 383 | 4346 | not finite/real on [-pi, pi] (2147); ill-conditioned IVP on [-pi, pi] (617); outside norm range (428) |
Table 12. Coverage of the standard benchmarks: representable items kept, items scanned, and the three main reasons for rejection
The ODE test items of Lample and Charton ask for the general solution of an equation . They are converted to initial-value problems by fixing the constants of the reference solution to the first values in (pairs for ODE2) for which the particular solution is representable, and by imposing (and ) at the first point where the equation can be solved for the highest derivative; for implicit equations, the IVP also selects the branch taken by the reference solution at . Since f-JEPA works from samples, we keep only the IVPs whose numerical solution (DOP853, relative tolerance ) reproduces the reference solution on to , which removes equations that are singular inside the interval. The pretrained solvers of Lample and Charton receive the original equation and return a general solution, whose constants are then solved from the initial conditions with SymPy; SymPy dsolve receives the initial conditions directly.
For symbolic regression, NeSymReS [7] is the released 100M model with the inference configuration of its repository (beam search and BFGS constant fitting), SNIP [34] is the released encoder-decoder model with latent space optimisation by Grey Wolf optimisation (population 50, 80 iterations, as in its example), MMSR [30] is the released model with the settings of its benchmark script (run in a separate TensorFlow environment; the released model is bivariate, so the second input is a dummy variable drawn uniformly from ), and Operon [9] uses our operator set and a budget of 20 seconds per function, as PySR. Every method is given at most 600 seconds per function, after which the function counts as unsolved. On ODEBench, every method searches first-order autonomous equations , the form of all one-dimensional ODEBench systems: the learned route of f-JEPA and SINDy use the library with floating-point coefficients (the constants of ODEBench are decimals), ODEFormer [16] receives the samples with their original times, and the derivative-estimation baseline fits PySR to on smoothed finite differences. The checkpoints of NeSymReS, SNIP and the ODE solvers of Lample and Charton contain only tensors (and, for NeSymReS, a checkpoint-callback object) and were loaded with weights_only=True after a static scan; the server that hosts the Lample and Charton files had an expired TLS certificate at the time of download, so their SHA-256 hashes are recorded and the files were only loaded in this restricted mode. The official code of these baselines required only compatibility fixes for NumPy 2, Python 3.11 and PyTorch 2 (deprecated aliases, a mutable dataclass default and byte masks), which do not change their computations. As context, Lample and Charton report accuracies of 95.6% to 99.6% on their complete integration test sets and general-solution accuracies of 97.2% (ODE1) and 88.2% (ODE2) with beam 10; their protocols differ from ours, so these numbers are not included in the tables. The reference solutions pass the numeric verification of Table 9 for 97.5% (ODE1) and 96% (ODE2) of the IVPs, where the residual is the maximum of over 256 points of relative to the size of the terms. ODEFormer reports its integration accuracies only sporadically and aggregated over all dimensions, so they are not reproduced. For the integration benchmarks of Table 8, the answer of f-JEPA combines, per integrand, the solver of Algorithm 1 with a second latent proposal obtained by inverting the derivative operator, , followed by two rounds of refinement on the residual ; the answer with the smaller residual is kept, which does not use the true antiderivative, and these settings were chosen on 150 validation integrands. The second proposal alone verifies far fewer exact antiderivatives than the solver of Algorithm 1, since it lacks the substitution level, so the combination only improves the approximate rows of the table. The timeout of the ODEFormer simplifier schedules an alarm signal that it never cancels; in the first evaluation this pending alarm shortened the 60-second limit of the next search of the learned route of f-JEPA to about one second, so that 67 of its 138 searches were interrupted. The harness now cancels any pending alarm after every ODEFormer call, and the learned route was re-evaluated on the same noisy samples; the affected evaluation gave 37%, 30%, and 65% instead of the 80%, 87%, and 89% of Table 27. Three inference-only variants of the learned route (falling back to the next decoded candidate, refitting the coefficients of the decoded support in the weak form, and a wider beam) were compared on 40 synthetic validation systems, and none was more accurate than the original route, which is therefore kept.
Evaluation metrics
Scale errors are median values of with inputs mult f. Linearity compares with for training-type combinations, and additivity compares with for random pairs. Derivative retrieval finds the nearest neighbour of among all distinct test derivatives. ODE annihilation is the residual of ODE-family functions relative to that obtained with a shuffled ODE (1 is chance level), and ODE precision at 20 ranks the pool by . Decoding is exact when the relative error of the decoded expression is below , and the shape is correct when the error is below after refitting all constants to the true values. For ODE discovery, valid@1 is the fraction of functions whose top-ranked equation is satisfied exactly by the true function.
The rows of Table 1 are defined as follows.
Geometry: Pearson correlation, over random pairs of test functions, between the cosine (inner product) of the two latents and the cosine (inner product) of the two functions in , and mean absolute difference between the two cosines; an exact isometry would give 1, 1, and 0.
Linearity: the scale error above, the same error for combinations of two functions of the same family (the type seen in pretraining), and for sums of two unrelated test functions.
Derivative: top-1 and top-5 retrieval of among 3,956 candidates by nearest neighbour of ; relative error and cosine between and ; ratio of their norms; and retrieval with the best linear map fitted to by ridge regression on 60k training pairs.
Zero-shot: second derivatives obtained by applying twice, or once to , retrieved among 3,640 candidates; relative errors of the eigenfunction relations for and for ; and ODE annihilation.
Structure: Spearman correlation between the order of a swept coefficient and the order of the latents along the sweep; distance between the latents of two rewritings of the same function, relative to random pairs; and accuracy of a linear probe for the family of a function and of a linear probe for its sampled values.
In the symbolic-regression comparison of Table 5, a formula is exact when its clean relative error is below and SymPy proves its equality with the target, and it is within the threshold when its relative error with respect to the clean target is at most 0.0037, the function-space analogue of (Appendix C.2), which also counts near-exact formulas with floating-point constants. The remaining columns are, at , the share of functions with a held-out NMSE below , the median NMSE on (Extrap.; 35 of the 50 formulas of f-JEPA coincide with the target), and the median number of nodes of the formula, and, at , the median relative error with respect to the clean function. The same function-space threshold is used for the approximate antiderivatives of Table 8, with the constant fixed by . In Tables 4 and 26, the definite-integral column counts the integrands for which . In Table 16, an input is not decoded when the relative error of its decoded formula is above , and the evaluation uses 4,000 functions for the derivative and 1,000 inner functions per composition operator.
Inference procedures
This section specifies the procedures used on the frozen encoder in Section 4. In them, is the scale-free latent residual of an ODE , and is the noise level estimated from the flat floor of the Legendre spectrum of the samples. Every procedure ends with an exact check, either a SymPy proof or a high-precision numerical comparison, and no procedure uses the ground truth to select its output.
Theoretical results
Proof of Proposition 1
Consider and , and let and . Then and . Since is symmetric in and , we have . Since is linear and is odd, the numerator of is odd, and so is its normalisation, . Therefore
where .
Input: integrand (tokens and values at the 64 nodes). Output: a SymPy-verified antiderivative , or none.
, ; shortlist the 16 training functions nearest to and the 16 with nearest to .
Decode with beam 16 and refit the constants of every candidate so that on the nodes.
Exact combination: orthogonal matching pursuit (at most 6 atoms) over decoded pieces, shortlisted functions and for ; coefficients snapped to fractions.
Term splitting: integrate each term of a sum separately and add the results.
Residual peeling (2 rounds): encode the residual , integrate it with and repeat steps 2–3.
Integration by parts: for each product in , build the chain of antiderivatives of with and the decoder (constants dropped) and add the atoms to a small basis before the full one.
Substitution and reduction: reduce powers and products of trigonometric functions to sums; for every inner function of , rewrite in , integrate it with steps 1–6 and substitute back.
Return the first candidate for which SymPy proves within the time limit.
Algorithm 1: Symbolic integration on the frozen stack (levels of Table 4)
Input: values at the 64 nodes. Output: a formula.
Estimate ; compute as the norm of with a spectral derivative (low-pass to degree 40 if ); .
Decode with beam 16; refit the constants of every candidate to by weighted least squares and snap them to fractions with denominator at most 12.
Exact combination of the candidates with the atoms .
Residual peeling (2 rounds): encode the residual values with , decode them and add the results to the basis.
Return the candidate minimising bits per operator + the bits of its constants (for , the shortest exact candidate).
Algorithm 2: Symbolic regression from samples
For only the first step is needed, since then ; this is the argument behind the scaling laws of the integrator and of the composition operators (Proposition 4), whose gains and directions are not symmetrised and which therefore satisfy their laws exactly only for positive scalars.
Why a finite latent vector can reproduce the Sobolev inner product
This section answers two questions in plain terms. First, can a vector of finite dimension behave exactly like a function in the Sobolev space, with dot products equal to Sobolev inner products and linear combinations equal to linear combinations of functions? Proposition 2 shows that it can: the coefficients of a function on a Sobolev-orthonormal polynomial basis form such a vector, up to a truncation error and a quadrature error that both vanish quickly for smooth functions, so the target of the pretraining is attainable. Second, how far can the learned encoder be from this ideal? Proposition 3 and Corollary 1 show that, because the norm of a latent is computed exactly, the error of every latent dot product and latent distance comes from only two sources: the angle error of the network, which training reduces, and the error of the 64-node quadrature, which is negligible for the functions used here. Hence, any remaining error of the latent geometry is the angular error of the encoder, which is measured in Appendix § 9.§ 3.
Let , let , and let be the -node Gauss–Legendre rule mapped to with weights normalised so , which is the rule used by the encoder (). is exact for polynomials of degree at most . Let map onto , and let denote the open Bernstein ellipse with parameter in the variable . We use the following classical bound [44 Theorem 19.3], divided by 2 because of the normalised measure.
Input: ODE , pool of functions with latents and stored values. Output: verified solutions.
For every pool function compute , with , and keep the with the lowest residual.
Re-rank the shortlist by the exact residual of the ODE on the stored values of , and .
Linear ODEs: for all subsets of at most 3 functions among the 16 best and , compute the exact null vector of the residual matrix and snap it to fractions.
Refit the constants of the 50 best skeletons so that the ODE holds (scale-free residual below , every term active).
Return the candidates for which SymPy proves that the ODE holds.
Algorithm 3: Solving ODEs with closed-form solutions
Input: ODE, initial conditions, solution values from a spectral Newton method. Output: formulas with atoms.
Build a dictionary of atoms. Learned: the 200 nearest latents to , the 200 best functions by the latent ODE residual, decoded composition outputs of the 30 best, and latent residual peeling ( of the current residual, 16 neighbours and 8 decoded beams). Controls: the 200 nearest functions in value space, 200 random pool functions, and value-space residual peeling.
Greedily add the atom that most decreases , refit all coefficients by nonlinear least squares, and refine the set by swaps.
Report the relative deviation of each formula from a high-precision numerical solution (not used in steps 1–2).
Algorithm 4: Compact formulas for initial-value problems without closed form
Quadrature bound If is analytic in and bounded there by , then
The bound decays geometrically with the number of nodes; for functions that are only times differentiable the decay is algebraic in instead. The data filter of Appendix A.1 additionally checks, for every stored function, that its norm agrees between 64 and 128 nodes to a relative tolerance of .
Reference embedding Let be the basis of polynomials of degree less than that is orthonormal in (Gram–Schmidt applied to ), let be the orthogonal projection onto their span, the truncation error, the exact coefficients, and
so that each coordinate of is the discrete quadrature of , and is linear in .
Proposition 2. For whose values and derivatives are defined at the nodes, let be the quadrature error of the coefficients. Then
Moreover, as ; if is a polynomial of degree with ; and if and are analytic in with there, then
Proof. Since and are orthogonal to the span of the , the cross terms vanish in
and by orthonormality , which is the Parseval identity on the span. The Cauchy–Schwarz inequality gives . Next, $L_K(f)^\mathsf{T}L_K(g)-
Input: 64 noisy samples. Output: an equation , .
Build the library of the route: spectral derivatives of the samples (strong form), weak-form projections on 40 test functions, or exact derivatives of the formula returned by Algorithm 2 (samples formula).
Fit all supports of at most 4 terms by least squares, rank them by relative residual, and snap each coefficient to the fraction with the smallest denominator that keeps the residual within 5%.
Return the top-ranked equation, on which valid@1 is scored.
Algorithm 5: ODE discovery from noisy samples (one route)
, whose absolute value is at most , and by the Bessel inequality. The triangle inequality gives (11). Since polynomials are dense in on a bounded interval, . If is a polynomial of degree , the integrands and have degree at most , so is exact and . Finally, applying (9) to each coordinate and summing the squares gives the analytic bound. □
Since is linear and reproduces the inner product up to (11), it has all the properties sought for the latent space, and for it can be embedded in by padding with zeros.
Learned encoder The encoder does not compute : its direction is produced by a transformer from the symbolic text. Its latent (Equation 3) uses the quadrature norm , computed from the values of and of its symbolic derivative at the nodes, so that no numerical differentiation is involved. Let be the quadrature Gram entry, the target cosine of Equation 5, and the angular error of the encoder.
Proposition 3. For the latent of (3),
and if is analytic in and bounded there by , the last term is at most .
Proof By (3), , since by definition. Subtracting and applying the triangle inequality gives (12), and (9) bounds the last term. □
Corollary 1. With ,
Proof Expand and , use , and apply Proposition 3 to the cross term. □
Thus, the only term in Equation 12 that depends on the network is the angular error , whose square, averaged over the pairs of a batch, is the first term of the geometry objective (Equation 5).
Scaling laws of the composition operators and of the integrator
The composition operators and the integrator, whose construction is given in Appendix A.3, satisfy the scaling laws of their targets exactly for positive scalars.
Proposition 4. For every and every , , , , , and the integrator satisfies .
Proof For , and . For these operators , so and are unchanged, and (8) gives , while the affine term of the logarithm becomes . The integrator is the case . □
Additional experimental results
A second encoder trained on
In order to check that learning a Sobolev space does not depend on the interval, a second encoder is trained once, from scratch, with the configuration of (Table 11) on 2.0M functions generated on , and its checkpoint is selected on its validation split at step 38,000 of 40,000. Table 13 compares both encoders, each on the held-out test split of its own interval; since the two test sets contain different functions, the differences mix the effect of the encoder with that of the data. matches in geometry, scaling, linearity and derivative retrieval, and it is even slightly better on several of these metrics (cosine correlation 0.996 against 0.991, scale error at of 0.037 against 0.054, error on random sums 0.141 against 0.216), so the approach does not depend on the interval . The values of in Table 13 are computed on a smaller sample than those of Table 1, hence the small differences between the two tables. It is weaker in the zero-shot operator algebra: the annihilation residual of ODEs is 2.5 times larger, the chain error of is 32% larger and the effective rank of the directions is lower. The family probe is also lower, which we attribute to the data rather than to the encoder, since 20% of the functions generated on are numerically close to linear, against 11% on . A shorter interval was discarded for the same reason: on , 44% of the generated functions are close to linear and 5.4% of random pairs have , against 2.4% for the training data of and 0.6% for that of .
| Metric | ||
| cosine correlation | 0.991 | 0.996 |
| inner-product correlation | 0.999 | 1.000 |
| effective rank of | 54.7 | 45.4 |
| scale error, | 0.015 | 0.012 |
| scale error, | 0.054 | 0.037 |
| linearity error (combinations) | 0.070 | 0.070 |
| error on random sums | 0.216 | 0.141 |
| derivative retrieval, top-1 (4,000) | 0.820 | 0.825 |
| derivative retrieval, top-5 (4,000) | 0.960 | 0.956 |
| relative error of (median) | 0.070 | 0.076 |
| gain ratio of | 1.004 | 0.991 |
| chain error | 0.240 | 0.316 |
| , top-1 | 0.458 | 0.398 |
| ODE annihilation | 0.208 | 0.530 |
| eigenfunction error, trigonometric | 0.174 | 0.258 |
| sweep ordering (Spearman) | 0.828 | 0.868 |
| linear probe, family accuracy | 0.742 | 0.445 |
| calibrated (own validation split) | 0.0051 | 0.0038 |
Table 13. and , each on the test split of its own interval (1,024 functions for the geometry, scaling and ODE rows, 4,000 for retrieval). Errors are relative
Figure 11 shows the latent of next to that of , with the projection described in Appendix C.8, trained in the same configuration on 120k training directions of : no angle term and radius equal to the rank of . On the test split, the mean of is 0.182 for and 0.080 for (median angular errors of and ), lower than for because the directions of have a lower effective rank (Table 13), and the validation family accuracy is 0.367 and 0.448, against 0.431 and 0.518 for .
Calibration of the -criterion
Figure 6 shows the data behind the threshold of § 3.2, computed on the 19,765 distinct functions of the validation split. The relative latent gap to the nearest distinct function and the corresponding gap have almost the same distribution (medians 0.128 and 0.121), and the latent and nearest neighbours coincide for 77% of the functions. The relative error of lies between the 5th percentile and the median of these gaps, which is why retrieval succeeds while the -ball is never reached. The absolute root mean square gaps span 2.5 orders of magnitude between their 5th and 95th percentiles, because the norms of the functions do, which rules out a single absolute threshold. Finally, latent distances reproduce distances on random pairs (Pearson 0.99997), but only to a median relative error of 7% on nearest-neighbour pairs, so that very close functions are not resolved as accurately as distant ones. When the criterion is applied to a formula rather than to a latent, the relative error of the formula is compared with the threshold obtained by the same rule from the function-space gaps, 0.0037.
Table 14 re-scores the predictions of every learned operator with this criterion. No prediction reaches : the smallest relative error of on the validation split is 0.013, whereas 67.1% of its predictions are within 0.1 of their target, against a top-1 accuracy of 82.0%.
Figure 6. Calibration of the -criterion on the validation split: (a) relative gaps to the nearest distinct function and relative error of , (b) the same quantities as absolute root mean square distances, and (c) latent against distance
Numerical check of the inner-product bound
Figure 7 measures both terms of Equation (12) for on all pairs of 2,000 test functions, against a reference inner product computed with 512 nodes. The quadrature error decays geometrically with the number of nodes and is negligible from (median relative to ), whereas the error of the encoder stays at its angular floor (median 0.0083) for every , so that the error of the latent inner product is the angular error of the network. Since the encoder with another latent dimension was trained, panel (b) uses the projection of the latents of on their top principal directions as a proxy: the error reaches the floor of the encoder at , whereas a polynomial truncation of degree below , as in Proposition 2, and the optimal rank- approximation of the Gram matrix reach machine precision at the same dimension, so the 256 dimensions of the latent space are not the limiting factor.
Figure 7. Median relative error of the latent inner product on all pairs of 2,000 test functions: (a) against the number of quadrature nodes, and (b) against the dimension of a truncated representation
The propositions bound the inner-product error by , but they do not bound itself, which depends on training and is measured empirically: its mean over held-out pairs is 0.017 (Table 1).
Decoder and module predictions
Figure 8 and Table 15 detail the decoder. Errors are dominated by constants and by long expressions, whereas the function shape is correct in most cases. Second derivatives decoded from have the correct shape for 39.9% of the functions, although they are never seen by the encoder or the decoder.
Figure 8. Decoder accuracy on the test set: (a) exact reconstruction and correct shape for from , from and from the predicted latent , and (b) exact reconstruction of by expression length
Table 15: Examples of test functions and their decoding from (beam search with round-trip re-ranking); the last column gives the relative error when the decoding is not exact
| True function | Decoded function | Error |
| ✓ | ||
| ✓ | ||
| ✓ | ||
| 4.0e-01 | ||
| ✓ | ||
| \frac{16x^2e^{-\frac{9x^2}{4}}}{\frac{13x}{5}+83\atan(4x)} | \frac{16x^2e^{-\frac{9x^2}{4}}}{3x+83\atan(4x)} | 6.4e-03 |
| 1.1e-01 | ||
| 2.8e-02 | ||
| ✓ |
Table 15. Examples of test functions and their decoding from (beam search with round-trip re-ranking); the last column gives the relative error when the decoding is not exact
Figure 9 compares predicted and true quantities for every module. The integrator predicts the magnitude of the antiderivative with a log-correlation of 0.995 (median absolute log error 0.021), the composition operators follow the true gains over five orders of magnitude, and the median cosine between predicted and true latents is 0.998 for , 0.985 for , between 0.964 (cos ) and 0.995 (log ) for the composition operators, and 0.994 for with and without noise.
Figure 9. Predicted against true quantities on the test sets: (a) gain of the integrator, (b) gain of the composition operators, and (c) distribution of between predicted and true latents for every module
Table 16 gives the comparison of the learned operators with the symbolic path discussed in § 4.2.
Symbolic regression: pipeline levels and additional comparisons
Table 17 reports the exact recovery of each level of Algorithm 2 on 300 test functions. The value encoder is robust to noise, with a median cosine similarity of 0.994 to the true latent, and under noise the additional search levels do not change the selected formula, since candidates with more atoms do not pass the description-length criterion. Excluding the test functions that lie within a relative distance of of a training function gives 0.80, 0.63, and 0.43 instead of 0.80, 0.63, and 0.45.
Table 18 is an earlier comparison on a fixed subset of 100 test functions with PySR [14] and the end-to-end transformer [23], with three accuracy criteria. PySR fits the samples accurately but recovers the exact formula only for 3% of the clean functions, and the end-to-end transformer, designed for numerical accuracy rather than exact constants, is within 10% of the target for 37% to 44% of the functions. The exact rates of f-JEPA differ between the subsets of 300, 100, and 50 functions (Table 17, Table 18, and Table 5) because of their size; on a rerun of the noisy pipeline, the rate at on the 300 functions changed from 0.447 to 0.473, which gives the order of the run-to-run variation.
Table 18. Symbolic regression from 64 samples on a fixed subset of 100 test functions: share of formulas recovered exactly and with a relative error below and below . The last row decodes the true latent of the formula, which does not depend on the samples
Table 19 gives the out-of-distribution comparison of § 4.7, on random trees generated like the integrands of Table 26 but disjoint from them, without poles on . PySR, whose genetic search is not tied to a training distribution, is the strongest method on clean samples, but at 1% noise its formulas fit the noise. On the 24 classic univariate benchmarks Nguyen [47], Keijzer [25], and Livermore [37], the 7 that require a positive domain being composed with an affine map to , PySR recovers 9 formulas exactly on clean samples but a single one at 1% noise, whereas f-JEPA recovers 8 at both noise levels.
Per-equation and per-integrand results
Table 20 to Table 24 report the per-equation and per-integrand results that are summarised in § 4. For instance, the best approximation of is plus harmonics, where is the mean value of .
Figure 10 complements Table 6: the deviation of the learned formulas decreases steadily with the number of atoms and stays one to three orders of magnitude below that of the Padé approximants for every . In median, the learned and network-free dictionaries are close for most , with the clearest advantage of the learned atoms at and ; at the learned formula is more accurate than the best network-free control on 11 of the 12 equations, and more accurate than Padé on all of them.
Figure 10. Initial-value problems without closed form: (a) median deviation from the numerical solution against the number of atoms , and (b) deviation of the best control and of the Padé approximant divided by that of the best learned method for each ODE at
As an independent check that does not rely on the numerical reference solution, Table 23 reports for the same four-atom formulas (stored with four significant digits) the scale-free ODE residual over 256 points of and the error in the initial conditions at . All formulas satisfy the initial conditions to about , and none of them solves the equations exactly, as expected for compact approximations; latent peeling and composed atoms have the smallest residuals (3.9% and 4.3%), against 7.2% for the best network-free control.
| Method | ODEs | Median residual | Median IC error |
| Random atoms (control) | 12 | 1.5e-01 | 1.0e-03 |
| Value-space neighbours (control) | 12 | 7.2e-02 | 2.0e-04 |
| Value-space peeling (control) | 12 | 1.0e-01 | 3.2e-04 |
| Latent neighbours | 12 | 1.1e-01 | 2.4e-04 |
| Latent ODE shortlist | 11 | 1.0e-01 | 2.9e-04 |
| Latent peeling | 12 | 3.9e-02 | 9.4e-05 |
| Latent shortlist + atoms | 11 | 1.0e-01 | 2.9e-04 |
| + composed atoms | 12 | 4.3e-02 | 2.6e-04 |
Table 23. Residual and initial-condition check of the four-atom formulas for the 12 initial-value problems without closed form (medians over the ODEs; formulas with four significant digits).
Table 25 lists the out-of-distribution integrands of Table 26 for which the learned levels of our solver return an antiderivative verified by SymPy, whereas SymPy’s own integrate returns none within 10 seconds. In every case the antiderivative is a nested composition, which the learned integrator reaches after the substitution or by-parts step has reduced the integrand to a function close to the training families.
Table 26 and Table 27 give the complete results of the out-of-distribution integration and of the ODEBench discovery discussed in § 4.7. On ODEBench, the stricter threshold gives the same ranking of the methods as .
Why few ODE solutions of Lample and Charton are verified
Table 28 analyses the 400 initial-value problems of Table 9 from their reference solutions and the formulas returned by the samples-to-formula route of f-JEPA (original decoder, no ODE-aware search) (‘numerically exact’: relative error below with respect to the reference solution on the 64 nodes, but not proven exact by SymPy). The reference solutions have the length of our training expressions and use only operators of our vocabulary, so the failures are not due to length or to unknown operators. Two causes account for them. First, a quarter to a third of the references contain irrational constants coming from the integration constants of the generator of Lample and Charton (for instance or ); f-JEPA recovers about 40% of these formulas with a rational approximation of the constant (, relative error ), which no exact test can accept. Second, for references with rational constants, about three quarters of the formulas have the wrong shape, because these solutions combine products and compositions of elementary functions that are not among the parameterised families of the pretraining data. The last row of Table 28 shows that the limitation lies in the pretrained encoder and decoder rather than in the value encoder: when the latent is computed from the symbolic form of the reference solution and given to the same solver, only 7.5% and 4.5% of the solutions are recovered exactly (34% and 32% within a relative error of ), against 87% for the functions of our test set. Both causes are addressed by the two additions of Table 9: the decoder and the value encoder are fine-tuned on expressions of the Lample and Charton training sets, with the encoder and frozen, and the selection of a single formula is replaced by an ODE-aware search. The search keeps the whole candidate pool of the samples-to-formula solver, refits the coefficients of every candidate on the numerical solution, identifies them as exact constants by integer-relation detection (PSLQ) against , , , , , and solves the initial conditions exactly for one or two coefficients, and ranks the resulting formulas by their residual in the equation. Under the stricter symbolic proof, the two additions raise the verified shares from 5% and 5.5% to 16.5% and 18%. Ranking by the latent residual under instead of the numeric residual was slightly worse in a preliminary run on 40 IVPs per set, since the noise floor of cannot separate the exact formula from near-exact ones. The ODE-aware search identifies the irrational constants exactly (for instance it returns and ), and fine-tuning the decoder on expressions of the Lample and Charton training sets raises the exact recovery from the true latent on the first 40 IVPs of each set from 15% to 25% (ODE1) and from 5% to 8% (ODE2). On ODE1, the remaining failures of the final route are formulas of the wrong shape (53.5% of the IVPs) or close to the reference without being exact (21.5%), and none of them solves the equation with initial conditions of another solution, so the IVP formulation itself is not a source of error.
| ODE1 | ODE2 | |
| Median prefix length of the reference solution (pretraining: 18) | 19 | 19 |
| Reference beyond the 99th percentile of pretraining lengths | 0.025 | 0.000 |
| Reference with irrational constants | 0.355 | 0.245 |
| f-JEPA verified | 0.000 | 0.020 |
| f-JEPA numerically exact, not provable (rel. error ) | 0.394 | 0.408 |
| Reference with rational constants only | 0.645 | 0.755 |
| f-JEPA verified | 0.078 | 0.066 |
| f-JEPA wrong shape (rel. error ) | 0.736 | 0.748 |
| True latent instead of : exact recovery | 0.075 | 0.045 |
Table 28. Diagnosis of the IVPs of Table 9 (200 per test set): reference solutions and outcome of f-JEPA; shares within each group of references
Projections of the latent space
The projection of Figure 3 and Figure 11 is built as follows. An autoencoder with layers and maps the unit directions to codes normalised to the unit sphere , and is trained for 60 epochs on the 120k training directions with the loss . The scale is then reintroduced as the radius, , and the display was chosen on the validation split by the 10-nearest-neighbour accuracy of the function family in the displayed coordinates, among the radii and and an additional loss term that preserves the angles. The angle term raises the correlation between code and latent cosines from 0.50 to 0.62 for and from 0.61 to 0.73 for , but it does not improve the family accuracy, so it is not used. On the test split, the mean of is 0.283 for and 0.115 for and ), and the validation family accuracy is 0.431 and 0.518, against 0.630 for the UMAP projection of the same functions, which preserves local neighbourhoods better but discards the norms and has no explicit map.
!Latents of and projected in two and three dimensions bbox=0.17,0.09,0.83,0.38
Figure 11. Latents of (left) and (right) projected in two (top) and three (bottom) dimensions, each for (a) 120k training functions and (b) 20.4k test functions of its own interval. The construction and colouring are those of Figure 3, which is the top left panel
Time and memory consumption
Table 29 reports the training cost of each component of f-JEPA; peak GPU memory during encoder training was 2.7 GB with gradient checkpointing. All components together have 120M parameters (encoder and 35.1M, decoder 17.7M, composition bank 52.9M, integrator 7.6M and value encoder 7.4M), and every one of them is trained on a single 8 GB laptop GPU.
| Component | Parameters | Steps | Time |
| Encoder and | 35.1M | 40k | 6.6 h (0.6 s/step) |
| Decoder | 17.7M | 105k | 3.5 h (0.11 s/step) |
| Composition bank (7 operators) | 52.9M | 40k | 1 h |
| Integrator | 7.6M | 60k | 13 min (0.013 s/step) |
| Value encoder (fine-tuning) | 7.4M | 20k | 39 min |
Table 29. Training cost of the components of f-JEPA on a single RTX 4070 Laptop GPU (8 GB)
Table 30 compares the inference cost and the size of f-JEPA with those of the other methods, measured in our own runs on the same machine (16 CPU cores and the same GPU). For f-JEPA, the parameters are those of the modules that the task uses: the value encoder and the decoder for symbolic regression and for the tasks that start from samples (ODE solving and discovery), and the encoder with , the integrator and the decoder for integration. On every task, f-JEPA has fewer parameters than every pretrained neural network it is compared with: 25.1M against 26.4M (NeSymReS) to 166.7M (SNIP) for symbolic regression, 60.4M against 185.0M for integration and for ODE solving with the models of Lample and Charton, and 25.1M against 60.6M for ODE discovery with ODEFormer. Even all the modules of f-JEPA together (120M) are smaller than a single integrator of Lample and Charton. For symbolic regression, f-JEPA is also the fastest method: a forward pass of the value encoder, a beam search of the decoder and a few least-squares refits take 0.44 seconds per function, whereas the search-based methods use their whole budget (PySR and Operon, 20 seconds), SNIP refines its latent by an evolutionary search (23 seconds) and MMSR needs 93 seconds; the pretrained transformers E2E and NeSymReS take 2.0 and 1.6 seconds. For integration and ODE solving, in contrast, f-JEPA is slower than the sequence-to-sequence models, since it explores several routes and verifies every candidate with SymPy: 3.4 seconds per integrand against 0.15 to 0.91 seconds for Lample and Charton, and 1.4 seconds per IVP against 0.29 seconds. In terms of memory, the integration solver peaks at 4.1 GB of GPU memory, most of which holds the latents of the 2M-function pool used for the shortlists.
| Task | Method | Seconds per item | Parameters |
| Symbolic regression | f-JEPA (value encoder and decoder) | 0.44 | 25.1M |
| Symbolic regression | NeSymReS [7] | 1.6 | 26.4M |
| Symbolic regression | E2E transformer [23] | 2.0 | 93.5M |
| Symbolic regression | Operon [9] (20 s budget) | 20.0 | – |
| Symbolic regression | PySR [14] (20 s budget) | 20.2 | – |
| Symbolic regression | SNIP [34] | 23.1 | 166.7M |
| Symbolic regression | MMSR [30] | 92.7 | 90.0M |
| Integration | f-JEPA (encoder, , integrator, decoder) | 3.4 | 60.4M |
| Integration | Lample and Charton [28] | 0.15–0.91 | 185.0M |
| Integration | SymPy integrate | 0.03–0.43 | – |
| ODE solving (IVPs) | f-JEPA (value encoder and decoder) | 1.43 | 25.1M |
| ODE solving (IVPs) | Lample and Charton [28] | 0.29 | 185.0M |
| ODE solving (IVPs) | SymPy dsolve | 4.34 | – |
| ODE discovery | f-JEPA (value encoder and decoder) | 3.21 | 25.1M |
| ODE discovery | ODEFormer [16] | 0.54 | 60.6M |
| ODE discovery | SINDy (PySINDy) | 0.03 | – |
| ODE discovery | Weak SINDy (PySINDy) | 0.35 | – |
| ODE discovery | Estimated derivatives + PySR | 20.3 | – |
Table 30. Inference time per item (median wall-clock seconds on the same machine) and number of parameters of the networks used by each method. For f-JEPA on symbolic regression and integration, the time is the mean over the 300 test items
Galleries of predictions
The following figures show unselected examples (the first test functions in a fixed random order, or every problem of a benchmark), except Figure 13, whose examples are selected as described in its caption; the examples already shown in Figure 4 and Figure 5 are not repeated. In Figure 4, the panels show the first exact (a) and the first inexact (b) reconstruction of , the first exact (c) and an inexact (d) decoding of with the correct shape, the first decoding of with the correct shape (e), and the first inexact decoding of (f), with antiderivatives shown with zero mean; in Figure 5, panels (a) and (b) are the first exact and the first inexact recovery. The decoded formula is the top candidate of beam search without re-ranking or constant refitting, so these examples include the failure modes of the latent operators and of the decoder. Below each panel, the true formula is written in black and the produced formula in orange (floating-point coefficients rounded to three significant digits; very long sums are truncated).
Figure 13. Composition outputs decoded to formulas, two per operator in the order sin, cos, log, , square, cube, and quadratic. These are the decodings with the smallest error among the first 150 random test pairs, and are not representative of the average decoding
In this unselected sample (Figure 12) no derivative is decoded exactly, in agreement with the 6.8% exact rate of § 4.2: 6 of the 11 decoded derivatives have the correct family and frequencies with relative errors below 0.25, whereas the other 5 miss a phase, a term or a scale, mostly for oscillatory functions with large coefficients.

Figure 12. Derivatives predicted in latent space and decoded, from , for the first test functions in a fixed random order, without the one shown in Figure 4 (titles give )
Decoding the outputs of the composition operators is much harder than retrieving them, since the decoder is not trained on composed latents and the composed expressions are long; nevertheless, Figure 13 shows that for every operator some outputs are decoded into formulas that follow the true function closely, two of them exactly ( and ), and that the typical residual error is a constant or a frequency slightly off, whereas the same latents retrieve the correct function for 57% to 92% of the pairs (Table 3).
Figure 14 to Figure 16 complete the picture for the applications: at 1% noise, symbolic regression errors are small constant errors, the five-term approximations of non-elementary antiderivatives deviate by at most 3.1% from the antiderivative obtained by quadrature (Table 24 reports the larger error of their derivative with respect to the integrand), and four-atom formulas track the numerical solutions of all twelve initial-value problems, with the Painlevé-I type equation of Figure 5 as the only case with a visible deviation.
Figure 14. Symbolic regression from 64 samples at for the first 12 test functions in a fixed random order that are not shown in Figure 5 (titles give the true function)

Figure 16. The ten initial-value problems without closed-form solution that are not shown in Figure 5: four-atom formula of the best learned method against the numerical solution
Figure 15. The integrands without elementary antiderivative of Table 24 that are not shown in Figure 5: five-term symbolic approximation against cumulative quadrature (relative error of the antiderivative, both curves with zero mean). Panels: , relative error 0.0079; formula: . , relative error 0.00016; formula: . , relative error 0.0031; formula: . , relative error 0.00086; formula: . , relative error 0.00029; formula: . , relative error 0.00016; formula: . , relative error 0.0022; formula: . , relative error 0.0067; formula: . , relative error 0.0085; formula: . , relative error 0.0003; formula: .
References
- [1]Miltiadis Allamanis, Pankajan Chanthirasegaram, Pushmeet Kohli, and Charles Sutton. Learning continuous semantic representations of symbolic expressions. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 80–88, 2017.arxiv.org/abs/1611.01423
- [2]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629, 2023.arxiv.org/abs/2301.08243
- [3]George A. Baker and Peter Graves-Morris. Padé Approximants. Cambridge University Press, 2 edition, 1996.
- [4]Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025.arxiv.org/abs/2511.08544
- [5]Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022.arxiv.org/abs/2105.04906
- [6]Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2404.08471
- [7]Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascondolo. Neural symbolic regression that scales. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 936–945, 2021.arxiv.org/abs/2106.06427
- [8]Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016.DOI
- [9]Bogdan Burlacu, Gabriel Kronberger, and Michael Kommenda. Operon C++: An efficient genetic programming framework for symbolic regression. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pages 1562–1570, ACM, 2020. doi: 10.1145/3377929.3398099.DOI
- [10]William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabrício Olivetti de França, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason H. Moore. Contemporary symbolic regression methods and their relative performance. In Advances in Neural Information Processing Systems, Track on Datasets and Benchmarks, 2021.arxiv.org/abs/2107.14351
- [11]Yang Chang, Kuang-Da Wang, Ping-Chun Hsieh, Cheng-Kuan Lin, and Wen-Chih Peng. DDOT: A derivative-directed dual-decoder ordinary differential equation transformer for dynamic system modeling. arXiv preprint arXiv:2506.18522, 2025.arxiv.org/abs/2506.18522
- [12]François Charton. Linear algebra with transformers. Transactions on Machine Learning Research, 2022.arxiv.org/abs/2112.01898
- [13]Delong Chen, Mustafa Shukor, Theo Moutakanni, Willy Chung, Jade Yu, Tejaswi Kasarla, Allen Bolourchi, Yann LeCun, and Pascale Fung. VL-JEPA: Joint embedding predictive architecture for vision-language. In International Conference on Learning Representations, 2026. arXiv:2512.10942.arxiv.org/abs/2512.10942
- [14]Miles Cranmer. Interpretable machine learning for science with PySR and SymbolicRegression.jl. arXiv preprint arXiv:2305.01582, 2023.arxiv.org/abs/2305.01582
- [15]Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Świszcz, and Razvan Pascanu. Sobolev training for neural networks. In Advances in Neural Information Processing Systems, volume 30, pages 4278–4287, 2017.arxiv.org/abs/1706.04859
- [16]Stéphane d’Ascoli, Sören Becker, Philippe Schwaller, Alexander Mathis, and Niki Kilbertus. ODEFormer: Symbolic regression of dynamical systems with transformers. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.05573
- [17]Brian M. de Silva, Kathleen Champion, Markus Quade, Jean-Christophe Loiseau, J. Nathan Kutz, and Steven L. Brunton. PySINDy: A Python package for the sparse identification of nonlinear dynamical systems from data. Journal of Open Source Software, 5(49):2104, 2020. doi: 10.21105/joss.02104.DOI
- [18]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, 2019.arxiv.org/abs/1810.04805
- [19]Neeraj Gangwar and Nickvash Kani. Semantic representations of mathematical expressions in a continuous vector space. Transactions on Machine Learning Research, 2023.arxiv.org/abs/2211.08142
- [20]Jean-Bastien Grill, Florian Strub, Florent Alché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2006.07733
- [21]Tyler Ingebrand, Adam J. Thorpe, and Ufuk Topcu. Zero-shot transfer of neural ODEs. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2405.08954
- [22]Divyansh Jha, Yuanfang Xie, Varan Mehra, and Brennen Yu. BRo-JEPA: Learning modular arithmetic in latent space. arXiv preprint arXiv:2606.01372, 2026.arxiv.org/abs/2606.01372
- [23]Pierre-Alexandre Kamienny, Stéphane d’Ascoli, Guillaume Lample, and François Charton. End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems, volume 35, 2022.
- [24]Alan A. Kaptanoglu, Brian M. de Silva, Urban Fasel, Kadierdan Kaheman, Andy J. Goldschmidt, Jared Callaham, Charles B. Delahunt, Zachary G. Nicolaou, Kathleen Champion, Jean-Christophe Loiseau, J. Nathan Kutz, and Steven L. Brunton. PySINDy: A comprehensive Python package for robust sparse system identification. Journal of Open Source Software, 7(69):3994, 2022. doi: 10.21105/joss.03994.DOI
- [25]Maarten Keijzer. Improving symbolic regression with interval arithmetic and linear scaling. In Genetic Programming: 6th European Conference, EuroGP 2003, volume 2610 of Lecture Notes in Computer Science, pages 70–82, 2003.DOI
- [26]Bernard O. Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences, 17(5):315–318, 1931.pubmed.ncbi.nlm.nih.gov/16577368
- [27]Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizadehnesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89):1–97, 2023.
- [28]Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations, 2020.arxiv.org/abs/1912.01412
- [29]Yann LeCun. A path towards autonomous machine intelligence. OpenReview preprint, 2022.
- [30]Yanjie Li, Jingyi Liu, Min Wu, Lina Yu, Weijun Li, Xin Ning, Wenqiang Li, Meilan Hao, Yusong Deng, and Shu Wei. MMSR: Symbolic regression is a multi-modal information fusion task. Information Fusion, 114:102681, 2025. doi: 10.1016/j.inffus.2024.102681.DOI
- [31]Zongyi Li, Nikola Kovachki, Kamyar Azizadehnesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
- [32]Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3:218–229, 2021.
- [33]Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9:4950, 2018.DOI
- [34]Kazem Meidani, Parshin Shojaee, Chandan K. Reddy, and Amir Barati Farimani. SNIP: Bridging mathematical symbolic and numeric realms with unified pre-training. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.02227
- [35]Daniel A. Messenger and David M. Bortz. Weak SINDy: Galerkin-based data-driven model selection. Multiscale Modeling & Simulation, 19(3):1474–1497, 2021.DOI
- [36]Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, et al. SymPy: Symbolic computing in Python. PeerJ Computer Science, 3:e103, 2017.
- [37]T. Nathan Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P. Santiago, Daniel Faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In Advances in Neural Information Processing Systems, volume 34, 2021.
- [38]Brenden K. Petersen, Mikel Landajuela Larma, T. Nathan Mundhenk, Claudio P. Santiago, Soo K. Kim, and Joanne T. Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations, 2021.
- [39]Albert Rich, Patrick Scheibe, and Nasser Abbasi. Rule-based integration: An extensive system of symbolic integration rules. Journal of Open Source Software, 3(32):1073, 2018. doi: 10.21105/joss.01073.DOI
- [40]Paul Saegert and Ullrich Köthe. Breaking the simplification bottleneck in amortized neural symbolic regression. arXiv preprint arXiv:2602.08885, 2026.arxiv.org/abs/2602.08885
- [41]Peter J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics, 656:5–28, 2010.
- [42]Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K. Reddy. LLM-SR: Scientific equation discovery via programming with large language models. In International Conference on Learning Representations, 2025.arxiv.org/abs/2404.18400
- [43]SymPy Development Team. integration-test-suites: A corpus of symbolic integration problems. https://github.com/sympy/integration-test-suites, 2026. Commit 871364f.
- [44]Lloyd N. Trefethen. Approximation Theory and Approximation Practice. SIAM, Philadelphia, 2013.
- [45]Silviu-Marian Udrescu and Max Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaba2631, 2020.DOI
- [46]Mert Ünsal, Timon Gehr, and Martin Vechev. AlphaIntegrator: Transformer action search for symbolic integration proofs. arXiv preprint arXiv:2410.02666, 2024.arxiv.org/abs/2410.02666
- [47]Nguyen Quang Uy, Nguyen Xuan Hoai, Michael O’Neill, R. I. McKay, and Edgar Galván-López. Semantically-based crossover in genetic programming: Application to real-valued symbolic regression. Genetic Programming and Evolvable Machines, 12(2):91–119, 2011.DOI
- [48]Marco Valentino, Jordan Meadows, Lan Zhang, and André Freitas. Multi-operational mathematical derivations in latent space. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1446–1458, 2024.
- [49]Martin Vastl, Jonáš Kulhánek, Jiří Kubalík, Erik Derner, and Robert Babuška. SymFormer: End-to-end symbolic regression using transformer-based architecture. IEEE Access, 12:37840–37849, 2024.arxiv.org/abs/2205.15764
- [50]Jarkko Venna and Samuel Kaski. Neighborhood preservation in nonlinear projection methods: An experimental study. In Artificial Neural Networks – ICANN 2001, volume 2130 of Lecture Notes in Computer Science, pages 485–491. Springer, 2001.DOI
- [51]Sean Welleck, Peter West, Jize Cao, and Yejin Choi. Symbolic brittleness in sequence models: On systematic generalization in symbolic mathematics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8629–8637, 2022.arxiv.org/abs/2109.13986
- [52]Hongbo Zheng, Suyuan Wang, Neeraj Gangwar, and Nickvash Kani. E-Gen: Leveraging E-graphs to improve continuous representations of symbolic expressions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 11772–11788, 2025.arxiv.org/abs/2501.14951










