WorksComputer Science
Observable-Specific Validity Tests for Scientific AI: Controlled Approximations and Intervention Response
Abstract
An approximation can reproduce an energy while missing its response, or satisfy an error tolerance without being certified by a given bound. We formulate observable-specific validity records for scientific artificial intelligence (AI) that link a reduction to its reference model, error measure, tolerance, parameter domain, and transformations or interventions. Explicit displacement and velocity certificates for a singularly perturbed oscillator recover distinct convergence orders and provide sufficient validity guarantees. Exact Hubbard dimer relations resolve the different coupling domains accepted by energy and response criteria. Across 16 repeated neural fits, imposing dimensional homogeneity reduces energy scale transfer error from 52.3% to 0.0768% and raises the median fraction jointly satisfying gap and response tolerances from zero to 100%. Paired transformations verify the protected scaling relation, while a coupling crossover identifies the additional information needed for response transfer. A causal model recovers a positive intervention response beneath a strong negative observational association. For two mechanisms observed with Gaussian noise whose responses first differ at order , we derive an exact fixed control query requirement scaling as and verify its full success curve numerically. Together with a 256-task evaluation interface, these results provide independently checkable tests of controlled approximation, physical transfer, and mechanism discrimination in scientific AI.
Introduction
A useful physical model often becomes more explanatory by retaining fewer details. In a strongly interacting electron system, virtual charge excitations can be replaced by a low energy spin exchange. In a strongly damped mechanical system, a rapidly relaxing momentum can be eliminated from the slow displacement dynamics. The resulting description succeeds when the retained variables and neglected processes have the right hierarchy for the observable being studied. Establishing this hierarchy gives numerical predictions their physical interpretation.
Scientific artificial intelligence (AI) increasingly supports this process through prediction, equation discovery, and the organization of scientific evidence [1, 2]. Recent perspectives place large language models (LLMs) within the broader cycle of hypothesis formulation, modeling, and experimental assessment [3]. In physics, language models can execute multistep Hartree–Fock calculations when supplied with an appropriate decomposition [4]. PhysReason, PHYBench, and the condensed matter benchmark CMPhysBench assess demanding physical problems [5, 6, 7], while CritPt and PRL-Bench extend evaluation toward research calculations [8, 9]. These capabilities motivate quantitative tests of the physical commitments that accompany a successful prediction.
The connection between prediction and understanding also has a substantial philosophical foundation. Contextual accounts relate understanding to knowledge that a scientific community can use and manipulate [10]. Analyses of machine learning emphasize the evidential connection between a model and its target [11, 12]. We give this connection an operational form. A system demonstrates a specified physical capability when its explanation makes correct, testable commitments about reductions, limits, and interventions. This definition ties the evaluated capability to explicit scientific consequences.
Several recent studies directly motivate such tests. PhySense evaluates principle-based reasoning, including symmetry, dimensional analysis, power counting, and limiting cases [13]. PhysGym controls the prior information supplied during interactive discovery. Newton-Bench changes physical laws and experimental environments, and LLM-SRBench tests equation discovery beyond common formula recall [14, 15, 16]. DiscoverPhysics separately evaluates predictive error and explanatory quality in noncanonical simulated worlds [17]. On the methodological side, KeplerAgent uses inferred physical structure to constrain symbolic regression, while explicit physical modeling has been separated from execution in physics reasoning [18, 19]. Our focus is the validity ity of an approximation for each observable. Reference quantities, acceptance tolerances, validity domains, and sufficient certificates become explicit evaluation objects. This adds an independently checkable validity decision to principle-based reasoning and interactive discovery.
Physical structure provides a productive basis for such commitments. Reviews of physics informed machine learning describe how equations, symmetries, and domain knowledge can guide learning[20]. Dimensional analysis constrains admissible functional forms[21, 22], and sparse discovery identifies compact dynamics within a chosen library[23]. Hamiltonian, Lagrangian, and equivariant neural architectures encode conservation or transformation relations[24, 25, 26]. Physics informed neural networks (PINNs) enforce differential equations through residual losses[27]; studies of their optimization identify conditions under which these constraints are successfully learned[28]. These approaches show the value of making a physical requirement explicit. Our framework extends this logic to the validity domain of a proposed reduction and to the observables it is intended to describe.
Our contribution is an evaluation record for individual observables linking a proposed reduction to its reference model, target quantities, validity domain, acceptance tolerance, and intervention consequences. We make the record executable through distinct tests for certified validity, measured error, transformation consistency, and joint observable acceptance. Explicit fast–slow certificates, exact dimer relations, paired neural experiments, and a known causal model instantiate these tests. An exact Gaussian measurement calculation then connects mechanism separation to query cost. The reported evidence consists of analytical models, neural surrogates, and controlled simulations; it supplies reference tests for subsequent language model evaluations.
Observable-specific approximation certificates
The object being evaluated
Let denote a reference physical model with parameters . A proposed reduction specifies its retained variables or projection, effective equations, neglected terms, and controlling dimensionless quantities. We represent its claimed scope by
where is the set of target observables, the parameter domain, the observation window, the normalization rules, the acceptance tolerances, the proposed error bounds, and the specified transformations or interventions. For observable , define
The positive scale function is declared before evaluation. The rule is fixed; its numerical value may depend on the parameters. Here displacement and dimensionless velocity use unit reference scales, gap error uses , and response-exponent error uses . An intervention difference uses a declared nonzero response scale; the dimensionless causal example below uses . A certificate establishes by analysis or a justified bound. For domain bookkeeping set outside its proof scope, while retaining an explicit out-of-scope label. Its certified domain satisfies
Equation (3) gives two complementary outputs. The certified domain provides a sufficient guarantee, while the reference error determines actual acceptance whenever it can be evaluated. Points with remain eligible for reference based verification. Physical tolerances such as describe the approximation itself; the numerical answer matching tolerance used by the evaluation interface measures the accuracy of a reported answer.
The record organizes the physical content of an explanation into the obligations in Table 1. An observation window specifies which fast processes must be retained, a transformation specifies which relations must be preserved, and an intervention specifies which response must change. Each successful test then supports a concrete part of the explanation. Alternative reductions are evaluated through the same declared quantities and tolerances.
| Obligation | Test object | Supported physical capability |
| Observable validity | Defined norm, scale, and observation window | Resolving displacement and initial velocity on their relevant time scales |
| Controlled reduction | Small parameter, retained variables, omitted terms | Locating the regime in which eliminated processes remain perturbative |
| Physical transformation | Covariance under the specified change | Preserving the physical relation under the specified transformation |
| Intervention response | A changed equation, coupling, or structural assignment | Predicting the response to a changed physical assignment |
| Mechanism discrimination | Information from allowed measurements at fixed cost | Selecting measurements that separate competing physical responses |
Table 1. Physical commitments made explicit by the evaluation record. Each row connects a scientific claim to a quantitative test and the capability it supports.
A residual certificate with an explicit physical scope
Residual-based error control and goal oriented estimates have a substantial numerical-analysis foundation[29]. Green function bounds also support validated computations for singularly perturbed boundary value problems[30]. We use the same residual-to-error principle in a known initial value problem, obtaining explicit constants for two observables and separating initial mismatch from fast residual layers. These constants make the approximation record directly evaluable.
Consider the dimensionless damped oscillator
where and for mass , stiffness , damping , and physical time . Primes denote derivatives with respect to . For , the causal Green function of the differential operator is nonnegative and has integral one on . Write . For ,
with by continuity.
Proposition 1 (Displacement certificate). For and , let be a candidate trajectory for Eq. (4), and let . On , with and ,
The norms of each residual component are taken over .
The proof is given in Appendix A. The bound separates initial data mismatch, persistent residuals, and short residual layers. A rapidly decaying residual can have a large instantaneous amplitude and a small integrated effect on displacement. The maximum norm controls persistent forcing, whereas the integrated norm captures a brief impulse. Choosing between them for each residual component retains the time scale of the eliminated process within an explicit displacement guarantee.
For the leading reduction , Eq. (6) gives
A matched first order displacement is
Its initial derivative mismatch is . Splitting the slow and fast residuals yields
Thus the certificate itself recovers and directly from the equation and the residual structure.
Proposition 2 (Velocity certificate). Under the assumptions of Proposition 1,
The derivative of the Green function weights the residual differently from the displacement kernel. For the two reductions, this gives
The first remains order unity, while the second is . Appendix A establishes the kernel identities and the bounds.
Transformations and interventions require different responses
For a physical transformation , the observable obeys the corresponding map . Unit changes, label permutations, and removable gauge phases belong in this class when their assumptions hold. An intervention changes a coupling, boundary condition, or structural assignment and specifies a corresponding response. We evaluate paired correctness alongside transformation consistency, so that the score records both the preserved relation and agreement with the physical reference.
For a specified intervention pair, we test
with a prespecified nonzero response scale. When observations admit several intervention responses, the record identifies the compatible mechanisms and an informative measurement that separates them[31, 32, 33]. In this way the explanation leads either to a verified response or to a concrete experimental question.
Physical models and computational methods
Fast–slow dynamics
The ratio in Eq. (4) compares the inertial relaxation time with the restoring time . We compare the exact solution, , and over . A grid of 220 logarithmically spaced parameters and 5000 time points combining logarithmic and uniform sampling on resolves the initial layer. Velocity is evaluated over both and the delayed window . Continuous displacement boundaries at tolerance 0.05 are located by optimizing over temporal error extrema and applying a bracketed root solver in . Independent Radau integration verifies the exact trajectories, including critical damping. Numerical integration and root finding use SciPy[34].
Effective exchange and learned representations
We use the symmetric two-site Hubbard model with two electrons,
There is one undirected bond, , and the spins are . The exact dimer provides a transparent reference for effective-model reasoning[35]. Virtual high-energy excitations motivate a low-energy transformation of the type formalized by Schrieffer and Wolff[36]. The triplet energy is zero and the relevant singlet block is
Here h.c. denotes the Hermitian conjugate. The positive singlet–triplet gap is
corresponding to . We also evaluate and . Numerical evaluation uses the rationalized expression to avoid cancellation.
We compare a raw multilayer perceptron (MLP) with a dimensionless multilayer perceptron (-MLP), where denotes dimensionless groups. Both networks have two hidden layers of 32 hyperbolic tangent units and a linear output. They use the limited memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimizer[37], implemented in scikit-learn[38]. The raw MLP maps to , where is a fixed reference energy. The -MLP maps to and reconstructs by multiplication by , imposing the dimensional homogeneity exactly. This comparison isolates the effect of a supplied physical representation within a common network architecture.
Each of 16 independent fits uses 240 physical training points sampled uniformly in logarithmic coordinates from and . Each fit has 512 test points in each of three distributions comprising the training range, an energy scale shift with , and a coupling ratio shift with . One range changes at a time, and the two representations share the physical samples. Training uses exact gap labels, with test points reserved for evaluation. Inputs are standardized using training statistics, and the natural logarithmic targets retain their original scale.
Besides relative gap error, we measure the response exponent at fixed , using centered differences with logarithmic step . Central values are medians of the per fit test medians; percentile ranges describe variation across the repeated fits.
A known observational and interventional model
To isolate causal response, we use centered, dimensionless structural equations driven by independent standard normal exogenous noises[31],
Here is a measured common cause, an intermediate quantity, and the response. Each of 400 independent repetitions contains 400 observational samples and 400 randomized samples. The randomized assignment is , independent of and both outcome noises. It produces , with a fresh independent standard-normal . Observational and randomized conditions share the same 400 draws of , but use independent -assignment and outcome noises. All ordinary least-squares regressions include an intercept; unadjusted and -adjusted estimates use the same observational samples. The coordinates are dimensionless and have zero population means, with no distribution-specific rescaling.
Two-mechanism measurement design
Consider equal-prior hypotheses
with independent observations , , , and . Gains, saturation scale, and noise are known, and all queries have equal cost. Bayesian design selects a query by expected posterior information gain[39]. We use 40-point Gauss–Hermite quadrature on 48 logarithmically spaced candidate controls. Under these equal-variance known hypotheses, the best query is for every nondegenerate posterior, so the information strategy repeats that control. The known hypotheses give an analytically tractable reference for measurement selection.
We compare it with uniform controls on , logarithmically uniform controls, and repeated measurements. The eight-query experiment has 500 noise repetitions, evenly divided between the two true hypotheses, with common noise sequences paired across policies. The outcome is the fraction with posterior probability at least 0.95 on the true hypothesis at each budget. Wilson intervals quantify finite repetition uncertainty[40]. The comparison isolates measurement selection for these two known physical mechanisms.
Results
Initial layer matching improves displacement and velocity at different orders
As shown in Figure 1(a,b), the exact dynamics separate slow displacement from the initial velocity layer. At , both approximate displacements are close to the exact trajectory. The leading solution nevertheless starts with , whereas the reference velocity is zero. For the full window,
Figure 1. Observable-dependent validity of the damped-oscillator reduction. (a) Displacement and (b) early velocity at . (c) Solid displacement-error curves cover the full plotted parameter range; dashed residual certificates apply only for . The horizontal line is tolerance 0.05 and the vertical line marks the onset of oscillatory exact motion. (d) Velocity errors on full (solid) and delayed (dashed) windows, with full window certificates (dash-dot) restricted to . All errors are deterministic, using the initial displacement and dimensionless time unit as reference scales.
for every positive . This persistent initial mismatch identifies the fast momentum relaxation that must be restored for a full window velocity approximation. Panel (d) shows how a delayed observation window instead selects the slow dynamics.
The mismatch originates in eliminating momentum before its initial relaxation. The fast displacement component has amplitude and duration , so its derivative contributes at order unity even as its displacement vanishes. Consequently, the zero-inertia and initial-time limits do not commute for velocity. The matched solution restores this component, leaving displacement and full window velocity errors of orders and , respectively. Fitted exponents over are 0.993 and 1.993 for the leading and matched displacement errors, and 0 and 1.000 for the respective full window velocity errors. The observed slopes confirm this separation of scales.
Figure 1(c,d) connects this hierarchy to usable parameter domains. At displacement tolerance 0.05, the continuously located validity boundaries are for the leading reduction and 0.16390 for the matched reduction. Their sufficient certified boundaries are 0.02593 and 0.10103. Matching therefore enlarges both the directly accepted displacement interval and the analytically guaranteed interval. The matched velocity has its own sufficient boundary, , because differentiating the fast layer amplifies its contribution. The gap between each bound and the measured error quantifies the conservatism introduced by separately bounding residual components.
These accuracy boundaries all precede the critical value , where the two real decay rates coalesce. Beyond this value, they form a complex conjugate pair. Consequently, an overdamped trajectory can already require a more accurate reduction before any oscillatory motion appears. The record distinguishes a physical change of motion from an observable dependent accuracy requirement. It also shows how the required measurement window determines the variables that an effective description must retain.
Energy agreement and response agreement have different domains
The dimer gap has the controlled expansion
The second order term reflects a virtual intermediate state with a doubly occupied site. Two hopping events connect the low-energy spin configurations, with an energy cost for the intermediate charge excitation. Singlet mixing lowers the energy, whereas Pauli exclusion prevents this lowering for the triplet, giving antiferromagnetic . The exact response exponent is
As illustrated in Figure 2(b), decreases from 2 in the superexchange regime toward 1 as charge hybridization becomes strong. For small , the two hopping events and their charge energy denominator give . For large , the exact singlet–triplet gap approaches , so the dominant energy dependence is linear in hopping. The exponent is a logarithmic susceptibility to hopping and tests the response dependence predicted by the proposed exchange mechanism.
Figure 2. Effective exchange and the scope of a superexchange explanation. (a) Exact gap and truncated expansions. The fourth order curve is shown only where positive; it becomes negative for . (b) Exact hopping-response exponent. (c) Relative gap error and absolute exponent error of the second order reduction, each divided by its declared tolerance 0.05. The shaded interval passes the gap criterion but fails the exponent criterion. Curves are exact analytical relations of the dimer model.
For , the exact relative gap error and absolute exponent error are
Consequently, permits
whereas requires
At , the respective limits are 0.1146 and 0.08217. The shaded interval in Figure 2(c) therefore isolates points accepted by the gap criterion that require a more accurate response description. The tolerances represent 5% relative gap error and 0.05 absolute exponent error. Expanding Eq. (21) gives and , explaining why response is the more sensitive diagnostic near the controlled limit. A small correction to the energy can contribute more strongly to its logarithmic derivative.
The fourth order curve in Figure 2(a) further identifies the role of expansion control. The Taylor series has radius of convergence in , and its fourth order truncation becomes negative at while the exact gap remains positive. In the coupling shift test its median relative error is about 130%, compared with 69.6% at second order. These analytical controls provide a physical interpretation of the crossover results. Resumming the hybridization or using the exact reduced function is appropriate when the expansion parameter reaches this regime.
The completed records in Table 2 translate these curves into individual decisions. At , direct verification accepts the leading displacement while the residual bound supplies a more conservative guaranteed domain. At , the same second order expression supports the gap value under its tolerance and calls for refinement of the hopping response. Recording these outcomes separately identifies both the usable prediction and the physical quantity that benefits from an improved approximation.
| Record field | Leading oscillator, | Second-order dimer, |
| Targets and scales | Displacement, | Gap ; exponent |
| Actual errors | ; | |
| Bound or reference | Exact error expressions, Eq. (21) | |
| Validity decisions | Displacement passes | Gap passes; response fails |
| Certificate decision | Not guaranteed by this bound | Exact expressions resolve both decisions |
Table 2. Filled evaluation records. All listed acceptance tolerances are 0.05. The oscillator uses ; the dimer quantities are static, with the derivative at fixed .
Homogeneity protects energy scale transfer
The exact relation , for , identifies the energy scale transformation. It leaves and the competition between hopping and repulsion unchanged. Table 3 and Figure 3 show how enforcing that relation changes the learned model. The raw network has a median same range gap error of 0.139%, but its error rises to 52.3% when the overall energy scale changes. The dimensionless network has errors of 0.0733% and 0.0768%, respectively. The improvement follows from a supplied physical prior: scale transfer requires no extrapolation of the dimensionless function when remains in its training range.
| Distribution | Representation | Gap error (%) | Exponent error |
| Same range | Raw MLP | 0.1388 | 0.006155 |
| Same range | Π-MLP | 0.07332 | 0.004088 |
| Energy-scale shift | Raw MLP | 52.33 | 0.5163 |
| Energy-scale shift | Π-MLP | 0.07675 | 0.004289 |
| Coupling-ratio shift | Raw MLP | 9.813 | 0.1356 |
| Coupling-ratio shift | Π-MLP | 8.71 | 0.1492 |
Table 3. Neural fits evaluated against the exact dimer. Each value is the median across 16 independent fits of a 512-point test median. Gap errors are percentages; response-exponent errors are absolute dimensionless differences.
Figure 3. Gap errors (top), hopping-response exponent errors (middle), and joint acceptance fractions (bottom) in 16 repeated network fits. A bottom-row point is the fraction of 512 test points simultaneously satisfying relative gap error and absolute exponent error ; other points are per fit error medians. Gray segments connect fits using the same physical samples. Black bars are medians across fits and colored vertical lines span their 2.5–97.5 percentiles, describing training and sampling variation.
The middle row of Figure 3 shows that the protected transfer extends to a physical derivative. The raw network’s median exponent error rises from 0.00616 to 0.516, whereas the dimensionless network changes from 0.00409 to 0.00429. Writing makes the reason explicit. Its hopping exponent is , independent of the overall energy scale. The same representation thus preserves the relation between energy and response along each scaling orbit. The Euler homogeneity identity provides a complementary check of this imposed structure.
Changing probes the shape of the crossover function. In , gap errors are 8.71% for the dimensionless network and 9.81% for the raw network, with corresponding exponent errors of 0.149 and 0.136. These results locate the additional learning problem within itself. In this range the exact exponent moves away from the quadratic superexchange limit toward linear hybridization, so data that resolve the crossover provide information beyond the scaling relation. Appendix § 11 makes this distinction precise through a family of functions sharing the same homogeneity, training values, and limiting laws while varying within an unobserved interval. Energy scale transfer rises from zero for the raw representation to 100% for the dimensionless representation, whose 2.5–97.5 percentile range is 89.4–100%. The bottom row of Figure 3 displays the paired changes directly. Under coupling ratio transfer, the dimensionless model has a median gap acceptance of 30.7% and joint acceptance of zero. The derivative criterion identifies a concrete target for additional training, namely the change in the physical response through hybridization.
Joint acceptance makes this distinction directly operational. For each trained fit and the declared test distribution, define
where probabilities are empirical fractions in each fit’s test set, summarized by their median across fits. As reported in Table 4, the median joint fraction under
| Distribution | Representation | (%) | (%) | (%) |
| Same range | Raw MLP | 100 | 98.9 | 98.9 |
| Same range | -MLP | 100 | 100 | 100 |
| Energy-scale shift | Raw MLP | 1.17 | 0 | 0 |
| Energy-scale shift | -MLP | 100 | 100 | 100 |
| Coupling-ratio shift | Raw MLP | 20.8 | 0.879 | 0.879 |
| Coupling-ratio shift | -MLP | 30.7 | 0 | 0 |
Table 4. Median acceptance fractions for the two physical tolerances in Eq. (24). Each fit uses 512 test points shared by the two representations. Variation across fits is shown in Figure 3.
Joint acceptance also gives a practical interpretation to the error summaries. A median prediction error describes a typical point, whereas estimates the probability mass, under the declared test distribution, on which both physical requirements are satisfied. Reporting , , and together therefore connects surrogate accuracy to its usable domain. The paired fits in the figure show how strongly this domain changes when the representation incorporates the correct scale relation.
We additionally transform each of the 8192 same range test points per representation by , keeping fixed. The maximum dimensionless-network defect is across all five transformations. At , the raw-network defect has a pooled median of 0.622. These strictly paired tests isolate covariance from the independent test sampling used in Table 3, explaining why the two dimensionless gap-error summaries can differ slightly while the covariance relation is exact to rounding precision.
Causal structure recovers the intervention response
For Eq. (16), , , and . The unadjusted population slope and explained variance are
An intervention on instead has
In the observational data, an increase of is usually accompanied by an increase of . The negative direct effect of on exceeds the positive pathway, producing the negative association. Randomization replaces the assignment of and cuts the link from , while preserving the response equation for . It therefore isolates the positive response coefficient .
Figure 4(c) confirms the population analysis over 400 repetitions. The observational regression has median and mean slope with standard deviation 0.0160. Adjustment for the measured common cause recovers a mean slope of 2.0017 with standard deviation 0.0504, while randomization gives 1.9941 with standard deviation 0.1532. For a fixed exogenous state and , the structural response is , compared with approximately from observational regression. Specifying the intervention thus changes both the sign and the physical meaning of the predicted response.
Figure 4. Observational and intervention response in a known synthetic system. (a) Structural graph and coefficients. (b) One repetition with 400 samples per condition and population regression lines; randomized is independent of . (c) Slopes over 400 repetitions; points are medians, segments are 2.5–97.5 percentiles, and the dashed vertical line is the true intervention slope. Variables use the dimensionless coordinates of the structural equations, with no distribution-specific rescaling.
The scatter and slope distributions clarify why precision and causal accuracy are distinct. The observational slope is tightly concentrated around the association generated by the common cause. Adjustment uses the measured to separate its direct contribution from the response, while the unadjusted randomized regression averages over variation in . This explains the smaller spread of the adjusted estimator in this supplied structural model. Both procedures recover the positive response through their respective identification conditions.
An identifiability test specifies which additional observation would determine the response. With independent standard Gaussian noises and , the two models
generate the same observational distribution and have -intervention slopes and zero, respectively. Temporal information, a justified physical direction, or an intervention can separate these mechanisms. Reporting this equivalence class together with a discriminating measurement turns observational ambiguity into an actionable experimental prediction.
Mechanism separation sets the measurement cost
For two known Gaussian hypotheses with mean difference and equal variance , a single measurement at fixed control has Kullback–Leibler (KL) divergence
After independent measurements at that control, the log posterior odds in favor of the true hypothesis are Gaussian with mean and variance . Equal prior odds are assumed.
Proposition 3 (Fixed-control query requirement). For and , the smallest integer budget ensuring posterior probability at least on the true hypothesis with probability at least is
where and .
Appendix § 12 derives this expression. If the first difference between mechanisms is with and , then at fixed , , and ,
The exponent connects the first distinguishing physical correction to the measurement cost for the specified Gaussian hypotheses and fixed control.
For Eq. (17), and . Both mechanisms share the linear response, with their first difference arising from the quadratic saturation correction. As shown in Figure 5(a,Figure 5b), this physical degeneracy produces an information scale and an discrimination cost. With and , Eq. (29) requires eight measurements at and 332,510 at . Moving the control into the nonlinear regime separates the mechanisms relative to the same observation noise.
Figure 5. Mechanism separation and query cost. (a) Exact per-query KL divergence and its small-control fourth order asymptote. (b) Integer query requirement from Eq. (29) for , at noise standard deviation 0.3. (c) Equal-budget measurement strategies over 500 balanced noise repetitions; shaded regions are pointwise Wilson 95% intervals. The black dashed curve is the analytical success probability at from Eq. (38). The known hypotheses make repeated measurement the maximum-information policy.
The policy comparison in Figure 5(c) gives the corresponding finite budget result. Eight measurements produce correct and sufficiently confident fractions of 97.2% for maximum information selection, 43.6% for uniform controls, and 7.8% for controls sampled uniformly in logarithmic coordinates. Repeated low control measurements remain below the posterior threshold. The exact success probability at rises from 17.53% at one query to 95.84% at eight; the simulated eight query fraction differs by 1.52 Monte Carlo standard errors. The analytical curve therefore provides a quantitative check across the full budget range.
The ordering of these strategies follows their access to the distinguishing saturation term. Uniform sampling allocates some measurements near the informative upper control, whereas logarithmic sampling concentrates more strongly where the shared linear response dominates. The maximum information policy uses the largest allowed separation at every measurement. In this known two mechanism problem, the result gives a direct physical reason for the gain from experimental design[39]. It also identifies a useful evaluation target for a scientific agent, namely selecting a control that reveals the relevant correction within a fixed measurement budget.
Implications for evaluating language models
The controlled examples define evaluation targets for approximation validity, transformation consistency, intervention response, and measurement selection. The development interface implements structured questions about prescribed reductions and physical models, with numerical and categorical outputs scored against analytical references.
The evaluation set contains 128 base systems and 256 paired prompts in four families. Oscillator tasks return actual displacement validity and certificate status separately; dimer tasks return distinct gap and response decisions. The distinction in Eq. (3) thus becomes a scored scientific output. Field correctness, strict correctness, paired joint correctness, and equivalence consistency quantify complementary aspects of the responses. Both variants of each system are presented independently and remain in the same data split. Missing or unparseable responses remain in the score denominators.
These questions support comparisons between structured physical prompts and general analysis prompts under the same response format and tool budget. Supplying the governing equations isolates the use of physical structure in validity and response calculations. A further evaluation condition can ask a model to construct equations from scenarios and data, following the control of prior information studied in PhysGym[14]. Separating these conditions identifies the contributions of model construction and reasoning within a supplied model.
Interactive evaluations can extend these tasks by allowing a model to choose successive measurements and tracking the information gained within a fixed query budget. The Gaussian calculation supplies an analytical reference for measurement selection with known candidate mechanisms. Analogous extensions can ask a model to propose a reduction and then test its accepted observables and domain. Appendix § 14 defines the current task families, data splits, scoring rules, and representative analytical answers. The task bank and scoring implementation are available from the corresponding author on request.
Discussion
Validity changes with the observable and time window because eliminating a fast variable affects derivatives more strongly than slowly varying coordinates. In the oscillator, the leading displacement converges uniformly as while the initial velocity mismatch remains unity. Matching the initial layer improves displacement to and velocity to , whereas delaying the observation window targets the response after fast relaxation. Propositions 1 and 2 express this distinction through the Green function norms appropriate to each observable. The dimer gives a corresponding static example. Near the exchange limit, the relative gap error is and the absolute response-exponent error is , producing different acceptance boundaries at the same numerical tolerance. These calculations identify which variables and corrections must be retained for a specified physical measurement.
The neural comparison separates energy scale transfer from changes in coupling. Writing the predicted gap as times a learned function of enforces homogeneity and preserves the response exponent along a scaling orbit. Under the energy scale shift, the dimensionless network has a median joint acceptance fraction of 100%. Under the coupling ratio shift, the corresponding fraction is zero, although the median gap acceptance fraction is 30.7%. The second test therefore identifies where additional training information is needed, namely the shape and derivative of the crossover function. As shown by the function family in Appendix § 11, crossover observations determine information independent of homogeneity and the limiting laws. New data in that range can be selected to improve both the gap and its hopping response.
Mechanism discrimination depends on the measured variables, allowed interventions, and observation noise. In the causal example, measuring the common cause or randomizing the intermediate variable recovers the positive response; the observationally equivalent models in Eq. (27) specify an intervention that distinguishes their responses. In the measurement example, the candidate mechanisms share a linear response and first separate through the quadratic saturation term. Their per-query information consequently scales as , giving an query requirement at fixed noise. Increasing the control from 0.05 to 1 reduces the required budget from 332,510 to eight measurements for the stated confidence targets. The gain follows from exposing the distinguishing nonlinear correction. Specifying the accessible control range, noise, and candidate responses thus connects a proposed mechanism to a quantitative measurement decision.
Conclusion
Validity records for individual observables make the scope of a physical explanation quantitatively testable. We linked reductions to declared scales and tolerances, separated certified domains from actual validity, and derived complementary displacement and velocity certificates. Exact dimer relations and paired neural tests show why energy homogeneity protects scale transfer while joint acceptance of gap and response identifies the coupling crossover as a separate learning target. A known causal model separates observational association from intervention response. The order at which two mechanisms separate then determines an exact fixed control query cost under Gaussian noise.
These results support evaluation that keeps prediction, approximation, and mechanism identification individually accountable. The analytical references and structured development interface connect each validity decision to a physical quantity, an acceptance criterion, and a reproducible calculation, providing a basis for evaluating physical reasoning in scientific AI.
Data and code availability
The data, code, trained model weights, task bank, and scoring implementation supporting this study are available from the corresponding author on request.
AI assistance disclosure
Generative AI assisted literature discovery, analytical development, code implementation, figure preparation, and manuscript drafting. Analytical expressions, explicit structural equations, and independently checked numerical calculations define the reference results. The authors take responsibility for the analysis, numerical results, references, and final manuscript.
Proof and scope of the residual certificate
For , the Green function is
where , are defined in Eq. (5) and . It is nonnegative and its integral is one. Its maximum occurs at , giving Eq. (5). At critical damping, and the same statements hold by continuity.
Let be the exact solution with initial displacement one and initial velocity zero. It decreases from one to zero in this parameter range. The error satisfies
The two convolution bounds and establish Proposition 1. The nonnegative kernel on establishes the stated scope of this certificate.
For , the residual separates as
Bounding in maximum norm and in integrated norm, together with the initial velocity error, gives Eq. (9). The fast residual has amplitude but integrated magnitude , explaining the importance of matching the residual norm to the eliminated process.
For velocity, and differentiation of the convolution gives
The kernel is unimodal and vanishes at both endpoints. The following kernel norms are taken over ; residual norms remain over .
The positive derivative maximum is . Its negative extremum occurs at and has magnitude smaller than ; the critical case follows by continuity. Young’s convolution inequalities now establish Proposition 2. For , use the maximum norm for and the integrated norm for . Their contributions are and , respectively. Together with , these give Eq. (11).
Independent information in the crossover function
Write , , with . Let a smooth nonnegative bump have compact support in a test interval disjoint from the training interval. Then
has the same homogeneity, hopping sign invariance on the single bond, and training values as . Since the bump vanishes near zero and infinity, both asymptotic regimes also remain unchanged. Varying changes the value and derivative within its support while preserving these constraints. This family explicitly isolates the information supplied by crossover observations from that supplied by symmetry, training values, and limiting laws.
Derivation of the Gaussian query requirement
At a fixed , the log likelihood ratio from one observation is
Under , it is normal with mean and variance ; under , reverse its sign. The distributions of evidence in favor of the true hypothesis are identical. Summing independent observations and using equal priors gives
Set the argument at least and solve the resulting quadratic in to obtain Eq. (29). The result quantifies the sampling probability of attaining the declared posterior threshold on the true mechanism.
Training and uncertainty
Both networks use double precision and natural logarithms. Input centering and scaling are fitted on the training samples, and logarithmic targets retain their original scale. Training uses an penalty of , a gradient tolerance of , and at most 2000 iterations. Each repeated fit varies the physical samples and initialization, with both representations using the same training and test points. This pairing compares the two representations under matched physical conditions.
The oscillator and dimer curves describe deterministic changes of physical parameters. The network percentiles describe training and sampling variation across repeated fits. Causal intervals summarize finite samples from the stated structural model, and measurement intervals summarize independent observation noise under the two hypotheses. Independent simulation with 50,000 balanced noise sequences verifies the analytical success curve over budgets 1–8, with maximum probability discrepancy 0.00389. These uncertainty descriptions give each plotted range a specific statistical interpretation.
Executable development interface
Each of four families has 32 base systems divided into 16 training, eight development, and eight test groups. Across families this gives 64, 32, and 32 groups, corresponding to 128, 64, and 64 prompts. Paired variants remain in the same split. Oscillator variants rescale all equation coefficients, and dimer variants rescale energies and reverse the hopping sign on the single bond. Causal and measurement variants test consistency under equivalent wording and parameter ordering. The oscillator is stratified into certified points, accepted points with a bound above tolerance, rejected points with a bound above tolerance, and points outside the proof domain. The dimer is stratified by joint gap and response acceptance, gap acceptance alone, and rejection of both. Each split contains every realizable stratum. For each sampled parameter , the boundary margin satisfies
where contains the relevant positive boundaries. For the oscillator, and the boundaries are the leading displacement certificate threshold, the actual displacement threshold on , and . For the dimer, and the boundaries are the gap and response acceptance thresholds. The generator uses unrounded boundary values; the sampling configuration specifies ranges, logarithmic sampling, boundary adjacent points, and randomization.
As summarized in Table 5, 16 oscillator prompts distinguish actual acceptance from a sufficient guarantee, and 22 dimer prompts distinguish energy acceptance from response acceptance. These cases make the two central validity decisions directly testable within every data split.
| Family | Physical region | Training | Development | Test |
| Oscillator | Certified | 4 | 2 | 2 |
| Oscillator | Valid, bound above tolerance | 4 | 2 | 2 |
| Oscillator | Invalid, bound above tolerance | 4 | 2 | 2 |
| Oscillator | Outside proof domain | 4 | 2 | 2 |
| Dimer | Gap and response accepted | 6 | 3 | 2 |
| Dimer | Gap accepted, response rejected | 5 | 3 | 3 |
| Dimer | Gap and response rejected | 5 | 2 | 3 |
Table 5. Stratified coverage of the oscillator and dimer task families, counted by base system. Each system contributes two equivalent prompts.
Let task have required fields and indicate a correct field. Let be strict task correctness, and let , . The evaluation scores are defined as
where when all paired answers agree after mapping to the common representation. The first score weights the 1280 required fields equally, the second requires every field of a task to be correct, and the third requires both variants of a system to be correct. The fourth measures equivalence consistency after validating field types and permitted certificate states. Reporting the last two together separates preserved relations from agreement with the analytical reference. Field scores are also grouped into validity decisions (256 fields), numerical inference (384), qualitative regime identification (64), and structural rules (576), giving each physical obligation an explicit contribution to the evaluation.
A numerical answer matches reference when . Paired equivalence uses the symmetric rule . Boolean answers use either true or false values. Certificate status distinguishes a sufficient guarantee, a bound above tolerance, and parameters outside the proof domain. Missing answers, invalid types, and nonfinite numbers receive zero for the corresponding fields, and all tasks remain in the score denominators. Different derivations are accepted when they produce the requested quantities and decisions. Explanatory reasoning and proposed alternative reductions can be assessed alongside these structured outputs.
Representative physical tasks
Oscillator. For , , , evaluate the leading and matched reductions on at tolerance 0.05. The reference gives , nonoscillatory motion, and displacement error powers 1 and 2. The leading displacement is accepted with , while its bound is above tolerance.
Hubbard dimer. For , , the reference gives , , leading hopping power 2, and invariance under hopping sign reversal. The second order gap error satisfies tolerance 0.05; its response error requires refinement.
Causal response. In and , take independent standard normal exogenous variables and , , , . The observational slope is and the intervention slope is . Adjustment for measured recovers the positive response.
Measurement design. Compare and with equal priors and , , . The maximum information control is the upper endpoint. The saturated response lies below the linear response for , and their difference vanishes as . At the likelihoods coincide and the posterior odds remain unchanged.
These analytical answers satisfy all required fields and receive full field and strict correctness scores. Paired variants yield the same answers after the specified physical mapping.
References
References
- [1]G. Carleo, I. Cirac, K. Cranmer, et al. “Machine learning and the physical sciences”. In: Reviews of Modern Physics 91.4, 045002 (2019). DOI: 10.1103/RevModPhys.91.045002.DOI
- [2]H. Wang, T. Fu, Y. Du, et al. “Scientific discovery in the age of artificial intelligence”. In: Nature 620.7972 (2023), pp. 47–60. DOI: 10.1038/s41586–023–06221–2.DOI
- [3]Y. Zhang, S. A. Khan, A. Mahmud, et al. “Exploring the role of large language models in the scientific method: from hypothesis to discovery”. In: npj Artificial Intelligence 1.1, 14 (2025). DOI: 10.1038/s44387–025–00019–5.
- [4]H. Pan, N. Mudur, W. Taranto, et al. “Quantum many-body physics calculations with large language models”. In: Communications Physics 8.1 (2025), p. 49. DOI: 10.1038/s42005–025–01956–y.
- [5]X. Zhang, Y. Dong, Y. Wu, et al. “PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning”. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 16593–16615. DOI: 10.18653/v1/2025.acl-long.811.DOI
- [6]S. Qiu, S. Guo, Z.-Y. Song, et al. PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models. 2025. arXiv: 2504.16074.arxiv.org/abs/2504.16074
- [7]W. Wang, D. Huang, J. Li, et al. CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics. 2025. arXiv: 2508.18124.arxiv.org/abs/2508.18124
- [8]M. Zhu, M. Tian, X. Yang, et al. Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark. 2025. arXiv: 2509.26574.arxiv.org/abs/2509.26574
- [9]T. Miao, W. Jin, M. Zhang, et al. PRL-Bench: A Comprehensive Benchmark Evaluating LLMs’ Capabilities in Frontier Physics Research. 2026. arXiv: 2604.15411.arxiv.org/abs/2604.15411
- [10]H. W. de Regt and D. Dieks. “A Contextual Approach to Scientific Understanding”. In: Synthese 144.1 (2005), pp. 137–170. DOI: 10.1007/s11229–005–5000–4.DOI
- [11]E. Sullivan. “Understanding from Machine Learning Models”. In: The British Journal for the Philosophy of Science 73.1 (2022), pp. 109–133. DOI: 10.1093/bjps/axz035.
- [12]E. Duede. “Deep Learning Opacity in Scientific Discovery”. In: Philosophy of Science 90.5 (2023), pp. 1089–1099. DOI: 10.1017/psa.2023.8.DOI
- [13]Y. Xu, Y. Liu, Z. Gao, et al. PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models. 2025. arXiv: 2505.24823.arxiv.org/abs/2505.24823
- [14]Y. Chen, P. Pikos, M. Ostaszewski, et al. PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors. 2025. arXiv: 2507.15550.arxiv.org/abs/2507.15550
- [15]T. Zheng, K. K.-W. Tam, N. H.-N. K. Nguyen, et al. NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents. 2025. arXiv: 2510.07172.arxiv.org/abs/2510.07172
- [16]P. Shojaee, N.-H. Nguyen, K. Meidani, et al. LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models. 2025. arXiv: 2504.10415.arxiv.org/abs/2504.10415
- [17]M. L. Wiemann, L. M. Smith, P. Melchior, et al. DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking. 2026. arXiv: 2605.26087.arxiv.org/abs/2605.26087
- [18]J. Yang, O. Venkatachalam, M. Kianezhad, et al. Think like a Scientist: Physics-guided LLM Agent for Equation Discovery. 2026. arXiv: 2602.12259.arxiv.org/abs/2602.12259
- [19]Y. Zhang, X. Guo, R. Pan, et al. “Decoupled Physical Modeling and Execution for Physics Reasoning”. In: Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2026, pp. 32112–32131. G. E. Karniadakis, I. G. Kevrekidis, L. Lu, et al. “Physics-informed machine learning”. In: Nature Reviews Physics 3.6 (2021), pp. 422–440. DOI: 10.1038/s42254-021-00314-5.
- [21]E. Buckingham. “On Physically Similar Systems; Illustrations of the Use of Dimensional Equations”. In: Physical Review 4.4 (1914), pp. 345–376. DOI: 10.1103/PhysRev.4.345.DOI
- [22]S.-M. Udrescu and M. Tegmark. “AI Feynman: A physics-inspired method for symbolic regression”. In: Science Advances 6.16 (2020), eaay2631. DOI: 10.1126/sciadv.aay2631.DOI
- [23]S. L. Brunton, J. L. Proctor, and J. N. Kutz. “Discovering governing equations from data by sparse identification of nonlinear dynamical systems”. In: Proceedings of the National Academy of Sciences 113.15 (2016), pp. 3932–3937. DOI: 10.1073/pnas.1517384113.DOI
- [24]S. Greydanus, M. Dzamba, and J. Yosinski. “Hamiltonian Neural Networks”. In: Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc., 2019. URL: https://proceedings.neurips.cc/paper/2019/hash/26cd8ecadce0d4efd6cc8a8725cbd1f8 - Abstract.html.arxiv.org/abs/1906.01563
- [25]M. Cramer, S. Greydanus, S. Hoyer, et al. Lagrangian Neural Networks. 2020. arXiv: 2003.04630.
- [26]V. G. Satorras, E. Hoogeboom, and M. Welling. “E(n) Equivariant Graph Neural Networks”. In: Proceedings of the 38th International Conference on Machine Learning. Vol. 139. Proceedings of Machine Learning Research. PMLR, 2021, pp. 9323–9332. URL: https://proceedings.mlr.press/v139/satorras21a.html.
- [27]M. Raissi, P. Perdikaris, and G. Karniadakis. “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations”. In: Journal of Computational Physics 378 (2019), pp. 686–707. DOI: 10.1016/j.jcp.2018.10.045.DOI
- [28]A. Krishnapriyan, A. Gholami, S. Zhe, et al. “Characterizing possible failure modes in physics-informed neural networks”. In: Advances in Neural Information Processing Systems. Vol. 34. Curran Associates, Inc., 2021, pp. 26548–26560. URL: https://proceedings.neurips.cc/paper/2021/hash/df438e5206f31600e6ae4af72f2725f1 - Abstract.html.arxiv.org/abs/2109.01050
- [29]Y. Cao and L. Petzold. “A Posteriori Error Estimation and Global Error Control for Ordinary Differential Equations by the Adjoint Method”. In: SIAM Journal on Scientific Computing 26.2 (2004), pp. 359–374. DOI: 10.1137/S1064827503420969.DOI
- [30]J. Birrell. “A Posteriori Error Bounds for Two Point Boundary Value Problems: A Green’s Function Approach”. In: Journal of Computational Dynamics 2.2 (2015), pp. 143–164. DOI: 10.3934/jcd.2015001.DOI
- [31]J. PEARL. “Causal diagrams for empirical research”. In: Biometrika 82.4 (1995), pp. 669–688. DOI: 10.1093/biomet/82.4.669.DOI
- [32]URL: https://aclanthology.org/2026.emnlp-main.1405/. the IEEE 109.5 (2021), pp. 612–634. DOI: 10.1109/JPROC.2021.3058954.DOI
- [33]J. Peters, P. Bühlmann, and N. Meinshausen. “Causal Inference by using Invariant Prediction: Identification and Confidence Intervals”. In: Journal of the Royal Statistical Society Series B: Statistical Methodology 78.5 (2016), pp. 947–1012. DOI: 10.1111/rssb.12167.
- [34]P. Virtanen, R. Gommers, T. E. Oliphant, et al. “SciPy 1.0: fundamental algorithms for scientific computing in Python”. In: Nature Methods 17.3 (2020), pp. 261–272. DOI: 10.1038/s41592-019-0686-2.DOI
- [35]D. J. Carrascal, J. Ferrer, J. C. Smith, et al. “The Hubbard dimer: a density functional case study of a many-body problem”. In: Journal of Physics: Condensed Matter 27.39 (2015), p. 393001. DOI: 10.1088/0953-8984/27/39/393001.DOI
- [36]J. R. Schrieffer and P. A. Wolff. “Relation between the Anderson and Kondo Hamiltonians”. In: Physical Review 149.2 (1966), pp. 491–492. DOI: 10.1103/PhysRev.149.491.DOI
- [37]D. C. Liu and J. Nocedal. “On the limited memory BFGS method for large scale optimization”. In: Mathematical Programming 45.1-3 (1989), pp. 503–528. DOI: 10.1007/bf01589116.
- [38]F. Pedregosa, G. Varoquaux, A. Gramfort, et al. “Scikit-learn: Machine Learning in Python”. In: Journal of Machine Learning Research 12.85 (2011), pp. 2825–2830. URL: https://jmlr.org/papers/v12/pedregosa11a.html.arxiv.org/abs/1201.0490
- [39]K. Chaloner and I. Verdinelli. “Bayesian Experimental Design: A Review”. In: Statistical Science 10.3 (1995), pp. 273–304. DOI: 10.1214/ss/1177009939.DOI
- [40]E. B. Wilson. “Probable Inference, the Law of Succession, and Statistical Inference”. In: Journal of the American Statistical Association 22.158 (1927), pp. 209–212. DOI: 10.1080/01621459.1927.10502953.DOI




