WorksComputer Science
Physics-informed neural networks in the small-data regime: A controlled comparison with classical PDE surrogates
Abstract
Neural surrogates for engineering models are often trained on large collections of simulated solutions. We investigate whether physics-informed neural networks (PINNs) can achieve accurate predictions from scarce labelled observations by incorporating governing equations and prescribed initial and boundary conditions. We introduce hierarchical loss weighting (H-PINN) and its gradient-direction-aware extension (HO-PINN), which adjust the emphasis on physical constraints according to higher-priority losses. Hierarchical PINNs are compared with Vanilla PINN, tuned gradient boosting, support-vector regression, and Gaussian-process regression on a heat equation and three heterogeneous two-phase-flow benchmarks. Experiments use 20–1000 labelled states and ten paired seeds. Each label is one coordinate–state observation, not a complete simulation. At least one PINN variant achieves the highest mean primary-target in each interpolation and equation-assisted forecasting comparison at 20 and 50 labels. The advantage for water saturation persists across all tested budgets. Hierarchical weighting reduces variability across runs in selected settings. These results demonstrate that known physics can support accurate neural surrogate predictions from very few solution observations, at substantially higher fitting cost than the tested classical regressors.
Introduction
Many subsurface decisions require a model to be evaluated far more often than a single forward study. History matching tests many geological hypotheses against production data. Uncertainty quantification propagates uncertain rock and fluid properties through the model. Optimization repeatedly changes wells or controls. In each case, many parameter or control hypotheses lead to many high-fidelity flow solves [1, 2, 3]. Even when one simulation is tractable, the complete many-query workflow can be prohibitively expensive.
Porous-media flow is also a difficult surrogate problem. Permeability is heterogeneous and may vary over several orders of magnitude. Pressure is comparatively smooth but globally coupled through an elliptic equation. Saturation is transported through the heterogeneous velocity field and develops moving fronts whose location depends nonlinearly on mobility ratios, relative-permeability parameters, and the rock field [3, 4]. This contrast motivates reporting pressure and saturation separately and using water saturation as the primary flow target.
Surrogates represent this response in different ways. Projection-based reduced-order models approximate dynamics in low-dimensional spaces [5], while learned nonlinear representations encode solution fields on reduced manifolds [6]. Convolutional encoder–decoders predict spatial fields [7, 3], and neural operators learn mappings between input and output functions [8, 9, 4]. Here we study coordinate-to-state regression from scattered observations. Gradient boosting, support-vector regression, and Gaussian-process regression provide the non-neural baselines [10, 11, 12]. This formulation compares methods at a fixed number of revealed solution values rather than complete training fields.
Physics-informed learning offers a different trade-off. Instead of learning only from solution values, a PINN differentiates a neural representation and evaluates the governing equations at coordinates that require no reference solution [13, 14]. Initial and boundary conditions provide further constraints. The central question of this study is therefore not whether PINNs can solve the equations in principle. We ask at which label budgets and for which solution structures this additional physical information outweighs the optimization difficulty and fitting cost of PINNs.
Small data has a precise meaning in our protocol. A labelled state is one coordinate–state pair. For the flow problems it has the form
where is absent in the fixed-parameter tests. Thus, denotes 500 individual states, not 500 complete reservoir simulations, permeability realizations, or time-dependent fields. Every method receives the same labels. The PINNs additionally receive the PDE, initial and boundary conditions, residual coordinates over the full time interval, and the permeability coefficient evaluated at residual points. Forecasting is consequently equation-assisted rather than purely data-driven.
The study makes four contributions. First, it compares six nested budgets from 20 to 1000 states using ten paired seeds and separate interpolation and forecasting regions. Second, it examines a smooth heat problem and heterogeneous two-phase-flow problems, including a seven-input parametric surrogate. Third, it compares a fixed PINN architecture with extensively tuned XGB, SVR, and GP baselines. Fourth, it introduces hierarchical exponential loss activation and its gradient-direction-aware extension, and assesses their accuracy and run-to-run reliability against unit-weight vanilla PINN. The experiments identify a PINN advantage at 20–50 labels and throughout the tested budgets for water saturation. They also identify settings where weighting reduces variability, settings where vanilla is more reliable, and the smooth, sufficiently sampled regime where GP is preferable.
Related work
Data-driven porous-flow surrogates are commonly constructed from complete simulation realizations. Convolutional encoder–decoders have been used for porous-media uncertainty quantification [7] and dynamic multiphase flow [3]. Recurrent and convolutional surrogates have also accelerated data assimilation and history matching in two- and three-dimensional reservoirs [1, 2]. These studies learn strong field-level representations but their training unit is typically a realization or a sequence of fields. Our training unit is instead one labelled state at one coordinate and time.
Operator-learning methods extend field prediction across coefficients and controls. DeepONet learns nonlinear operators from paired functions [8]. The Fourier neural operator performs global spectral mixing for parametric PDEs [9]. U-FNO augments this construction with a U-Net path and has been applied to multiphase flow and geological carbon storage [4]. Such models are designed for reusable maps between fields. The present study addresses the complementary setting in which only tens to hundreds of scattered state values are exposed to the learner while the differential operator is known.
PINNs incorporate this operator by automatic differentiation and penalize residuals at unlabelled coordinates [13, 14]. Sun et al. constructed fluid-flow surrogates without simulation labels by imposing conservation equations [15]. He et al. combined multiphysics constraints with sparse observations for subsurface transport data assimilation [16]. Chen et al. used physical constraints to identify governing equations from scarce measurements [17]. For two-phase flow, residual adaptivity and finite-volume-informed formulations have been used to improve front resolution in heterogeneous media [18, 19]. Coupled pressure and saturation studies nevertheless report persistent difficulty near sharp fronts [20].
Physics information does not remove numerical and optimization limitations. Standard PINNs can fail even on relatively simple PDEs [21]. Broad benchmarks show strong dependence on the equation, architecture, sampling, and optimizer [22]. Finite elements can also be more accurate and efficient when the task is simply to solve one forward problem [23]. We therefore keep the reference solver as ground truth, define the task as surrogate reconstruction after labels have been revealed, and report fitting cost rather than presenting PINNs as replacements for mature discretizations.
Loss balancing is a central part of reliable PINN training. Gradient-statistics annealing balances back-propagated gradients across loss terms [24]. Self-adaptive PINNs learn pointwise emphasis [25]. Inverse-Dirichlet weighting estimates imbalance from gradient statistics [26], while relative-progress methods compare the rates at which objectives improve [27]. Our H-PINN instead activates IBC and PDE objectives as higher-priority losses approach prescribed scales. HO-PINN further reduces scalar weight for gradient directions already represented by earlier objectives. Unlike PCGrad [28], projection is used only to form weights and does not replace the gradient in the final backward pass.
The classical baselines provide complementary inductive biases. Gradient boosting uses additive trees [10] [29]. Support-vector regression uses a regularized kernel representation [11]. We use an RBF kernel in our experiments. GP regression uses a covariance-based prior over functions [12]. We examine when known physics improves prediction relative to these tuned regressors at matched label budgets, and whether hierarchical weighting improves run-torun reliability. Nested labels, paired seeds, and separate interpolation and equation-assisted forecasting sets support this comparison.
Small-data surrogate problem
Learning task and architecture
Let collect space, time, and any physical parameters, and let denote the solution. All models receive
Here counts labelled solution states rather than complete simulations or fields. In Test 1, and . In the fixed-parameter flow tests, , while Test 4 appends the four relative-permeability parameters. The flow target is . PINNs additionally evaluate equation residuals at unlabelled coordinates. These coordinates contain no reference-solver targets. Permeability is evaluated there as a known PDE coefficient and is not passed to the network as an input feature.
All tests use a gated residual multilayer perceptron with six hidden layers of width 128. Inputs are mapped to using known domain bounds. Two learned tanh transforms are multiplied elementwise before the GELU hidden layers, with residual connections after every second layer and Xavier initialization. The heat model returns temperature. The flow model returns pressure, two saturations, and four auxiliary phase-velocity components. Sigmoid transformations and saturation normalization enforce
Loss components and hierarchical weighting
The objective combines mean-square data, initial/boundary-condition (IBC), and PDE errors,
Each epoch uses all fixed labels and freshly sampled residual points. The data loss compares temperature in Test 1 and in Tests 2–4. Vanilla PINN uses constant weights , hence . H-PINN and HO-PINN normalize their weights to sum to one. Architecture, prescribed physics, labels, residual-sampling stream, and training schedule are shared across the three variants. The objective-scale difference is part of this implementation and may interact with gradient clipping.
H-PINN activates the IBC and PDE terms as preceding errors approach prescribed scales. With sg denoting stop-gradient, define
The exponential gates and weights are
The target scales control the transition between objectives. We use and .
HO-PINN also accounts for repeated gradient directions. Write and
The implementation removes projections sequentially and computes
It then uses
We set . Repeated directions receive less weight, while the positive floor preserves every active objective. At , HO-PINN reduces to H-PINN. All gates and gradient-direction factors are detached before the backward pass. Algorithm 1 summarizes the training procedure for both variants. Appendix A describes parameter selection.
PDE benchmarks and reference solutions
Periodic heat equation
Test 1 uses
with periodic boundaries and . Its exact solution is
This benchmark evaluates smooth reconstruction without numerical reference error.
Heterogeneous two-phase flow
Tests 2–4 model incompressible, immiscible water–oil displacement on , neglecting gravity and capillary pressure. Phase saturation is the fraction of pore volume occupied by phase . Conservation and closure give
where is porosity and is Darcy flux. The phases share pressure and obey
with absolute permeability , relative permeability , and viscosity . Corey-type laws are used [30],
Adding the phase equations yields
Equivalently, transport satisfies
We fix , , and . Initially and . Pressure is imposed as and . Both phases have zero normal flux on . At the inlet , zero oil flux specifies pure-water injection. Water inflow follows from the pressure difference. Outflow composition follows the interior upwind state.
The PINN predicts phase velocities directly. Its residual loss is the sample mean of the squared residual norm for
The IBC loss enforces the conditions above. The data loss averages the summed squared errors of , , .
Tests 2 and 3 use . Their permeability contrasts are approximately 24 and , respectively. Test 4 uses a third fixed heterogeneous field and 100 parameter combinations with
Its input is . Permeability is supplied only as a PDE coefficient at residual points. Test 1 has two inputs and final time 0.05. Tests 2 and 3 have three inputs and final times 0.30 and 0.05. Test 4 has seven inputs and final time 0.20. The heat reference is analytical. Each flow reference uses a finite-volume grid. Tests 2 and 3 each contain one flow realization, while Test 4 contains 100 parameter combinations. Test 4 holds out coordinates, not entire parameter combinations, so it does not establish generalization to unseen parameter realizations. Temperature is the primary Test 1 target and water saturation is the primary flow target.
Reference discretization
The cell-centred finite-volume solver recomputes pressure at every adaptive transport substep. Harmonic face transmissibilities handle permeability jumps. Phase fluxes use upstream fractional flow, and explicit saturation updates are divided by cell pore volume , with a CFL limit of 0.2. Pressure and saturation are stored synchronously at 101 times including . Final times retain developed displacement fronts. The PINN residuals and reference solver enforce the same continuum equations through different discretizations.
Experimental protocol
Training and evaluation sets
Budgets are , with seeds 101–110 controlling label selection and PINN initialization. Training sets are nested within each seed and shared by all methods. Labels occupy and are balanced over ten temporal bins.
Fixed, disjoint test sets measure interpolation at and forecasting at . Test 1 has 29,041 interpolation and 12,336 forecasting points. Each flow test uses 1,500 points per region. Test 4 draws these sets from a deterministic pool of 100,000 states covering all 100 parameter combinations.
PINN residuals cover the full time interval, excluding current labels and evaluation coordinates. They contain no reference values. Forecasting therefore uses knowledge of the future differential operator but no future solution labels. Initial and boundary conditions are prescribed physical information.
Optimization and baseline tuning
All PINNs use 50,000 epochs of Adam with initial learning rate , cosine decay to , and gradient clipping at one. The checkpoint with the minimum balanced training loss is retained. A separate set of 512 unlabelled physics-validation points provides diagnostics, not checkpoint selection. The fixed architecture and weighting parameters were selected in staged development runs described in Appendix § 14.
Classical models fit each output independently. XGB uses histogram-based squared-error boosting. SVR uses a radial-basis kernel with input and target scaling. GP uses standardized inputs and constant-scaled stationary kernels with white noise. At every budget and seed, 96 XGB configurations, 96 SVR configurations, and 20 GP kernel–regularization combinations are evaluated using three strictly forward-time folds. GP also optimizes kernel parameters by marginal likelihood with three restarts. Preprocessing is fitted inside each fold, and negative MSE determines the selected parameters. Neither evaluation set is used for tuning.
Metrics
For evaluation points, we report
Metrics are computed per run, followed by the mean and sample standard deviation across ten seeds. Test 1 uses temperature. Flow tests report water saturation and pressure separately, with saturation defining the primary ranking. Error bars describe run-to-run variability, not confidence intervals. We assess predictive reliability within each fixed test–budget–region condition using the std ratio or . Smaller ratios indicate less sensitivity to the joint change in labels and initialization. We also inspect lower-tail scores and negative evaluations. These diagnostics describe predictive variability rather than smoothness of the training loss. Conditions share seeds and nested labels, so counts across conditions are descriptive and not independent significance tests.
Results
Table 1 reports all six methods at the two smallest budgets. Figure 1 extends the comparison to all six budgets by subtracting the strongest classical mean from the strongest of the three PINN means. The primary target is temperature in Test 1 and water saturation in Tests 2–4. Field figures show the fixed , seed-101 H-PINN and classical runs to illustrate spatial structure.
| Test | Vanilla | H-PINN | HO-PINN | XGB | SVR | GP | |
| T1 | 20 | ||||||
| T1 | 50 | ||||||
| T2 | 20 | ||||||
| T2 | 50 | ||||||
| T3 | 20 | ||||||
| T3 | 50 | ||||||
| T4 | 20 | ||||||
| T4 | 50 | ||||||
| Test | Vanilla | H-PINN | HO-PINN | XGB | SVR | GP | |
| T1 | 20 | ||||||
| T1 | 50 | ||||||
| T2 | 20 | ||||||
| T2 | 50 | ||||||
| T3 | 20 | ||||||
| T3 | 50 | ||||||
| T4 | 20 | ||||||
| T4 | 50 |
Table 1. Primary ↑ at 20 and 50 labels (mean ± std over ten seeds). Test 1 uses temperature and Tests 2–4 use water saturation. Vanilla denotes unit-weight PINN. Bold marks the highest mean.
Figure 1. Difference between the best PINN and best classical mean for the primary target. Positive cells favour PINNs. Each value is computed from ten paired seeds. Forecasting uses future residual coordinates but no future solution labels.
Tables B.1–B.4 in Appendix B give primary-target and MSE for every budget and all six methods. Supplementary Tables S1–S3 provide the corresponding pressure metrics for Tests 2–4.
Test 1: periodic heat equation
Figure 2 compares temperature profiles at four times. H-PINN follows the analytical solution throughout the interval. Boosting predictions retain a piecewise-constant form, while SVR and GP smooth the sparsely sampled profile.
Figure 2. Test 1 temperature for and seed 101 at . Reference denotes the analytical solution.
Both weighted PINNs are effectively exact in interpolation at and 50. In forecasting, HO-PINN reaches at , compared with for vanilla and for GP. GP leads forecasting from and interpolation from , but its largest mean advantage over the best PINN is 0.014 in . This test identifies a regime in which enough labels make a classical smoothness prior sufficient.
Test 2: moderate permeability contrast
The final-time fields in Figure 3 show the difference between the smooth pressure response and the transported saturation front. H-PINN preserves the front location and heterogeneous shape. The classical models recover the broad pressure trend but substantially smooth or discretize the saturation field.
Figure 3. Test 2 pressure and water saturation at , with and seed 101. All predictions share the reference colour range within each row.
At , H-PINN obtains saturation in interpolation and in forecasting. The strongest classical means are 0.541 and 0.230. At , the best PINN means increase to 0.977 and 0.948, with substantially smaller dispersion than the strongest classical entries. The advantage remains at , where the best PINN forecasting mean is 0.993 and the best classical mean is 0.852.
Test 3: high permeability contrast
Test 3 increases the permeability contrast to approximately . Figure 4 shows that H-PINN reconstructs the main saturation pathway and pressure transition. XGB produces block artefacts, while SVR and GP miss much of the front geometry.
Figure 4. Test 3 pressure and water saturation at , with and seed 101.
At , weighted PINNs have substantially greater seed variability than vanilla, although their means remain above the classical baselines. At , H-PINN reaches forecasting , compared with for vanilla and for GP. H-PINN and GP have mean MSE values of and , respectively. The PINN advantage over classical models persists through .
Test 4: seven-input parametric surrogate
Test 4 trains one model across a pool of 100 relative-permeability parameter combinations. Figure 5 displays one fixed combination at the final forecasting time. H-PINN follows the pressure interface and the principal saturation branches, while the three classical predictions lose much of the front structure.
Figure 5. Test 4 pressure and water saturation at for parameter combination 7, with and seed 101. The total label budget is shared across the pool of 100 combinations. No parameter combination is deliberately withheld.
The advantage persists in seven input dimensions. H-PINN forecasting saturation improves from at to at . At the latter budget, vanilla gives , while the best classical mean is with std . H-PINN therefore improves both the mean and dispersion in this condition. At , however, vanilla has smaller forecasting std than either weighted variant.
Run-to-run reliability of loss weighting
Table 2 covers all budgets and both regions. H-PINN and HO-PINN each reduce std relative to vanilla in 27 of 48 conditions. The clearest benefit is
| Test | Median std ratio ↓ H / Vanilla | Median std ratio ↓ HO / Vanilla | Lower std, out of 12 ↑ H-PINN | Lower std, out of 12 ↑ HO-PINN |
| Test 1 | 0.19 | 0.16 | 11 | 12 |
| Test 2 | 1.11 | 1.14 | 4 | 3 |
| Test 3 | 0.98 | 0.95 | 7 | 6 |
| Test 4 | 1.03 | 1.01 | 5 | 6 |
Table 2. Run-to-run variability relative to vanilla PINN after 50,000-epoch training. Each test has 12 conditions (six budgets, two regions), each with ten paired seeds. Ratios below one favour weighting. Counts and medians are descriptive, not independent significance tests.
Test 1, with median ratios 0.19 and 0.16. At , forecasting std decreases from 0.053 for vanilla to 0.006 for H and 0.005 for HO, while the mean also improves. In Test 4 at , H reduces forecasting std by a factor of 5.1 and raises the worst observed seed score from 0.632 to 0.795. HO does not reproduce this benefit in that condition.
The benefit is conditional. Vanilla has smaller std in most Test 2 conditions. In Test 3 at , its forecasting score is , whereas H and HO give and . Across all 480 evaluations per PINN, H and HO have eight and seven negative primary values, all in Tests 3–4 at , versus zero for vanilla. The two regions may flag the same trained model. Weighting can thus improve reliability markedly, but does not uniformly protect against low-budget failures. Appendix § 14.§ 5 reports the separate fixed-epoch comparison.
Cross-test result and computational cost
At 20 and 50 labels, a PINN leads all 16 test–region–budget comparisons. Over , a PINN leads in 23 of 24 comparisons. Across Tests 2–4, a PINN leads all 36 saturation comparisons, and across all tests it leads 42 of 48. The six classical wins are confined to the heat equation. Among all methods, vanilla leads 17 conditions, H leads 10 and HO leads 15.
This label efficiency has a substantial fitting cost. At , average GPU fitting times across the four tests range from 39–96 minutes for vanilla, 41–97 for H, and 64–158 for HO. Classical fitting including tuning takes approximately 0.8–66 seconds on CPU.
Discussion
The ten-seed study supports a solution-label advantage under known governing equations and prescribed initial and boundary conditions. At 20–50 labels, a PINN leads every comparison with classical models. For water saturation, this advantage persists through 1000 labels. Pressure is smoother and generally easier to reconstruct, so averaging it with saturation would obscure front-prediction errors. Vanilla PINN demonstrates that much of the gain comes from physical supervision and the shared constrained architecture. Weighting adds a substantial reliability benefit for the heat problem and selected flow conditions, but can also suppress useful objectives for too long when labels are extremely sparse.
The comparison holds solution labels fixed while allowing PINNs additional physical information. Residuals cover future times, which makes the forecasting task equation-assisted. It is not equivalent to unconstrained extrapolation from past labels. The results do not establish the same ranking when the PDE is unavailable, uncertain, or enforced only within the labelled interval.
Three limitations affect generalization. First, development inspected predictive metrics and reused seeds 101–103. Seeds 104–110 were added after configuration selection, but the protocol is not fully nested model selection. Second, the flow references use one spatial resolution and fixed permeability fields. Grid refinement and independent fields are needed to assess discretization sensitivity and transfer. Test 4 also does not hold out entire parameter combinations. Third, the comparison includes vanilla PINN and non-neural surrogates, but not specialized numerical solvers or other adaptive loss-weighting algorithms. The findings therefore concern these implementations rather than a ranking of weighting methods in general.
The heat-equation crossover shows that additional labels can make a conventional kernel surrogate preferable. PINN fitting is also substantially more expensive. The practical benefit depends on whether the reduction in required observations compensates for this cost. Our protocol establishes efficiency with respect to revealed state labels. It does not claim that labels correspond to full simulator runs. When sparse labels are extracted from already computed fields, reducing their number does not automatically reduce the offline simulation cost in the same proportion.
Conclusions
This study demonstrates that physics-informed neural networks can achieve accurate surrogate predictions from scarce labelled observations, including for strongly nonlinear, heterogeneous two-phase flow with moving saturation fronts. The benchmarks include permeability contrasts approaching five orders of magnitude and a seven-input parametric problem. Despite these challenges, a PINN achieves the highest mean primary-target accuracy among the tested methods in every interpolation and equation-assisted forecasting comparison at 20 and 50 labelled states. For water saturation, the advantage over tuned classical regression persists across all tested budgets up to 1000 states. These results show that useful neural surrogate predictions do not necessarily require large labelled datasets, even for challenging nonlinear flow problems, when reliable governing equations and prescribed initial and boundary conditions are available.
We introduce hierarchical loss weighting (H-PINN), which progressively activates physical constraints, and its gradient-direction-aware extension (HO-PINN). Both reduce variability across runs markedly for the heat equation and in selected flow settings. Their benefits remain problem dependent: neither consistently outperforms vanilla PINN, which is more reliable in some extremely sparse flow cases.
For engineering applications, the principal implication is that physical knowledge can supplement scarce solution observations. This label efficiency entails substantially higher fitting cost, and Gaussian-process regression becomes preferable for the sufficiently sampled smooth heat solution. Because each label represents one coordinate–state observation, the results establish efficiency with respect to revealed solution values rather than proportional savings in full simulations.
Data and code availability
The code and data are maintained in a single GitHub repository:
https://github.com/kirillkatsuba/pinn-small-data-jcp
CRediT authorship contribution statement
Kirill Katsuba: Conceptualization, Methodology, Software, Investigation, Formal analysis, Data curation, Validation, Visualization, Writing – original draft, Writing – review & editing.
Oleg Rudenko: Conceptualization, Software, Validation, Writing – review & editing.
Egor Bugaev: Software, Validation, Writing – review & editing.
Alexander Tarakanov: Conceptualization, Supervision, Writing – review & editing.
Acknowledgments
The supercomputer of HSE University was used to conduct the computational research [31].
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Declaration of generative AI and AI-assisted technologies in the writing process
During preparation of this work, the authors used OpenAI Codex for code-assisted analysis, LaTeX drafting, and language editing. The authors reviewed and edited all generated material and take full responsibility for the content of the publication.
Model selection and reproducibility
Staged PINN development
Development used all four tests, , seeds 101–103, both regions and 5000 epochs. Primary was supplemented by pressure accuracy, physics-validation losses, seed variability and fitting time. Selection was sequential, first capacity, then activation, weighting scales, HO floor, residual density and learning rate. Previously chosen settings were fixed. Table A.1 lists the candidates. Width 128 and depth six provided the preferred accuracy–cost compromise. GELU had the highest pooled development . This limited screen was not an exhaustive architecture search. Both weighted variants were retained.
| Component | Candidates | Selected |
| Width, depth | (64, 4), (128, 6), (128, 8) | (128, 6) |
| Activation | tanh, SiLU, GELU | GELU |
| 0.01, 0.03, 0.1, 0.3 | 0.03 | |
| 0.001, 0.003, 0.01, 0.03 | 0.003 | |
| HO floor | 0, 0.25, 0.5, 0.75, 1 | 0.25 |
| Residual ratio | 1, 2, 4 | 2 |
| Initial learning rate | , , |
Table A.1. Sequential PINN development screen. Candidate sets on different rows were not evaluated as one Cartesian product.
Selection used joint diagnostics without a preregistered scalar score. The optimum in was shallow around 0.25. Although slightly improved the initial pooled predictive score, 0.003 better balanced IBC and PDE errors in later screens and was retained. Final settings were shared across tests, budgets and seeds without retuning.
Optimization and checkpointing
Final runs use 50,000 Adam epochs, cosine decay from to , and gradient-norm clipping at one. Adam was fixed after pilot inspection. L-BFGS was excluded because fresh residuals change closure objectives. Each epoch uses all labels and fresh residual points over the full horizon. Labels and evaluation coordinates are excluded from the residual pool. The 512 fixed physics-validation points are diagnostic. The checkpoint minimizes the balanced training objective, without predictive or physics-validation epoch selection. Known domain bounds normalize inputs. Initialization, labels and residual streams are paired across PINNs. Of seeds 101–110, 101–103 were reused from development and 104–110 were added after settings were frozen. Dispersion combines label and optimization variability, with partial development-seed reuse acknowledged.
Algorithm 1: Training with H-PINN and HO-PINN
Data: Labelled states ; prescribed IBC and PDE; residual sampler; variant ; epochs ; scales , ; floor .
Result: Selected network parameters .
Initialize , Adam, and the cosine learning-rate schedule; set ;
for to do
Sample residual coordinates over the full time horizon, excluding labels and evaluation states;
Evaluate , , and at the current ;
;
;
, , ;
if then
for ;
;
;
;
;
, ;
for ;
; ;
with held fixed;
Clip to at most one and update with Adam;
Advance the cosine learning-rate schedule;
if then
; save the updated parameters as ;
return ;
Here denotes stop-gradient, clip truncates to the stated interval, and is the projection-removal operator in Section 3. We use and . Projections determine scalar weights only. As in the training code, checkpoint selection uses the pre-update objective and saves the corresponding post-update parameters.
Classical-model search
Classical methods use the same labels, with one regressor per output. Training times form four ordered blocks , with folds
Mean fold MSE selects the candidate, which is refitted on all labels. Scaling is fitted within each fold. Neither evaluation set enters tuning.
The procedure is not a greedy search. XGB and SVR use reproducible randomized searches over fixed discrete spaces, while GP uses an exhaustive grid. XGB evaluates 96 configurations per output, budget, and seed. Its space contains , maximum depth in , learning rate in , child weight in , row and feature fractions in , split penalty in , and and penalties in and , respectively. Histogram-based squared-error boosting is used.
RBF-SVR also evaluates 96 configurations. Inputs and targets are standardized inside the cross-validation pipeline. The candidates are
The randomized candidate sequence is fixed by the experiment seed and output index.
GP evaluates all 20 combinations of five covariance families and four nugget values. The families are RBF, Matérn with smoothness , , or , and rational quadratic. Each is multiplied by a learned constant and augmented by a learned white-noise term. The nugget is . For each candidate, kernel amplitude, length scales, and noise level are optimized by marginal likelihood with three restarts. Inputs are standardized and targets are normalized. Neither the interpolation nor extrapolation evaluation set is accessed by any of these searches.
Reference-solution generation
Test 1 uses the exact analytical solution at 161 times and 257 spatial coordinates. Labels are sampled continuously across ten temporal bins. Flow references use cell-centred grids and 101 synchronized snapshots, including the initial state. Final times 0.30, 0.05 and 0.20 for Tests 2–4 accommodate different permeability scales and retain developed fronts before outflow.
At every adaptive transport substep, pressure is recomputed from the current total mobility. Face permeability is the harmonic mean of adjacent cells. Phase transport uses first-order upstream fractional flow, and the conservative update is divided by the cell pore volume. The step size obeys CFL = 0.2. Generation aborts if saturation leaves beyond numerical tolerance or if the pressure solve exceeds its residual tolerance. The stored runs have pressure-balance errors below and phase mass-balance errors below .
Fixed permeability fields have contrasts of about 24 and in Tests 2–3. Test 4 uses a third field and 100 fixed parameter tuples. All fields and tuples are shared across methods. Training candidates cover the first 70% of time, with 100,000 pooled states for Test 4. Temporal interleaving yields nested budgets. Evaluation states are removed before label or residual sampling. Flow PINNs satisfy saturation bounds and have closure errors near , which do not determine predictive rankings.
Fixed-epoch convergence comparison
An additional 720 PINN fits cover all four tests, six budgets, ten seeds and three variants. They use 10,000 epochs and a cosine horizon of 10,000 epochs, with checkpoints at 1000, 2500, 5000, 7500 and 10,000. These are independent short-budget runs, not prefixes of the main 50,000-epoch schedule. At epoch 1000, H and HO exceed vanilla in mean primary in 24 and 23 of the 48 conditions. At epoch 10,000 these counts are 26 and 24, while smaller standard deviations occur in 25 and 23 conditions. Evaluation uses exact epochs, not selected checkpoints. These mixed rankings do not establish universally faster convergence or a wall-clock advantage, particularly because HO requires additional gradient evaluations.
Exploratory architecture and hierarchy ablations
CPU pilots at , seed 101 and 10,000 epochs remove all output transformations while retaining the backbone, optimizer and losses. Against exact final checkpoints of the constrained GPU controls, forecasting saturation decreases for every variant in Tests 2–3. For H-PINN it falls from 0.862 to 0.845 and from 0.588 to 0.470, respectively. All direct Test 4 runs have nonfinite weights by epoch 1000. Negative saturation raised to a noninteger Corey exponent is undefined in real arithmetic, making unrestricted outputs incompatible with the unmodified residual outside the physical domain. Single-seed, cross-device pilots cannot establish multi-seed robustness.
A second H-PINN pilot prioritizes IBC, PDE, then data. With and , its normalized weights are proportional to . It gives no consistent improvement across Tests 1–4 and a final data weight of only in Test 3. We retain data–IBC–PDE priority.
Complete primary-target metrics across label budgets
Tables B.1–B.4 report all six methods at . Interpolation and equation-assisted forecasting are reported separately. Test 1 uses temperature and Tests 2–4 use water saturation. Supplementary Tables S1–S3 give pressure metrics for the same budgets and methods. Every cell uses the same ten paired seeds as the main comparison. Bold identifies the best unrounded mean within each row, so equally rounded entries need not share highlighting. All std values remain visible without separate highlighting. MSE mantissas use one decimal place and uses three.
Table B.1: Test 1 temperature, all budgets (mean ± sample std over ten seeds). Bold marks the best unrounded mean. MSE entries use .
| Vanilla | H-PINN | HO-PINN | XGB | SVR | GP | |
| Temperature, — Interpolation | ||||||
| 20 | 0.993±0.003 | 1.000±0.000 | 1.000±0.000 | 0.385±0.131 | 0.461±0.303 | 0.797±0.098 |
| 50 | 0.997±0.001 | 1.000±0.000 | 1.000±0.000 | 0.639±0.117 | 0.691±0.228 | 0.932±0.055 |
| 100 | 0.999±0.001 | 1.000±0.000 | 1.000±0.000 | 0.684±0.152 | 0.777±0.222 | 0.989±0.009 |
| 200 | 0.999±0.000 | 1.000±0.000 | 1.000±0.000 | 0.766±0.123 | 0.971±0.071 | 0.998±0.001 |
| 500 | 1.000±0.000 | 1.000±0.000 | 1.000±0.000 | 0.862±0.124 | 0.908±0.146 | 1.000±0.000 |
| 1000 | 1.000±0.000 | 1.000±0.000 | 1.000±0.000 | 0.927±0.063 | 0.891±0.130 | 1.000±0.000 |
| Temperature, — Equation-assisted forecasting | ||||||
| 20 | 0.946±0.069 | 0.992±0.008 | 0.994±0.003 | −0.069±0.468 | −0.274±0.542 | 0.267±0.817 |
| 50 | 0.884±0.053 | 0.993±0.006 | 0.993±0.005 | 0.276±0.438 | 0.479±0.524 | 0.973±0.028 |
| 100 | 0.937±0.051 | 0.993±0.008 | 0.994±0.005 | 0.409±0.404 | 0.527±0.430 | 0.998±0.002 |
| 200 | 0.947±0.033 | 0.990±0.007 | 0.987±0.011 | 0.707±0.241 | 0.779±0.175 | 0.999±0.001 |
| 500 | 0.965±0.024 | 0.986±0.023 | 0.984±0.020 | 0.844±0.146 | 0.848±0.120 | 1.000±0.000 |
| 1000 | 0.982±0.019 | 0.990±0.008 | 0.994±0.004 | 0.881±0.035 | 0.738±0.246 | 1.000±0.000 |
| Vanilla | H-PINN | HO-PINN | XGB | SVR | GP | |
| Temperature, MSE — Interpolation | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | ||||||
| 1000 | ||||||
| Temperature, MSE — Equation-assisted forecasting | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | ||||||
| 1000 |
Table B.1.
| Vanilla | H-PINN | HO-PINN | XGB | SVR | GP | |
| Water saturation, — Interpolation | ||||||
| 20 | 0.954±0.007 | 0.954±0.008 | 0.937±0.052 | 0.362±0.241 | 0.396±0.276 | 0.541±0.157 |
| 50 | 0.977±0.003 | 0.975±0.004 | 0.977±0.003 | 0.632±0.046 | 0.685±0.092 | 0.745±0.089 |
| 100 | 0.987±0.002 | 0.988±0.003 | 0.988±0.003 | 0.747±0.052 | 0.815±0.039 | 0.876±0.031 |
| 200 | 0.995±0.001 | 0.994±0.001 | 0.994±0.001 | 0.836±0.033 | 0.840±0.047 | 0.916±0.017 |
| 500 | 0.998±0.001 | 0.998±0.001 | 0.998±0.001 | 0.908±0.025 | 0.878±0.048 | 0.960±0.003 |
| 1000 | 1.000±0.000 | 1.000±0.000 | 1.000±0.000 | 0.945±0.019 | 0.898±0.035 | 0.975±0.002 |
| Water saturation, — Equation-assisted forecasting | ||||||
| 20 | 0.917±0.015 | 0.917±0.016 | 0.893±0.079 | 0.230±0.287 | −0.864±1.240 | 0.013±0.392 |
| 50 | 0.948±0.009 | 0.947±0.010 | 0.948±0.010 | 0.397±0.136 | 0.279±0.250 | 0.486±0.197 |
| 100 | 0.965±0.005 | 0.966±0.004 | 0.966±0.005 | 0.503±0.095 | 0.429±0.198 | 0.626±0.150 |
| 200 | 0.975±0.003 | 0.974±0.004 | 0.974±0.004 | 0.612±0.080 | 0.559±0.203 | 0.739±0.093 |
| 500 | 0.983±0.004 | 0.985±0.003 | 0.985±0.002 | 0.686±0.056 | 0.676±0.142 | 0.842±0.014 |
| 1000 | 0.993±0.001 | 0.992±0.001 | 0.993±0.001 | 0.726±0.043 | 0.730±0.091 | 0.852±0.016 |
| Water saturation, MSE — Interpolation | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | ||||||
| 1000 | ||||||
| Water saturation, MSE — Equation-assisted forecasting | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | ||||||
| 1000 |
Table B.2. Test 2 water saturation, all budgets (mean ± sample std over ten seeds). Bold marks the best unrounded mean. MSE entries use .
| Vanilla | H-PINN | HO-PINN | XGB | SVR | GP | |
| \multicolumn{7}{c}{Water saturation, — Interpolation} | ||||||
| 20 | 0.755±0.018 | 0.461±0.360 | 0.567±0.312 | 0.019±0.350 | 0.084±0.482 | 0.265±0.401 |
| 50 | 0.847±0.015 | 0.841±0.015 | 0.842±0.010 | 0.344±0.302 | 0.377±0.304 | 0.544±0.247 |
| 100 | 0.908±0.023 | 0.905±0.020 | 0.905±0.020 | 0.604±0.122 | 0.596±0.171 | 0.748±0.211 |
| 200 | 0.947±0.009 | 0.946±0.010 | 0.946±0.010 | 0.770±0.067 | 0.711±0.049 | 0.891±0.019 |
| 500 | 0.979±0.005 | 0.980±0.005 | 0.980±0.005 | 0.888±0.018 | 0.779±0.063 | 0.949±0.006 |
| 1000 | 0.990±0.002 | 0.991±0.002 | 0.991±0.002 | 0.931±0.019 | 0.766±0.065 | 0.969±0.004 |
| \multicolumn{7}{c}{Water saturation, — Equation-assisted forecasting} | ||||||
| 20 | 0.642±0.042 | 0.334±0.401 | 0.445±0.365 | −0.089±0.269 | −2.148±4.703 | 0.084±0.284 |
| 50 | 0.768±0.033 | 0.762±0.026 | 0.759±0.025 | 0.172±0.394 | −1.011±2.289 | 0.293±0.269 |
| 100 | 0.857±0.029 | 0.855±0.026 | 0.853±0.026 | 0.440±0.137 | −0.636±2.185 | 0.551±0.214 |
| 200 | 0.915±0.015 | 0.915±0.015 | 0.916±0.015 | 0.614±0.050 | −0.389±1.808 | 0.678±0.131 |
| 500 | 0.954±0.005 | 0.956±0.005 | 0.957±0.006 | 0.740±0.043 | 0.231±0.487 | 0.802±0.036 |
| 1000 | 0.972±0.004 | 0.974±0.003 | 0.974±0.003 | 0.809±0.032 | 0.422±0.186 | 0.838±0.031 |
| \multicolumn{7}{c}{Water saturation, MSE — Interpolation} | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | ||||||
| 1000 | ||||||
| \multicolumn{7}{c}{Water saturation, MSE — Equation-assisted forecasting} | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | ||||||
| 1000 |
Table B.3. Test 3 water saturation, all budgets (mean ± sample std over ten seeds). Bold marks the best unrounded mean. MSE entries use .
| Vanilla | H-PINN | HO-PINN | XGB | SVR | GP | |
| \multicolumn{7}{c}{Water saturation, — Interpolation} | ||||||
| 20 | 0.495±0.251 | 0.522±0.305 | 0.426±0.358 | −0.058±0.268 | −0.187±0.294 | −0.077±0.339 |
| 50 | 0.777±0.100 | 0.807±0.015 | 0.779±0.109 | 0.182±0.160 | 0.083±0.109 | 0.282±0.192 |
| 100 | 0.864±0.013 | 0.867±0.013 | 0.865±0.013 | 0.419±0.065 | 0.230±0.094 | 0.575±0.210 |
| 200 | 0.912±0.010 | 0.913±0.013 | 0.915±0.009 | 0.591±0.059 | 0.263±0.124 | 0.783±0.090 |
| 500 | 0.963±0.006 | 0.962±0.006 | 0.963±0.005 | 0.738±0.042 | 0.450±0.074 | 0.885±0.021 |
| 1000 | 0.983±0.005 | 0.983±0.004 | 0.982±0.004 | 0.831±0.018 | 0.527±0.032 | 0.933±0.006 |
| \multicolumn{7}{c}{Water saturation, — Equation-assisted forecasting} | ||||||
| 20 | 0.558±0.207 | 0.550±0.360 | 0.435±0.454 | −0.124±0.314 | −0.325±0.364 | −0.084±0.280 |
| 50 | 0.794±0.059 | 0.815±0.011 | 0.799±0.067 | 0.198±0.269 | 0.019±0.180 | 0.216±0.214 |
| 100 | 0.864±0.012 | 0.866±0.013 | 0.863±0.015 | 0.405±0.064 | 0.231±0.120 | 0.529±0.189 |
| 200 | 0.905±0.011 | 0.903±0.015 | 0.907±0.011 | 0.521±0.100 | 0.258±0.195 | 0.704±0.099 |
| 500 | 0.951±0.007 | 0.950±0.006 | 0.951±0.006 | 0.663±0.040 | 0.405±0.118 | 0.797±0.048 |
| Vanilla | H-PINN | HO-PINN | XGB | SVR | GP | |
| 1000 | ||||||
| Water saturation, MSE — Interpolation | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | ||||||
| 1000 | ||||||
| Water saturation, MSE — Equation-assisted forecasting | ||||||
| 20 | ||||||
| 50 | ||||||
| 100 | ||||||
| 200 | ||||||
| 500 | ||||||
| 1000 |
Table B.4. Test 4 water saturation, all budgets (mean ± sample std over ten seeds). Bold marks the best unrounded mean. MSE entries use .
References
- [1]M. Tang, Y. Liu, L. J. Durlofsky, A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems, Journal of Computational Physics 413 (2020) 109456. doi:10.1016/j.jcp.2020.109456.DOI
- [2]M. Tang, Y. Liu, L. J. Durlofsky, Deep-learning-based surrogate flow modeling and geological parameterization for data assimilation in 3d subsurface flow, Computer Methods in Applied Mechanics and Engineering 376 (2021) 113636. doi:10.1016/j.cma.2020.113636.DOI
- [3]S. Mo, Y. Zhu, N. Zabaras, X. Shi, J. Wu, Deep convolutional encoder–decoder networks for uncertainty quantification of dynamic multiphase flow in heterogeneous media, Water Resources Research 55 (1) (2019) 703–728. doi:10.1029/2018WR023528.DOI
- [4]G. Wen, Z. Li, K. Azizadehnesheli, A. Anandkumar, S. M. Benson, U-FNO—an enhanced Fourier neural operator-based deep-learning model for multiphase flow, Advances in Water Resources 163 (2022) 104180. doi:10.1016/j.advwatres.2022.104180.DOI
- [5]P. Benner, S. Gugercin, K. Willcox, A survey of projection-based model reduction methods for parametric dynamical systems, SIAM Review 57 (4) (2015) 483–531. doi:10.1137/130932715.DOI
- [6]S. Fresca, L. Dedè, A. Manzoni, A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs, Journal of Scientific Computing 87 (2) (2021) 61. doi:10.1007/s10915-021-01462-7.DOI
- [7]Y. Zhu, N. Zabaras, Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification, Journal of Computational Physics 366 (2018) 415–447. doi:10.1016/j.jcp.2018.04.018.DOI
- [8]L. Lu, P. Jin, G. Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence 3 (3) (2021) 218–229. doi:10.1038/s42256-021-00302-5.DOI
- [9]Z. Li, N. Kovachki, K. Azizadehnesheli, B. Liu, K. Bhattacharya, A. Stuart, A. Anandkumar, Fourier neural operator for parametric partial differential equations, in: International Conference on Learning Representations, 2021, accessed 8 November 2026. URL https://openreview.net/forum?id=c8P9NQvTmn0
- [10]T. Chen, C. Guestrin, XGBoost: A scalable tree boosting system, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2016, pp. 785–794. doi:10.1145/2939672.2939785.DOI
- [11]H. Drucker, C. J. C. Burges, L. Kaufman, A. J. Smola, V. Vapnik, Support vector machines, in: Advances in Neural Information Processing Systems, Vol. 9, 1996, pp. 155–161.
- [12]C. E. Rasmussen, C. K. I. Williams, Gaussian Processes for Machine Learning, MIT Press, Cambridge, MA, 2006. doi:10.7551/mitpress/3206.001.0001.DOI
- [13]M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378 (2019) 686–707. doi:10.1016/j.jcp.2018.10.045.DOI
- [14]G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, L. Yang, Physics- informed machine learning. Nature Reviews Physics 3 (2021) 422–440. doi:10.1038/s42254-021-00314-5.
- [15]L. Sun, H. Gao, S. Pan, J.-X. Wang, Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data, Computer Methods in Applied Mechanics and Engineering 361 (2020) 112732. doi:10.1016/j.cma.2019.112732.DOI
- [16]Q. He, D. Barajas-Solano, G. Tartakovsky, A. M. Tartakovsky, Physics-informed neural networks for multiphysics data assimilation with application to subsurface transport, Advances in Water Resources 141 (2020) 103610. doi:10.1016/j.advwatres.2020.103610.DOI
- [17]Z. Chen, Y. Liu, H. Sun, Physics-informed learning of governing equations from scarce data, Nature Communications 12 (1) (2021) 6136. doi:10.1038/s41467-021-26434-1.DOI
- [18]J. M. Hanna, J. V. Aguado, S. Comas-Cardona, R. Askri, D. Borzacchiello, Residual-based adaptivity for two-phase flow simulation in porous media using physics-informed neural networks, Computer Methods in Applied Mechanics and Engineering 396 (2022) 115100. doi:10.1016/j.cma.2022.115100.DOI
- [19]X. Yan, J. Lin, S. Wang, Z. Zhang, P. Liu, S. Sun, J. Yao, K. Zhang, Physics-informed neural network simulation of two-phase flow in heterogeneous and fractured porous media, Advances in Water Resources 189 (2024) 104731. doi:10.1016/j.advwatres.2024.104731.DOI
- [20]Y. Kenzhebek, T. Imankulov, S. D. Bekele, I. Panfilova, D. Akhmed-Zaki, Coupled pressure and saturation prediction for two-phase flow in porous media using physics-informed neural networks (PINNs), Computer Methods in Applied Mechanics and Engineering 446 (2025) 118229. doi:10.1016/j.cma.2025.118229.DOI
- [21]A. S. Krishnapriyan, A. Gholami, S. Zhe, R. M. Kirby, M. W. Mahoney, Characterizing possible failure modes in physics-informed neural networks, in: Advances in Neural Information Processing Systems, Vol. 34, 2021, pp. 26548–26560.arxiv.org/abs/2109.01050
- [22]Z. Hao, J. Yao, C. Su, H. Su, Z. Wang, F. Lu, Z. Xia, Y. Zhang, S. Liu, L. Lu, J. Zhu, PINNacle: A comprehensive benchmark of physics-informed neural networks for solving PDEs, in: Advances in Neural Information Processing Systems, Vol. 37, 2024, pp. 76721–76774. doi:10.52202/079017-2442.DOI
- [23]T. G. Grossmann, U. J. Komorowska, J. Latz, C.-B. Schönlieb, Can physics-informed neural networks beat the finite element method?, IMA Journal of Applied Mathematics 89 (1) (2024) 143–174. doi:10.1093/imamat/hxae011.DOI
- [24]S. Wang, Y. Teng, P. Perdikaris, Understanding and mitigating gradient flow pathologies in physics-informed neural networks, SIAM Journal on Scientific Computing 43 (5) (2021) A3055–A3081. doi:10.1137/20M1318043.DOI
- [25]L. D. McClenny, U. M. Braga-Neto, Self-adaptive physics-informed neural networks, Journal of Computational Physics 474 (2023) 111722. doi:10.1016/j.jcp.2022.111722.
- [26]S. Maddu, D. Sturm, C. L. Müller, I. F. Sbalzarini, Inverse Dirichlet weighting enables reliable training of physics informed neural networks, Machine Learning: Science and Technology 3 (1) (2022) 015026. doi:10.1088/2632-2153/ac3712.DOI
- [27]R. Bischof, M. A. Kraus, Multi-objective loss balancing for physics-informed deep learning, Computer Methods in Applied Mechanics and Engineering 439 (2025) 117914.




