WorksPhysics and Astronomy
OPINE: Orthogonal Projection in Invertible Nonlinear Embeddings for Turbulent Flows
Abstract
Proper orthogonal decomposition (POD) represents physical fields with orthogonal modes ordered by variance, so that one decomposition gives reduced coordinates in every dimension, but it needs many modes when the data lie near a curved manifold. Nonlinear autoencoders are more compact, but their information loss happens inside a nonlinear encoder, and a model trained with one latent dimension gives no model with a smaller one. We introduce Orthogonal Projection in Invertible Nonlinear Embeddings (OPINE), which learns an invertible nonlinear change of coordinates and performs POD in the transformed coordinates. Because the map is invertible, all information loss occurs in the POD projection, and ordering its basis after training gives reduced coordinates of every smaller dimension from a single model. Training alternates gradient updates of the map with a damped update that tracks the evolving POD subspace, and we prove that this update converges locally to the exact POD subspace. We evaluate OPINE on synthetic manifolds, chaotic Kuramoto-Sivashinsky dynamics, forced Burgers turbulence, two-dimensional Kolmogorov flow, and three-dimensional minimal channel flow. OPINE is more accurate than POD at every tested retained dimension above one, and on the Kuramoto-Sivashinsky and Burgers dynamics, it is more accurate than convolutional and fully connected autoencoders by more than two orders of magnitude at the largest retained dimension.
Introduction
A snapshot of a turbulent flow has thousands to millions of degrees of freedom, yet its dominant dynamics often lie near a much lower-dimensional set. Reduced coordinates that describe this set are the basis of data compression, flow analysis, and reduced-order models. The standard method for obtaining them is proper orthogonal decomposition (POD), known in statistics as principal component analysis. POD modes are orthogonal and ordered by the variance they capture, and for every the leading modes span the best -dimensional linear subspace [39, 4, 51]. One decomposition therefore provides reduced models of every size, and each coordinate has a clear physical meaning.
This optimality, however, holds only among linear subspaces. Curved and transport-dominated solution sets, such as traveling waves, shocks, and the chaotic attractors of turbulent flows, can need many modes even when they have low-dimensional nonlinear parametrizations [20, 35]. Autoencoders overcome this limitation with nonlinear encoders and decoders, and they have improved on POD for fluid fields [33, 43, 45]. However, a standard autoencoder has neither orthogonal modes nor an ordering, its encoder can map different states to the same latent vector, and a model trained with one latent dimension gives no model with a smaller one. Nested dropout, Matryoshka representations, and hierarchical autoencoders order the latent variables [48, 34, 17], but the truncation still happens inside a nonlinear encoder.
Our key idea is to keep POD as the only bottleneck, but to apply it in transformed coordinates. In Orthogonal Projection in Invertible Nonlinear Embeddings (OPINE), an invertible map, parametrized by a neural network, transforms the field without reducing its dimension, POD removes dimensions in the transformed coordinates, and the inverse map returns the result to physical space (Figure 1). Because the map is invertible and keeps the dimension of the field, no information is lost before the projection, and the whole model reduces to the identity when all coordinates are kept. OPINE can therefore be viewed as a search for an observable of the field, that is, a nonlinear change of variables in which the data lie close to a linear subspace, similar to the feature map of kernel PCA ([49]) but invertible, so that no pre-image problem arises ([42]). Once this observable is found, the properties of POD, including orthogonal, ordered, and nested modes, come at no additional cost. Training is a coupled problem, because the POD basis changes whenever the map changes, and a sudden change of the basis would also change the loss suddenly. We therefore alternate gradient updates of the map with a damped, aligned step of orthogonal iteration. Our contributions are:
Figure 1. OPINE for a scalar field: only the rank- POD projection of loses information, and and its exact inverse share . Training alternates the two steps at bottom right (Section 3.4). For multi-component fields, is omitted and .
Method: a factorization of nonlinear dimensionality reduction into an invertible change of coordinates and an explicit POD projection, which confines all information loss to the projection and gives reduced coordinates of every smaller dimension from one model (§ 3);
Training and theory: a scheme that tracks the POD subspace without differentiating through an eigendecomposition, together with a proof that the damped update converges to the exact POD subspace and follows it closely while the map changes (§ 3.4; Appendix § 8);
Experiments: from synthetic manifolds and one-dimensional chaotic dynamics to two- and three-dimensional turbulence, in which OPINE improves on POD at every retained dimension above one, and ablations that support the choice of the POD projection (§ 4).
Related Work
OPINE is related to three lines of work: autoencoders and nonlinear model reduction, ordered and invertible representations, and subspace tracking.
Autoencoders and nonlinear model reduction. Autoencoders learn a nonlinear encoder and decoder through a low-dimensional bottleneck and are the standard nonlinear generalization of POD ([33, 21]); variational autoencoders add a probabilistic latent space ([30]). In fluid mechanics, autoencoders have compressed near-wall turbulence ([43]), decomposed flows into nonlinear modes ([45]), built hierarchical or nearly orthogonal latent modes ([17, 14]), and learned coordinates for Koopman and inertial-manifold models ([40, 7, 37]). In reduced-order modeling, convolutional autoencoders provide nonlinear trial manifolds for advection-dominated systems ([35, 41]), and POD precompression makes them cheaper to train ([16]). Classical alternatives transport the data before a linear projection, as in shifted POD and registration ([47, 50]), or add a polynomial correction to a POD subspace, as in quadratic manifolds ([18]); these fix the form of the transformation. OPINE instead learns a general invertible map and applies POD after it, so that POD remains the only bottleneck.
Ordered and invertible representations. Invertible networks such as Real NVP, Glow, and invertible residual networks provide flexible bijective maps with tractable inverses ([11, 29, 2]), and injective or manifold flows use such maps to learn low-dimensional manifolds ([5, 6, 32, 9]). Closest to OPINE are nested-dropout flows, which order the coordinates of an invertible network and reconstruct by zeroing its trailing coordinates ([48, 3]), and Neural-PCA, which couples a flow to a PCA layer for likelihood-based representation learning ([36]). In principle, an unrestricted invertible map could absorb a global rotation, but stacks of local coupling layers cannot represent one cheaply. OPINE therefore keeps its bottleneck close to the POD subspace of the transformed data and updates it in closed form, which we show matters in practice. It is also trained for field reconstruction rather than likelihood.
Subspace tracking. Online PCA and subspace-tracking algorithms update dominant eigenspaces as the data evolve [46, 53], whereas backpropagating through an eigendecomposition [25, 52] is fragile near the repeated eigenvalues of paired Fourier modes. We use damped, aligned orthogonal iteration.
Method
Sections 3.1–3.3 present our first contribution, the factorization, and Section 3.4 the second, the training scheme. Let be a discretized field with components on a grid of points, and thus degrees of freedom. OPINE places an explicit projection inside an invertible change of coordinates:
Because is bijective, the projection is the only step that loses information.
Representation
We first define the reduced coordinates. Let , let be the mean of the transformed data, and let be an orthonormal basis of the learned -dimensional subspace. For and , the reduced coordinates and the reconstruction are
Here , and counts real coefficients. The retained dimension of the experiments equals , and each model is trained with . When all transformed coordinates are kept (), the reconstruction is exact. Because is a projection and is bijective, is a nonlinear projection onto an -dimensional manifold, the kind of trial manifold used by manifold Galerkin and least-squares Petrov–Galerkin models [35].
Invertible Map
The representation above holds for any invertible ; we now describe the one we use. It is a stack of blocks , where is an invertible channel mixing [29] and is an affine coupling layer [11] that splits the channels into and maps
This layer can be inverted in closed form for any conditioner network . Multi-component fields use their physical components as channels, so that . Scalar fields first pass through an invertible space-to-depth rearrangement , which folds small blocks of neighboring grid points into channels, so that . We also tried invertible residual networks and monotone piecewise-quadratic maps, but affine coupling layers were the most accurate and the fastest to invert (Appendix § 10).
POD Bottleneck
The bottleneck is the POD of the transformed data. For fixed and transformed training states with mean and covariance , the rank- POD basis
spans the leading -dimensional eigenspace of , whose eigenvalues we denote .
Training
It remains to train the map and the basis together. The map minimizes the reconstruction error, while and must follow the POD of the transformed data, which changes as changes. We keep tied to that POD, rather than learning it as a free parameter, so that the retained subspace can be ordered by variance. Training alternates two updates, neither of which differentiates through an eigendecomposition. First, with and fixed, we take a round of gradient steps on
over minibatches , with from Eq. (2). Second, with fixed, we recompute and take one damped, aligned step of orthogonal iteration:
where qf is the orthonormal factor of a thin QR factorization and is an SVD. Orthogonal iteration moves the subspace toward the dominant eigenspace but may rotate the basis within it, and the Procrustes rotation undoes this rotation so that the old and new bases can be interpolated. Since , the update is well defined for every , and a smaller gives stronger damping. Appendix § 8 proves that this update converges to the exact POD subspace and follows it closely while changes.
After training, we diagonalize and set . This leaves the projection unchanged and orders the basis by variance, so each prefix spans the most energetic -dimensional subspace of , and one model gives reduced coordinates for every (§ 4.6).
Experiments
Our third contribution is an evaluation of OPINE on datasets of increasing dimension, from synthetic one-dimensional profiles to three-dimensional turbulence. We ask four questions: whether OPINE finds transformed coordinates in which data of known low dimension have low rank (Q1; § 4.2); whether it improves on POD and autoencoders with the same number of retained coefficients (Q2; § 4.3–§ 4.5); whether its improvement over POD comes from the transformed coordinates or from the choice of projection (Q3; § 4.2); and whether one trained model is accurate at every smaller (Q4; § 4.6).
Baselines and Evaluation
To keep the comparison fair, every method stores the same number of real coefficients per snapshot: for POD and OPINE, and a -dimensional latent vector for the autoencoders. POD is Eq. (2) with the identity. The CNN autoencoder compresses the field to its latent vector with strided convolutions and boundary-aware padding, and reconstructs it with a mirrored decoder (Figure 7 in Appendix § 10). For the one-dimensional problems we also report a fully connected (MLP) autoencoder. OPINE also has far fewer parameters: for the turbulent flows, it has to parameters, whereas the CNN autoencoders have to . All learned models are trained separately at each and selected on validation data (Appendix § 10). We report the test normalized mean squared error (NMSE) on a held-out set shared by all methods.
Figure 7. CNN autoencoder baseline, illustrated for a two-dimensional field with three velocity components: strided convolutions with boundary-aware padding compress the field to a -dimensional latent vector, and a mirrored decoder reconstructs it. The three-dimensional experiments use its volumetric counterpart.
Synthetic Manifolds and Ablations
We begin with synthetic data whose low-dimensional parametrization is known exactly. The family on a periodic grid, with a random phase , is a one-parameter set of traveling profiles. It is not low rank in the original coordinates, because the generates all odd harmonics, but it is exactly rank 2 in the invertible coordinates . We train at with 32 samples and ten seeds. OPINE finds such coordinates (Q1): two POD modes of its transformed coordinates hold 99.99% of the transformed variance, and its test error is lower than that of POD on every seed and 28 times lower on average, whereas the CNN autoencoder overfits the small training set.
We then use four synthetic families of profiles on 128 points of to identify the source of the improvement over POD (Q3). In Pulse, , where is the periodic distance from a random center ; its states form a closed loop. Pulse width is the same pulse with a random width in instead of 0.3. In Front, with a random position and steepness . In Torus, with two random phases and the logistic function ; its states form a torus. POD needs many modes for all four, because the parameters move or deform the profiles. Table 1 retrains the same invertible map with different projections and otherwise identical training. Keeping the first entries of , as nested-dropout flows do at a fixed dimension, is usually no more accurate than plain POD. A fixed Fourier basis improves on POD on three of the four families, but OPINE is the most accurate on all four. In Figure 2, the pointwise error of OPINE is about two orders of magnitude below that of POD along the whole profile, whereas keeping the first entries of gives noisy reconstructions. For these translation-invariant families, POD of the raw data is itself a Fourier basis ordered by variance. The fixed Fourier basis differs from OPINE because the invertible map is not translation-equivariant, so the POD modes of are not Fourier modes, and because the basis is ordered by wavenumber rather than by variance. Table 1 is an ablation of the projection with the architecture and training of the invertible map held fixed; OPINE is compared with the autoencoders on the dynamical systems below, where the benefit of the transformed coordinates is largest.
| POD | first entries | Fourier basis | OPINE | |
| Torus | ||||
| Front | ||||
| Pulse | ||||
| Pulse width |
Table 1. Bottleneck ablation at : median test NMSE. POD projects the raw data. The other three columns retrain the same invertible map with a different projection: keeping the first entries of and setting the others to their training mean, onto a fixed Fourier basis, or onto the POD subspace of the transformed field, which is OPINE. Best in bold. Figure 2 shows sample reconstructions.
Figure 2. Sample reconstructions for the bottleneck ablation of Table 1 at . Top: a test profile (grey) and its reconstructions; the profile is the test sample with the median OPINE error. Bottom: pointwise error on a logarithmic scale. POD misses the shape of each profile, and keeping the first entries of gives noisy profiles. The error of OPINE is about two orders of magnitude below that of POD, and on Front and Torus it is also well below that of the fixed Fourier basis.
On the same families, damped tracking is as accurate as exact recomputation of the POD, consistent with Appendix § 8, whereas freezing the subspace at its initial POD roughly triples the error on Pulse. Tracking is also much cheaper than differentiating through an eigendecomposition (Appendix § 9).
One-Dimensional Dynamics
Having confirmed that OPINE finds known low-dimensional coordinates, we turn to two one-dimensional systems whose dynamics concentrate near a low-dimensional set that is not known in advance. The Kuramoto–Sivashinsky (KS) equation on a periodic domain of length 22 models flame fronts and thin-film flows, and its chaotic attractor lies close to an inertial manifold of dimension about 8 [37]. Forced Burgers turbulence, , is driven by random forcing of the largest scales that is uncorrelated in time. It produces a statistically stationary field of steep, moving shocks, a standard transport-dominated test case for linear reduction. Each system has eight independent trajectories, which are split by trajectory (Appendix § 9).
Figure 3 summarizes both systems. The left panels show a test trajectory in space and time: KS forms a slowly drifting cellular pattern, and Burgers a field of sharp fronts. The middle panels show where each method errs at a fixed , with darker regions marking larger pointwise error. POD has large errors everywhere, the autoencoders have errors concentrated along the cells and shocks, and OPINE’s error is small everywhere. The right panels show the test NMSE over . From onward, OPINE is at least as accurate as every other method, and its error keeps falling where the autoencoders level off, so that at the largest it is more than two orders of magnitude below them. On KS, its error drops sharply near , the manifold dimension estimated by Linot & Graham [37].
Figure 3. Kuramoto–Sivashinsky (a) and forced Burgers turbulence (b). Left: a test trajectory in space and time. Middle: pointwise error of each method at the marked by the dashed line on the right, on a shared logarithmic scale (darker is worse); titles give the test NMSE of the run shown. Right: test NMSE versus (median and range over three seeds).
Two-Dimensional Kolmogorov Flow
We now move from one to two space dimensions. Kolmogorov flow is two-dimensional Navier–Stokes flow driven by a sinusoidal body force, , on a doubly periodic domain. At with forcing wavenumber , the flow is chaotic, with coherent vortices that form, merge, and decay. We reduce vorticity snapshots on a 128 × 128 grid from 100 independent trajectories, split by trajectory so that test snapshots are selected in time to training snapshots (Appendix § 10).
Figure 4 shows that OPINE improves on POD at every , by up to 75%, and that the margin grows with (Q2). The CNN autoencoder, which has 20 to 300 times more parameters, is more accurate at small , and OPINE is more accurate at the largest . The reconstructions on the right show the source of the larger POD error: with the same number of coefficients, POD blurs the vortex cores and the thin filaments between them, which the nonlinear methods retain. Figure 5 decodes single reduced coordinates back to vorticity. The POD and OPINE coordinates come in phase-shifted pairs of the same spatial scale, as expected for a statistically homogeneous direction, and OPINE’s are sharper; the CNN latent coordinates are not paired.

Figure 4. Kolmogorov flow. Left: test NMSE versus (and train NMSE of the learned models). Right: vorticity of a test snapshot and its reconstructions at ; POD blurs the vortex cores and filaments that the CNN and OPINE retain.
Figure 5. Decoded reduced coordinates for Kolmogorov flow at , each activated at one training standard deviation. POD and OPINE directions form phase-shifted pairs (: index in variance order), and OPINE’s are sharper. CNN latent coordinates have no ordering or pairing, so for each pair we show the two most correlated ones (#: latent index).
Three-Dimensional Minimal Channel Flow
Finally, we turn to wall-bounded turbulence, which is organized around elongated streaks of slow and fast fluid and the vortices that regenerate them near the wall. We reduce full three-dimensional velocity fields in a minimal channel at friction Reynolds number , as in the direct numerical simulation of Kim et al. [28]. This small periodic box, in the spirit of the minimal flow unit of Jiménez & Moin [26], holds only a few near-wall streaks and the vortices that sustain them. Each snapshot has about degrees of freedom, and the 20,000 snapshots are split chronologically, so that the test period follows the training period (Appendix § 10).
Figure 6a shows the accuracy of each method from to 2048. OPINE improves on POD at every , by up to 14%, and its advantage grows with . The CNN autoencoder follows POD at small , but from its training and test errors both level off, whereas OPINE’s error keeps decreasing, with 20 to 200 times fewer parameters than the CNN. In a wall-parallel plane Figure 6b, POD and the CNN smooth the near-wall streaks, whereas OPINE preserves more of their pattern and of the smaller-scale fluctuations.

Figure 6. Three-dimensional minimal channel. (a) Test NMSE versus retained dimension , with the train NMSE of the CNN and OPINE dashed. (b) Streamwise fluctuation in a wall-parallel plane of a test snapshot and its reconstructions at . (c) -criterion isosurfaces at a common isovalue for in another test snapshot, and (d) streamwise-fluctuation isosurfaces (: friction velocity) at in the snapshot of (b); color indicates wall-normal position.
Figure 6c,Figure 6d shows the reconstructed vortices and streaks in three dimensions. Isosurfaces of the -criterion [24], which mark vortex cores, show that POD recovers only a few smoothed vortices, while the CNN and OPINE recover larger sets. Isosurfaces of the streamwise fluctuation show the low-speed streak that dominates the box: all methods recover it, but OPINE reproduces more of its undulation and of the smaller high-speed regions near the wall.
Nested Reduced Coordinates
In the comparisons above, OPINE was trained separately at each . Because its POD basis is ordered, a single model trained at the largest also gives every smaller by keeping only the leading modes (§ 3.4; Q4). The prefixes are more accurate than POD at every on Kolmogorov flow and the minimal channel, and from on the synthetic manifolds; on the minimal channel they come within 4% of separately trained models from onward (Table 7; Appendix B). As with POD, one training run thus gives accurate models of every size. For the minimal channel, the prefixes of one model replace seven separately trained models, one for each from 32 to 2048. As in POD, the accuracy of each prefix in the transformed coordinates is also known without any reconstruction: the mean squared error of the prefix of dimension on the training data equals , with from Section 3.4 in decreasing order, so the eigenvalues show how much variance each smaller retains.
| Kolmogorov | prefix | 0.357 | 0.190 | 0.0743 | 0.0182 | 0.00152 | ||
| trained at | 0.209 | 0.108 | 0.0340 | 0.00923 | 0.00152 | |||
| POD | 0.358 | 0.195 | 0.0804 | 0.0261 | 0.00610 | |||
| Minimal channel | prefix | 0.710 | 0.634 | 0.547 | 0.460 | 0.369 | 0.280 | 0.204 |
| trained at | 0.691 | 0.608 | 0.522 | 0.440 | 0.356 | 0.277 | 0.204 | |
| POD | 0.713 | 0.641 | 0.561 | 0.480 | 0.397 | 0.314 | 0.236 |
Table 7. Test NMSE of the ordered prefixes of one OPINE model trained at the largest , of OPINE models trained at each , and of POD.
Conclusion
This paper makes three contributions. First, OPINE factors nonlinear dimensionality reduction into an invertible change of coordinates and an explicit, ordered POD projection, so that all information loss is confined to the projection and one model gives reduced coordinates of every smaller dimension. Second, its training tracks the POD subspace with a damped, aligned step of orthogonal iteration, which avoids differentiating through an eigendecomposition, provably converges locally to the exact POD subspace, and needs only minibatch products with an basis. Third, from synthetic manifolds to three-dimensional turbulence, OPINE improves on POD at every retained dimension above one, with far fewer parameters than the CNN autoencoders; on the KS and Burgers dynamics it is more than two orders of magnitude more accurate than autoencoders at the largest retained dimension, and the ablations support the choice of the POD projection. OPINE is not designed to outperform unconstrained autoencoders on every problem; its purpose is an interpretable reduction built only from an invertible change of variables and POD, and it is most beneficial when the states lie near a low-dimensional set that a linear subspace cannot capture. Its coordinates also remain interpretable: decoded one at a time, they form phase-shifted pairs like POD modes but sharper (Figure 5). The same construction, an invertible nonlinear map followed by an ordered POD projection, can be used wherever POD is used, for example for compression, modal analysis, and reduced-order modeling of other physical fields.
Limitations and future work. Training reduced dynamical models in OPINE coordinates is a natural next step beyond the reconstruction studied here; such models will need hyper-reduction because acts on the full field. The subspace could also be optimized directly on the Grassmann manifold ([13, 1]) instead of by damped orthogonal iteration. Projections constrained by symmetry, with POD performed within Fourier blocks of a homogeneous direction, are a further extension that we leave for future work. The analysis of the subspace update is local and does not cover the joint training of the map and the basis (Appendix § 8), and the flow datasets are simulations at moderate Reynolds numbers.
AI Use Statement
Generative AI tools were used to assist with editing and polishing the manuscript, including improving clarity, grammar, and presentation, and to help draw parts of the schematics. The analysis of the subspace update in Appendix A, including the statements and proofs of Theorem 1 and Proposition 2, was drafted mainly with the help of generative AI; the authors checked every step of the derivations. All AI-assisted content was reviewed and verified by the authors, who take full responsibility for the final content of the paper.
Reproducibility Statement
Section 3 specifies the model and the training algorithm, and Appendix A states the assumptions of the analysis. Appendices B and C give the architectures, optimizer settings, data generation, preprocessing, and data splits of every experiment. Code and data-generation scripts will be released upon publication.
Analysis of the Subspace Update
During training, OPINE does not recompute the POD basis exactly. After each round of gradient updates of , it moves the basis only a fraction of the way toward one step of orthogonal iteration, Eq. (7). This appendix shows that this damped update has the right target. For a fixed map with a spectral gap, it converges locally to the exact POD subspace, the only nearby fixed point, for every (Theorem 1). While changes, it trails that subspace by a bounded angle (Proposition 2). The method itself does not require the exact basis: every quantity in § 3 is defined for any orthonormal , which only makes the transformed projection error larger than its minimum. We use two textbook facts without proof: for fixed , the minimizers of Eq. (4) span a leading -dimensional eigenspace of [15, 22], and plain orthogonal iteration, the case , converges to it at rate [19 Sec. 8.2.4]. The joint, nonconvex optimization of and the basis is beyond the scope of this analysis.
Notation. We write for the basis and for of § 3.4, and for the spectral norm of a matrix. Let with orthogonal, , eigenvalues , and . For with orthonormal columns and nonsingular, write with ; then for the principal angles between and , and with is an orthonormal basis of that span [19 Sec. 6.4.3]. The update of Eq. (7) is , with and the Procrustes rotation of Eq. (6).
Theorem 1 (Convergence of the aligned, damped update). Let and . Near , the subspace depends only on , and there are constants such that implies
Consequently, for every , iterates started close enough to satisfy , where are the principal angles after updates, and , the exact POD subspace, is the only fixed point of in this neighborhood, for every .
Proof. Span only. Replacing by with orthogonal leaves unchanged, and is the orthonormal basis of that span closest to in Frobenius norm, which is unique near the fixed point, where is nonsingular. Since , the basis closest to is , and the argument of qf becomes , with the same span. We may therefore take .
Linearization. Since with , the span of is that of , so , with the orthogonal polar factor of . Hence , , and . So and . All maps involved are smooth near , which gives and .
Contraction and fixed point. . If with , then , and induction gives the rate. A fixed span in this neighborhood satisfies , hence ; conversely, because .
The alignment is necessary. Without it, may differ from by an arbitrary rotation even at the fixed point: if the columns of are eigenvectors of , an QR factorization without sign normalization may return , and then is rank deficient at .
Proposition 2 (Tracking a moving covariance). Let be the transformed covariance at round , with gap and drift , and let be the largest principal angle between after round and the leading eigenspace of . If each update contracts this angle by a factor whenever it is at most , as Theorem 1 provides
for , and if and , then for all and .
Proof. By Weyl’s inequality and the Davis–Kahan sin theorem [10], consecutive leading eigenspaces differ by an angle of at most . The largest principal angle is a metric on subspaces of equal dimension, so , which stays below by induction and unrolls to the bound.
A larger therefore tracks a moving covariance more closely, whereas a smaller changes the bottleneck, and hence the objective seen by the gradient updates, more gradually; Appendix B compares several choices.
One-Dimensional Study
The ablations of § 4.2 and the experiments of § 4.3 concern scalar fields on a periodic one-dimensional grid and use a compact training configuration, described below, that differs from that of the other experiments (Appendix § 10).
Implementation. The spatial coupling follows § 3.2: a space-to-depth rearrangement with factor 4, followed by blocks of invertible channel mixing and affine coupling, whose conditioners are three-layer circularly padded convolutional networks of width with kernel size 5 and log-scales bounded by . The spectral coupling is an exactly translation-equivariant alternative to the space-to-depth rearrangement, which breaks translation symmetry. Each spectral-split coupling layer partitions the wavenumbers into sets and , keeps the band-limited part of the field, and updates the Fourier coefficients of the other part as , where returns one real log-scale per wavenumber in from the Fourier moduli of , is a circularly padded convolutional network, and masks the wavenumbers in . Its inverse is available in closed form, and because the scales depend only on Fourier moduli and is a circular convolution, the layer commutes with grid shifts. Successive layers alternate between partitions by and by , which divide both odd and even harmonics between the two halves, whereas a partition into even and odd wavenumbers leaves one half empty for data with only odd harmonics. Unless stated otherwise, and . The bottleneck is initialized with the exact POD of the transformed training data and updated after each of 20 rounds of 150 AdamW steps (learning rate , cosine schedule, gradient clipping at 1) with , and the checkpoint with the lowest validation NMSE is reported. The CNN autoencoder has four stride-2 circularly padded convolutions and a mirrored decoder, and the MLP autoencoder has two hidden layers of width 256 in the encoder and in the decoder. Both are trained for the same 3000 steps. This study uses three seeds.
Datasets. The four synthetic families, defined in § 4.2, have 512 training, 256 validation, and 2000 test samples each, with all parameters drawn uniformly. The states of Pulse form a circle, those of Torus a torus, and those of Pulse width and Front a cylinder. KS data are generated with ETDRK4 [27] on 64 points with time step 0.25, and Burgers data with ETDRK4 on 256 points, viscosity 0.07, forcing of amplitude 0.5 that is white in time and acts on wavenumbers 1 to 3, and storage on 128 points. Each system provides eight independent trajectories of 1000 snapshots (spacing 1 for KS and 0.5 for Burgers, after a transient), split by trajectory into six for training, one for validation, and one for testing, and normalized by the training standard deviation.
Full results. Table 2 reports all retained dimensions. Three patterns stand out. First, OPINE improves on POD at every above one, with error reductions of more than three orders of magnitude for the spatial coupling and four for the spectral coupling. At , no method can be exact, because each synthetic dataset contains a closed loop, which cannot be mapped continuously and injectively onto a line. OPINE, whose encoder is an invertible map followed by a linear projection, improves on POD only modestly there, whereas the autoencoders reach errors of two to seven percent. Second, for Torus, the errors of OPINE and the autoencoders all drop sharply at , although a torus can be embedded in three dimensions. Third, the CNN autoencoder is more accurate on Pulse and Pulse width, which contain a single traveling pulse, whereas OPINE is more accurate than both autoencoders on Front from and at least as accurate as them on KS and Burgers from onward. The spectral coupling is exactly translation-equivariant but less accurate than the spatial coupling on KS and Burgers.
| Dataset | POD | CNN | MLP | OPINE | OPINE-spectral | |
| Torus | 1 | |||||
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| 6 | ||||||
| Pulse | 1 | |||||
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| 6 | ||||||
| Pulse width | 1 | |||||
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| 6 | ||||||
| Front | 1 | |||||
| 2 | ||||||
| 3 | ||||||
| 4 | ||||||
| 6 | ||||||
| KS | 2 | |||||
| 4 | ||||||
| 6 | ||||||
| 8 | ||||||
| 12 | ||||||
| 16 | ||||||
| Burgers | 4 | |||||
| 8 | ||||||
| 16 | ||||||
| 32 |
Table 2. Median test NMSE of the one-dimensional study. OPINE uses the spatial coupling, OPINE-spectral the spectral-split coupling described above. Best in bold; “–” marks runs not completed.
Bottleneck ablation. Table 3 retrains the same spatial invertible map with each bottleneck, keeping the training procedure fixed. Keeping the first entries of the flattened transformed field, the truncation used by nested-dropout flows, fails on every dataset. A fixed Fourier basis is competitive only when the retained Fourier modes happen to suffice, and is up to four orders of magnitude less accurate than OPINE. Figure 2 shows a typical test profile of each dataset and its reconstructions at .
| Dataset | first entries | fixed Fourier basis | OPINE | |
| Torus | 3 | |||
| Torus | 4 | |||
| Pulse | 2 | |||
| Pulse | 4 | |||
| Pulse width | 2 | |||
| Pulse width | 4 | |||
| Front | 2 | |||
| Front | 4 |
Table 3. Bottleneck ablation with the same spatial invertible map: median test NMSE.
Subspace-update ablation. Table 4 compares the update of Section 3.4 with a frozen subspace, weaker damping, exact recomputation of the POD after every round, and differentiation through an eigendecomposition of the transformed covariance at every step. Adapting the subspace matters on Pulse, where freezing it increases the error by a factor of about 3 relative to the default and 11 relative to the best tracking variant, and matters little on Torus, consistent with Appendix A: the damping changes the path of training rather than its target. Among the tracking variants, the weaker damping is the most accurate, so the default is conservative. Differentiating through an eigendecomposition of the full transformed covariance at every step is the most accurate option where it runs, but it is fragile and does not scale: with a GPU eigensolver, all six runs failed because the solver did not converge on the nearly repeated eigenvalues of the transformed covariance, and the results shown were obtained with a CPU eigensolver at several times the cost per step. Each step also requires the covariance of the entire training set, which is impractical at the scale of the minimal channel (). The tracking scheme avoids both problems.
| Dataset | frozen | exact | eigendecomposition | |||
| Torus | 3 | |||||
| Pulse | 2 |
Table 4. Subspace-update ablation with the spatial invertible map: median test NMSE. The eigendecomposition variant was run with a CPU eigensolver; with a GPU eigensolver, all six runs failed.
Size and cost of the coupling stack. Table 5 varies the number of blocks and the conditioner width . A compact spatial stack with and comes within a factor of 1.4 of a stack with and , the depth and width used for the turbulent flows, with fewer parameters, and remains within a factor of 2. Measured on a single CPU thread under identical conditions for fields of points, a training step of the spatial stack costs 13, 27, and 43 ms for , , and , and one of the spectral stack 22, 49, and 103 ms, about twice as much, because of its Fourier transforms and per-wavenumber scale network.
| Dataset | coupling | parameters | test NMSE | ||
| Torus () | spatial | 2 | 32 | 12,328 | |
| Torus () | spatial | 4 | 32 | 24,656 | |
| Torus () | spatial | 8 | 64 | 180,384 | |
| Torus () | spectral | 2 | 32 | 19,524 | |
| Torus () | spectral | 4 | 32 | 39,048 | |
| KS () | spatial | 4 | 32 | 24,656 | |
| KS () | spatial | 8 | 64 | 180,384 | |
| KS () | spectral | 4 | 32 | 30,728 |
Table 5. Coupling-stack size: parameters and median test NMSE.
Nested prefixes. Table 6 truncates a single model trained at to its ordered prefixes (§ 3.4). As on the turbulent flows (§ 4.6), the errors grow monotonically as the prefixes shorten and remain below POD from onward, but do not match models trained at each .
Experimental Details
This appendix gives the training settings, the synthetic problem, the additional ablations and results, the baselines, and the datasets of the experiments outside the one-dimensional study.
| Torus, prefix | ||||||
| trained at | – | |||||
| Pulse width, prefix | ||||||
| trained at | – |
Table 6. Ordered prefixes of one model trained at versus models trained at each : median test NMSE (no model was trained at ). The two rows come from separate training runs.
OPINE training. Outside the one-dimensional study of Appendix § 9, every OPINE model is trained at a single prescribed retained dimension. The coupling stack uses affine coupling blocks with conditioner width and is optimized with AdamW [38] at learning rate , with a cosine-annealed schedule and gradient-norm clipping at 1. Training runs for 25 rounds, each consisting of one epoch of stochastic optimization followed by one damped subspace update (§ 3.4) with . The log-scales of the coupling layers are bounded by . For the minimal channel, the basis is initialized with 12 steps of orthogonal iteration (Eq. (6)); for Kolmogorov flow, it is initialized from the exact POD basis of the training data. The product in Eq. (6) is accumulated over minibatches of centered transformed states, so is never formed; for Kolmogorov flow it uses a random subset of 20,000 training states. The batch size is 16 for the minimal channel and 64 for Kolmogorov flow. Training is performed in single precision, and invertibility is additionally checked in double precision. We report the checkpoint with the lowest validation NMSE. All turbulent-flow models use a single seed. OPINE has 61,760 and 96,192 parameters for Kolmogorov flow and the minimal channel, and the CNN autoencoders have 1.3–17.6 and 2.5–19.1 million parameters, increasing with .
Synthetic problem. For each of ten seeds, we sample 32 training, 256 validation, and 2000 test phases. OPINE is trained for 30 rounds with learning rate and batch size 256, and the CNN autoencoder ( parameters) for 400 epochs. The test NMSE at is for POD and for OPINE (mean and standard deviation over seeds); the ratio for individual seeds ranges from 11 upward. The CNN autoencoder reaches a training NMSE of , but its test NMSE ranges from to across seeds, with median .
Nested prefixes of the turbulent-flow models. Table 7 compares the ordered prefixes of one model trained at the largest with models trained at each and with POD (§ 4.6).
Choice of the invertible map. Besides affine coupling layers, we tested invertible residual networks [2, 8] and monotone elementwise maps built from piecewise-quadratic and spline functions [23, 44, 12]. Invertible residual networks were less expressive at a comparable number of parameters, because the Lipschitz constraint that guarantees invertibility limits each block, and their inverse requires a fixed-point iteration, which made the evaluation of slow. Monotone maps needed deep stacks to couple distant grid points, and these deep stacks suffered from vanishing gradients. Affine coupling layers are inverted in closed form and gave the most accurate models among the maps we tried.
CNN baseline training. The convolutional autoencoders are optimized with AdamW at learning rate with a cosine-annealed schedule, and the reported checkpoint is again selected by validation NMSE alone. The minimal channel uses 50 epochs with batch size 32, except that the run at stopped early at epoch 30 when its validation error stalled; Kolmogorov flow uses 30 epochs with batch size 64. Padding follows the boundary conditions of each dataset: circular in the spanwise direction and zero in the wall-normal direction for the minimal channel, and circular in both directions for Kolmogorov flow.
Datasets. The Kolmogorov dataset is generated with the pseudospectral JAX-CFD solver [31] on the domain , using 2/3 dealiasing and combined Crank–Nicolson/Runge–Kutta time stepping; we verified that the trajectories are spectrally resolved and statistically stationary after burn-in. After a burn-in of 100 time units, each of the 100 trajectories provides 1000 snapshots at unit time intervals, and the trajectories are split 80/10/10. The three-dimensional minimal channel spans in the streamwise, spanwise, and wall-normal directions, with the channel half-height, a no-slip floor, and a stress-free top, on a grid, and uses 20,000 snapshots of the three velocity components at spacing with the chronological 16,000/2,000/2,000 split of § 4.5. The minimal-channel data are preprocessed as follows: for each velocity component and each wall-normal level, the training-set mean profile is subtracted and the result is divided by the training-set RMS profile.
References
- [1]P.-A. Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008.DOI
- [2]Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning, pp. 573–582, 2019.arxiv.org/abs/1811.00995
- [3]Artur Bekasov and Iain Murray. Ordering dimensions with nested dropout normalizing flows. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2020.arxiv.org/abs/2006.08777
- [4]Gal Berkooz, Philip Holmes, and John L. Lumley. The proper orthogonal decomposition in the analysis of turbulent flows. Annual Review of Fluid Mechanics, 25:539–575, 1993. doi: 10.1146/annurev.fl.25.010193.002543.DOI
- [5]Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2003.13913
- [6]Anthony L. Caterini, Gabriel Loaiza-Ganem, Geoff Pleiss, and John P. Cunningham. Rectangular flows for manifold learning. In Advances in Neural Information Processing Systems, volume 34, 2021.arxiv.org/abs/2106.01413
- [7]Kathleen Champion, Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116 (45):22445–22451, 2019.DOI
- [8]Ricky T. Q. Chen, Jens Behrmann, David K. Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1906.02735
- [9]Edmond Cunningham, Adam D. Cobb, and Susmit Jha. Principal component flows. In International Conference on Machine Learning, 2022.
- [10]Chandler Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. III. SIAM Journal on Numerical Analysis, 7(1):1–46, 1970.
- [11]Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In International Conference on Learning Representations, 2017.arxiv.org/abs/1605.08803
- [12]Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1906.04032
- [13]Alan Edelman, Tomás A. Arias, and Steven T. Smith. The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2):303–353, 1998.arxiv.org/abs/physics/9806030
- [14]Hamidreza Eivazi, Soledad Le Clainche, Sergio Hoyas, and Ricardo Vinuesa. Towards extraction of orthogonal and parsimonious non-linear modes from turbulent flows. Expert Systems with Applications, 202:117038, 2022.arxiv.org/abs/2109.01514
- [15]Ky Fan. On a theorem of Weyl concerning eigenvalues of linear transformations I. Proceedings of the National Academy of Sciences, 35(11):652–655, 1949.pubmed.ncbi.nlm.nih.gov/16578320
- [16]Stefania Fresca and Andrea Manzoni. POD-DL-ROM: Enhancing deep learning-based reduced order models for nonlinear parametrized PDEs by proper orthogonal decomposition. Computer Methods in Applied Mechanics and Engineering, 388:114181, 2022. doi: 10.1016/j.cma.2021.114181.DOI
- [17]Kai Fukami, Taichi Nakamura, and Koji Fukagata. Convolutional neural network based hierarchical autoencoder for nonlinear mode decomposition of fluid field data. Physics of Fluids, 32(9):095110, 2020. doi: 10.1063/5.0020721.DOI
- [18]Rudy Geelen, Stephen Wright, and Karen Willcox. Operator inference for non-intrusive model reduction with quadratic manifolds. Computer Methods in Applied Mechanics and Engineering, 403:115717, 2023.DOI
- [19]Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, MD, 4th edition, 2013.
- [20]Constantin Greif and Karsten Urban. Decay of the Kolmogorov η-width for wave problems. Applied Mathematics Letters, 96:216–222, 2019. doi: 10.1016/j.aml.2019.05.013.DOI
- [21]Geoffrey E. Hinton and Ruslan R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.pubmed.ncbi.nlm.nih.gov/16873662
- [22]Philip Holmes, John L. Lumley, Gal Berkooz, and Clarence W. Rowley. Turbulence, Coherent Structures, Dynamical Systems and Symmetry. Cambridge University Press, 2nd edition, 2012.
- [23]Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078–2087, 2018.arxiv.org/abs/1804.00779
- [24]Julian C. R. Hunt, Alan A. Wray, and Parviz Moin. Eddies, streams, and convergence zones in turbulent flows. In Proceedings of the Summer Program, Center for Turbulence Research, pp. 193–208, 1988. CTR-S88.
- [25]Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix backpropagation for deep networks with structured layers. In IEEE International Conference on Computer Vision, 2015.DOI
- [26]Javier Jiménez and Parviz Moin. The minimal flow unit in near-wall turbulence. Journal of Fluid Mechanics, 225:213–240, 1991. doi: 10.1017/S0022112091002033.DOI
- [27]Aly-Khan Kassam and Lloyd N. Trefethen. Fourth-order time-stepping for stiff PDEs. SIAM Journal on Scientific Computing, 26(4):1214–1233, 2005.DOI
- [28]John Kim, Parviz Moin, and Robert Moser. Turbulence statistics in fully developed channel flow at low reynolds number. Journal of Fluid Mechanics, 177:133–166, 1987. doi: 10.1017/S0022112087000892.DOI
- [29]Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, volume 31, 2018.arxiv.org/abs/1807.03039
- [30]Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.arxiv.org/abs/1312.6114
- [31]Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine learning-accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21):e2101784118, 2021. doi: 10.1073/pnas.2101784118.DOI
- [32]Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanić. Trumpets: Injective flows for inference and inverse problems. In Conference on Uncertainty in Artificial Intelligence, 2021.arxiv.org/abs/2102.10461
- [33]Mark A. Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChe Journal, 37(2):233–243, 1991. doi: 10.1002/aic.690370209.DOI
- [34]Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2205.13147
- [35]Kookjin Lee and Kevin T. Carlberg. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973, 2020. doi: 10.1016/j.jcp.2019.108973.
- [36]Shen Li and Bryan Hooï. Neural PCA for flow-based representation learning. In International Joint Conference on Artificial Intelligence, pp. 3229–3235, 2022. doi: 10.24963/ijcai.2022/448.
- [37]Alec J. Linot and Michael D. Graham. Deep learning to discover and predict dynamics on an inertial manifold. Physical Review E, 101:062209, 2020.DOI
- [38]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
- [39]John L. Lumley. Stochastic Tools in Turbulence. Academic Press, 1970.
- [40]Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9:4950, 2018.DOI
- [41]Romit Maulik, Bethany Lusch, and Prasanna Balaprakash. Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders. Physics of Fluids, 33(3):037106, 2021.DOI
- [42]Sebastian Mika, Bernhard Schölkopf, Alex Smola, Klaus-Robert Müller, Matthias Scholz, and Gunnar Rätsch. Kernel PCA and de-noising in feature spaces. In Advances in Neural Information Processing Systems, volume 11, 1999.
- [43]Michele Milano and Petros Koumoutsakos. Neural network modeling for near wall turbulent flow. Journal of Computational Physics, 182(1):1–26, 2002.DOI
- [44]Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural importance sampling. ACM Transactions on Graphics, 38(5):145:1–145:19, 2019.
- [45]Takaaki Murata, Kai Fukami, and Koji Fukagata. Nonlinear mode decomposition with convolutional neural networks for fluid dynamics. Journal of Fluid Mechanics, 882:A13, 2020. doi: 10.1017/jfm.2019.822.DOI
- [46]Erkki Oja. A simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15(3):267–273, 1982. doi: 10.1007/BF00275687.
- [47]Julius Reiss, Philipp Schulze, Jörn Sesterhenn, and Volker Mehrmann. The shifted proper orthogonal decomposition: A mode decomposition for multiple transport phenomena. SIAM Journal on Scientific Computing, 40(3):A1322–A1344, 2018.arxiv.org/abs/1512.01985
- [48]Oren Rippel, Michael A. Gelbart, and Ryan P. Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pp. 1746–1754, 2014.arxiv.org/abs/1402.0915
- [49]Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5):1299–1319, 1998.DOI
- [50]Tommaso Taddei. A registration method for model order reduction: Data compression and geometry reduction. SIAM Journal on Scientific Computing, 42(2):A997–A1027, 2020.arxiv.org/abs/1906.11008
- [51]Kunihiko Taira, Steven L. Brunton, Scott T. M. Dawson, Clarence W. Rowley, Tim Colonius, Beverley J. McKeon, Oliver T. Schmidt, Stanislav Gordeyev, Vassilios Theofilis, and Lawrence S. Ukeiley. Modal analysis of fluid flows: An overview. AIAA Journal, 55(12):4013–4041, 2017.arxiv.org/abs/1702.01453
- [52]Wei Wang, Zheng Dang, Yinlin Hu, Pascal Fua, and Mathieu Salzmann. Backpropagation-friendly eigendecomposition. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1906.09023
- [53]Bin Yang. Projection approximation subspace tracking. IEEE Transactions on Signal Processing, 43(1):95–107, 1995. doi: 10.1109/78.365290.DOI




