betaFeedback

WorksAgricultural and Biological Sciences

Self-Supervised Learning of Local Predictive Directions as a Neural Computation

Abstract

Natural sensory dynamics are globally nonlinear but often exhibit simple local geometry, with nearby trajectories evolving preferentially along a small number of predictive directions. We ask how neural circuits could discover this geometry directly from temporal experience. We introduce Local Predictive Directions (LPD), a self-supervised framework that learns local directions whose projections maximally predict near-future trajectory evolution. LPD applies past-future canonical correlation analysis to temporal differences, making the learned representation invariant to translations of the underlying state while avoiding estimation of equilibria or a global dynamical model. To represent systems with multiple predictive regimes, we introduce a non-negative similarity-matching objective, with offline and online algorithms, that partitions trajectory segments into predictive domains, each associated with a distinct local predictive direction. The online algorithm admits a natural neural-network interpretation: the resulting activities provide soft domain assignments, while synaptic weights encode the corresponding local predictive filters. On stochastic nonlinear dynamical systems, LPD recovers the local predictive geometry and partitions trajectories into distinct dynamical regimes. On naturalistic sensory streams, the online algorithm produces center-surround spatial receptive fields and ON/OFF sustained and transient temporal filters resembling those of early visual neurons. Our results suggest that predictive learning can provide a self-supervised mechanism by which neural circuits discover the local geometry of environmental dynamics and represent it in receptive fields.

© 2026 M. Zenari*, X. Ding*, D. Burnham, A.R. Ellis-Mohr, A.M. Sengupta & D.B. Chklovskii.

Introduction

High-dimensional time series from physical, biological, and neural systems often evolve on structured nonlinear manifolds. Global dynamics may be complex, yet short-time evolution is typically low-dimensional: within a small neighborhood, state changes are captured by a few tangent directions — the local-linear view underlying classical nonlinear dynamical systems theory (Strogatz, [52]), Koopman and dynamic mode decomposition (Schmid, [49]; Brunton et al., [8]), and local models of nonlinear time series (Farmer and Sidorowich, [14]; Casdagli, 1991). Some tangent directions are more predictive than others: perturbations along them set the system’s evolution over the next few steps, while orthogonal ones decay, fluctuate, or say little about the future. This suggests a self-supervised problem: from an unlabeled trajectory alone, can one discover tangent directions that vary slowly along it and whose projections predict future evolution?

Near a saddle point, for instance, unstable manifolds organize future trajectories: projecting the state, relative to the saddle, onto an unstable direction determines which branch is followed (Strogatz, [52]). Such saddle-organized dynamics are central to stable heteroclinic channels, winnerless competition, and transient neural dynamics (Rabinovich et al., [42, 43, 44]; Cvitanović and Lippolis, [13]). Away from fixed points, short trajectory segments still sample a local linearization, so the predictive objects are not global coordinates but local tangent-space directions varying across state space.

Existing approaches fall into three families (details in Appendix § 6). Switching linear dynamical systems and spectral HMM methods (Ackerson and Fu, [1]; Fox et al., [15]; Linderman et al., [28]; Hsu et al., [19]) require explicit inference over latent discrete states under a generative model. Koopman operator theory, dynamic mode decomposition and VAMP (Brunton et al., [8]; Schmid, [49]; Wu and Noé, [56]), and their deep variants (Lusch et al., [30]; Otto and Rowley, [33]; Takeishi et al., [53]), seek a globally linearizing feature space whose leading modes span broad regions of state space and may not resolve local saddle-to-saddle geometry. Deep predictive objectives (Mardt et al., [31]; Oord et al., [32]; Zerveas et al., [57]) learn global embeddings, but are heavily parameterized and less interpretable.

Past–future correlation offers a principled definition of such predictive directions. Canonical correlation analysis (CCA) finds projections of paired variables with maximal correlation (Hotelling, [18]); related past–future covariance methods underlie subspace system identification and balanced stochastic realization (Arun and Kung, [3]; Katayama, [24]), and connect to predictive information and the Gaussian information bottleneck (Bialek et al., [6]; Chechik et al., [11]). Applied to past and future observations, it finds projections of the present maximally informative about subsequent evolution, recovering dominant predictive modes in locally linear dynamics. This motivates the Rectified Spectral Unit (ReSU), a biologically inspired self-supervised rule combining past–future CCA with rectification to learn predictive features from dynamical data (Pughe-Sanford et al., [40]; Qin et al., [41]). Existing CCA- and ReSU-based methods are nonetheless global or single-regime: they assume one predictive structure dominates, or that local neighborhoods are supplied externally, and so do not discover multiple local predictive directions whose locations and domains are unknown.

We introduce a self-supervised objective and algorithm for learning such local predictive directions (LPDs) directly from trajectories, Figure 1. Temporal differences give access to local tangent-space information: applying past–future CCA to finite differences rather than absolute states makes the method invariant to translations of the local coordinate system, such as a saddle’s unknown location, so predictive directions are learned from trajectory increments without estimating equilibria or fitting a global dynamical model. To discover multiple local predictive regimes, we replace canonical projection followed by rectification with a non-negative similarity-matching formulation. Similarity matching provides a normative route from similarity-preserving objectives to online neural learning algorithms [36, 35, 38, 29], while non-negativity promotes localized, parts-based, and clustering-like representations [27, 34, 50]. Here, similarity matching preserves the predictive correlation structure of the data while non-negativity induces competition and locality, Figure 1b; output units therefore specialize to different regions of state space, learning distinct local predictive directions, Figure 1c, and their non-negative activities act as soft domain assignments, Figure 1d. Unlike clustering by geometric proximity or variance [47], the proposed objective clusters trajectory segments by short-time predictive structure. The resulting algorithm admits offline and online implementations, the online one having a natural neural interpretation in terms of feedforward weights, lateral competition, and rectified activity.

Overview of the LPD algorithm

Figure 1. Overview of the LPD algorithm. (a) Clusters of slowly varying flow directions in the tangent space of a nonlinear dynamical system (colors), with a sample trajectory in black. (b) Similarity of temporal differences (left) and of the non-negative outputs (right) across time. (c) Online implementation as a self-supervised neural network, with feedforward weights W\mathbf{W} and lateral weights M\mathbf{M} (see Online Algorithm and Neural Interpretation, § 2). (d) Soft domain assignments over time for the trajectory in (a).

We validate LPD on stochastic nonlinear systems with known local structure and on sensory streams from translated models of natural images, where the learned temporal filters resemble those of biological neurons and pooling across pixels yields localized filters that tile space like biological receptive fields. These results extend previous predictive interpretations of neural computation [51, 46, 10, 45].

Learning Local Predictive Directions from Trajectory Data

Local Predictive Structure in Tangent Space. Our goal is to identify local directions in state space whose projections are maximally informative about near-future evolution of a stochastic discrete-time dynamical system, xt+ζ=f(xt)+ωtx_{t+\zeta} = f(x_t) + \omega_t, where xt∈Rnx_t \in\mathbb{R}^n and ωt\omega_t denotes process noise. Over a sufficiently short trajectory segment, the dynamics can be approximated locally by an affine map. Around a reference point x∗x_* in state space,

xt+ζ−f(x∗)≈Ax∗(xt−x∗)+ωt,Ax∗:=∂f∂x∣x∗.(1)x_{t+\zeta} - f(x_*) \approx A_{x_*}(x_t - x_*) + \omega_t,\qquad A_{x_*} := \left.\frac{\partial f}{\partial x}\right|_{x_*}. \tag*{(1)}

Locally predictive directions in tangent space determine the subsequent displacement of the trajectory and can be viewed as low-rank components of the local linear map Ax∗A_{x_*}. In nonlinear systems, these directions generically vary across state space. A trajectory may pass through multiple regions in which different tangent directions are predictive (Figure 1a). The learning problem is therefore twofold: to discover a finite set of local predictive directions, and to assign trajectory segments to the regions in which each direction is active. A direct approach based on Eq. (1), however, requires representing states relative to the appropriate local reference point x∗x_*, which is generally unknown from data.

Translation-Invariant Predictive Learning from Temporal Differences. To remove the dependence of Eq. (1) on unknown local reference points, we represent local dynamics using temporal differences. For a fixed lag τ\tau, define δxt:=xt+τ−xt\delta x_t := x_{t+\tau} - x_t. If the dynamics are approximately affine over the corresponding local trajectory segment, then

δxt+ζ=xt+ζ+τ−xt+ζ≈Ax∗(xt+τ−xt)+(ωt+τ−ωt)=Ax∗δxt+ηt,(2)\delta x_{t+\zeta} = x_{t+\zeta+\tau} - x_{t+\zeta} \approx A_{x_*}(x_{t+\tau} - x_t) + (\omega_{t+\tau} - \omega_t) = A_{x_*}\delta x_t + \eta_t, \tag*{(2)}

where ηt:=ωt+τ−ωt\eta_t := \omega_{t+\tau} - \omega_t is an effective differenced-noise term. Importantly, the affine offset cancels. Thus, temporal differences preserve the local linear action of Ax∗A_{x_*} on tangent displacements while removing the dependence on unknown translations of the local coordinate system. This inspires us to define past and future descriptors from tangent space rather than absolute states, pt:=δxtp_t := \delta x_t and ft:=δxt+ζf_t := \delta x_{t+\zeta}, which is translation invariant, and in fact invariant to any invertible affine change of coordinates (Appendix § 9).

Predictive Directions via Past–Future Correlation. Global predictive directions can be found via past–future canonical correlation analysis (CCA) ([3, 11]). Here, to find local predictive directions we apply CCA to the temporal differences. Specifically, we seek directions viv_i and uiu_i that maximize the correlation between linear projections of ptp_t and ftf_t:

u1,v1=arg⁡max⁡u,v E[(vTpt)(uTft)]s.t.E[(vTpt)2]=E[(uTft)2]=1,(3)u_1, v_1 = \underset{u,v}{\arg\max}\ \mathbb{E}\left[(v^\mathsf{T}p_t)(u^\mathsf{T}f_t)\right]\qquad\text{s.t.}\qquad\mathbb{E}\left[(v^\mathsf{T}p_t)^2\right] = \mathbb{E}\left[(u^\mathsf{T}f_t)^2\right] = 1, \tag*{(3)}

with additional orthogonality constraints for subsequent components. Define the covariance matrices Cpp:=E[ptptT]C_{pp} := \mathbb{E}[p_t p_t^\mathsf{T}], Cff:=E[ftftT]C_{ff} := \mathbb{E}[f_t f_t^\mathsf{T}], Cpf:=E[ptftT]C_{pf} := \mathbb{E}[p_t f_t^\mathsf{T}] and Cfp:=E[ftptT]C_{fp} := \mathbb{E}[f_t p_t^\mathsf{T}]. The CCA directions solve

Cpp−1CpfCff−1Cfpvi=ρi2vi,Cff−1CfpCpp−1Cpfui=ρi2ui,(4)C_{pp}^{-1}C_{pf}C_{ff}^{-1}C_{fp}v_i = \rho_i^2v_i,\qquad C_{ff}^{-1}C_{fp}C_{pp}^{-1}C_{pf}u_i = \rho_i^2u_i, \tag*{(4)}

where ρi\rho_i is the iith canonical correlation coefficient. When the local dynamics are dominated by a single expanding or otherwise predictive mode, the leading canonical direction captures the strongest evolving tangent-space direction (Appendix § 7). Near a saddle dominated by one unstable mode, the leading past filter extracts the unstable coordinate (the corresponding left eigenvector), while the associated future mode aligns with the unstable tangent direction (Pughe-Sanford et al., [40]).

CCA as a Similarity-Matching Objective. To construct an online algorithm with a natural neural interpretation, we now turn the predictive-correlation principle into a self-supervised learning objective. Let P:=[p1,…,pT]∈Rn×T\mathbf{P} := [\mathbf{p}_1,\ldots,\mathbf{p}_T] \in\mathbb{R}^{n \times T} and F:=[f1,…,fT]∈Rn×T\mathbf{F} := [\mathbf{f}_1,\ldots,\mathbf{f}_T] \in\mathbb{R}^{n \times T} denote matrices of past and future increment descriptors. For the top kk canonical components, define Zp:=V⊤P\mathbf{Z}_p := \mathbf{V}^{\top}\mathbf{P} and Zf:=U⊤F\mathbf{Z}_f := \mathbf{U}^{\top}\mathbf{F}, where V=[v1,…,vk]\mathbf{V} = [\mathbf{v}_1,\ldots,\mathbf{v}_k] and U=[u1,…,uk]\mathbf{U} = [\mathbf{u}_1,\ldots,\mathbf{u}_k] contain the CCA directions. Define the predictive similarity matrix

G:=P⊤Cpp−1CpfCff−1F=Zp⊤RZf,(5)\mathbf{G} := \mathbf{P}^{\top}\mathbf{C}_{pp}^{-1}\mathbf{C}_{pf}\mathbf{C}_{ff}^{-1}\mathbf{F} = \mathbf{Z}_p^{\top}\mathbf{R}\mathbf{Z}_f, \tag*{(5)}

where Rij=ρiδij\mathbf{R}_{ij} = \rho_i\delta_{ij} is the diagonal matrix of canonical correlations and the second equality is derived in Appendix § 8 for the full-rank case k=nk = n, becoming a rank-kk approximation for k<nk < n. This matrix measures how strongly each past sample predicts each future sample after whitening the past and future representations (Appendix § 12). G\mathbf{G} requires only sample covariances of the increments; the Jacobian Ax∗\mathbf{A}_{\mathbf{x}_*} is never estimated. For the unconstrained objective ((6)) below, the leading canonical coordinates span the dominant singular subspace of this predictive similarity matrix (Appendix § 11). Thus, rather than solving the CCA eigenproblem explicitly, we can recover the predictive representation by solving the following similarity-matching problem over trajectory segments:

Z∗=arg⁡min⁡Z∈Rk×T∥GG⊤−Z⊤Z∥F2.(6)\mathbf{Z}^* = \underset{\mathbf{Z}\in\mathbb{R}^{k\times T}}{\arg\min}\left\|\mathbf{G}\mathbf{G}^{\top}-\mathbf{Z}^{\top}\mathbf{Z}\right\|_F^2. \tag*{(6)}

The minimizer Z∗\mathbf{Z}^* spans the same canonical subspace as Zp\mathbf{Z}_p, up to an orthogonal transformation; its component scaling is set by the spectrum of GG⊤\mathbf{G}\mathbf{G}^{\top} and differs from the conventional CCA normalization. Intuitively, two time points should have similar output representations if their past increments predict future increments in similar ways. This similarity matching formulation of CCA is different from previous ones ([38, 29]) in that it allows a causal online implementation (Algorithm 1) that computes the output from the past input alone while using future input for learning only.

Learning Multiple Local Predictive Directions with Non-Negative Similarity Matching. The preceding objective ((6)) learns a global predictive subspace. However, nonlinear dynamics typically contain multiple local predictive regimes: different regions of state space can have different tangent directions that predict future motion (Figure 1a). A single unconstrained CCA subspace can therefore mix together distinct local dynamical structures. To separate these regimes, we impose elementwise non-negativity on the output representation:

min⁡Z≥0∥GG⊤−Z⊤Z∥F2.(7)\underset{\mathbf{Z}\geq0}{\min}\left\|\mathbf{G}\mathbf{G}^{\top}-\mathbf{Z}^{\top}\mathbf{Z}\right\|_F^2. \tag*{(7)}

Non-negativity changes the interpretation of the output ([34, 37, 50]). Instead of signed canonical coordinates, the columns of Z\mathbf{Z} become soft domain assignments: {Z}it\{\mathbf{Z}\}_{it} indicates the degree to which trajectory segment tt belongs to predictive regime ii (Figure 1d). The similarity-matching term strives to preserve past–future predictive structure, while non-negativity and lateral competition encourage different units to specialize to different local regions of state space. Unlike clustering in state space (Pehlevan and Chklovskii, 2014; Pehlevan et al., 2017; Sengupta Algorithm 1: Learning Local Predictive Directions Online. C+:=(C+ϵI)−1C^{+} := (C+\epsilon I)^{-1} denotes a Tikhonov-regularized inverse and [⋅]+[\cdot]_{+} elementwise rectification.

Initialize W∈Rk×n\mathbf{W} \in\mathbb{R}^{k \times n} and M∈Rk×k\mathbf{M} \in\mathbb{R}^{k \times k} with M≻0\mathbf{M} \succ0; Initialize covariance estimates Cpp,Cff,Cpf,Cfp,Czp,Czz\mathbf{C}_{pp}, \mathbf{C}_{ff}, \mathbf{C}_{pf}, \mathbf{C}_{fp}, \mathbf{C}_{zp}, \mathbf{C}_{zz}; for t←1t \leftarrow1 to TT do &nbsp;&nbsp;&nbsp;&nbsp;Receive increment descriptors (pt,ft)(\mathbf{p}_t,\mathbf{f}_t); &nbsp;&nbsp;&nbsp;&nbsp;Update Cpp,Cff,Cpf,Cfp\mathbf{C}_{pp}, \mathbf{C}_{ff}, \mathbf{C}_{pf}, \mathbf{C}_{fp}; &nbsp;&nbsp;&nbsp;&nbsp;Inference: update zt\mathbf{z}_t for fixed W,M\mathbf{W},\mathbf{M}; &nbsp;&nbsp;&nbsp;&nbsp;repeat &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;zt←[zt+ηZ(Wpt−Mzt)]+\mathbf{z}_t \leftarrow[\mathbf{z}_t + \eta_Z(\mathbf{W}\mathbf{p}_t - \mathbf{M}\mathbf{z}_t)]_{+}; &nbsp;&nbsp;&nbsp;&nbsp;until inference convergence; &nbsp;&nbsp;&nbsp;&nbsp;Update Czp\mathbf{C}_{zp} and Czz\mathbf{C}_{zz}; &nbsp;&nbsp;&nbsp;&nbsp;Update W←CzpCpp+CpfCff+CfpCpp+\mathbf{W} \leftarrow\mathbf{C}_{zp}\mathbf{C}_{pp}^{+}\mathbf{C}_{pf}\mathbf{C}_{ff}^{+}\mathbf{C}_{fp}\mathbf{C}_{pp}^{+}; &nbsp;&nbsp;&nbsp;&nbsp;Update M←Czz\mathbf{M} \leftarrow\mathbf{C}_{zz}; end

et al., 2018), it groups trajectory segments whose past increments predict their future increments in similar ways, discovering domains with shared local predictive geometry. Applying the Legendre transform, we derive from Eq. (7) the following equivalent min–max problem over auxiliary variables N∈Rk×T\mathbf{N} \in\mathbb{R}^{k \times T} and M∈Rk×k\mathbf{M} \in\mathbb{R}^{k \times k}, whose optima are N∗=ZG\mathbf{N}^{*} = \mathbf{Z}\mathbf{G} and M∗=ZZT\mathbf{M}^{*} = \mathbf{Z}\mathbf{Z}^{\mathsf{T}} (Appendix E).

min⁡Z≥0min⁡Nmax⁡MTr⁡[−4ZTNGT+2ZTMZ+2NTN−MTM].(8)\min_{\mathbf{Z} \ge0}\min_{\mathbf{N}}\max_{\mathbf{M}}\operatorname{Tr}\left[-4\mathbf{Z}^{\mathsf{T}}\mathbf{N}\mathbf{G}^{\mathsf{T}}+2\mathbf{Z}^{\mathsf{T}}\mathbf{M}\mathbf{Z}+2\mathbf{N}^{\mathsf{T}}\mathbf{N}-\mathbf{M}^{\mathsf{T}}\mathbf{M}\right]. \tag*{(8)}

Optimization of Eq. (8) using gradient descent alternates an inference step for Z\mathbf{Z} at fixed W,M\mathbf{W},\mathbf{M} with updates of W\mathbf{W} and M\mathbf{M} from empirical covariances, where the weight matrix W\mathbf{W} is defined by WP=NGT\mathbf{W}\mathbf{P}=\mathbf{N}\mathbf{G}^{\mathsf{T}}; the resulting batch algorithm is given in Appendix H.

Online Algorithm and Neural Interpretation. Since all covariances can be updated recursively, objective (7) also admits an online algorithm: at each step it receives (pt,ft)(\mathbf{p}_t,\mathbf{f}_t), infers a non-negative activity zt\mathbf{z}_t, and updates the covariances, W\mathbf{W} and M\mathbf{M} (Algorithm 1). This online algorithm has a natural neural interpretation (Figure 1c). The matrix W\mathbf{W} corresponds to feedforward synaptic weights that define local predictive filters over past increments. The matrix M\mathbf{M} corresponds to lateral interactions that implement competition between units. The circuit interpretation applies to the inference dynamics; the weight updates, which are non-local, provide an efficient streaming implementation. Rectification enforces non-negative activity, so each unit reports the presence of a particular local predictive direction rather than a signed canonical coordinate. The resulting activity vector zt\mathbf{z}_t therefore acts as a soft domain assignment of the current trajectory segment to one of several locally predictive tangent-space regimes (Figure 1d).

Experiments on Stochastic Dynamical Systems

Affine saddle-point dynamics. For affine dynamics near a single saddle point, Figure 2a, ReSU ([40]) fails to recover the unstable directions unless the data are provided relative to the fixed-point location. In contrast, our algorithm correctly identifies the unstable directions even when the fixed-point location is unknown (see Appendices § 7 and I.§ 1). The output activity, zt∈R+2\mathbf{z}_t \in\mathbb{R}^{2}_{+}, represents the rectified projection of the past finite differences pt∈R2\mathbf{p}_t \in\mathbb{R}^{2} onto the unstable manifold.

Online LPD applied to stochastic dynamical systems

Figure 2. Online LPD (Alg. 1) applied to stochastic dynamical systems. Gray arrows show the flow; trajectories are colored by the most active output, saturation by its magnitude; insets show the learned tangent-space projections. (a) Affine dynamics near a saddle at (−1,1)(-1,1), k=2k=2; insets (i)–(ii) compare the output with the expected activation (black dashed) for two trajectories. (b) gLV with N=3N=3 species, k=3k=3. (c) Two-saddle dynamics on a sphere, k=4k=4.

Generalized Lotka–Volterra Systems. To test multiple saddle points, we apply LPD to a stochastic generalized Lotka–Volterra (gLV) system, widely used to study stable heteroclinic channels (SHC) ([7, 2]). We consider N=3N=3 species arranged to form an SHC, producing sequential transitions between saddles at which one species dominates and the others are nearly extinct (Appendix I). The resulting dynamics generate a limit-cycle-like sequence of metastable states, as shown in Figure 2b. The results of Algorithm 1 show that each of the k=3k=3 output units successfully captures the predictive direction associated with one transition between saddles, thereby predicting the identity of the next dominant species. These findings remain robust for a greater number of species (N≤100N \le100) and across different noise levels (see Appendix I.§ 3).

Two Saddle Points on a Sphere. On a curved manifold, we consider a heteroclinic flow on the sphere ([39, 5]) with saddle points at the north and south poles (Appendix I). LPD recovers the predictive geometry without prior knowledge of the saddle locations, Figure 2c. The symmetry of this system also reveals a limitation: the unstable direction at one pole aligns with the stable direction in the antipodal region, leading to redundant activations. Such aligned regimes cannot be distinguished from tangent information alone and require additional state or temporal context.

Model components and baselines. The objective (7) itself provides an internal criterion for the number of output components kk, unlike the external criteria of kk-means clustering or Hidden Markov Models: the objective plateaus once kk reaches the intrinsic dimensionality of the predictive structure, and this heuristic elbow coincides with the precision and recall optima of the learned directions (Appendix § 15; Appendix § 15.§ 1 for natural stimuli). To isolate the roles of temporal differencing and non-negativity, we compare LPD on the gLV system (Figure 2b) against past–future CCA (PF), non-negative PF (NN-PF) and finite-differences CCA (FD), scoring held-out trajectories by the Normalized Mutual Information (NMI) between the maximally active component and the next saddle visited. At low and intermediate noise LPD attains higher NMI than all three, converging with FD only at high noise, so both ingredients matter (Appendix § 17). LPD also carries more information about the future tangent vector than a Gaussian Mixture Model fit to the same finite differences, partly via its unnormalized magnitude (Appendix § 18).

Learning Neuronal Responses from Natural Stimuli

Here, we apply Algorithm 1 to sensory stimuli, viewing them as observations of an unknown nonlinear dynamical system. Over short timescales, much of the variation perceived by the retina arises from self-motion and saccadic eye movements. This motivates modeling natural scenes as time series formed by sliding windows over static images. For simplicity, we use the one-dimensional “dead leaves” model of natural images ([20, 48]), which partitions space into patches of different but uniform contrast with sharp boundaries (Figure 3a). First, we apply LPD to the past and future temporal differences of pairs of translated images assuming they represent dynamical states, Figure 3a. The LPDs qualitatively resemble electrophysiologically measured neuronal receptive fields: i) The non-negative constraint along with the short-range correlations in natural images drives each filter to be localized and tile the pixel space (Figure 3b, left), which corresponds to the retinal mosaic ([55, 23]). ii) The center–surround structure of each filter in Figure 3b, left resembles the spatial receptive field of bipolar cells ([17, 26, 4]), Figure 3c, left. iii) Filters with positive and negative peaks correspond to ON and OFF bipolar cells. Each class tiles the pixel space independently of the other ((Kolb, 1976)). Unlike previous explanations of the ON/OFF center–surround receptive fields based on efficient coding ([22, 21, 16]), our approach does not rely on explicitly minimizing metabolic costs. Second, we apply LPD to the past–future temporal difference pairs of time-delay embedding of a scalar time series obtained from the same “dead leaves” model. Such time-delay embedding is necessary if the dynamical system is only partially observed ([54]). Within this framework, the learned weights define linear temporal filters mapping past finite-difference patterns to future ones. Resulting filters (Figure 3b, right) qualitatively resemble ON/OFF sustained/transient filters of retinal bipolar cells (Figure 3c, right) ([26, 55, 12]). Both filter types can be interpreted as LPDs of the stimulus dynamics, which alternate between two local regimes: plateaus of nearly constant luminance, and transitions where contrast boundaries move across the receptive field. Plateaus yield LPDs that integrate the input over time (sustained filters); transitions yield LPDs that compute temporal derivatives (transient filters) and spatial derivatives (center–surround receptive fields), reflecting the inertia of moving boundaries: the recent direction of motion predicts the near future. Spatial and temporal filters differ because motion directions are nearly isotropic in space, whereas time has a fixed arrow. These results suggest that sensory neurons can be viewed as detectors of LPDs governing stimulus dynamics.

LPDs learned on naturalistic stimuli resemble biological receptive fields and temporal filters

Figure 3. LPDs learned on naturalistic stimuli resemble biological receptive fields and temporal filters. (a) Translated “dead leaves” stimulus; past and future samples are temporal differences between segments. (b) Left: center–surround ON (solid) and OFF (dashed) LPDs tile space (k=20k = 20, τ=1\tau= 1, ζ=2\zeta= 2). Right: LPDs on lag vectors (k=4k = 4, τ=1\tau= 1, ζ=n+1\zeta= n + 1, nn the embedding dimension) resemble sustained ON/OFF (green/blue) and transient ON/OFF (red/yellow) cells. (c) Measured receptive field and temporal filter of a retinal bipolar cell (courtesy of D.B. Kastner and S.A. Baccus).

Discussion

We introduced a data-driven framework for discovering LPDs in dynamical systems. Across synthetic and naturalistic trajectory data, this approach identifies LPDs without explicit system identification or latent-state inference, providing a simple primitive for identifying predictive structure (LPD does not itself produce forecasts). Because learning relies only on second-order short-time statistics and a compact set of feedforward and lateral weights, LPD is substantially less parameterized than typical deep predictive architectures, an advantage when data are limited. The current framework assumes that dynamics are approximately linear within local neighborhoods and that predictive directions associated with distinct regimes are sufficiently separated to be resolved by competition. Systems with extremely fast switching, strong nonlinearities, or heavy-tailed noise may violate these assumptions and require extensions of the objective. An important direction for future work is to establish theoretical conditions under which non-negative similarity matching provably recovers LPDs. Finally, the learned LPDs correspond to modes where small perturbations influence future trajectories, suggesting a natural interface with control. Low-rank controllers acting along these directions may enable efficient steering between metastable regions, analogous to control along unstable manifolds in classical dynamical systems. Incorporating reward-weighted or goal-conditioned variants of the predictive objective could therefore yield a self-supervised route to learning control-relevant representations without explicit system identification. Our framework may thus help model neural computation beyond the sensory periphery.

Code availability. Code to reproduce all experiments is available at https://github.com/MarcoZenari/local-predictive-directions.

Extended Related Work

Our method is related to existing approaches for modeling dynamical systems, but differs in key aspects. Switching Linear Dynamical Systems (SLDS) [1, 15, 28] model nonlinear dynamics as collections of locally linear regimes indexed by a latent discrete state that evolves according to a Markov chain. Hsu et al. [19] use an SVD of past–future observation probabilities to recover a global observable representation of a Hidden Markov Model (HMM). Whereas these approaches require explicit inference over latent states and optimize the likelihood of observations under a generative model, we discover LPDs directly from short-time correlations and output soft domain assignments online.

An alternative to explicitly partitioning state space is to approximate nonlinear dynamics with linear dynamics in a transformed feature space. Koopman operator theory [8], Dynamic Mode Decomposition (DMD) [49], and the Variational Approach for Markov Processes (VAMP) [56] all pursue this strategy, aiming to represent nonlinear dynamics with global modes of a linear operator acting on a diverse, and potentially high dimensional, set of observables. Classic finite-rank approaches, such as DMD and VAMP, are powerful and interpretable, but often emphasize leading eigen- or singular modes, which in many applications correspond to slow, large-scale, or long-timescale structure. As a result, the learned dynamic modes can span broad regions of state space and may not resolve the local saddle-to-saddle geometry that governs transitions in multistable systems. Recent deep Koopman methods go beyond fixed dictionaries and leading spectral truncations by learning expressive nonlinear observables or encoder–decoder representations in which the dynamics are approximately linear [30, 33, 53]. However, these methods still primarily seek a globally valid transformed state-space representation. Most relevant to our setting, Lusch et al. [30] learn state-dependent latent eigenvalues through an auxiliary network, enabling local variation in the learned Koopman dynamics. This construction is especially natural for low-dimensional oscillatory or continuous-spectrum systems, where latent dynamics can be represented by structured rotating or spiraling modes, but may be less directly suited to transient dynamics that move between unstable centers. In contrast, LPD is local in both space and time: by operating on finite differences, it directly decomposes trajectory segments into localized predictive directions and non-negative soft domain assignments, readily accommodating multiple unstable centers.

In addition, deep learning methods have extended prior operator-based and variational frameworks (e.g., VAMPnets [31]) and have been applied to multivariate time series forecasting via Transformer architectures. Representative Transformer-style objectives include contrastive predictive coding (CPC) [32] and masked modeling [57] which learn global predictive embeddings. These models are highly parameterized and trained on large datasets with task-specific training procedures, often requiring significant computational costs and careful hyperparameter tuning. While these models can achieve strong predictive performance, the resulting representations are typically global and less interpretable, encompassing complex dependencies across the input sequence. In contrast to global latent-space models and Gaussian Mixture Models [47], the self-supervised method proposed here identifies LPDs from the system’s short time correlations.

CCA Recovers the Most Predictive Direction of a Saddle Fixed Point

We start from Eq. (4) in the main text:

Cpp−1CpfCff−1Cfpvi=ρi2vi,Cff−1CfpCpp−1Cpfui=ρi2ui.(9)C_{pp}^{-1}C_{pf}C_{ff}^{-1}C_{fp}v_i=\rho_i^2v_i,\qquad C_{ff}^{-1}C_{fp}C_{pp}^{-1}C_{pf}u_i=\rho_i^2u_i. \tag*{(9)}

Under the linearized dynamics,

Cpf=E[pt(Apt+ωt)⊤]=CppA⊤,C_{pf}=\mathbb{E}[p_t(Ap_t+\omega_t)^\top]=C_{pp}A^\top,
Cfp=E[(Apt+ωt)pt⊤]=ACpp,(10)C_{fp}=\mathbb{E}[(Ap_t+\omega_t)p_t^\top]=AC_{pp}, \tag*{(10)}

which reduces Eq. (4) to

A⊤Cff−1ACppvi=ρi2vi,A^\top C_{ff}^{-1}AC_{pp}v_i=\rho_i^2v_i,
Cff−1ACppA⊤ui=ρi2ui.(11)C_{ff}^{-1}AC_{pp}A^\top u_i=\rho_i^2u_i. \tag*{(11)}

Notice that

(A⊤Cff−1ACpp)⊤Cppvi=CppA⊤Cff−1ACppvi=ρi2Cppvi.(12)\begin{aligned} (A^\top C_{ff}^{-1}AC_{pp})^\top C_{pp}v_i \\ &=C_{pp}A^\top C_{ff}^{-1}AC_{pp}v_i \\ &=\rho_i^2C_{pp}v_i. \tag*{(12)} \end{aligned}

Thus, {vi,Cppvi}\{v_i,C_{pp}v_i\} is a pair of right and left eigenvectors of A⊤Cff−1ACppA^\top C_{ff}^{-1}AC_{pp}. Similarly, {ui,Cffui}\{u_i,C_{ff}u_i\} is a pair of right and left eigenvectors of Cff−1ACppA⊤C_{ff}^{-1}AC_{pp}A^\top. Therefore, ui⊤Cffuj=vi⊤Cppvj=0u_i^\top C_{ff}u_j=v_i^\top C_{pp}v_j=0 for i≠ji\ne j.

Also notice that

A⊤(Cff−1ACppA⊤ui)=(A⊤Cff−1ACpp)A⊤ui=ρi2A⊤ui.(13)\begin{aligned} A^\top(C_{ff}^{-1}AC_{pp}A^\top u_i) \\ &=(A^\top C_{ff}^{-1}AC_{pp})A^\top u_i \\ &=\rho_i^2A^\top u_i. \tag*{(13)} \end{aligned}

Thus, vi∝A⊤uiv_i\propto A^\top u_i. Under proper normalization, these results can be summarized by

V⊤CppV=I,U⊤CffU=I,A⊤U=VR,(14)V^\top C_{pp}V=I,\qquad U^\top C_{ff}U=I,\qquad A^\top U=VR, \tag*{(14)}

where U[:,i]:=uiU[:,i]:=u_i, V[:,i]:=viV[:,i]:=v_i, Rij:=δijρiR_{ij}:=\delta_{ij}\rho_i. Thus, we have

A⊤=VRU−1=VRU⊤Cff,(15)A^\top=VRU^{-1}=VRU^\top C_{ff}, \tag*{(15)}
A=CffURV⊤=∑iρiCffuivi⊤.(16)A=C_{ff}URV^\top=\sum_i\rho_iC_{ff}u_iv_i^\top. \tag*{(16)}

On the other hand, we have the eigen-decomposition

A=∑iλirili⊤,(17)A=\sum_i\lambda_ir_il_i^\top, \tag*{(17)}

where λi\lambda_i is the eigenvalue of AA, rir_i and lil_i are right and left eigenvectors of AA, respectively. When a single unstable mode dominates, λ1\lambda_1 is real and ∣λ1∣≫∣λ2∣\lvert\lambda_1\rvert\gg\lvert\lambda_2\rvert. That is, the leading rank-1 term of A\mathbf{A} dominates. Comparing the leading terms of Eq. (16) and Eq. (17), we have

v1∝l1,Cffu1∝r1.(18)\mathbf{v}_1 \propto\mathbf{l}_1,\quad\mathbf{C}_{ff}\mathbf{u}_1 \propto\mathbf{r}_1. \tag*{(18)}

Thus, the top predictive mode is orthogonal to every eigen-direction except the most unstable direction.

Normalization of output components

We start from the eigenvalue problems for CCA (Eq. (4) in the main text).

Cpp−1CpfCff−1Cfpvi=ρi2vi,\mathbf{C}_{pp}^{-1}\mathbf{C}_{pf}\mathbf{C}_{ff}^{-1}\mathbf{C}_{fp}\mathbf{v}_i = \rho_i^2\mathbf{v}_i,
Cff−1CfpCpp−1Cpfui=ρi2ui.\mathbf{C}_{ff}^{-1}\mathbf{C}_{fp}\mathbf{C}_{pp}^{-1}\mathbf{C}_{pf}\mathbf{u}_i = \rho_i^2\mathbf{u}_i.

It follows that

GZp[i]=PTCpp−1CpfCff−1F(PTvi)=PTCpp−1CpfCff−1Cfpvi=ρi2Zp[i].(19)\begin{aligned} \mathbf{G}\mathbf{Z}_p[i] &= \mathbf{P}^{\mathsf T}\mathbf{C}_{pp}^{-1}\mathbf{C}_{pf}\mathbf{C}_{ff}^{-1}\mathbf{F}(\mathbf{P}^{\mathsf T}\mathbf{v}_i) \\ &= \mathbf{P}^{\mathsf T}\mathbf{C}_{pp}^{-1}\mathbf{C}_{pf}\mathbf{C}_{ff}^{-1}\mathbf{C}_{fp}\mathbf{v}_i \\ &= \rho_i^2\mathbf{Z}_p[i]. \tag*{(19)} \end{aligned}

Thus, Zp[i]\mathbf{Z}_p[i] is a right eigenvector of G\mathbf{G} with eigenvalue ρi2\rho_i^2. Similarly, Zf[i]\mathbf{Z}_f[i] is a left eigenvector of G\mathbf{G} with eigenvalue ρi2\rho_i^2. Notice that

Zp[i]TZf[j]=E[(viTpt)(ujTft)]=ρiδij.(20)\mathbf{Z}_p[i]^{\mathsf T}\mathbf{Z}_f[j] = \mathbb{E}[(\mathbf{v}_i^{\mathsf T}\mathbf{p}_t)(\mathbf{u}_j^{\mathsf T}\mathbf{f}_t)] = \rho_i\delta_{ij}. \tag*{(20)}

Therefore, when k=nk=n is the rank of G\mathbf{G}, we can eigen-decompose G\mathbf{G} as follows.

G=∑i=1kρi2Zp[i]Zf[i]T1ρi=ZpTRZf.(21)\begin{aligned} \mathbf{G} &= \sum_{i=1}^{k} \rho_i^2\mathbf{Z}_p[i]\mathbf{Z}_f[i]^{\mathsf T}\frac{1}{\rho_i} \\ &= \mathbf{Z}_p^{\mathsf T}\mathbf{R}\mathbf{Z}_f. \tag*{(21)} \end{aligned}

Invariance to Affine Changes of Coordinates

Consider an invertible linear change of coordinates of the state, xt′=Bxt+c\mathbf{x}'_t=\mathbf{B}\mathbf{x}_t+\mathbf{c}, with B\mathbf{B} an invertible n×nn \times n matrix and c∈Rn\mathbf{c}\in\mathbb{R}^n, and denote by primes all quantities computed from the transformed trajectory. Temporal differencing annihilates the offset,

δxt′=xt+τ′−xt′=B(xt+τ−xt)=B δxt,(22)\delta\mathbf{x}'_t=\mathbf{x}'_{t+\tau}-\mathbf{x}'_t=\mathbf{B}(\mathbf{x}_{t+\tau}-\mathbf{x}_t)=\mathbf{B}\,\delta\mathbf{x}_t, \tag*{(22)}

so P′=BP\mathbf{P}'=\mathbf{B}\mathbf{P}, F′=BF\mathbf{F}'=\mathbf{B}\mathbf{F} and Cαβ′=BCαβBT\mathbf{C}'_{\alpha\beta}=\mathbf{B}\mathbf{C}_{\alpha\beta}\mathbf{B}^{\mathsf T} for α,β∈{p,f}\alpha,\beta\in\{p,f\}. Substituting into the predictive similarity matrix of the main text,

G′=PTBT(BCppBT)−1(BCpfBT)(BCffBT)−1BF=PT(BTB−T)Cpp−1(B−1B)Cpf(BTB−T)Cff−1(B−1B)F=G.(23)\begin{aligned} \mathbf{G}' &= \mathbf{P}^{\mathsf T}\mathbf{B}^{\mathsf T}\left(\mathbf{B}\mathbf{C}_{pp}\mathbf{B}^{\mathsf T}\right)^{-1}\left(\mathbf{B}\mathbf{C}_{pf}\mathbf{B}^{\mathsf T}\right)\left(\mathbf{B}\mathbf{C}_{ff}\mathbf{B}^{\mathsf T}\right)^{-1}\mathbf{B}\mathbf{F} \\ &= \mathbf{P}^{\mathsf T}\left(\mathbf{B}^{\mathsf T}\mathbf{B}^{-\mathsf T}\right)\mathbf{C}_{pp}^{-1}\left(\mathbf{B}^{-1}\mathbf{B}\right)\mathbf{C}_{pf}\left(\mathbf{B}^{\mathsf T}\mathbf{B}^{-\mathsf T}\right)\mathbf{C}_{ff}^{-1}\left(\mathbf{B}^{-1}\mathbf{B}\right)\mathbf{F}=\mathbf{G}. \tag*{(23)} \end{aligned}

Since the objectives ((6) and (7)) depend on the data only through G\mathbf{G}, the learned outputs Z\mathbf{Z} are unchanged. The feedforward weights are not: with N=ZG\mathbf{N}=\mathbf{Z}\mathbf{G} also unchanged, W′P′=NG′T=WP\mathbf{W}'\mathbf{P}'=\mathbf{N}\mathbf{G}'^{\mathsf T}=\mathbf{W}\mathbf{P}, and P′=BP\mathbf{P}'=\mathbf{B}\mathbf{P} gives W′=WB−1\mathbf{W}'=\mathbf{W}\mathbf{B}^{-1}, as also follows from the update rule for W\mathbf{W} in Algorithm 1. Temporal differencing removes the translation, while whitening by Cpp\mathbf{C}_{pp} and Cff\mathbf{C}_{ff} removes the remaining linear part, leaving the outputs invariant and the filters equivariant.

Two remarks. The derivation uses only Cαβ′=BCαβBTC'_{\alpha\beta} = B C_{\alpha\beta} B^{\mathsf{T}}, which holds for the empirical covariances computed from a single trajectory as well as in the population limit, so the invariance is exact at finite sample. The numerical implementation replaces the inverses by Tikhonov-regularized ones, S+=(S+ϵI)−1S^{+} = (S + \epsilon I)^{-1} (Appendix § 13); since ϵI\epsilon I does not transform with the data, the invariance then holds exactly for orthogonal BB and only approximately for general invertible BB, including pure rescalings B=sIB = sI.

Legendre Transform of Similarity Matching Objective

We apply the Legendre transform to the similarity matching objective.

arg min⁡Z ∥GGT−ZTZ∥F2=arg min⁡Z[−2Tr⁡(ZTZGGT)+Tr⁡(ZTZZTZ)]=arg min⁡Z min⁡N max⁡M[−4Tr⁡(ZTNGT)+2Tr⁡(ZTMZ)+2Tr⁡(NTN)−Tr⁡(MTM)].(24)\begin{aligned} \underset{Z}{\operatorname{arg\,min}}\ \lVert GG^{\mathsf{T}} - Z^{\mathsf{T}}Z\rVert_{F}^{2} &= \underset{Z}{\operatorname{arg\,min}}\left[-2\operatorname{Tr}\left(Z^{\mathsf{T}}ZGG^{\mathsf{T}}\right) + \operatorname{Tr}\left(Z^{\mathsf{T}}ZZ^{\mathsf{T}}Z\right)\right] \\ &= \underset{Z}{\operatorname{arg\,min}}\ \underset{N}{\operatorname{min}}\ \underset{M}{\operatorname{max}}\left[-4\operatorname{Tr}\left(Z^{\mathsf{T}}NG^{\mathsf{T}}\right) + 2\operatorname{Tr}\left(Z^{\mathsf{T}}MZ\right) \right.\\ &\qquad\left.{}+ 2\operatorname{Tr}\left(N^{\mathsf{T}}N\right) - \operatorname{Tr}\left(M^{\mathsf{T}}M\right)\right]. \tag*{(24)} \end{aligned}

To see why the second line and the third line are equal, one can set the derivatives of the last line with respect to NN and MM to zero to find the optima

N∗=ZG,M∗=ZZT.(25)N^{*} = ZG,\qquad M^{*} = ZZ^{\mathsf{T}}. \tag*{(25)}

Substituting the optima back into the last line finishes the proof.

Similarity Matching Recovers the Low-Dimensional Principal Subspace

The solution Z∗Z^{*} obtained from the unconstrained problem

Z∗=arg min⁡Z∈Rk×T ∥GGT−ZTZ∥F2(26)Z^{*} = \underset{Z\in\mathbb{R}^{k\times T}}{\operatorname{arg\,min}}\ \lVert GG^{\mathsf{T}} - Z^{\mathsf{T}}Z\rVert_{F}^{2} \tag*{(26)}
Z∗=M−1NGT=M−1NFTCff−1CfpCpp−1P=M−1WP≃VTP,(27)Z^{*} = M^{-1}NG^{\mathsf{T}} = M^{-1}NF^{\mathsf{T}}C_{ff}^{-1}C_{fp}C_{pp}^{-1}P = M^{-1}WP \simeq V^{\mathsf{T}}P, \tag*{(27)}

where ≃\simeq denotes equality up to an orthogonal transformation and component scaling (§ 2), and recovers the kk-dimensional subspace spanned by the first kk canonical components of the CCA problem between PP and FF. A proof that similarity matching objectives successfully recover the low-dimensional subspace defined by a similarity matrix, such as GGTGG^{\mathsf{T}} in Eq. (26), is provided by Pehlevan et al. [36]. Here, we validate this method empirically by applying the unconstrained algorithm defined in Eq. (27) to a standard benchmark commonly used for evaluating CCA algorithms. We generate synthetic paired datasets (X,Y)(X, Y) with known latent correlations. Specifically, we consider a one-dimensional latent variable Zlatent∈R1×dZ_{\mathrm{latent}} \in\mathbb{R}^{1\times d} and project it onto high-dimensional spaces via random linear mixing matrices AA and BB to produce

X=ZlatentAT+ϵX,Y=ZlatentBT+ϵY,(28)X = Z_{\mathrm{latent}}A^{\mathsf{T}} + \epsilon_{X},\qquad Y = Z_{\mathrm{latent}}B^{\mathsf{T}} + \epsilon_{Y}, \tag*{(28)}

where ϵX\epsilon_{X} and ϵY\epsilon_{Y} are Gaussian noise terms. The algorithm in Eq. (27) is applied directly to (X,Y)(X, Y), without computing finite differences, to demonstrate that the similarity matching solution can successfully recover the underlying low-dimensional subspace, which in this case corresponds to the directions of maximal correlation identified by CCA. Finally, the canonical variable (Z\mathbf{Z}) computed by the algorithm is compared against the known latent variable ZlatentZ_{\mathrm{latent}}. The result is shown in Figure 4a, also compared with the standard CCA results. For comparison, Figure 4b shows the results of the non-negative algorithm defined in the main text for the constrained problem with k=2k = 2 output neurons, Z1\mathbf{Z}_1 and Z2\mathbf{Z}_2.

Gaussian latent variable plots

Figure 4. Gaussian latent variable. A 1D1D Gaussian latent variable, Zlatent\mathbf{Z}_{\mathrm{latent}}, is projected onto two high-dimensional variables, X\mathbf{X} and Y\mathbf{Y}. (a) Standard CCA and unconstrained algorithm with k=1k = 1 correctly retrieve the latent structure. (b) The non-negative algorithm with k=2k = 2 learns positive and negative projections of the latent structure.

Whitening and Similarity-Matching Interpretation

We consider a generic stochastic process with paired variables

pt:=xt,ft:=xt+1,xt∈Rn.\mathbf{p}_t := \mathbf{x}_t,\qquad\mathbf{f}_t := \mathbf{x}_{t+1},\qquad\mathbf{x}_t \in\mathbb{R}^{n}.

We define the second-order statistics

Cpp=E[ptptT],Cff=E[ftftT],\mathbf{C}_{pp} = \mathbb{E}[\mathbf{p}_t\mathbf{p}_t^{\mathsf{T}}],\qquad\mathbf{C}_{ff} = \mathbb{E}[\mathbf{f}_t\mathbf{f}_t^{\mathsf{T}}],

and the cross-covariances

Cpf=E[ptftT],Cfp=CpfT.\mathbf{C}_{pf} = \mathbb{E}[\mathbf{p}_t\mathbf{f}_t^{\mathsf{T}}],\qquad\mathbf{C}_{fp} = \mathbf{C}_{pf}^{\mathsf{T}}.

All matrices lie in Rn×n\mathbb{R}^{n \times n}. To isolate interaction structure independently of marginal statistics, we introduce whitened variables

p~t=Cpp−1/2pt,f~t=Cff−1/2ft,\widetilde{\mathbf{p}}_t = \mathbf{C}_{pp}^{-1/2}\mathbf{p}_t,\qquad\widetilde{\mathbf{f}}_t = \mathbf{C}_{ff}^{-1/2}\mathbf{f}_t,

and the corresponding cross-covariance in whitened coordinates

C~pf=Cpp−1/2CpfCff−1/2,C~fp=C~pfT.\widetilde{\mathbf{C}}_{pf} = \mathbf{C}_{pp}^{-1/2}\mathbf{C}_{pf}\mathbf{C}_{ff}^{-1/2},\qquad\widetilde{\mathbf{C}}_{fp} = \widetilde{\mathbf{C}}_{pf}^{\mathsf{T}}.

This transformation removes the influence of marginal covariances, retaining only the interaction structure between past and future variables. We now consider the matrix

G=PTCpp−1CpfCff−1F,\mathbf{G} = \mathbf{P}^{\mathsf{T}}\mathbf{C}_{pp}^{-1}\mathbf{C}_{pf}\mathbf{C}_{ff}^{-1}\mathbf{F},

where P\mathbf{P} and F\mathbf{F} collect samples of pt\mathbf{p}_t and ft\mathbf{f}_t over time. A direct computation of its Gram matrix yields

GGT=PTCpp−1CpfCff−1FFTCff−1CfpCpp−1P.\mathbf{G}\mathbf{G}^{\mathsf{T}}=\mathbf{P}^{\mathsf{T}}\mathbf{C}_{pp}^{-1}\mathbf{C}_{pf}\mathbf{C}_{ff}^{-1}\mathbf{F}\mathbf{F}^{\mathsf{T}}\mathbf{C}_{ff}^{-1}\mathbf{C}_{fp}\mathbf{C}_{pp}^{-1}\mathbf{P}.

For long trajectories, we use the empirical approximation

FFT≈TCff,\mathbf{F}\mathbf{F}^{\mathsf{T}}\approx T\mathbf{C}_{ff},

which gives

GGT=TPTCpp−1CpfCff−1CfpCpp−1P.\mathbf{G}\mathbf{G}^{\mathsf{T}}=T\mathbf{P}^{\mathsf{T}}\mathbf{C}_{pp}^{-1}\mathbf{C}_{pf}\mathbf{C}_{ff}^{-1}\mathbf{C}_{fp}\mathbf{C}_{pp}^{-1}\mathbf{P}.

Rewriting in whitened coordinates yields

GGT=TP~TC~pfC~pfTP~.\mathbf{G}\mathbf{G}^{\mathsf{T}}=T\widetilde{\mathbf{P}}^{\mathsf{T}}\widetilde{\mathbf{C}}_{pf}\widetilde{\mathbf{C}}_{pf}^{\mathsf{T}}\widetilde{\mathbf{P}}.

This shows that the relevant structure is fully captured by the whitened cross-covariance acting on whitened inputs. Defining

X‾:=C~fpP~,\overline{\mathbf{X}}:=\widetilde{\mathbf{C}}_{fp}\widetilde{\mathbf{P}},

we obtain the compact similarity form

GGT=X‾TX‾,\mathbf{G}\mathbf{G}^{\mathsf{T}}=\overline{\mathbf{X}}^{\mathsf{T}}\overline{\mathbf{X}},

which connects this construction to standard similarity-matching objectives. Special case: linear dynamics. If the dynamics are linear,

ft=Apt+wt,\mathbf{f}_t=A\mathbf{p}_t+\mathbf{w}_t,

then the cross-covariance satisfies

Cfp=ACpp.\mathbf{C}_{fp}=A\mathbf{C}_{pp}.

Substituting into the whitened expression gives

C~fp=Cff−1/2CfpCpp−1/2=Cff−1/2ACpp1/2=:A~.\begin{aligned} \widetilde{\mathbf{C}}_{fp}=\mathbf{C}_{ff}^{-1/2}\mathbf{C}_{fp}\mathbf{C}_{pp}^{-1/2} \\ &=\mathbf{C}_{ff}^{-1/2}A\mathbf{C}_{pp}^{1/2}=:\widetilde{A}. \end{aligned}

Note that in whitened coordinates, the dynamics reduce to

f~t=A~p~t.\widetilde{\mathbf{f}}_t=\widetilde{A}\widetilde{\mathbf{p}}_t.

Details on Numerical Implementation of LPD

The offline implementation of LPD solves the min–max problem of Eq. (8) by alternating gradient updates: an inference step that iterates a rectified update of Z\mathbf{Z} at fixed W,M\mathbf{W},\mathbf{M}, followed by a learning step that updates W\mathbf{W} and M\mathbf{M} from empirical covariance estimates (Algorithm 2), where [⋅]+[\cdot]_+ denotes elementwise rectification. The covariance matrices are defined as

Czp:=ZPT,Czz:=ZZT,(29)\mathbf{C}_{zp}:=\mathbf{Z}\mathbf{P}^{\mathsf{T}},\qquad\mathbf{C}_{zz}:=\mathbf{Z}\mathbf{Z}^{\mathsf{T}}, \tag*{(29)}

and similarly for the remaining terms. Regularization is applied through Tikhonov-regularized inverses, S+:=(S+ϵI)−1\mathbf{S}^{+}:=(\mathbf{S}+\epsilon\mathbf{I})^{-1}, to ensure numerical stability. Convergence of the offline algorithm is assessed by monitoring the relative parameter changes

∥W(r+1)−W(r)∥F∥W(r)∥F,∥M(r+1)−M(r)∥F∥M(r)∥F,(30)\frac{\lVert\mathbf{W}^{(r+1)}-\mathbf{W}^{(r)}\rVert_F}{\lVert\mathbf{W}^{(r)}\rVert_F},\qquad \frac{\lVert\mathbf{M}^{(r+1)}-\mathbf{M}^{(r)}\rVert_F}{\lVert\mathbf{M}^{(r)}\rVert_F}, \tag*{(30)}

Algorithm 2: Learning Local Predictive Directions Offline

Input: past and future increment matrices P,F\mathbf{P},\mathbf{F}; number of output units kk

Initialize W∈Rk×n\mathbf{W} \in\mathbb{R}^{k \times n} and M∈Rk×k\mathbf{M} \in\mathbb{R}^{k \times k} with M≻0\mathbf{M} \succ0;

repeat

Inference: update Z\mathbf{Z} for fixed W,M\mathbf{W},\mathbf{M};

repeat

Z←[Z+ηZ(WP−MZ)]+\mathbf{Z} \leftarrow[\mathbf{Z} + \eta_Z(\mathbf{W}\mathbf{P} - \mathbf{M}\mathbf{Z})]_+;

until inference convergence;

Learning: update empirical covariances Cpp,Cff,Cpf,Cfp,Czp,Czz\mathbf{C}_{pp}, \mathbf{C}_{ff}, \mathbf{C}_{pf}, \mathbf{C}_{fp}, \mathbf{C}_{zp}, \mathbf{C}_{zz};

W←W+ηW(CzpCpp+CpfCff+CfpCpp+−W)\mathbf{W} \leftarrow\mathbf{W} + \eta_W\left(\mathbf{C}_{zp}\mathbf{C}_{pp}^{+}\mathbf{C}_{pf}\mathbf{C}_{ff}^{+}\mathbf{C}_{fp}\mathbf{C}_{pp}^{+} - \mathbf{W}\right);

M←M+ηM(Czz−M)\mathbf{M} \leftarrow\mathbf{M} + \eta_M(\mathbf{C}_{zz} - \mathbf{M});

until relative changes in W\mathbf{W} and M\mathbf{M} fall below threshold;

and terminating the iterations once both quantities fall below a predefined threshold. Both the online and offline implementations of the algorithm can be run on a single standard GPU for all experiments reported in the paper. Moreover, we note that applying the offline algorithm to the same data reproduces the results obtained with the online algorithm, with no significant differences. More generally, when the dataset is sufficiently large and the underlying dynamical process is stationary, the online and offline implementations yield equivalent results.

Details on stochastic dynamical systems

Affine Saddle-Point Dynamics — fig:2a of the Main Text

Let xt∈R2\mathbf{x}_t \in\mathbb{R}^{2} denote the system state at time tt, and let x∗∈R2\mathbf{x}^{*} \in\mathbb{R}^{2} represent the saddle point. The dynamics are given by

xt+1=x∗+A(xt−x∗)+σξt,ξt∼N(0,I),(31)\mathbf{x}_{t+1} = \mathbf{x}^{*} + \mathbf{A}\left(\mathbf{x}_t - \mathbf{x}^{*}\right) + \sigma\boldsymbol{\xi}_t,\qquad\boldsymbol{\xi}_t \sim\mathcal{N}(0,\mathbf{I}), \tag*{(31)}

where A=diag⁡(λu,λs)\mathbf{A} = \operatorname{diag}(\lambda_u,\lambda_s) is a diagonal matrix with one unstable eigenvalue λu>1\lambda_u > 1 and one stable eigenvalue 0<λs<10 < \lambda_s < 1, and σ\sigma controls the level of stochasticity. In this case the unstable direction is along the x1x_1 axis, dunstable=(1,0)\mathbf{d}_{\mathrm{unstable}} = (1,0).

Generalized Lotka–Volterra Systems — fig:2b of the Main Text

To validate our algorithm in the presence of multiple saddle points, we apply it to trajectories generated by a stochastic population dynamics model known as the generalized Lotka–Volterra (gLV) system, commonly used to illustrate stable heteroclinic channels and winnerless competition:

x˙i=xi(1−∑jAjixj)+σξi(t),ξi(t)∼N(0,1),(32)\dot{x}_i = x_i\left(1 - \sum_j A_{ji}x_j\right) + \sigma\xi_i(t),\qquad\xi_i(t) \sim\mathcal{N}(0,1), \tag*{(32)}

where xix_i denotes the abundance of species ii, AijA_{ij} encodes the effect of species ii on species jj, and σ\sigma sets the noise amplitude. A positive AijA_{ij} indicates that species ii has a detrimental effect on species jj; if AjiA_{ji} is also positive, the two species interact competitively. By tuning these interactions, the system can enter regimes exhibiting stable heteroclinic channels [7, 2].

A convenient set of saddle points can be constructed by considering fixed points in which a single species is active (xi>0x_i > 0) while all others are extinct (xj=0x_j = 0 for j≠ij \ne i). Setting the self-interactions to Aii=1A_{ii} = 1 for all species ii places these fixed points at xi=1x_i = 1, so that linearizing Eq. (32) about the saddle dominated by species ii gives x˙j≈(1−Aij)xj\dot{x}_j \approx(1 - A_{ij})x_j for every j≠ij \ne i. Each row of the interaction matrix therefore collects the transverse eigenvalues around the corresponding equilibrium: entries with Aij<1A_{ij} < 1 generate unstable directions, guiding the system from the saddle dominated by species ii toward the saddle where species jj becomes active.

We study a system with N=3N = 3 species arranged to form a stable heteroclinic channel (SHC), producing sequential transitions between saddle points. Transitions from the saddle where species ii is active to the saddle where species jj becomes active are induced by setting Aij=0.7A_{ij} = 0.7, while all other off-diagonal entries of the interaction matrix are set to 1.5.

Scalability in High-Dimensional Generalized Lotka–Volterra Systems

We evaluate the scalability of LPD with respect to system dimensionality using generalized Lotka–Volterra (gLV) dynamics. We consider varying system sizes N∈{3,5,10,25,50,100}N \in\{3, 5, 10, 25, 50, 100\}, across noise levels σ∈{0.01,0.05,0.1}\sigma\in\{0.01, 0.05, 0.1\}.

For each configuration, we simulate trajectories and apply LPD to learn predictive representations. The generalized Lotka–Volterra dynamics are given by

x˙=x⊙(1−ATx)+σξ(t),(33)\dot{x} = x \odot\left(1 - A^{\mathsf{T}}x\right) + \sigma\xi(t), \tag*{(33)}

where A∈RN×NA \in\mathbb{R}^{N \times N} induces sequential switching dynamics (same convention as Eq. (32)) and ξ(t)\xi(t) is Gaussian white noise. Simulations are performed using Euler integration with dt=0.1dt = 0.1 over T=5000T = 5000, starting near single-species dominance with small perturbations. After removing an initial transient of 10310^3 steps, we construct Ndata=19,000N_{\mathrm{data}} = 19{,}000 training samples.

Performance is evaluated by comparing the inferred transition structure to the ground-truth transition graph induced by AA. Since components are unordered, we align learned and ground-truth transitions using the Hungarian algorithm applied to a similarity matrix between normalized rows of M−1WM^{-1}W and the theoretical transition matrix. The resulting agreement is quantified using NMI.

We observe that LPD remains robust as system dimensionality increases, with consistently high NMI across all tested values of NN, without requiring any re-tuning of optimization hyperparameters, and across all noise levels.

A Sphere with Two Saddle Points — fig:2c of the Main Text

To further test the algorithm on a nonlinear, curved manifold, we consider a two-dimensional flow on the surface of a sphere. Flows on the sphere defined by stream functions in spherical coordinates are a standard tool in dynamical systems and fluid mechanics and naturally exhibit heteroclinic dynamics [39, 5]. We consider stochastic dynamics constrained to the surface of the unit sphere, where the state x(t)∈R3\mathbf{x}(t) \in\mathbb{R}^{3} evolves under a combination of deterministic flow and isotropic noise tangent to the sphere. The deterministic component is derived from a stream function

Scalability in gLV systems

Figure 5. Scalability in gLV systems. Normalized Mutual Information (NMI) as a function of system size NN under different noise levels, reported as mean ±\pm standard deviation over 10 realizations. Results are for the offline implementation Alg. 2.

ψ(θ,ϕ)=sin⁡2θcos⁡(2ϕ),(34)\psi(\theta,\phi) = \sin^{2}\theta\cos(2\phi), \tag*{(34)}

defined in spherical coordinates (θ,ϕ)(\theta,\phi), with θ\theta the polar angle from the positive zz-axis and ϕ\phi the azimuthal angle in the xyxy-plane. The gradient of the stream function defines a divergence-free flow on the sphere, with velocity components

vθ=−1sin⁡θ∂ψ∂ϕ,vϕ=∂ψ∂θv_{\theta} = -\frac{1}{\sin\theta}\frac{\partial\psi}{\partial\phi}, \qquad v_{\phi} = \frac{\partial\psi}{\partial\theta}

along the local tangent vectors eθ\mathbf{e}_{\theta} and eϕ\mathbf{e}_{\phi}. Stochastic perturbations are added as Gaussian noise projected onto the tangent plane via the projection matrix Π=I−xxT\mathbf{\Pi} = \mathbf{I} - \mathbf{x}\mathbf{x}^{\mathsf{T}}, ensuring that the dynamics remain on the sphere. The resulting stochastic differential equation is

dx=v(x) dt+σΠ(x) dwt,d\mathbf{x} = \mathbf{v}(\mathbf{x})\,dt + \sigma\mathbf{\Pi}(\mathbf{x})\,d\mathbf{w}_{t},

where v(x)=vθeθ+vϕeϕ\mathbf{v}(\mathbf{x}) = v_{\theta}\mathbf{e}_{\theta} + v_{\phi}\mathbf{e}_{\phi} and σ\sigma controls the noise amplitude. Trajectories are integrated numerically with an Euler–Maruyama scheme, followed by renormalization to maintain the unit-norm constraint. This formulation captures both structured, divergence-free flows on curved manifolds and stochastic variability confined to the manifold. This system contains two saddle points located at the north (θ=0)(\theta= 0) and south (θ=π)(\theta= \pi) poles of the sphere. The unstable manifolds correspond to directions in the tangent plane along which trajectories locally depart from these poles, as determined by linearizing the velocity field around the fixed points:

  • North pole (θ=0)(\theta= 0): unstable directions along (1,1)(1,1) and (−1,−1)(-1,-1) in the tangent plane spanned by x1x_{1} and x2x_{2} at x3=1x_{3} = 1.

  • South pole (θ=π)(\theta= \pi): unstable directions along (1,−1)(1,-1) and (−1,1)(-1,1) in the tangent plane spanned by x1x_{1} and x2x_{2} at x3=−1x_{3} = -1.

Model Selection and Validation

The proposed normative framework provides an internal, task-aligned criterion for selecting the number of output components kk, evaluated using the same objective optimized during learning rather than an external criterion such as kk-means clustering or hidden Markov models. Because the objective is non-increasing in kk, the plateau is a heuristic rather than an explicit complexity penalty; we therefore validate it below against ground-truth directions. In practice, the algorithm is run for a range of candidate values of kk, and the corresponding objective is evaluated. Once kk reaches the intrinsic dimensionality of the unstable structure, the objective exhibits a clear plateau; the resulting elbow provides a natural estimate of the number of dynamical modes (Figure 6a). We further validate this criterion by computing precision and recall of the alignment between learned and ground-truth directions. These results confirm the expected behavior: for k<ktruek < k_{\mathrm{true}}, the method recovers correct directions but misses components, whereas for k>ktruek > k_{\mathrm{true}}, all true directions are recovered but redundant or spurious components are introduced. To quantify model selection more precisely, we evaluate the objective

Objective, precision, and recall as functions of output components across noise levels

Figure 6. Model selection and validation for Section 3 (SHC in gLV system). Results are shown as mean ±\pm standard error over 10 trials. (a) Objective function as a function of the number of components kk across noise levels. (b–c) Precision and recall (Eq. (39)) across kk and noise levels.

1T2∥GGT−ZTZ∥F2,(35)\frac{1}{T^2}\left\lVert\mathbf{G}\mathbf{G}^{\mathsf{T}}-\mathbf{Z}^{\mathsf{T}}\mathbf{Z}\right\rVert_{\mathrm{F}}^2, \tag*{(35)}

where TT is the number of samples used to compute G\mathbf{G} (matching the sample size of Z\mathbf{Z}). Since the derivation of Z\mathbf{Z} neglects constant terms (see Appendix § 8), these must be reinstated when evaluating the objective. We do so by rescaling each row of Z\mathbf{Z} with a constant factor, introducing kk free parameters, which are optimized to obtain a corrected estimate of Eq. (35). To further validate the selection of kk, we compute matching-based precision and recall between learned directions and ground-truth unstable directions. Let

{vi}i=1kand{dj}j=1ktrue(36)\left\{\mathbf{v}_i\right\}_{i=1}^{k}\quad\text{and}\quad\left\{\mathbf{d}_j\right\}_{j=1}^{k_{\mathrm{true}}} \tag*{(36)}

denote the learned and ground-truth directions, respectively, each normalized to unit norm, ∥vi∥=∥dj∥=1\lVert\mathbf{v}_i\rVert=\lVert\mathbf{d}_j\rVert=1. We define the similarity matrix

Cij=max⁡(⟨vi,dj⟩,0),(37)C_{ij}=\max\left(\left\langle\mathbf{v}_i,\mathbf{d}_j\right\rangle,0\right), \tag*{(37)}

and compute the optimal one-to-one matching

M=arg max⁡1–1 matchings ∑(i,j)∈MCij.(38)\mathcal{M}=\underset{\text{1--1 matchings}}{\operatorname{arg\,max}}\ \sum_{(i,j)\in\mathcal{M}} C_{ij}. \tag*{(38)}

Precision and recall are then defined as

Precision⁡=1k∑(i,j)∈MCij,Recall⁡=1ktrue∑(i,j)∈MCij.(39)\operatorname{Precision}=\frac{1}{k}\sum_{(i,j)\in\mathcal{M}}C_{ij},\qquad \operatorname{Recall}=\frac{1}{k_{\mathrm{true}}}\sum_{(i,j)\in\mathcal{M}}C_{ij}. \tag*{(39)}

Precision quantifies the quality of learned directions while penalizing spurious components, whereas recall measures how many ground-truth directions are recovered. Together, these metrics characterize the effect of under- and over-estimating kk.

Model Selection for Naturalistic Stimuli

The same criterion applies to the naturalistic stimulus ensemble of § 4. We run LPD on past–future temporal-difference pairs obtained from the one-dimensional dead-leaves model and evaluate the objective (35) over a range of candidate kk (Figure 7). As for the dynamical systems of § 3, the objective decreases steeply while additional components still capture distinct predictive structure, and exhibits a plateau once kk exceeds the intrinsic dimensionality of the ensemble; the elbow again provides a natural estimate of the number of predictive components, which is the criterion used to select the number of output units for the filters reported in Figure 3. Unlike the generalized Lotka–Volterra case, this stimulus ensemble has no ground-truth set of predictive directions, so the precision and recall of Eq. (39) cannot be evaluated and the objective curve is the only available criterion.

Objective as a function of output components for the dead-leaves stimulus ensemble

Figure 7. Model selection for the dead-leaves stimulus ensemble. Objective (35) as a function of the number of output components kk, for LPD applied to past–future temporal differences of the one-dimensional dead-leaves model of § 4. The elbow provides the estimate of the number of predictive components used to learn the filters in Figure 3.

Robustness to Finite-Difference Lag

We assess the sensitivity of LPD to the finite-difference lag τ\tau, which controls the temporal resolution of the tangent-space approximation. For a fixed system size (N=10N = 10), we construct finite differences as

δxt=xt+τ−xt,(40)\delta x_t = x_{t+\tau} - x_t, \tag*{(40)}

with τ∈{1,5,10,15,20,25}\tau\in\{1, 5, 10, 15, 20, 25\}, and repeat the full learning and evaluation pipeline across different noise levels. Performance is measured using normalized mutual information (NMI) between the inferred and ground-truth transition structures. We find that performance is only weakly affected by the choice of τ\tau, as long as it remains below the characteristic timescales of the system dynamics. This indicates that LPD is robust to moderate changes in temporal discretization.

NMI as a function of finite-difference lag for different noise levels

Figure 8. Robustness to finite-difference lag. NMI as a function of lag τ\tau for N=10N = 10, across different noise levels. Results are reported as mean ±\pm standard deviation over 10 independent trials.

Ablation Studies: Finite Differences and Non-Negativity

We study the contribution of finite differences and the non-negativity constraint in LPD by comparing against three baselines: standard past–future CCA (PF), non-negative PF (NN-PF), and unconstrained finite-differences CCA (FD). All models are trained on gLV trajectories (Figure 2b) and evaluated on independent test trajectories with matching noise. Ground-truth labels correspond to the next visited saddle, while predictions are obtained from the maximal activation of each model’s output. Performance is measured using normalized mutual information (NMI). As shown in Figure 9, LPD achieves higher NMI at low and intermediate noise, converging with FD only at high noise, indicating that both finite differences and non-negativity are essential for optimal performance. We also evaluate clustering performance using Adjusted Rand Index (ARI) and NMI across noise levels and number of components kk. LPD combines two key ingredients: finite differences for translational invariance and a non-negativity constraint. We compare against PF, NN-PF, and FD baselines under varying kk and noise levels. For evaluation, we segment trajectories into ground-truth labels corresponding to the next saddle visited in the SHC dynamics (gLV system). Each time point is assigned the label of the next saddle based on alignment of the finite difference

Clustering performance across methods under different noise levels

Figure 9. Clustering performance across methods. Adjusted Rand Index (ARI, top) and Normalized Mutual Information (NMI, bottom) as a function of kk under different noise levels. Each curve corresponds to a specific combination of non-negativity (ReLU) and finite-difference preprocessing, i.e., PF, NN-PF, FD, or LPD. Results are reported as mean ±\pm standard deviation over 10 independent realizations.

δxt=xt+τ−xt,τ=1,\delta x_t = x_{t+\tau} - x_t,\qquad\tau= 1,

with known unstable directions. Predicted labels are obtained by selecting the maximal component of each output vector zt\mathbf{z}_t. Performance is quantified using ARI and NMI, measuring clustering agreement with ground truth.

Comparison with Soft Clustering in Tangent Space

We compare LPD with Gaussian Mixture Models (GMMs) in tangent space to assess whether LPD captures predictive structure beyond standard unsupervised clustering. Both methods operate on identical finite-difference features. In particular, for each method we learn a representation Z(t)\mathbf{Z}(t). We run the experiment on the sphere with two-saddle dynamical system (Figure 2c) with noise level σ=0.01\sigma= 0.01. LPD produces embeddings zLPD∈Rk\mathbf{z}_{\mathrm{LPD}} \in\mathbb{R}^{k}, while GMM yields soft assignments zGMM\mathbf{z}_{\mathrm{GMM}} with k=4k = 4 components. In Figure 10, we show an example of trajectories of the dynamical system, the corresponding finite differences, the representations learned by LPD, and the soft assignments learned by the Gaussian Mixture

Example trajectories of the dynamical system, corresponding finite-difference features, and learned representations

Figure 10. Example trajectories of the dynamical system, corresponding finite-difference features, and learned representations. We show embeddings learned by LPD, soft assignments from a Gaussian Mixture Model (GMM) in tangent space, and normalized LPD obtained by enforcing simplex normalization at each time step. This normalization makes the LPD representation directly comparable to probabilistic soft assignments.

Model in tangent space. Finally, we also show the normalized LPD, where at each time step we enforce the components of ZZ to sum to 1, matching the normalization constraint of GMM. We evaluate the predictive information captured by the learned representations by

computing the time-lagged mutual information

I(zt;δxt+h),I(z_t; \delta x_{t+h}),

as a function of the prediction horizon hh, where δxt=xt+τ−xt\delta x_t = x_{t+\tau} - x_t with τ=1\tau= 1. The results are shown in Figure 11, comparing LPD, GMM, normalized LPD, and their corresponding hard assignments obtained by taking the argmax component at each time step. The results

Time-lagged mutual information between learned representations and future finite-difference dynamics

Figure 11. Time-lagged mutual information between learned representations and future finite-difference dynamics as a function of the prediction horizon hh. We compare LPD, GMM, normalized LPD, and their corresponding hard assignments obtained via argmax discretization. Error bars indicate standard deviation across 10 independent runs.

show that LPD captures more predictive information at small lags, while normalized LPD exhibits performance comparable to the Gaussian Mixture Model, as well as to the hard clustering obtained from both LPD and GMM. In particular, we emphasize that LPD is able to capture additional information because it also learns the magnitude of the finite differences, whereas GMM and normalized LPD are constrained by their normalization conditions. At the north and south poles, for example, all output components are close to zero; nevertheless, the neighborhood retains information about which of the two poles the trajectory is near, due to distinct activation patterns that either drive the dynamics toward or away from each saddle. GMM, instead, classifies the north and south poles in the same way, since it only depends on instantaneous values of the finite differences and does not account for the temporal correlations that connect these states to their past and future evolution.

References

  1. [1]George A. Ackerson and King-Sun Fu. On state estimation in switching environments. IEEE Transactions on Automatic Control, 15(1):10–17, 1970.
  2. [2]Valentin Afraimovich, Irma Tristan, Ramon Huerta, and Mikhail I Rabinovich. Winnerless competition principle and prediction of the transient dynamics in a lotka–volterra model. Chaos: An Interdisciplinary Journal of Nonlinear Science, 18(4), 2008.DOI
  3. [3]KS Arun and SY Kung. Balanced approximation of stochastic systems. SIAM journal on matrix analysis and applications, 11(1):42–68, 1990.DOI
  4. [4]Horace B Barlow. Summation and inhibition in the frog’s retina. The Journal of physiology, 119(1):69, 1953.pubmed.ncbi.nlm.nih.gov/13035718
  5. [5]George Keith Batchelor. An introduction to fluid dynamics. Cambridge university press, 2000.
  6. [6]William Bialek, Ilya Nemenman, and Naftali Tishby. Predictability, complexity, and learning. Neural Computation, 13(11):2409–2463, 2001.
  7. [7]Christian Bick and Mikhail I Rabinovich. On the occurrence of stable heteroclinic channels in lotka–volterra models. Dynamical Systems, 25(1):97–110, 2010.DOI
  8. [8]Steven L Brunton, Marko Budišić, Eurika Kaiser, and J Nathan Kutz. Modern koopman theory for dynamical systems. arXiv preprint arXiv:2102.12086, 2021.arxiv.org/abs/2102.12086
  9. [10]Matthew Chalk, Olivier Marre, and Gašper Tkačik. Toward a unified theory of efficient, predictive, and sparse coding. Proceedings of the national academy of sciences, 115(1):186–191, 2018.DOI
  10. [11]Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for gaussian variables. Journal of Machine Learning Research, 6:165–188, 2005.
  11. [12]BG Cleland, WR Levick, and KJ Sanderson. Properties of sustained and transient ganglion cells in the cat retina. The Journal of physiology, 228(3):649–680, 1973.pubmed.ncbi.nlm.nih.gov/4702151
  12. [13]Predrag Cvitanović and Domenico Lippolis. Knowing when to stop: How noise frees us from determinism. In AIP Conference Proceedings, volume 1468, pages 82–126. American Institute of Physics, 2012.DOI
  13. [14]J. Doyne Farmer and John J. Sidorowich. Predicting chaotic time series. Physical Review Letters, 59(8):845–848, 1987.pubmed.ncbi.nlm.nih.gov/10035887
  14. [15]Emily B Fox, Erik B Sudderth, Michael I Jordan, and Alan S Willsky. Nonparametric bayesian learning of switching linear dynamical systems. In Advances in neural information processing systems, pages 457–464, 2009.
  15. [16]Julijana Gjorgjieva, Markus Meister, and Haim Sompolinsky. Functional diversity among sensory neurons from efficient coding principles. PLoS computational biology, 15(11): e1007476, 2019.DOI
  16. [17]Haldan Keffer Hartline. The response of single optic nerve fibers of the vertebrate eye to illumination of the retina. American Journal of Physiology-Legacy Content, 121(2): 400–415, 1938.DOI
  17. [18]Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3/4):321–377, 1936.
  18. [19]Daniel Hsu, Sham M Kakade, and Tong Zhang. A spectral algorithm for learning hidden markov models. Journal of Computer and System Sciences, 78(5):1460–1480, 2012.arxiv.org/abs/0811.4413
  19. [20]D Jeulin. Dead leaves models: from space tesselation to random functions. In Symposium on the Advances in the Theory and Applications of Random Sets, pages 137–156. World Scientific, 1996.
  20. [21]Na Young Jun, Greg Field, and John Pearson. Efficient coding, channel capacity, and the emergence of retinal mosaics. Advances in neural information processing systems, 35: 32311–32324, 2022.DOI
  21. [22]Yan Karklin and Eero Simoncelli. Efficient coding of natural images with a population of noisy linear-nonlinear neurons. Advances in neural information processing systems, 24, 2011.pubmed.ncbi.nlm.nih.gov/26273180
  22. [23]David B Kastner and Stephen A Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature neuroscience, 14(10):1317–1322, 2011.DOI
  23. [24]Tohru Katayama. Subspace methods for system identification. Springer, 2005.
  24. [26]Stephen W Kuffler. Discharge patterns and functional organization of mammalian retina. Journal of neurophysiology, 16(1):37–68, 1953.pubmed.ncbi.nlm.nih.gov/13035466
  25. [27]Daniel D. Lee and H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401:788–791, 1999.pubmed.ncbi.nlm.nih.gov/10548103
  26. [28]Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial intelligence and statistics, pages 914–922. PMLR, 2017.
  27. [29]David Lipshutz, Yanis Bahroun, Siavash Golkar, Anirvan M Sengupta, and Dmitri B Chklovskii. A biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation, 33(9):2309–2352, 2021.DOI
  28. [30]Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):4950, 2018.DOI
  29. [31]Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noé. Vampnets for deep learning of molecular kinetics. Nature communications, 9(1):5, 2018.
  30. [32]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.arxiv.org/abs/1807.03748
  31. [33]Samuel E Otto and Clarence W Rowley. Linearly recurrent autoencoder networks for learning dynamics. SIAM Journal on Applied Dynamical Systems, 18(1):558–593, 2019.arxiv.org/abs/1712.01378
  32. [34]Cengiz Pehlevan and Dmitri B Chklovskii. A hebbian/anti-hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features. In 2014 48th Asilomar Conference on Signals, Systems and Computers, pages 769–775. IEEE, 2014.DOI
  33. [35]Cengiz Pehlevan and Dmitri B Chklovskii. Neuroscience-inspired online unsupervised learning algorithms: Artificial neural networks. IEEE Signal Processing Magazine, 36(6):88–96, 2019.DOI
  34. [36]Cengiz Pehlevan, Tao Hu, and Dmitri B Chklovskii. A hebbian/anti-hebbian neural network for linear subspace learning: A derivation from multidimensional scaling of streaming data. Neural computation, 27(7):1461–1495, 2015.DOI
  35. [37]Cengiz Pehlevan, Alexander Genkin, and Dmitri B Chklovskii. A clustering neural network model of insect olfaction. In 2017 51st Asilomar Conference on Signals, Systems, and Computers, page 593–600. IEEE, IEEE, 2017.DOI
  36. [38]Cengiz Pehlevan, Xinyuan Zhao, Anirvan M Sengupta, and Dmitri Chklovskii. Neurons as canonical correlation analyzers. Frontiers in computational neuroscience, 14:55, 2020.DOI
  37. [39]Maurício Matos Peixoto. Structural stability on two-dimensional manifolds. Topology, 1(2):101–120, 1962.
  38. [40]Joshua L Pughe-Sanford, Xuehao Ding, Jason Moore, Anirvan Sengupta, Charles Epstein, Philip Greengard, and Dmitri Chklovskii. Neurons as detectors of coherent sets in sensory dynamics. Advances in Neural Information Processing Systems, 38:158904–158924, 2026.
  39. [41]Shanshan Qin, Joshua L Pughe-Sanford, Alexander Genkin, Pembe Gizem Ozdil, Philip Greengard, Anirvan M Sengupta, and Dmitri Chklovskii. A network of biologically inspired rectified spectral units (resus) learns hierarchical features without error backpropagation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 2019–2028, 2026.
  40. [42]M Rabinovich, A Volkovskii, P Lecanda, Ramón Huerta, Henry DI Abarbanel, and Gilles Laurent. Dynamical encoding by networks of competing neuron groups: winnerless competition. Physical review letters, 87(6):068102, 2001.
  41. [43]Mikhail I Rabinovich, Pablo Varona, Allen I Selverston, and Henry DI Abarbanel. Dynamical principles in neuroscience. Reviews of modern physics, 78(4):1213–1265, 2006.DOI
  42. [44]Mikhail I. Rabinovich, Ramón Huerta, and Gilles Laurent. Transient dynamics for neural processing. Science, 321(5885):48–50, 2008.DOI
  43. [45]Rajesh PN Rao. A sensory–motor theory of the neocortex. Nature neuroscience, 27(7):1221–1235, 2024.DOI
  44. [46]Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79–87, 1999.pubmed.ncbi.nlm.nih.gov/10195184
  45. [47]Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741 (659-663):3, 2009.
  46. [48]Daniel L Ruderman. Origins of scaling in natural images. Vision research, 37(23):3385–3398, 1997.
  47. [49]Peter J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics, 656:5–28, 2010.
  48. [50]Anirvan Sengupta, Cengiz Pehle­van, Mariano Tepper, Alexander Genkin, and Dmitri Chklovskii. Manifold-tiling localized receptive fields are optimal in similarity-preserving neural networks. Advances in neural information processing systems, 31, 2018.DOI
  49. [51]Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: a fresh view of inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences, 216(1205):427–459, 1982.pubmed.ncbi.nlm.nih.gov/6129637
  50. [52]Steven H. Strogatz. Nonlinear Dynamics and Chaos. Westview Press, 1994.
  51. [53]Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1710.04340
  52. [54]Floris Takens. Detecting strange attractors in turbulence. In David Rand and Lai-Sang Young, editors, Dynamical Systems and Turbulence, Warwick 1980, volume 898 of Lecture Notes in Mathematics, pages 366–381. Springer, 1981.DOI
  53. [55]Frank S Werblin and John E Dowling. Organization of the retina of the mudpuppy, necturus maculosus. ii. intracellular recording. Journal of neurophysiology, 32(3):339–355, 1969.pubmed.ncbi.nlm.nih.gov/4306897
  54. [56]Hao Wu and Frank Noé. Variational approach for learning markov processes from time series data. Journal of Nonlinear Science, 30(1):23–66, 2020.arxiv.org/abs/1707.04659
  55. [57]George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2114–2124, 2021.arxiv.org/abs/2010.02803

Paper details

vlnc:W_jwvnwjm4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Agricultural and Biological Sciences

Not recorded for this paper

View statement
Looking for related papers…
Contents