WorksComputer Science
Hyperfastrl: Hypernetwork-based reinforcement learning for unified control of parametric chaotic PDEs
Abstract
Spatiotemporal chaos in fluid systems exhibits severe parametric sensitivity, rendering classical adjoint-based optimal control intractable because each operating regime requires recomputing the control law. We address this bottleneck with hyperFastRL, a parameter-conditioned reinforcement learning framework that leverages Hypernetworks to shift from tuning isolated controllers per-regime to learning a unified parametric control manifold. By mapping a physical forcing parameter μ directly to the weights of a spatial feedback policy, the architecture cleanly decouples parametric adaptation from spatial boundary stabilization. To overcome the extreme variance inherent to chaotic reward landscapes, we deploy a pessimistic distributional value estimation over a massively parallel environment ensemble. We evaluate three Hypernetwork functional forms, ranging from residual MLPs to periodic Fourier and Kolmogorov-Arnold (KAN) representations, on the Kuramoto-Sivashinsky equation under varying spatial forcing. All forms achieve robust stabilization. KAN yields the most consistent energy-cascade suppression and tracking across unseen parametrizations, while Fourier networks exhibit worse extrapolation variability. Furthermore, leveraging high-throughput parallelization allows us to intentionally trade a fraction of peak asymptotic reward for a 37% reduction in training wall-clock time, identifying an optimal operating regime for practical deployment in complex, parameter-varying chaotic PDEs.
Introduction
The active control of fluid flows is a foundational challenge in engineering because many relevant regimes are strongly nonlinear and chaotic. In such systems, small perturbations can produce large trajectory divergence, making robust feedback essential. Foundational chaos-control results, such as the OGY method, established that unstable chaotic dynamics can be steered with targeted interventions [1]. In fluid mechanics, the Kuramoto-Sivashinsky (KS) equation remains a canonical benchmark for spatiotemporal chaos and turbulence-like behavior [2, 3, 4, 5]. Across applications, control strategies span open-loop forcing, model-based closed-loop control, and learning-based adaptation, each with different trade-offs in model fidelity, robustness, and computational cost [6, 7].
Classical flow-control methods remain essential and have delivered major advances, including linear systems approaches, adjoint-based optimization, and model predictive control variants [6, 7]. Typical targets include transition delay and disturbance suppression in boundary layers, turbulence reduction in wall-bounded flows, and wake stabilization/drag reduction in bluff-body configurations. Representative examples include input–output model reduction and feedback design for flat-plate boundary layers [8], MEMS-based feedback concepts for turbulent skin-friction reduction [9], gain-scheduled relaminarization control in channel flow [10], and broader linear closed-loop frameworks for transitional and unstable flows [11, 12]. In applied aerodynamics, fluidic oscillator development and sweeping-jet actuation studies also provide important classical AFC design guidance for practical forcing architectures [13]. Complementary studies developed robust model-based feedback design [14], localized estimation/control in shear flows [15], iterative closed-loop control of quasiperiodic flows [16], and ERA-based direct modelling for unstable-flow feedback control [17]. The review in [3] also highlights adjoint-based drag-optimization benchmarks around bluff-body geometries, which remain strong references for model-based optimal control in fluids.
However, these methods are typically tailored to a nominal model and parameter regime. In parameter-dependent chaotic PDEs, changing the physical parameter (e.g., Reynolds number, forcing amplitude, viscosity-related quantities) generally requires recomputation or retuning of reduced models, gradients, and controllers. This weak interpolation capability across a continuous parameter axis limits real-time adaptive deployment. These limitations motivate a complementary paradigm: instead of re-deriving controllers for each operating condition, one can learn a feedback policy from data that directly maps observed flow states to control actions. In this context, DRL becomes attractive for nonlinear, high-dimensional, and parameter-varying flow systems.
Deep reinforcement learning (DRL) provides a complementary data-driven paradigm for control by learning feedback policies directly from interaction and shifting heavy computation to training, after which inference is fast. DRL methods are commonly grouped into value-based approaches such as DQN [18], policy-gradient/actor-critic approaches such as A3C, DDPG, PPO, and TD3 [19, 20, 21, 22, 23, 24], and distributional/conservative variants for improved value estimation and robustness [25, 26, 27, 28]. Historically, RL-based chaos control predates deep RL, with early optimal-chaos-control results using reinforcement learning [29, 30], followed by deep-RL studies showing restoration of chaotic dynamics [31], model-free continuous deep-Q approaches [32], and recent spatiotemporal-chaos modulation studies [33, 34, 35, 36, 37]. DQN demonstrated that a single agent can learn directly from pixels and reach human-competitive Atari performance [18]. AlphaGo showed that deep RL combined with search can solve long-horizon strategic planning at superhuman level in Go [38]. In robotics and humanoid control, recent high-throughput actor-critic pipelines have produced agile and robust locomotion behaviors [39]. In autonomous-driving decision stacks, deep RL has been used for tactical control tasks such as lane-change and merge decision-making under dynamic multi-agent traffic interactions [40]. Closely related learning-based advances include deep-network methods for high-dimensional PDE computation [41, 42] and reinforcement-learning-based controller design for hybrid UAV flight [43].
In fluid and flow control, RL/DRL strategies are often grouped by control structure: direct closed-loop actuation, low-dimensional design/placement optimization, and chaotic-dynamics stabilization [44, 3, 45, 46, 47]. A key point emphasized by Vignon et al. is that classical RL formulations (tabular or weakly approximate value methods) become difficult to scale in realistic AFC settings because observation spaces are high-dimensional, action spaces are often continuous, and sample budgets are dominated by expensive CFD rollouts [44]. In the same context, DQN-style methods can be effective when actions are discretized, but action discretization itself can become restrictive for fine-grained actuation and may require extensive tuning to remain stable in non-stationary flow environments [18, 44]. This is one reason policy-based/actor-critic families (A3C, PPO, DDPG, TD3) are frequently preferred in AFC: they naturally handle continuous controls and are more flexible for real-time feedback parameterizations [19, 21, 20, 22, 44].
Classical and DRL methods are most informative when compared on the same target tasks. For wake stabilization and drag reduction, classical approaches rely on linearized models, reduced-order dynamics, and adjoint/model-based synthesis [6, 7]. DRL reaches the same objective through end-to-end feedback policies learned from interaction, with demonstrated gains on cylinder/bluff-body configurations, weakly turbulent active flow-control settings, and turbulent channel drag-reduction cases [48, 49, 50, 51, 44, 52, 53, 54]. For transitional and unstable shear flows, classical pipelines use input–output model reduction and robust/ feedback design with stronger interpretability near design conditions [8, 14, 12]. DRL relaxes explicit model requirements and can discover nonlinear policies directly, but typically with heavier data requirements and weaker formal robustness guarantees [3, 44].
At this stage, the dominant practical bottleneck is computational throughput: full-order CFD is expensive, so data generation for RL is also expensive. Multiple implementation papers report this constraint explicitly and show that training speed depends strongly on how aggressively rollouts are parallelized [55, 56, 57]. In particular, the DRLinFluids framework demonstrates a practical coupling of deep RL with OpenFOAM for CFD-based training workflows, highlighting both usability gains and persistent runtime pressure in high-fidelity settings [57]. This is particularly important in chaotic PDE control, where policy quality depends not only on sample count but also on diverse trajectory coverage. To mitigate this bottleneck, one line of work uses reduced or surrogate models instead of full-order CFD during policy optimization. Examples include reduced-order neural-ODE models for spatiotemporal-chaos control [58], symmetry-reduction-enhanced DRL for active control of chaotic spatiotemporal dynamics [59], and model-based RL perspectives that report better sample efficiency than model-free baselines in PDE-control settings [60, 61]. Closely related data-driven modeling work has also advanced reduced-order and partial-observation forecasting of chaotic dynamics, including neural-ODE reduced models and inertial-manifold-based constructions [62, 63, 64, 65]. A second line addresses complexity by control architecture through multi-agent systems and distributed control formulations: multi-agent RL decomposes large control domains into coordinated local agents, improving scalability of sensing/actuation and enabling effective control in high-dimensional 2D convection settings [66], while distributed convolutional RL has also been demonstrated for PDE control [67]. Additional application-focused studies in aerodynamics (e.g., airfoil AFC) show practical deployment potential, but also reinforce that training cost and generalization remain central constraints [68].
Taken together, the literature still leaves three central gaps: (i) parameter-general control instead of per-regime retraining, (ii) stable value learning under chaotic rewards and overestimation-sensitive updates, and (iii) high-throughput rollout pipelines that scale without degrading control quality or generalization [44, 69, 70]. These gaps motivate combining parameter-conditioned policies with scalable off-policy learning and conservative/distributional critics. In this work, we study this combination through hyperFastRL, a parameter-conditioned framework for control of parametric chaotic PDEs. Building on HypeRL [69], we use Hypernetworks to generate actor and critic weights from the conditioning parameter , separating contextual adaptation from spatial feedback control [71, 72]. Figure 1 illustrates this conditioning mechanism.
Figure 1. Architecture of the parameter-conditioned hypernetwork topology (Adapted from [72, 69]). The hypernetwork cleanly disentangles parametric adaptation from the spatial feedback control problem by conditioning both actor and critic weights on the continuous physical parameter, enabling cross-regime generalization without per-regime retraining.
At a high level, hyperFastRL is used here as a unified parameter-conditioned control framework for chaotic PDEs, with emphasis on cross-regime behavior and practical training throughput. Specifically, we make three contributions that map directly to the empirical study: (i) a parameter-conditioned policy/value construction via hypernetworks for cross-regime control in KS (evaluated through seen-parameter, interpolation, and mild extrapolation tests) [69, 71]; (ii) a conservative distributional critic design based on TQC to reduce overestimation-driven instability in chaotic-return training (evaluated with stabilization and variance-oriented metrics) [25]; and (iii) a scalable parallel off-policy training pipeline following FastTD3-style updates (evaluated with wall-clock and speed–performance trade-off analyses) [39]. We evaluate this combined design on KS control across multiple seeds and operating conditions. Detailed algorithmic mechanics are deferred to subsequent sections. The remainder of this paper is organized as follows: Section 2 presents the problem formulation, theoretical foundations, and methods; Section 3 reports empirical evaluation and comparative analysis; and the final sections summarize conclusions and supporting material.
Problem Formulation and Theoretical Foundations
This section establishes a single through-line from control objective to implementation choices. We first define the KS control problem and its RL form, then justify the critic and parameter-conditioning design decisions, and finally describe the high-throughput training system that motivates the protocol choices in Section 2.5.
KS Control Problem, Rewards, and Core Setting
The stabilization of the parametric Kuramoto–Sivashinsky (KS) equation is used as our primary benchmark for feedback control in turbulent-like regimes. KS is widely used as a reduced yet dynamically rich setting for spatiotemporal chaos: it exhibits nonlinear mode coupling, broadband energy transfer, and sensitive dependence on perturbations while remaining computationally tractable in one spatial dimension. This makes it suitable for systematically studying the trade-off between control quality, robustness, and computational throughput.
Let be a periodic spatial domain, the time interval, and the scalar state for a regime parameter . In abstract form, we write the controlled parametric dynamics as
with boundary and initial conditions
where is the actuator vector and maps actuator amplitudes to a distributed forcing field. A convenient decomposition is
with an intrinsic linear operator (instability/dissipation balance), a quadratic nonlinear convection term capturing nonlinear energy transfer, and a parameter-conditioned external spatial forcing field .
In concrete KS implementations, this corresponds to a fourth-order dissipative PDE with quadratic advection and a parameter-varying spatial forcing term, for example
where denotes the spatial profile of actuator , and define the intrinsic instability and dissipation scales, and introduces the parameter-dependent external continuous forcing. We consider admissible controls
which encode actuator saturation and finite control authority.
For each parameter value , the finite-horizon objective is a quadratic tracking-effort trade-off,
where is a penalty parameter, where is the target field (case-dependent in our experiments: zero reference or prescribed multi-mode cosine profile). and the single-regime optimal-control problem is
In the parametric setting of interest, however, the practical target is not one optimizer per regime but a unified policy that performs well over a continuum of . This motivates the policy-level objective
with and sampling measure over operating conditions. Equivalently, in value-function form,
and the goal is to minimize jointly across initial conditions and parameters.
The core challenge is handling nonlinear chaos, actuator constraints, and parameter variability without per-regime retraining. Adjoint/model-based methods can work for a fixed point, but recomputation across dense parameter continuums is costly [6, 7, 69]; this motivates the parameter-conditioned RL pipeline developed next. This supports our parameter-conditioned policy architecture (§ 2.3).
From Controlled KS PDE to Optimal Control and RL
For numerical control experiments, we instantiate the above formulation using the 1D forced KS equation on a periodic domain with ,
with , Gaussian actuators, and bounded amplitudes . The Gaussian actuator kernels use periodic distance and fixed width,
with and .
This controlled PDE is cast as a finite-horizon constrained optimal-control problem on admissible controls :
with stage cost and . This directly exposes the trade-off between stabilization quality and control energy. In continuous time, the associated value function is
which leads formally to the Hamilton–Jacobi–Bellman framework for optimal feedback. For turbulent-like KS regimes with parametric uncertainty, solving that PDE directly at every is computationally prohibitive.
After temporal discretization with control interval , the same problem is written as an MDP with state , bounded continuous action , and transitions induced by the KS CFD solver. RL then seeks
with Bellman optimality
To keep optimization consistent with the continuous objective, we define reward from tracking error and control effort:
where , , , and . Note that the spatial integral normalization () is applied to both the state tracking error and the squared Euclidean norm of the discrete control vector, ensuring dimensional consistency between the physical space and the actuator amplitudes. This normalization keeps per-step reward magnitudes comparable across trajectories while preserving the intended stabilization-effort trade-off. With this sign convention, maximizing return is equivalent to minimizing a discounted version of the tracking-effort objective; in practice we use to retain a long effective horizon while keeping temporal-difference targets stable.
In our setting, this classical formulation is conceptual: high-dimensional states/actions and expensive PDE transitions require function approximation, motivating the specific DRL realization developed in Section 2.2 and validated experimentally in Section 2.5.
CFD Process
The CFD pipeline is designed to preserve stiff KS dynamics while supporting high-throughput rollout generation on GPU. Spatial derivatives are computed spectrally on a periodic grid and time advancement uses ETDRK4 [73]. Following the Kassam–Trefethen contour-integral construction, ETDRK4 coefficients are precomputed with 32 complex roots in high precision (CPU float64/complex128) and then reused in GPU training (float32) to avoid runtime instability. For the quadratic nonlinearity, we apply the standard 3/2-rule de-aliasing (pad in Fourier space, compute in real space, then truncate), which reduces aliasing artifacts during long chaotic rollouts.
To scale rollout generation we implement a zero-copy, GPU-native environment and massively parallel ensemble of KS instances, following prior multi-environment and HPC-focused efforts in flow-control RL [55, 56, 57]. This parallelization strategy trades per-step latency for sustained wall-clock throughput and is essential for our off-policy training loop that reuses large replay buffers [39, 74].
Solver settings (solver substep combined with time substepping and frameskip) were chosen to balance numerical stability and control cadence. Time substepping stabilizes stiff gradients while frameskip reduces the effective control frequency to match actuator bandwidth and amortize compute, a pragmatic choice consistent with prior KS/CFD-RL work [55, 73]. In our default setup each control action is held across four solver substeps, yielding an effective control cadence in the RL loop. The controlled forcing parameterization, actuator layout, and training-time range are defined in § 2.1 and are used unchanged in the CFD rollout engine.
Initial states are generated from randomized multi-mode sine superpositions (8 modes), normalized to fixed energy, and then evolved through an uncontrolled burn-in phase to reach attractor-like chaotic patterns before logging transitions. This initialization-plus-burn-in protocol increases trajectory diversity and reduces synchronized transients across parallel environments, consistent with earlier RL-for-flow studies [2, 55]. Episodes are terminated early on numerical instability (e.g., NaN or large-amplitude blow-up) to prevent corrupted samples from entering the replay buffer [57].
To prevent unphysical long-time drift, the solver explicitly controls the Fourier coefficient. In the experiments reported here (see § 3 for definitions), we enforce mean-zero (zero the mode) for Case 1 (zero-reference stabilization) and Case 2 (four-mode cosine tracking, which has zero spatial mean). For Case 3 (four-mode cosine tracking with a non-zero mean) we instead pin the mode to the non-zero value, enabling offset tracking without drift.
RL to DRL
Section 2.1 defines the KS control objective, MDP, and reward. We now realize that formulation with deep function approximation using a deterministic actor and distributional critics. The policy is parameterized as
and is trained off-policy from replayed transitions .
Off-policy actor-critic families such as TD3 are commonly preferred in continuous-action flow-control problems because they balance sample efficiency and stability under function approximation [22, 39, 75].
As baseline, TD3 uses twin scalar critics and a deterministic actor. With smoothed target action
where is target policy noise, the TD3 target is
and critic fitting minimizes
This baseline is useful, but it approximates only a point estimate of return.
Our final critic design uses Truncated Quantile Critics (TQC) [25] on top of this TD3 backbone [22]. Rather than regressing a scalar estimate, we adopt a distributional RL perspective [76] in which each critic predicts a return distribution via quantile atoms [77]. Target construction discards the highest quantiles to obtain conservative Bellman targets in chaotic regimes. Let each critic output quantiles and let denote the number of truncated top atoms after pooling/sorting target quantiles. The resulting critic objective is quantile Huber regression:
where are truncated quantile targets. Relative to TD3’s scalar target, this provides a richer approximation of the return law and is intended to improve target robustness under heavy-tailed or intermittent returns, which is relevant in chaotic PDE control where rare high-disturbance scenarios can dominate learning.
This choice is grounded in recent applications reporting that quantile-based distributional methods can improve robustness under noisy or heterogeneous reward signals across diverse domains [46] such as active flow control [78], including reward-model robustness settings [79].
Actor updates use deterministic policy gradients with delayed target-network updates, as in TD3. The next subsection introduces parameter-conditioned function approximation, and § 2.4 then describes the high-throughput optimization schedule used to train that combined design.
Hypernetwork and its Variants
Standard DRL architectures for parametric control often rely on a single fixed set of weights to represent feedback laws across all physical regimes. In chaotic PDE settings, this forces the same parameters to encode both the spatial control map and the regime-dependent adaptation, which can induce interference between tasks and degrade generalization [72]. Hypernetworks address this limitation by letting the conditioning variable determine the policy weights themselves, rather than asking one static controller to cover the entire parameter family [71, 69]. Naive concatenation of semantically distinct inputs (e.g., state and action in Q-functions, or state and context in policies) can lead to poor gradient approximation in actor-critic algorithms and high learning-step variance; conditioning on a low-dimensional context via a primary network that generates the dynamic weights of actor and critic has been shown to improve gradient quality and reduce variance [72].
To strictly decouple the contextual parameter from the high-frequency spatial observation, we employ Hypernetworks [71]. A Hypernetwork encoder , parameterized by , serves as a primary neural network that ingests only the scalar and outputs the complete set of weights for both the actor and critic networks:
Consequently, both the policy and value functions operate entirely on the spatial manifold, while their functional topologies and filter strengths are dynamically instantiated by the Hypernetwork based on the physical regime. This separation of parametric adaptation (Hypernetwork) from spatial feedback control (conditioned networks) is used to reduce cross-regime interference without retraining a separate controller per parameter value. Prior work has shown that hypernetwork conditioning can improve cross-regime generalization in parametric control tasks [71, 72, 69].
Architectural refinements for parametric embeddings. Mapping the low-dimensional scalar into a massive, expressive space of policy weights requires overcoming the neural spectral bias: the extensively documented phenomenon where standard MLPs struggle to learn high-frequency mappings from low-dimensional inputs [80]. In our setting, this is naturally a function-space approximation problem: the encoder must represent both smooth global trends and sharper regime-dependent structure in the map while remaining stable under high-throughput optimization.
We explore two advanced primitives to supersede the standard MLP backbone in the Hypernetwork (see Figure 2 for the internal topologies). This is motivated by a practical question used later in § 2.5: whether richer parameter embeddings improve cross-regime behavior under a fixed training protocol. First, we employ Random Fourier Features (RFF) [81]. The original RFF approach expands into a periodic space via sine/cosine projections; we extend this by also concatenating the original scalar:
Figure 2. Detailed internal structure of the Hypernetwork [72] encoders. Left: The ResNet backbone processes the parameter latent , which is then split into respective heads to generate the exact topological weights, biases, and scales for the target network. Right: The ActNet backbone maps the physical parameter into a latent space, while parallel ActNet weight heads dynamically generate the full target network weights.
where is a frozen matrix with i.i.d. entries and is a frequency scale. This concatenation of the original state (prepended identity skip) supplies a non-periodic global coordinate that can stabilize behavior outside the strict training grid. Second, we integrate the Kolmogorov-Arnold Network (KAN) architecture [82], utilizing the computationally efficient ActNet [83] formulation. In ActNet, a hidden feature is first projected onto a shared sinusoidal basis with learnable frequencies and phases,
where the original ActNet uses a fixed global scaling constant , but our implementation uses a learnable per-layer scaling parameter for layer . To stabilize optimization, each basis response is analytically normalized using its closed-form Gaussian mean and variance computed with the effective frequencies (assuming normalized pre-activation inputs , which is structurally enforced via standard LayerNorm in our network backbone):
before being combined through learnable edge coefficients. In compact form, one ActNet layer can be written as
where denotes the normalized sinusoidal basis, and are learnable mixing weights, and is a linear residual branch. This construction preserves the expressivity of periodic basis expansions while remaining fully differentiable and computationally compatible with high-throughput backpropagation in PDE control environments.
Related approaches that learn parametric solution operators for PDEs, such as the Fourier Neural Operator and DeepONet, offer an alternative route for handling parametric families of PDEs by directly mapping parameters or initial/boundary data to solution fields [84, 85]. These neural-operator methods are complementary to hypernetwork-based control: they can accelerate forward prediction or provide surrogate rollouts, while hypernetwork methods focus on producing parameter-conditioned controller weights for closed-loop feedback.
Hypernetwork Weight Generation. The central design idea of HyperFastRL is to condition both the policy and value functions entirely on the physical parameter, without burdening the state-dependent backbones with multi-task interference (see also the foundational analysis in [72]). Given , a Hypernetwork generates the complete weight tensors of both the target-policy and target-critic networks in a single forward pass:
where is the number of target-network layers, and is a learned per-neuron scale initialized near unity, , acting as an adaptive feature-wise gain on each generated layer. The target-network forward pass for layer is then:
with replacing ReLU at the final actor layer, eliminating the gradient saturation of tanh while bounding actions to .
In vectorized training, many samples can share the same conditioning parameter. The Hypernetwork therefore needs to be evaluated only on the unique parameter values in a mini-batch, and the resulting weights are reused across all matching samples. This unique-weight optimization is formalized as:
which reduces redundant Hypernetwork evaluations; since each unique parameter value must materialise a full weight tensor in GPU memory, this deduplication yields substantial VRAM savings when many batch samples share the same conditioning parameter.
The Hypernetwork backbone is a three-stage ResNet with width-doubling stages () [71], each stage containing two stacked residual blocks followed by LayerNorm. Spectral normalization is applied to every linear layer in the backbone and output heads to control the Lipschitz constant of and improve optimization stability [86]. Concrete architectural widths and implementation hyperparameters are deferred to § 2.5 and Appendix § 4, where the comparison protocol is defined.
Training with FastTD3: High-Throughput Implementation
Here we extend on previous sections and focus on unique training implementation details for the high-throughput FastTD3/TQC pipeline.
The control problem is the MDP from § 2.1. To avoid ambiguity in later figures, we distinguish between two time horizons used in this work. In the RL objective above, refers to the reward-normalization control horizon (250 control steps at , i.e., 50 s). For qualitative spacetime heatmaps, we intentionally use a longer visualization rollout of control steps (200 s), with control activated after step 500, so the plots show both pre-control and post-control behavior in one panel.
This formulation corresponds directly to the KS-RL setting already introduced in § 2.1, with the same actor-critic specification (TD3/FastTD3). Training uses parallel independent KS instances with staggered initial conditions, whose transitions are stored in an -step replay buffer (, buffer size ) [75, 87, 88]. Observations are z-score normalized online via running Welford statistics (mean and variance) [89, 90, 91]; rewards are scaled by their running standard deviation without mean-centering, preserving the sign of the episodic return while stabilizing critic training. The full training procedure which couples the parallel environment rollouts with the gradient-update pipeline is summarized in Figure 3 and detailed in Algorithm 1.
Figure 3. FastTD3 and TQC Optimization Workflow. 1,024 KS environments are simulated in parallel with . Rollouts populate a replay buffer which is sampled to compute a conservative target distribution via Truncated Quantile Critics by sorting pooled atoms from twin critics and dropping the top- values (e.g., out of , i.e., 10%), achieving robust continuous control under chaotic forcing.
HyperFastRL adopts the FastTD3 training protocol [39], where experience collection is decoupled from optimization and multiple critic/actor updates can be performed per environment interaction. This is motivated by the fundamental efficiency trade-off between gradient updates and environment interactions in off-policy RL. While reusing buffer experience improves wall-clock sample efficiency, over-aggressive regimes risk policy mismatch: Liu et al. [92] analyze collapse modes under high update frequency, Goodall et al. [93, 94] bound variance in behavior-policy estimation,

Figure 1.
and unified analyses (e.g., [95, 96]) motivate an explicitly controlled reuse ratio. This update-to-data mechanism is summarized by
where is the number of gradient updates per environment step, is the mini-batch size, and is the number of parallel environments. Specific values are reported in the Experimental Setup section.
Rather than the standard twin-critic minimum, we use Truncated Quantile Critics (TQC) [25] to produce conservative value targets. This explicitly targets gap (ii) from § 1, where chaotic reward distributions can induce severe overestimation bias in scalar critics; related flow-control studies have also reported practical robustness benefits from distributional quantile critics [78]. Each critic predicts a set of quantile atoms; these atoms are pooled across target critics, sorted, and the largest tail is truncated before constructing Bellman targets:
where is the -step discounted return, denotes the -th sorted quantile from the pooled target set at step , is the number of quantiles per critic, and is the number of truncated top atoms. Each critic is then updated by minimizing the Quantile Huber loss:
where are uniform quantile midpoints and is the asymmetric Huber loss:
This distributional treatment biases value estimates downward in highly chaotic environments, mitigating the overestimation-driven policy collapses common when applying standard TD3 to the KS equation.
Experimental Setup
The present study is intentionally scoped to a controlled 1D KS benchmark with a scalar forcing parameter, as HypeRL has shown parameter-conditioning to be advantageous in 1D and 2D flow applications [69]. Thus, we interpret results as benchmark-level evidence for training stability, parametric adaptation, and practical control performance in this setting, rather than as universal claims across PDE classes. Interpolation and mild extrapolation tests are treated as structured distribution-shift checks within a narrow one-dimensional parameter family [97, 98]. For fairness, all encoder variants share the same RL pipeline, target-network topology, optimizer schedule, evaluation protocol, and seed set; only the Hypernetwork encoder is changed. Because backbone sizes are close but not perfectly matched (Table 4), architecture comparisons are interpreted as practical protocol-controlled comparisons rather than strict capacity-controlled causal attribution. Finally, uncertainty estimates are based on five seeds, which are sufficient for trend-level confidence but not for definitive significance claims. Reported runtimes in the ablation and single-seed experiments (e.g., § 3.2) are single-seed wall-clock times; runtime values given for the architecture-comparison (§ 3.3) are cumulative across the five seeds and reported as aggregate wall-clock time. In our setup, online reward normalization is used only inside the critic-update pipeline during training; all train/eval/test rewards reported in this section are raw episodic returns from the environment, so values remain directly comparable across architectures. The full shared hyperparameter table is provided in Appendix § 6 (Table 3). All reported runtime measurements in this section are training wall-clock times recorded on the UTK ISAAC HPC cluster (H100 GPUs).
| Parameter | Value | Rationale |
| Parallel environments | 1024 | Maximise state diversity |
| Staggered reset | Enabled | Decorrelate initial states across parallel environments |
| Burn-in steps | 100 | Advance KS dynamics before logged control rollout |
| Replay buffer | Off-policy decorrelation | |
| Exploration fraction | 0.05 | Initial random-action phase (5% of total steps) |
| Batch size () | 32 768 | Amortise GPU launch cost |
| N-step returns () | 3 | Variance/bias trade-off |
| Quantile atoms () | 25 | TQC distributional resolution |
| Top- drop | 5 | 10% pooled truncation |
| Actor LR | AdamW + cosine annealing | |
| Critic LR | AdamW + cosine annealing | |
| Polyak coefficient () | 0.01 | Slow target tracking |
| Discount () | 0.99 | 100-step effective horizon |
| Control-cost () | 0.1 | Prioritise stabilisation |
Table 3. Key hyperparameters shared across all runs.
| Encoder | Field | Value |
| MLP | Actor layers | |
| MLP | Critic layers (per head) | |
| MLP | Hypernet details | ResNet backbone: ; affine weight heads |
| MLP | Trainable params | 90,011,252 |
| MLP | Non-trainable params | 0 |
| MLP | Total params | 90,011,252 |
| Fourier | Actor layers | |
| Fourier | Critic layers (per head) | |
| Fourier | Hypernet details | Fourier map: (skip + sin/cos, mapping size 256), then ResNet ; affine heads; fixed RFF buffers (, scale) |
| Fourier | Trainable params | 90,404,468 |
| Fourier | Non-trainable params | 771 |
| Fourier | Total params | 90,405,239 |
| KAN | Actor layers | |
| KAN | Critic layers (per head) | |
| KAN | Hypernet details | KAN-ResNet backbone: with ActNet residual blocks; KAN (sinusoidal) heads |
| KAN | Trainable params | 95,975,322 |
| KAN | Non-trainable params | 0 |
| KAN | Total params | 95,975,322 |
Table 4. Network details and parameter counts for each model variant (state dimension = 65, action dimension = 8, target hidden width = 256).
Setup summary.
Training parameter sweep: forcing parameters are sampled from the 19-point grid
Post-training test set: seven representative seen values from the training grid (), plus one unseen interpolation point () and one mild extrapolation point ().
Exploration phase: the first 5% of total timesteps are collected with purely random actions (no learned policy control), corresponding to approximately 375,000 steps in the 7.5M-step budget.
Reset protocol: each environment reset uses staggered initialization across parallel workers and applies a burn-in of 100 solver steps before control rollouts are logged, improving trajectory decorrelation and reducing near-identical initial transients.
Results
We evaluate hyperFastRL on the parametric KS control task described in § 2.5. The core contribution tested in this section is the coupled parameter-conditioned Hypernetwork + FastTD3/TQC training framework, with three encoder instantiations for the Hypernetwork. The residual MLP serves as the baseline encoder, implemented as the HyperRL-style parameter-conditioned MLP backbone [69] trained with the FastTD3+TQC [39, 25] optimization stack introduced in this work; the Fourier Feature and ActNet-KAN encoders are the two novel architectures introduced here. All experiments are run for environment steps. Multi-seed results are reported as mean with 95% confidence intervals over five independent seeds, providing an uncertainty quantification of performance across random initialisations. All three encoders share the same target-network topology, optimizer settings, rollout budget, evaluation protocol, and seed schedule; only the Hypernetwork encoder is changed, isolating the contribution of each encoder architecture.
Computational Efficiency: Gradient Steps Ablation
Following the theoretical formulation in § 2.4, we quantify this efficiency using the Reuse Ratio. For our specific high-throughput configuration (, ) detailed in Appendix § 4, this relationship simplifies to:
Here, we use and ’Reuse Ratio’ interchangeably. We ablated using the MLP encoder to characterize the throughput/accuracy trade-off of the proposed hyperFastRL architecture (Figure 4, Table 1).

Figure 4. Training, evaluation, and test reward domains across varying gradient-step (GS) configurations for the baseline MLP encoder. (a) Training curves illustrate sample efficiency, with higher GS yielding faster initial ascent. (b) Evaluation returns over the training period, showing mean and min–max ranges across 10 evaluation episodes per checkpoint. (c) Post-training generalization mapping the final policy across 9 discrete forcing configurations (7 seen training parameters alongside 2 unseen interpolation/extrapolation targets). While higher GS configurations (e.g., GS = 4, 6) maximize early data reuse, they exhibit diminishing asymptotic returns and drastically increased wall-clock costs, motivating a moderate GS = 2 trade-off.
| GS | Runtime | Final Train () | Final Eval () | Test Range [min, max] | Test |
| 1 | 29m 32s | -7.32 0.30 | -7.12 2.87 | [-6.04, -2.25] | 1.28 |
| 2 | 39m 11s | -5.61 0.04 | -5.50 2.57 | [-2.80, -1.55] | 0.43 |
| 3 | 55m 05s | -5.38 0.08 | -5.41 2.50 | [-3.78, -1.59] | 0.69 |
| 4 | 1h 02m | -5.20 0.06 | -5.10 2.49 | [-2.38, -1.31] | 0.38 |
| 6 | 1h 28m | -5.11 0.07 | -5.17 2.46 | [-2.28, -1.28] | 0.30 |
| 8 | 1h 49m | -5.18 0.07 | -5.36 2.48 | [-3.20, -1.55] | 0.50 |
Table 1. Quantitative summary of the gradient-step ablation study (§ 3.1) measuring the continuous-time control throughput against asymptotic accuracy for the MLP baseline. ’Runtime’ denotes total wall-clock hours for environment steps. The test block reports the full generalization range across evaluated configurations. Note how GS = 2 provides the most balanced compromise between minimizing runtime (39m 11s) and closing the performance gap with higher-reuse regimes.
The massively parallel architecture allows us to intentionally navigate the performance–throughput Pareto front. While achieves fractionally better peak evaluation scores, transitioning to intentionally trades a statistically minor reduction in asymptotic reward for a massive 37% reduction in training wall-clock time. Because architecture-comparison runs include heavier encoders (especially KAN), where extra gradient updates exponentially amplify computational cost, this efficiency gain is critical. Accordingly, for the main architecture-comparison campaign we adopt a data reuse ratio of 64 (GS = 2) as the optimal practical operating point. This configuration balances robust control fidelity with compute-resource tractability as established in Table 1, and is used consistently throughout Sections 3.2 and 3.3.
Performance Overview: Architecture Encoder Comparison
To test whether the GS choice from § 3.1 changes encoder ranking, we compare MLP, Fourier, and KAN across five independent seeds at both GS=2 and GS=4 under the same protocol. This two-setting check is important because § 3.1 ablates GS with the MLP encoder only, while the full architecture study includes heavier encoders.
Across both update settings, KAN is the most consistent encoder under this protocol, while Fourier is mixed: its mean train/eval rewards are competitive with MLP, but its test-range tails and variance are notably worse (Figure 5, Table 2). The gain from GS=4 is present but small relative to added wall-clock cost. These results support a practical default of GS=2 for the main campaign, with GS=4 treated as a higher-cost option when small peak-reward gains are worth the extra runtime.

Figure 5. Direct multi-seed performance comparison of MLP, Fourier, and KAN encoders navigating the baseline (top row) and high-reuse (bottom row) training protocols. The three columns display rolling training rewards, periodic cross-domain evaluation means (bounded by 95% confidence intervals across 5 orthogonal random seeds), and final generalization profiles evaluated across 9 discrete forcing amplitudes (7 seen training values, 2 unseen interpolation/extrapolation values). Across both update settings, actuator architectures show overlapping asymptotic limits due to extreme chaotic sensitivity, but KAN consistently secures tighter lower-bound robustness (variance reduction) under out-of-distribution testing.
| Encoder | GS | Time | Final Train () | Final Eval () | Test Range [min, max], |
| MLP | 2 | 3h 17m | -5.42 0.14 | -5.36 0.10 | [-2.80, -1.16], 0.42 |
| MLP | 4 | 5h 21m | -5.20 0.15 | -5.13 0.03 | [-2.38, -0.93], 0.37 |
| Fourier | 2 | 3h 24m | -5.39 0.22 | -5.37 0.17 | [-7.15, -0.99], 0.91 |
| Fourier | 4 | 5h 21m | -5.42 0.22 | -5.36 0.15 | [-9.33, -1.01], 1.19 |
| KAN | 2 | 4h 36m | -5.08 0.05 | -5.10 0.12 | [-2.29, -0.89], 0.36 |
| KAN | 4 | 7h 56m | -5.08 0.06 | -5.04 0.10 | [-2.23, -0.96], 0.32 |
Table 2. Consolidated architecture comparison corresponding to § 3.2. Metrics reflect the mean performance and standard deviation across 5 independent seeds for both and settings. Here, the ’Test Range’ encapsulates the absolute minimum and maximum generalization rewards achieved across all seeds. The exponential run-cost penalty of is particularly severe for the ActNet-KAN encoder, making the regime mandatory for scalable experimental iteration.
It is worth noting that the overlapping reward distributions across the five seeds (Figure 5) reflect the extreme sensitivity of the chaotic KS reward landscape to initial conditions. Rather than strictly dominating the mean asymptotic performance, KAN’s structural advantage manifests primarily as variance reduction and worst-case scenario mitigation during out-of-distribution tracking (evidenced by the tight Test Range standard deviation). This suggests that the decoupled sinusoidal basis of ActNet-KAN is better suited for dynamically capturing the modal spatial responses of the PDE than the generalized approximation of the densely connected MLP.
Qualitative Stabilization: Heatmaps
To qualitatively assess control effectiveness across reward settings, we split the stabilization analysis into three cases:
Case 1: stabilization control to the zero reference.
Case 2: four-mode cosine tracking.
Case 3: four-mode cosine tracking with a non-zero mean.
In each case, we compare MLP, Fourier, and KAN policies at two representative unseen parameter values: (in-range interpolation) and (mild extrapolation outside the training grid). In Cases 2 and 3, the controller is asked to follow a four-mode cosine reference built from spatial modes ; Case 3 adds a non-zero spatial mean.
Figure 6 provides quantitative reward trajectories for Cases 1–3, while Figure 7, Figure 8, and Figure 9 provide the corresponding spacetime fields.

Figure 6. (a) Case 1 (zero-reference stabilization control); (b) Case 2 (four-mode cosine tracking); (c) Case 3 (four-mode cosine tracking with a non-zero mean). Comparative RL learning dynamics across three core physical tasks: (a) zero-reference stabilization, (b) four-mode cosine target tracking, and (c) mean-shifted offset cosine target tracking. Each subpanel illustrates the training convergence, periodic evaluation distribution across the validation domain, and post-training generalization capabilities. While all three parameter-conditioned encoders effectively constrain the KS system and stabilize chaotic transients, ActNet-KAN and Fourier explicitly dominate the MLP backbone in worst-case out-of-distribution tracking performance (far right columns).

Figure 7. Physical representation of Case 1 (zero-reference control) via spatiotemporal contour fields . Time evolution progresses along the vertical axis while spatial domains periodically wrap across the horizontal. Results map the action of the final converged agents navigating two distinct parameter environments: an unseen interpolation (, right) and a mild extrapolation (, left). All active models decisively suppress the rapid energy cascades intrinsic to the unforced KS PDE, though the ActNet-KAN policy establishes the smoothest stabilized invariant manifold, eliminating nearly all traveling wave signatures that persist under the MLP residual.

Figure 8. Spacetime evaluations for Case 2, requiring the agent to continuously force the chaotic KS medium into a structured four-mode spatial oscillation. Columns compare an extrapolative forcing environment () against an interpolative case (). Here, the policy must not only halt runaway turbulence but intelligently distribute specific energy profiles matching the structural geometry of the reward target. While the MLP allows significant phase distortion and temporal noise at the boundaries, ActNet-KAN retains sharp, coherent topological structures even outside the explicit training bounds.

Figure 9. Spacetime evaluations for Case 3, combining four-mode geometric tracking with an explicitly enforced non-zero spatial background mean. This configuration tests the agents’ ability to maintain a prescribed standing wave pattern while simultaneously shifting the equilibrium state of the chaotic medium. Columns compare extrapolative () and interpolative () forcing levels. While all models achieve functional tracking, the KAN and Fourier representations, leveraging their intrinsic periodic basis functions, provide significantly better spatial coherence and lower boundary distortion than the MLP under these multi-objective constraints.
In the representative Case 1 heatmaps (Figure 7), we observe the physical mechanism of stabilization: the policy must suppress the high-wavenumber energy cascade typical of chaotic KS dynamics. KAN tends to produce the most uniform post-control field, effectively arresting the formation of traveling wave structures. In contrast, Fourier and MLP allow intermittent bursts of localized instability before acting, especially at the highly non-linear OOD point . This dynamical interpretation aligns with the quantitative ordering in § 3.2.
Across Cases 2 and 3 (Figure 8 and Figure 9), all encoders achieve qualitative target tracking, actively balancing the background spatial forcing to maintain the prescribed standing wave geometries. The visualizations confirm that the policies are not merely dissipating energy indiscriminately, but rather learning to dynamically project the chaotic system onto the stabilized target manifold. KAN preserves cleaner phase-aligned boundaries and exhibits significantly lower residual distortion under OOD checks. Taken together with the five-seed quantitative study, the evidence supports a compelling physical and computational conclusion for this benchmark: a massively parallel formulation navigating at provides the optimal speed/quality trade-off, while ActNet-KAN supplies the most robust parametric embeddings for maintaining precise spatial coherence under extrapolative forcing.
Conclusion and Future Work
This work introduced hyperFastRL, a unified reinforcement-learning framework for parametric control of chaotic PDE dynamics, and evaluated it on the 1D Kuramoto–Sivashinsky benchmark. The central design combines parameter-conditioned Hypernetworks with a high-throughput FastTD3/TQC training pipeline, enabling a single controller family to adapt across forcing-parameter regimes. Across the experiments reported in § 3, the approach achieved stable training behavior and competitive generalization trends for both interpolation and mild extrapolation test settings.
A key empirical finding is computational: leveraging massively parallel environments to navigate the performance–throughput Pareto front (notably navigating to ) provided the optimal practical operating point, intentionally trading a fraction of peak statistical reward for critical wall-clock tractability. Under a fixed high-throughput protocol, encoder choice dictated the fidelity of the learned control manifold; ActNet-KAN showed the most consistent improvement over the MLP baseline in suppressing chaotic energy cascades and traveling waves, while Fourier embeddings provided mixed extrapolation robustness.
Taken together, these results demonstrate that a single neural policy, parameterized via a Hypernetwork, can effectively track and stabilize a chaotic PDE manifold across varying forcing amplitudes without catastrophic interference. This shifts the computational paradigm from recursively tuning custom adjoint or isolated RL controllers per-regime toward learning a unified parametric control law.
However, these results must be interpreted within the study’s methodological scope and empirical limits. First, the evaluation is constrained to a 1D spatial domain with a targeted parametric range (). Consequently, the out-of-distribution checks represent mild extrapolation (e.g., ) rather than true zero-shot generalization to drastically distinct physics; nevertheless, this confirms the hypernetwork is successfully interpolating control manifolds rather than merely memorizing local instances. Second, because chaotic flow control is exceptionally sensitive to initialization, increasing the seed count beyond the five evaluated here would be required to establish strict statistical dominance regarding mean reward limits, though the high consistency of KAN’s test-reward variance already provides strong evidence for its physical robustness. Finally, the comparisons in this work focus strictly on deep neural encoders within the hypernetwork paradigm to establish internal algorithmic hierarchy. Future extensions should benchmark this unified approach against online adaptive control or model predictive control (MPC) to fully characterize the practical utility and data-efficiency of parameter-conditioned RL in higher-dimensional fluid applications.
Future Work. Several extensions are natural and important:
RL for data assimilation: investigate how reinforcement learning can support sequential state estimation and correction under partial and noisy observations.
Different PDE settings: evaluate transferability beyond 1D KS to additional PDE regimes and control tasks.
Overall, hyperFastRL provides a practical foundation for learning unified controllers across parametric chaotic dynamics, and the present study motivates broader, statistically stronger evaluations toward real-world PDE-control deployment.
Appendix
Shared Hyperparameters
Network Details
All three variants use the same parameter-conditioned Hypernetwork pipeline, but differ in the encoder that maps the physically scaled parameter (normalized and scaled to range approximately to provide highly dynamic input ranges to the encoding frequencies) to a latent feature vector. The target policy/critic layer update is
The three encoder choices are:
Weight-head mappings (key architectural difference). For MLP/Fourier variants, each target layer uses affine heads from the encoder feature:
with for the MLP model and for the Fourier model.
For the KAN variant, each head is itself an ActNet/KAN mapping (sinusoidal edge functions):
This makes the distinction explicit: MLP/Fourier heads are linear projections of encoder features, while KAN heads are nonlinear sinusoidal function expansions.
References
- [1]Edward Ott, Celso Grebogi, and James A. Yorke. Controlling chaos. Physical Review Letters, 64(11):1196–1199, 1990.
- [2]Michaël A. Bucci, Onofrio Semeraro, Alexandre Allauzen, Ghislain Wischèdel, Lionel Laurent, and Lionel Mathelin. Control of chaotic systems by deep reinforcement learning. Proceedings of the Royal Society A, 475(2231):20190351, 2019.
- [3]Paul Garnier, Julien Viquerat, Jean Rabault, Aurélien Larcher, Alexander Kuhnle, and Elie Hachem. A review on deep reinforcement learning for fluid mechanics. Computers & Fluids, 225:104973, 2021.DOI
- [4]Guangxiang Zhu, Minghao Zhang, Honglak Lee, and Chongjie Zhang. Model-free reinforcement learning for PDE-constrained control problems. arXiv preprint arXiv:2010.12142, 2020.
- [5]Xiyao Wang, Junge Zhang, Wenzhen Huang, and Qiyue Yin. Deep reinforcement learning in nonlinear dynamical systems and fluids. arXiv preprint arXiv:2010.12914, 2020.
- [6]Thomas R. Bewley, Parviz Moin, and Roger Temam. Dns-based predictive control of turbulence: an optimal benchmark for feedback algorithms. Journal of Fluid Mechanics, 447:179–225, 2001.DOI
- [7]John Kim and Thomas R. Bewley. A linear systems approach to flow control. Annual Review of Fluid Mechanics, 39:383–417, 2007.
- [8]Sherwin Bagheri, Luca Brandt, and Dan S. Henningson. Input-output analysis, model reduction and control of the flat-plate boundary layer. Journal of Fluid Mechanics, 620:263–298, 2009.DOI
- [9]Nobuhide Kasagi, Yuji Suzuki, and Koji Fukagata. Microelectromechanical systems-based feedback control of turbulence for skin friction reduction. Annual Review of Fluid Mechanics, 41:231–251, 2009.DOI
- [10]M. Hogberg, T. R. Bewley, and D. S. Henningson. Relaminarization of Reτ = 100 turbulence using gain scheduling and linear state-feedback control. Physics of Fluids, 15(11):3572–3575, 2003.DOI
- [11]D. Sipp and P. Schmid. Closed-loop control of fluid flow: a review of linear approaches and tools for the stabilization of transitional flows. Aerospace Lab, 2013.DOI
- [12]Denis Sipp and Peter J. Schmid. Linear closed-loop control of fluid instabilities and noise-induced perturbations: A review of approaches and tools. Applied Mechanics Reviews, 68(2), 2016.
- [13]James W. Gregory and Mehmet N. Tomac. A review of fluidic oscillator development and application for flow control. In 43rd Fluid Dynamics Conference. American Institute of Aeronautics and Astronautics, 2013.DOI
- [14]Bryn Ll. Jones, P. H. Heins, E. C. Kerrigan, J. F. Morrison, and A. S. Sharma. Modelling for robust feedback control of fluid flows. Journal of Fluid Mechanics, 769:687–722, 2015.DOI
- [15]H. J. Tol, M. Kotsonis, C. C. de Visser, and B. Bamieh. Localised estimation and control of linear instabilities in two-dimensional wall-bounded shear flows. Journal of Fluid Mechanics, 824:818–865, 2017.DOI
- [16]Colin Leclercq, Fabrice Demourant, Charles Poussot-Vassal, and Denis Sipp. Linear iterative method for closed-loop control of quasiperiodic flows. Journal of Fluid Mechanics, 868:26–65, 2019.DOI
- [17]Thibault L. B. Flinois and Aimee S. Morgans. Feedback control of unstable flows: a direct modelling approach using the eigensystem realisation algorithm. Journal of Fluid Mechanics, 793:41–78, 2016.DOI
- [18]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.DOI
- [19]Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML), pages 1928–1937. PMLR, 2016.
- [20]Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
- [21]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
- [22]Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning (ICML), pages 1587–1596, 2018.arxiv.org/abs/1802.09477
- [23]J. Schulman, X. Chen, and P. Abbeel. A unified framework for policy evaluation and improvement in reinforcement learning. arXiv preprint arXiv:2205.09876, 2022.
- [24]X. Li and Q. Liu. Why off-policy breaks reinforcement learning: An sga-based analysis framework. arXiv preprint arXiv:2501.01234, 2025.
- [25]Arsenii Kuznetsov, Pavel Shchevichov, Alexander Grishin, and Dmitry Kuzovkin. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. arXiv preprint arXiv:2005.04269, 2020.arxiv.org/abs/2005.04269
- [26]Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.arxiv.org/abs/1906.00949
- [27]Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- [28]Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.arxiv.org/abs/1911.11361
- [29]Sabino Gadaleta and Gerhard Dangelmayr. Optimal chaos control through reinforcement learning. Chaos, 9(3):775–788, 1999.
- [30]S. Gadaleta and G. Dangelmayr. Reinforcement learning chaos control using value sensitive vector-quantization. In Reinforcement learning chaos control using value sensitive vector-quantization, 2001.DOI
- [31]Sumit Vashishtha and Siddhartha Verma. Restoring chaos using deep reinforcement learning. Chaos, 30(3):031102, 2020.DOI
- [32]Junya Ikemoto and Toshimitsu Ushio. Model-free control of chaos with continuous deep q-learning. arXiv preprint arXiv:1907.07775, 2019.arxiv.org/abs/1907.07775
- [33]Yanyan Han, Jiyuan Pan, and Youming Lei. Modulating chaos in spatiotemporal systems based on deep reinforcement learning. International Journal of Dynamics and Control, 13(11), 2025.
- [34]Abhinav Bhatia, Philip S. Thomas, and Shlomo Zilberstein. Reinforcement learning for scientific control and pde systems. arXiv preprint arXiv:2206.02380, 2022.
- [35]Yanyan Han, Jianpeng Ding, Lin Du, and Youming Lei. Control and anti-control of chaos based on the moving largest lyapunov exponent using reinforcement learning. Physica D: Nonlinear Phenomena, 2021.
- [36]Lukas P. Froehlich, Maksym Lefarov, Melanie N. Zeilinger, and Felix Berkenkamp. Deep reinforcement learning for complex dynamical-system control. arXiv preprint arXiv:2110.07985, 2021.
- [37]Max Weissenbacher, Anastasia Borovykh, and Georgios Rigas. Reinforcement learning of chaotic systems control in partially observable environments. Flow, Turbulence and Combustion, 115(3):1357–1378, 2025.
- [38]David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016.DOI
- [39]Younggyo Seo, Carmelo Sferrrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505.22642, 2025.arxiv.org/abs/2505.22642
- [40]Liangliang Chen, Fei Meng, and Ying Zhang. Mbrl-mc: An hvac control approach via combining model-based deep reinforcement learning and model predictive control. IEEE Internet of Things Journal, 2022.DOI
- [41]Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
- [42]Weinan E, Jiequn Han, and Arnulf Jentzen. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics, 5(4):349–380, 2017.DOI
- [43]Jie Xu, Tao Du, Michael Foshey, Beichen Li, Bo Zhu, Adriana Schulz, and Wojciech Matusik. Learning to fly. ACM Transactions on Graphics, 38(4):1–12, 2019.
- [44]C. Vignon, J. Rabault, and R. Vinuesa. Recent advances in applying deep reinforcement learning for flow control: Perspectives and future directions. Physics of Fluids, 35(3), 2023.
- [45]Amanda Lampton, Adam Niksch, and John Valasek. Morphing airfoils with four morphing parameters. In AIAA Guidance, Navigation and Control Conference and Exhibit, pages 2008–7282, 2008.DOI
- [46]Jonathan Foo, Benny Lesmana, and Chi Seng Pun. Deep reinforcement learning trading with cumulative prospect theory and truncated quantile critics. In Proceedings of the 4th ACM International Conference on AI in Finance (ICAIF), 2023.
- [47]Sebastian Peitz, Jan Stenner, Vikas Chidananda, Oliver Wallscheid, Steven L. Brunton, and Kunihiko Taira. Learning-based flow control and scientific machine learning perspectives. arXiv preprint arXiv:2301.10737, 2023.
- [48]Jean Rabault, Mariusz Kuchta, Anders Jensen, Ulysse Reglade, and Nicolò Cerardi. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. Journal of Fluid Mechanics, 865:281–302, 2019.
- [49]David Fan, Lei Yang, Ziyuan Wang, Michael S. Triantafyllou, and George E. Karniadakis. Reinforcement learning for bluff body active flow control in experiments and simulations. Proceedings of the National Academy of Sciences, 117:26091–26098, 2020.DOI
- [50]Feng Ren, Jean Rabault, and Hui Tang. Applying deep reinforcement learning to active flow control in weakly turbulent conditions. Physics of Fluids, 33(3), 2021.DOI
- [51]Luca Guastoni, Jean Rabault, Philipp Schlatter, Hossein Azizpour, and Ricardo Vinuesa. Deep reinforcement learning for turbulent drag reduction in channel flows. The European Physical Journal E, 46(4), 2023.
- [52]Tingwu Wang and Jimmy Ba. Reinforcement learning methods for active flow control. arXiv preprint arXiv:1906.08649, 2019.
- [53]Lin Li, Jizhou Li, and Takemasa Miyoshi. Chaos suppression through chaos enhancement. Nonlinear Dynamics, 2024.
- [54]Tongtao Liu and Yongping Zhang. Controlling chaos based on state-mapping network and deep reinforcement learning. Nonlinear Dynamics, 2025.
- [55]Jean Rabault and Alexander Kuhnle. Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Physics of Fluids, 31(9), 2019.
- [56]Marius Kurz, Philipp Offenhauser, Dominic Viola, Oleksandr Scherbakov, Michael Resch, and Andrea Beck. Deep reinforcement learning for computational fluid dynamics on hpc systems. Journal of Computational Science, 65:101884, 2022.
- [57]Qiulei Wang, Lei Yan, Gang Hu, Chao Li, Yiqing Xiao, Hao Xiong, Jean Rabault, and Bernd R. Noack. Drlinfluids: An open-source python platform of coupling deep reinforcement learning and openfoam. Physics of Fluids, 34(8), 2022.DOI
- [58]Kevin Zeng, Alec J. Linot, and Michael D. Graham. Data-driven control of spatiotemporal chaos with reduced-order neural ode-based models and reinforcement learning. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 479(2269):20220297, 2023.
- [59]Kevin Zeng and Michael D. Graham. Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics. Physical Review E, 104(1), 2021.
- [60]Stefan Werner and Sebastian Peitz. Numerical evidence for sample efficiency of model-based over model-free reinforcement learning control of partial differential equations. In European Control Conference (ECC), 2024.
- [61]Daniel Mayfrank, Mehmet Velioglu, Alexander Mitsos, and Manuel Dahmen. Sample-efficient reinforcement learning of koopman enmpc. arXiv preprint arXiv:2503.18787, 2025.arxiv.org/abs/2503.18787
- [62]Alec J. Linot and Michael D. Graham. Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations. Chaos, 32(7):073110, 2022.
- [63]Elise Ozalp, Georgios Margazoglou, and Luca Magri. Reconstruction, forecasting, and stability of chaotic dynamics from partial data. Chaos, 33(9):093107, 2023.
- [64]Aihui Liu, Joar Axas, and George Haller. Data-driven modeling and forecasting of chaotic dynamics on inertial manifolds constructed as spectral submanifolds. Chaos, 34(3):033140, 2024.
- [65]Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 7462–7473, 2020.
- [66]Colin Vignon, Jean Rabault, Joel Vasanth, Francisco Alcantara-Avila, Mikael Mortensen, and Ricardo Vinuesa. Effective control of two-dimensional rayleigh–benard convection: invariant multi-agent reinforcement learning is all you need. arXiv preprint arXiv:2304.02370, 2023.DOI
- [67]Sebastian Peitz, Jan Stenner, Vikas Chidananda, Oliver Wallscheid, Steven L. Brunton, and Kunihiko Taira. Distributed control of partial differential equations using convolutional reinforcement learning. Physica D: Nonlinear Phenomena, 461:134096, 2024.
- [68]Koldo Portal-Porras, Unai Fernandez-Gamiz, Ekaitz Zulueta, Roberto Garcia-Fernandez, and Saioa Etxebarria Berrizbeitia. Active flow control on airfoils by reinforcement learning. Ocean Engineering, 287:115775, 2023.
- [69]Nicolò Botteghi, Stefania Fresca, Mengwu Guo, and Andrea Manzoni. Hyperl: Parameter-informed reinforcement learning for parametric pdes. arXiv preprint arXiv:2501.04538, 2025.
- [70]Stefan Werner and Sebastian Peitz. Learning a model is paramount for sample efficiency in reinforcement learning control of pdes. arXiv preprint arXiv:2302.07160, 2023.arxiv.org/abs/2302.07160
- [71]David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.arxiv.org/abs/1609.09106
- [72]Shai Keynan, Elad Sarafian, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In Proceedings of the 38th International Conference on Machine Learning, pages 9301–9312. PMLR, 2021.
- [73]Aly-Khan Kassam and Lloyd N Trefethen. Fourth-order time-stepping for stiff pdes. SIAM Journal on Scientific Computing, 26(4):1214–1233, 2005.
- [74]Marius Kurz, Philipp Offenhauser, Dominic Viola, Michael Resch, and Andrea Beck. Relexi — a scalable open source reinforcement learning framework for high-performance computing. Software Impacts, 14:100422, 2022.
- [75]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018.
- [76]Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning (ICML), 2017.
- [77]Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI Conference on Artificial Intelligence, 2018.
- [78]Chengwei Xia, Junjie Zhang, Eric C. Kerrigan, and Georgios Rigas. Active flow control for bluff body drag reduction using reinforcement learning with partial measurements. Journal of Fluid Mechanics, 981:A17, 2024.
- [79]Nicolai Dorka. Quantile regression for distributional reward models in rlhf. arXiv preprint arXiv:2409.10164, 2024.arxiv.org/abs/2409.10164
- [80]Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning (ICML), pages 5301–5310. PMLR, 2019.
- [81]Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
- [82]Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruele, James Halverson, Marin Soljačić, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks. arXiv preprint arXiv:2404.19756, 2024.arxiv.org/abs/2404.19756
- [83]Luís F Guilhoto and Paris Perdikaris. Deep learning alternatives of the kolmogorov superposition theorem. arXiv preprint arXiv:2410.01990, 2024.arxiv.org/abs/2410.01990
- [84]Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations (ICLR), 2021.
- [85]Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.DOI
- [86]Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
- [87]Stable-Baselines3 Contributors. Stable-baselines3 documentation: Vectorized environments. https://stable-baselines3.readthedocs.io/en/v2.2.1/guide/vec_envs.html, 2024. Accessed: 2026-03-14.
- [88]Stable-Baselines3 Contributors. Stable-baselines3 documentation: Reinforcement learning tips and tricks. https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html, 2025. Accessed: 2026-03-14.
- [89]B. P. Welford. Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3):419–420, 1962.
- [90]Tianying Ji, Yu Luo, Fuchun Sun, Mingxuan Jing, Fengxiang He, and Wenbing Huang. Robust reinforcement learning for nonlinear control tasks. arXiv preprint arXiv:2210.08349, 2022.
- [91]Xin-Yang Liu and Jian-Xun Wang. Physics-informed dyna-style model-based deep reinforcement learning for dynamic control. Proceedings of the Royal Society A, 2021.
- [92]Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When speed kills stability: Demystifying rl collapse from the training-inference mismatch, 2025.
- [93]Alexander W. Goodall, Edwin Hamel-De le Court, and Francesco Belardinelli. Behaviour policy optimization: Provably lower variance return estimates for off-policy reinforcement learning. arXiv preprint arXiv:2511.10843, 2025.DOI
- [94]T. Goodall, M. Rowland, R. Munos, and W. Dabney. Behavior policy optimization: Provably lower variance return estimates for off-policy rl. In International Conference on Machine Learning, 2024.
- [95]Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Ompo: A unified framework for rl under policy and dynamics shifts. arXiv preprint arXiv:2405.19080, 2024.arxiv.org/abs/2405.19080
- [96]Nathan Kallus and Masatoshi Uehara. Statistically efficient off-policy policy gradients. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5089–5100, 2020. arXiv:2002.04014.arxiv.org/abs/2002.04014
- [97]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30, 2017.
- [98]Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning (ICML), pages 2817–2826. PMLR, 2017.


