betaFeedback

WorksPhysics and Astronomy

Policy-DRIFT: Dynamic Reward-Informed Flow Trajectory Steering

Abstract

Skin-friction drag induced by wall-bounded turbulent flows accounts for a substantial fraction of energy consumption across commercial aerospace, wind energy, and marine transport. Its active reduction is one of the highest-value targets in engineering fluid dynamics. Deep reinforcement learning (DRL) has emerged as the leading approach for real-time flow control, yet its performance ceiling is set not by algorithmic capability but by reward structure, the naive scalar objective does not optimally reflect the underlying physics. Policy-DRIFT bypasses this ceiling by relocating reward information from policy gradients to generative model inference: a conditional flow matching model (CFM) constructs a physically-grounded manifold of realisable flow states spanning multiple control regimes, Terminal Reward Guidance (TRG) steers samples toward reward-maximising targets at inference, and a lightweight DRL policy, structurally decoupled from reward quality, tracks these full-field targets via root-mean-squared error (RMSE) minimisation. The test case is turbulent channel flow simulated using direct numerical simulation (DNS) at friction Reynolds number of Reτ=180\mathrm{Re}_τ= 180, which is the canonical benchmark for wall-bounded turbulence. Policy-DRIFT achieves 49%49\% drag reduction approaching the theoretical upper bound, which is ≈16%\approx 16\% higher than the DRL benchmark, while consuming 37×\times less actuation energy. Our approach combines generative methods with active flow control, marking a paradigm shift towards controlling complex physical systems efficiently.

Introduction

Active control of complex physical systems is central to modern engineering: from stabilising plasma in tokamak reactors to regulating flow in pipeline networks and cardiovascular devices, the ability to manipulate dynamical processes in real time translates directly into performance gains and energy savings. Deep reinforcement learning (DRL) has emerged as the leading paradigm for this challenge, delivering state-of-the-art controllers across robotic manipulation Luo et al., [30], molecular design Olivecrona et al., [33], plasma stabilisation Degrave et al., [10], and turbulent flow control Guastoni et al., [15]; Beneitez et al. [3]. Wall-bounded turbulent flows represent a particularly high-value target: skin-friction drag accounts for roughly half of total

Preprint. aerodynamic drag on commercial aircraft [44], and even a few percent reduction helps towards improving global fuel consumption efficiencies and CO2CO_2 emissions significantly. Recently, DRL controllers have demonstrated sustained drag reduction systematically outperforming classical heuristics such as opposition control [9].

Yet a fundamental limitation persists across DRL applications regardless of algorithmic advances, the reward signal remains a proxy of the true objective, and the policy ceiling is set entirely by how well this surrogate reflects what optimal actually looks like [39, 34]. In high-fidelity physical simulation, this is compounded by the prohibitive cost of sustained online DNS interaction during training, restricting policy improvement to what the proxy reward allows [12]. Prior remedies through inverse reward design [17] and reward-hacking mitigation [24] refine the proxy without eliminating the underlying problem: as long as reward gradients flow directly through the policy, learned behaviour remains fundamentally coupled to surrogate quality.

Policy-DRIFT addresses reward misspecification by relocating where reward acts: not as a per-step gradient shaping the DRL policy, but as guidance over a generative model of the controlled system. A conditional flow matching (CFM) model trained across a spectrum of control strategies constructs a physically-grounded manifold of realisable flow states. Terminal Reward Guidance (TRG) then navigates this manifold at inference, steering ODE trajectories toward terminal states that maximise a designer-specified reward, fully decoupling the policy’s learning signal from reward quality. A lightweight DRL policy learns exclusively to track these generative targets via RMSE minimisation. Its learning signal is a full-field spatial objective, never a spatially averaged reward gradient, so that optimal behaviour emerges from target tracking rather than reward engineering. Crucially, the CFM model requires no retraining when the control objective changes.

Instantiated with a cost-aware objective on turbulent channel flow DNS at a moderate friction Reynolds number of Reτ=180Re_\tau= 180 (Reτ=uτh/νRe_\tau= u_\tau h/\nu where uτu_\tau is the friction velocity, hh is the channel height and ν\nu is the viscosity.), Policy-DRIFT outperforms all available baselines: it achieves 48.95% drag reduction, being 16.2% higher than vanilla DRL and 38.9% higher than opposition control, while consuming approximately ∼\sim37×\times less actuation energy than DRL. Unlike standard DRL, which reacts to instantaneous scalar reward signals, Policy-DRIFT operates as a receding-horizon controller: at each control horizon, CFM with TRG computes the physically-realisable, reward-maximising target one horizon ahead; the DRL policy then executes 8 actuation steps to track it. Note that neither drag reduction nor energy savings are directly optimised by the policy, which emerge implicitly from target tracking.

Our contributions are as follows:

  • Policy-DRIFT framework. We introduce Policy-DRIFT, a generative control framework that replaces naive DRL reward signals with physically-grounded target states via CFM trained jointly on uncontrolled, opposition-controlled, and DRL-controlled flow regimes, enabling flexible reward specification without reward gradients entering the policy.

  • Terminal Reward Guidance (TRG). We propose TRG, which steers CFM ODE trajectories toward reward-maximising terminal states via gradients of a learned reward predictor, extending inference-time guidance to full-field PDE-governed state spaces with a manifold-regularising pre-placement design that prevents reward hacking without constraining the reward objective.

  • Generative target generation for turbulence control. CFM combined with TRG provides physically-grounded, reward-maximising flow targets during training, decoupling target discovery from policy execution such that the deployed policy acts reactively from wall-parallel sensing alone, without generative model queries at inference time.

  • Empirical validation on DNS benchmark. On Reτ=180Re_\tau= 180 turbulent channel flow, Policy-DRIFT simultaneously achieves 16.2% higher drag reduction than standard DRL and 38.9% higher than opposition control, while using approximately ∼\sim37×\times less actuation energy than DRL, with both gains emerging without the policy directly optimising either quantity.

Related Work

Turbulent flow control. Bewley et al. [ [4] established that receding-horizon optimal control with full DNS access achieves over 50% drag reduction and complete flow relaminarisation in turbulent channel flow at Reτ=180Re_\tau= 180, setting the physical upper bound for wall-transpiration control, but at the cost of iterative full-state DNS at every control step, identified by the authors as impossible to implement in practice. Deep reinforcement learning addressed this intractability by replacing per-step DNS with learned policies trained on custom reward signals, delivering state-of-the-art drag reduction under realistic partial-state sensing [15]. Yet sustained online interaction with high-fidelity DNS during training remains computationally expensive [12], and the reliance on hand-crafted reward proxies introduces a compounding limitation: proxy objectives routinely diverge from the true control goal [39, 34], with prior remedies through inverse reward design [17] and occupancy-measure regularisation [24] narrowing but not closing the gap.

Generative models for physical systems. Score-based diffusion [40] established generative modelling over continuous high-dimensional state spaces. Subsequently, several applications confirmed that generative models can represent the complex statistics of turbulent PDE state spaces [26, 45]. Flow matching [28, 2, 29, 42] emerged as a simulation-free, ODE-based alternative, achieving comparable expressiveness at substantially lower inference cost through direct velocity field regression, with applications spanning protein backbones [6] and crystalline materials [31]. Harder et al. [ [18] applied flow matching to turbulent PDE forecasting, comparing efficient sampling variants for autoregressive state prediction, but without any mechanism to steer generated states toward targeted reward configurations.

Reward guidance in generative models. Classifier guidance [11] and its classifier-free variant [19] established inference-time reward steering as a tractable alternative to retraining, subsequently extended to sequential decision-making through value-guided diffusion planners for offline RL [20, 1] and reward alignment via direct generative model fine-tuning [5, 37, 13]. In scientific design, Uehara et al. [2025] applied inference-time guidance over diffusion to protein and DNA sequences, and Jensen et al. [2026] proposed reward-guided flow matching via online value-function learning for image generation and molecular design. Both works generate standalone configurations in discrete or image domains. The value-matching formulation of Jensen et al. [2026] cannot be directly applied to turbulent velocity fields, as the required noise injection destroys the spatial coherence of physical snapshots. Inference-time guidance has not been applied over PDE-governed physical state spaces or as targets for an active control policy.

Learned surrogates and policy distillation. Learned dynamics models have replaced expensive simulators inside planning loops across robotics [32, 48], decoupling planning quality from per-step ground-truth access and enabling tractable receding-horizon control in low-dimensional systems. Policy distillation [38] established the complementary pattern of compressing an expensive planner into a fast reactive policy, recently extended to offline behavioural supervision [35]. Existing approaches rely on deterministic surrogate dynamics in low-dimensional settings; no prior work has deployed either pattern in high-dimensional turbulent flow with a generative planner capable of reward-steered inference.

Methods

Problem Formulation

We consider active drag reduction in turbulent open-channel flow of height hh at constant mass flux, driven by a time-varying pressure gradient. Streamwise, wall-normal, and spanwise coordinates are (x,y,z)(x,y,z) with corresponding velocity components u=(u,v,w)\mathbf{u} = (u,v,w). The flow is governed by the incompressible Navier–Stokes equations,

∂tu+(u⋅∇)u=−∇p+1ReclΔu,(1)\partial_t \mathbf{u} + (\mathbf{u} \cdot\nabla)\mathbf{u} = -\nabla p + \frac{1}{Re_{\mathrm{cl}}}\Delta\mathbf{u}, \tag*{(1)}
∇⋅u=0,(2)\nabla\cdot\mathbf{u} = 0, \tag*{(2)}

Figure 1. Policy-DRIFT framework. A CFM model trained on opposition control [9] and DRL-WSE [15] data is steered via Terminal Reward Guidance (TRG) to produce reward-maximising flow targets. A DRL agent tracks these targets via RMSE minimisation, achieving the highest drag reduction at optimised actuation energy (top right).

non-dimensionalised by hh and the laminar centreline velocity UclU_{cl} (Recl=Uclh/νRe_{cl}=U_{cl}h/\nu). No-slip and impermeability conditions are imposed at the lower wall, with wall-normal actuation vwv_w applied as a Dirichlet boundary condition. Simulations use the spectral solver Dedalus [7], with Fourier modes in xx and zz and Chebyshev polynomials in yy, at resolution (Nx,Nz,Ny)=(16,16,64)(N_x,N_z,N_y)=(16,16,64), with time integration via a third-order DIRK+ERK scheme at δtsim+≈0.039\delta t_{\mathrm{sim}}^+ \approx0.039. The friction Reynolds number is Reτ=uτh/ν=180Re_\tau=u_\tau h/\nu=180, where uτ=τw/ρu_\tau=\sqrt{\tau_w/\rho} is the friction velocity and ν\nu the kinematic viscosity, in a minimal channel (Lx×Ly×Lz=2.67h×h×0.8hL_x \times L_y \times L_z=2.67h \times h \times0.8h).

The full flow state at any instant is u∈R3×Nx×Nz×Ny\mathbf{u}\in\mathbb{R}^{3\times N_x\times N_z\times N_y}. Long simulations (t+>20,000t^+>20{,}000) under three distinct control strategies yield snapshot datasets {Dk}k=1K\{D_k\}_{k=1}^{K}, each consisting of consecutive state pairs (u0,u1)∈Dk(\mathbf{u}_0,\mathbf{u}_1)\in D_k separated by a control horizon Δt+=5\Delta t^+=5 viscous time units (t+=t/t∗t^+=t/t^*, t∗=ν/uτ2t^*=\nu/u_\tau^2), with joint dataset D=⋃k=1KDk\mathcal{D}=\bigcup_{k=1}^{K}D_k. In this work K=3K=3: D1D_1 (uncontrolled), D2D_2 (opposition-controlled, Choi et al. 1994), and D3D_3 (wall-shear-stress DRL-controlled, Guastoni et al. 2023), with ∣Dk∣=7,350|D_k|=7{,}350 snapshot pairs each.

Multi-Regime Flow Manifold

We construct a generative model of the conditional distribution pD(u1∣u0)p_{\mathcal{D}}(\mathbf{u}_1 \mid\mathbf{u}_0) across all KK control regimes jointly. For each target u1\mathbf{u}_1, define the conditional probability path

ps(u~∣u1)=N(u~;su1,(1−s)2I),s∈[0,1],(3)p_s(\tilde{\mathbf{u}}\mid\mathbf{u}_1)=\mathcal{N}\left(\tilde{\mathbf{u}};s\mathbf{u}_1,(1-s)^2I\right),\quad s\in[0,1], \tag*{(3)}

which satisfies p0=N(0,I)p_0=\mathcal{N}(0,I) and lim⁡s→1ps(⋅∣u1)=δu1\lim_{s\to1}p_s(\cdot\mid\mathbf{u}_1)=\delta_{\mathbf{u}_1}. A sample u~s∼ps(⋅∣u1)\tilde{\mathbf{u}}_s\sim p_s(\cdot\mid\mathbf{u}_1) is realised as u~s=(1−s)η+su1\tilde{\mathbf{u}}_s=(1-s)\boldsymbol{\eta}+s\mathbf{u}_1 with η∼N(0,I)\boldsymbol{\eta}\sim\mathcal{N}(0,I), and the associated conditional vector field is

us(u~s∣u1)=u1−u~s1−s=u1−η,(4)\mathbf{u}_s(\tilde{\mathbf{u}}_s\mid\mathbf{u}_1)=\frac{\mathbf{u}_1-\tilde{\mathbf{u}}_s}{1-s}=\mathbf{u}_1-\boldsymbol{\eta}, \tag*{(4)}

which is constant along each straight-line trajectory. Conditioning further on the current state u0\mathbf{u}_0, the marginal probability path is

ps(u~∣u0)=∫ps(u~∣u1) pD(u1∣u0) du1,(5)p_s(\tilde{\mathbf{u}} \mid\mathbf{u}_0) = \int p_s(\tilde{\mathbf{u}} \mid\mathbf{u}_1)\,p_D(\mathbf{u}_1 \mid\mathbf{u}_0)\,d\mathbf{u}_1, \tag*{(5)}

with marginal vector field us(u~∣u0)=E[us(u~s∣u1)∣u~s=u~,u0]u_s(\tilde{\mathbf{u}} \mid\mathbf{u}_0) = \mathbb{E}[u_s(\tilde{\mathbf{u}}_s \mid\mathbf{u}_1) \mid\tilde{\mathbf{u}}_s = \tilde{\mathbf{u}}, \mathbf{u}_0], which is intractable because it requires integrating over the unknown pD(u1∣u0)p_D(\mathbf{u}_1 \mid\mathbf{u}_0). We parameterise vθ(u~s,s,u0)≈us(u~∣u0)v_\theta(\tilde{\mathbf{u}}_s, s, \mathbf{u}_0) \approx u_s(\tilde{\mathbf{u}} \mid\mathbf{u}_0) and train it via the following result.

The conditional flow matching [28, 42] objective

LCFM(θ)=E(u0,u1)∼Dη∼N(0,I), s∼U[0,1][∥vθ(u~s,s,u0)−(u1−η)∥2](6)\mathcal{L}_{\mathrm{CFM}}(\theta) = \mathbb{E}_{(\mathbf{u}_0,\mathbf{u}_1)\sim\mathcal{D}}^{\eta\sim\mathcal{N}(0,I),\,s\sim\mathcal{U}[0,1]}\left[\left\|v_\theta(\tilde{\mathbf{u}}_s,s,\mathbf{u}_0)-(\mathbf{u}_1-\eta)\right\|^2\right] \tag*{(6)}

and the marginal objective LFM(θ)=E(u0,s)Eu~∼ps(⋅∣u0)[∥vθ(u~,s,u0)−us(u~∣u0)∥2]\mathcal{L}_{\mathrm{FM}}(\theta) = \mathbb{E}_{(\mathbf{u}_0,s)}\mathbb{E}_{\tilde{\mathbf{u}}\sim p_s(\cdot\mid\mathbf{u}_0)}\left[\left\|v_\theta(\tilde{\mathbf{u}},s,\mathbf{u}_0)-u_s(\tilde{\mathbf{u}}\mid\mathbf{u}_0)\right\|^2\right] satisfy

∇θLCFM=∇θLFM.\nabla_\theta\mathcal{L}_{\mathrm{CFM}} = \nabla_\theta\mathcal{L}_{\mathrm{FM}}.

Minimising LCFM\mathcal{L}_{\mathrm{CFM}} against the tractable conditional targets (u1−η)(\mathbf{u}_1-\eta) therefore recovers us(⋅∣u0)u_s(\cdot\mid\mathbf{u}_0) without computing the intractable integral in (5). Since D=⋃k=1KDk\mathcal{D}=\bigcup_{k=1}^{K}\mathcal{D}_k, the expectation in (6) marginalises over all KK control regimes simultaneously; no per-regime model, classifier head, or importance weight is required. At inference, integrating du~/ds=vθ(u~s,s,u0)d\tilde{\mathbf{u}}/ds=v_\theta(\tilde{\mathbf{u}}_s,s,\mathbf{u}_0) from s=0s=0 to s=1s=1 maps η∼N(0,I)\eta\sim\mathcal{N}(0,I) to a sample u^1∼pD(⋅∣u0)\hat{\mathbf{u}}_1\sim p_D(\cdot\mid\mathbf{u}_0) drawn from the joint multi-regime distribution.

Terminal Reward Guidance

With vθv_\theta frozen, the goal of TRG is to steer ODE trajectories toward terminal states u^1\hat{\mathbf{u}}_1 that maximise the scalar reward

R(u^1)=DR(u^1)−Eact(u^1),(7)R(\hat{\mathbf{u}}_1)=D_R(\hat{\mathbf{u}}_1)-E_{\mathrm{act}}(\hat{\mathbf{u}}_1), \tag*{(7)}

where

DR=1−τwτw,0(8)D_R=1-\frac{\tau_w}{\tau_{w,0}} \tag*{(8)}

is the drag-reduction fraction relative to uncontrolled flow (τw=ρν⟨∂yu⟩x,z∣wall\tau_w=\rho\nu\langle\partial_y u\rangle_{x,z}|_{\mathrm{wall}} is the mean wall-shear stress), and

Eact=12⟨∣vw∣3⟩x,z/uτ3(9)E_{\mathrm{act}}=\frac{1}{2}\langle|v_w|^3\rangle_{x,z}/u_\tau^3 \tag*{(9)}

is the normalised pumping energy of the wall actuation [22]. This requires a model that can translate a gradient signal defined at the terminal state back to intermediate ODE states, where it can influence the trajectory.

Reward Predictor. With vθv_\theta fixed, generate a dataset of unguided ODE rollouts: for each (u0,η)(\mathbf{u}_0,\eta), integrate du~/ds=vθ(u~s,s,u0)d\tilde{\mathbf{u}}/ds=v_\theta(\tilde{\mathbf{u}}_s,s,\mathbf{u}_0) to obtain the trajectory {u~si}i=1T\{\tilde{\mathbf{u}}_{s_i}\}_{i=1}^{T} and terminal state u^1=u~1\hat{\mathbf{u}}_1=\tilde{\mathbf{u}}_1, then compute the terminal reward rT=R(u^1)r_T=R(\hat{\mathbf{u}}_1). We train a predictor Rψ:R3×Nx×Nz×Ny×[0,1]→RR_\psi:\mathbb{R}^{3\times N_x\times N_z\times N_y}\times[0,1]\to\mathbb{R} to map any intermediate ODE state to rTr_T:

LRP(ψ)=Ei[si⋅(Rψ(u~si,si)−rT)2].(10)\mathcal{L}_{\mathrm{RP}}(\psi)=\mathbb{E}_i\left[s_i\cdot\left(R_\psi(\tilde{\mathbf{u}}_{s_i},s_i)-r_T\right)^2\right]. \tag*{(10)}

The ramp weight w(s)=sw(s)=s down-weights ODE states near s=0s=0 (pure noise) and assigns full weight near s=1s=1 (physical field), where RψR_\psi can be most reliably trained. Crucially, the same terminal rTr_T serves as the regression target for every intermediate state along the trajectory, with no one-step look-ahead or Bellman bootstrap. This eliminates compounding bootstrapping error at the cost of requiring RψR_\psi to learn long-range reward prediction from early noisy states; the ramp weighting mitigates this by concentrating supervision near the terminal end.

Guided ODE integration. At each ODE step from ss to s+δss+\delta s, TRG applies a pre-placement correction before the velocity-model (VM) step:

u~s+=u~s+γδs ∇u~sRψ(u~s,s),(11)\tilde{\mathbf{u}}_s^+=\tilde{\mathbf{u}}_s+\gamma\delta s\,\nabla_{\tilde{\mathbf{u}}_s}R_\psi(\tilde{\mathbf{u}}_s,s), \tag*{(11)}
u~s+δs=u~s++δs⋅vθ(u~s+,s,u0),(12)\tilde{\mathbf{u}}_{s+\delta s}=\tilde{\mathbf{u}}_s^++\delta s\cdot v_\theta(\tilde{\mathbf{u}}_s^+,s,\mathbf{u}_0), \tag*{(12)}

where γ>0\gamma>0 is the guidance scale. The nudged intermediate state u~s+\tilde{\mathbf{u}}_s^+ is passed as input to vθv_\theta, which propagates it through one learned velocity step. The alternative post-placement scheme applies the correction after the VM step: u~s+δs=u~s+δs⋅vθ(u~s,s,u0)+γ δs ∇Rψ\tilde{\mathbf{u}}_{s+\delta s} = \tilde{\mathbf{u}}_s + \delta s \cdot v_\theta(\tilde{\mathbf{u}}_s, s, \mathbf{u}_0) + \gamma\,\delta s\,\nabla R_\psi. In our experiments, post-placement causes reward hacking: drag reduction clamps at DR→1.0\mathrm{DR} \to1.0 within the first few guidance steps, producing states that score high on RψR_\psi but are physically unrealisable. Pre-placement avoids this because vθv_\theta acts on u~s+\tilde{\mathbf{u}}_s^+ directly: the learned dynamics map the nudged intermediate back toward the support of ps(⋅∣u0)p_s(\cdot\mid\mathbf{u}_0), keeping the trajectory on the physical flow manifold at every step. This use of the generative model as an implicit manifold projector is the domain-specific design choice that distinguishes TRG from classifier guidance applied directly to the terminal state.

Reinforcement-learning framework

Framework formulation We formulate closed-loop flow control as a reinforcement learning (RL) problem, in which several agents collectively optimise a shared global reward rr through distributed interaction with a fluid environment. A policy network is shared by agents, which maps partial flow observations oo to continuous control actions aa. This implementation reduces the effective parameter count and promotes generalisation through repeated exposure to diverse local flow states. We adopt TD3 [14] as the policy algorithm for all reward formulation comparisons; SAC [16] is included as an additional off-policy baseline under the WSE formulation, with algorithm selection justified in Appendix § 11.

The environment is partitioned into a set of pseudo-environments, each assigned to a single agent that observes and actuates its local subdomain independently. Here, the number of pseudo-environments is set equal to the number of active wall grid points Nx×NzN_x \times N_z. This yields a scalar action space a∈R1×1a \in\mathbb{R}^{1 \times1} and an observation space of fixed dimension o∈RNfeat×1o \in\mathbb{R}^{N_{\mathrm{feat}} \times1}, where NfeatN_{\mathrm{feat}} is the number of input features.

Each agent observes the inner-scaled fluctuation components of the streamwise (u′+u^{\prime+}) and wall-normal (v′+v^{\prime+}) velocity at an off-wall sensing plane y+=15y^+ = 15, giving Nfeat=2N_{\mathrm{feat}} = 2. Prior to input, the spatial mean is subtracted from each feature over streamwise and spanwise directions in channel flow. Actions correspond to wall-normal velocity Dirichlet boundary conditions, bounded by the local friction velocity ∣a∣≤uτ|a| \leq u_\tau, modelling localised blowing and suction actuation. A zero-net-mass-flux (ZNMF) constraint is enforced by subtracting the spatial mean of the wall actuation before application, preventing spurious mass injection due to the incompressibility constraint.

Reward formulations. We evaluate three reward functions. WSE uses r=DRr = \mathrm{DR} (Eq. (8)), the instantaneous drag-reduction fraction relative to uncontrolled flow, following prior work [15]. WEN extends this by penalising actuation energy: r=DR−Eactr = \mathrm{DR} - E_{\mathrm{act}} (Eq. (9)). TRG (Policy–DRIFT) replaces the scalar reward entirely: the signal is the RMSE between the current flow state and the TRG-guided CFM target u^1\hat{\mathbf{u}}_1, computed once at the start of each 8-step horizon (Δt+=5\Delta t^+ = 5) and held fixed throughout. Tracking a full-field spatial target also eliminates a pathological failure mode of scalar rewards: optimising instantaneous mean drag can exploit spatial averaging, yielding high rewards without genuine flow control, whereas RMSE to a full-field target enforces spatially uniform tracking without additional regularisation.

Results

Policy–DRIFT is evaluated on turbulent channel flow DNS at Reτ=180\mathrm{Re}_\tau= 180, a canonical flow control configuration. Baselines include opposition control, the wall-shear-stress DRL controller (TD3-WSE) and a DRL controller trained directly on the cost-aware objective in Equation 7 (TD3-WEN), providing a direct comparison under identical reward specification. All methods are evaluated on six initial conditions held out from training. Results are reported sequentially: CFM fidelity, TRG guidance quality, and finally the closed-loop DRL performance.

Conditional Flow Matching

A single conditional flow matching model vθv_\theta, trained without any regime labels or per-strategy supervision, learns to jointly represent the stochastic dynamics pD(u1∣u0)p_{\mathcal{D}}(\mathbf{u}_1 \mid\mathbf{u}_0) across three qualitatively distinct control trajectories: passive uncontrolled evolution, opposition control, which selectively attenuates near-wall ejections and sweeps, and the aggressive wall-shear-stress DRL strategy that produces substantially different flow statistics. That these co-exist within a single learned manifold is not guaranteed: each strategy induces a different conditional distribution over successor states u1\mathbf{u}_1, and the model must resolve this multi-modal structure from the shared data D\mathcal{D}. Quantitatively, vθv_\theta achieves a mean squared error of order 10−410^{-4} on held-out snapshots from all three regimes, and the continuity residuals ∣∇⋅u^1∣\lvert\nabla\cdot\hat{u}_1\rvert of the generated fields are of the same order of magnitude as those of the DNS snapshots, confirming that the joint training objective recovers the conditional vector field without per-regime supervision and that the generated states are physically realisable. The resulting manifold is richer than any individual strategy alone: by spanning the continuum between control regimes, it provides TRG with a space of physically realisable flow states that no single trajectory could supply.

Terminal Reward Guidance (TRG)

With vθv_\theta fixed, TRG navigates the learned manifold toward its high-reward region at inference. Across all three held-out uncontrolled snapshots (Figure 2), guided rollouts consistently reach higher drag reduction, by ΔDR≈0.026\Delta\mathrm{DR} \approx0.026–0.0300.030, relative to their unguided counterparts, while maintaining comparably low actuation energy, demonstrating that TRG extracts reward-maximising configurations already latent in the multi-regime manifold without introducing additional actuation cost. This behaviour is consistent across all three training regimes (Appendix § 10), confirming that the manifold spans reward-maximising states beyond what any individual control trajectory supplies.

Streamwise velocity fluctuations in the x-y plane for three held-out uncontrolled snapshots

Figure 2. Streamwise velocity fluctuations u′(x,y)u'(x,y) in the xx-yy plane for three held-out uncontrolled (D1)(D_1) snapshots. Columns show the conditioning snapshot u0\mathbf{u}_0, the unguided CFM terminal state u1CFM\mathbf{u}_1^{\mathrm{CFM}}, the TRG-guided terminal state u1TRG\mathbf{u}_1^{\mathrm{TRG}}, and the absolute difference u1TRG−u1CFM\mathbf{u}_1^{\mathrm{TRG}}-\mathbf{u}_1^{\mathrm{CFM}}. TRG consistently improves drag reduction by ΔDR≈0.026\Delta\mathrm{DR} \approx0.026–0.0300.030 with negligible change in actuation energy. Corrections concentrate near the wall (y/h≈0.4y/h \approx0.4), where turbulent production peaks, while the bulk flow remains largely undisturbed, the perturbation magnitude is two orders smaller than the velocity fluctuations, confirming physically-constrained guidance. Guidance scale γ=5\gamma= 5.

The difference fields (u1TRG−u1CFM)(u_1^{\mathrm{TRG}} - u_1^{\mathrm{CFM}}) are informative about the mechanism: corrections concentrate near the wall (y/h≈0.4y/h \approx0.4), precisely where turbulent production and skin-friction drag originate, while the bulk flow is largely undisturbed. The perturbation magnitude is two orders of magnitude smaller than the velocity fluctuations themselves, confirming that TRG acts as a targeted, physically-constrained nudge rather than a wholesale restructuring, a direct consequence of the pre-placement design, which routes guidance through vθv_\theta at every step and keeps trajectories on the physical manifold.

Closed-Loop Control Performance

Table 1 and Figure 3 summarise closed-loop performance across all methods. Policy-DRIFT achieves 48.95% drag reduction, approaching the theoretical upper bound of > 50% established by [4] using full DNS access at every control step, while using only offline data and a single wall-parallel sensing plane at inference. Against the state-of-the-art DRL controller for flow control Guastoni et al. [ [15] (TD3-WSE), Policy-DRIFT gains 16.2% in drag reduction while consuming ∼37×\sim37\times less actuation energy, improvements that emerge without the policy directly optimising either quantity. Compared to opposition control [9], the classical physics-based strategy that directly mitigates the near-wall momentum using full wall-normal velocity measurements, Policy-DRIFT achieves 38.9% higher drag reduction at comparable actuation energy, demonstrating that a data-driven generative approach substantially outperforms classical heuristic flow control strategies. Figure 3 shows that this separation from baselines is established early in the transient (t+≈200t^{+} \approx200) and maintained with low variance across all six out-of-distribution initial conditions, while TD3-WSE sustains actuation energy an order of magnitude above all other methods throughout.

Figure 3. Drag reduction (top) and actuation energy EactE_{\mathrm{act}} (bottom, log scale) over t+∈[0,3000]t^{+} \in[0, 3000] for 6 out-of-distribution initial conditions (shaded band: min–max across trajectories; scalar values averaged over t+∈[500,3000]t^{+} \in[500, 3000]). Policy-DRIFT separates from all baselines by t+≈200t^{+} \approx200 and sustains the highest drag reduction with low variance throughout.

The most direct comparison is against TD3-WEN, which optimises the identical cost-aware objective DR−Eact\mathrm{DR} - E_{\mathrm{act}} through policy gradients. Despite sharing the same reward, TD3-WEN achieves only 42.86% DR, a 14.2% gap closed by Policy-DRIFT for a modest 1.6×1.6\times increase in actuation energy. This gap directly isolates the cost of routing reward through policy gradients: the energy penalty enters every critic update, driving the policy toward suppressing actuation globally at the expense of drag reduction. In Policy-DRIFT, energy efficiency is never penalised by the policy; it emerges from the structure of the generative targets themselves. The reward acts where it is most effective, at target selection on the physical manifold, not at gradient computation.

Physical Interpretation of Control Policies

The joint PDF of (u′+,v′+)(u^{\prime+}, v^{\prime+}) at the sensing plane y+=15y^+ = 15, where turbulence production peaks [23], provides a physical picture of how each strategy acts on the near-wall cycle. The uncontrolled flow exhibits the characteristic broad distribution with pronounced ejection (u′+<0u^{\prime+} < 0, v′+>0v^{\prime+} > 0) and sweep (u′+>0u^{\prime+} > 0, v′+<0v^{\prime+} < 0) tails, the primary events responsible for skin-friction drag in wall-bounded turbulence. Opposition control narrows these tails substantially, confirming near-wall event suppression as its operating mechanism, but remains limited by its reliance on instantaneous wall-normal measurements without any learned flow representation. TD3-WSE pushes ejection and sweep suppression further but at the cost of a strongly bimodal v′+v^{\prime+} distribution extending to ±1\pm1, a clear over-actuation signature in which wall-normal advection amplifies rather than selectively attenuating turbulent events. TD3-WEN recovers a compact v′+v^{\prime+} distribution by penalising actuation energy, but the energy penalty suppresses actuation globally, weakening ejection and sweep attenuation relative to WSE. Policy-DRIFT resolves both failure modes simultaneously: the v′+v^{\prime+} footprint is as narrow as TD3-WEN while ejection and sweep suppression matches or exceeds TD3-WSE, combining strong drag reduction with energy efficiency without directly optimising either.

Figure 4. Joint PDFs of (u′+,v′+)(u^{\prime+}, v^{\prime+}) at y+≈15y^+ \approx15 for the uncontrolled flow and all controlled methods. Dashed lines mark the ejection (u′+<0u^{\prime+} < 0, v′+>0v^{\prime+} > 0) and sweep (u′+>0u^{\prime+} > 0, v′+<0v^{\prime+} < 0) quadrants. Policy-DRIFT achieves the most compact distribution with the strongest attenuation of both quadrants.

Conclusions

Policy-DRIFT demonstrates that relocating reward information from policy gradients to generative inference systematically breaks the performance ceiling imposed by reward misspecification. By tracking physically-grounded full-field targets rather than scalar proxies, the policy avoids the structural failure modes of reward-based DRL such as over-actuation, uninformative exploration, and surrogate-bounded performance without any direct optimisation of the quantities it improves. On turbulent channel flow at Reτ=180Re_{\tau} = 180, Policy-DRIFT approaches the theoretical upper bound of Bewley et al. [ [4] at 48.95% drag reduction, consuming ∼37×\sim37\times less actuation energy than the DRL baseline. The CFM model requires no retraining when the control objective changes, only RψR_{\psi} need be updated, and the framework extends directly to complex flow configurations such as aircraft wings [47], opening a zero-shot pathway to drag reduction in geometries beyond the training distribution.

Limitations and future work. The present study is validated on a channel at Reτ=180Re_{\tau} = 180; natural extensions include higher Reynolds numbers, complex wall geometries, and application to experimental data, a step toward digital-twin integration in fluid dynamics. Online adaptation of RψR_{\psi} during closed-loop control and deployment across other high-fidelity physical domains where the same reward-structure bottleneck applies, such as plasma stabilisation, molecular dynamics, and robotics, are equally promising directions.

Acknowledgements

This research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-26-1-A053. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

Data and Normalisation

Dataset. The training data consists of three subsets of consecutive DNS snapshot pairs (u0,u1)(\mathbf{u}_0,\mathbf{u}_1) at horizon Δt+=5\Delta t^{+}=5: D1\mathcal{D}_1 (uncontrolled flow), D2\mathcal{D}_2 (opposition control), and D3\mathcal{D}_3 (DRL wall-shear-stress control). Each subset contains 7,350 pairs, yielding 22,050 training pairs in the joint dataset D\mathcal{D}. An 80/20 train–test split is applied independently within each regime; consecutive pairs never cross source boundaries.

Preprocessing. Raw DNS snapshots contain the instantaneous velocity field u=(u,v,w)∈R3×16×16×64\mathbf{u}=(u,v,w)\in\mathbb{R}^{3\times16\times16\times64} (dimensions: components, NxN_x, NzN_z, NyN_y, with Ny=64N_y=64 wall-normal points). Velocity fluctuations are obtained by subtracting the per-voxel time-mean U‾(x,y,z)\overline{\mathbf{U}}(x,y,z) estimated from the full dataset: u′=u−U‾\mathbf{u}^{\prime}=\mathbf{u}-\overline{\mathbf{U}}. Fluctuations are then normalised channel-wise with min-max scaling to [0,1][0,1] using statistics computed over the training set. Table 2 reports the per-channel statistics.

Channelσ\sigmaminmax
u′\mathbf{u}^{\prime}0.0548−0.37260.5168
v′\mathbf{v}^{\prime}0.0196−0.21820.2335
w′\mathbf{w}^{\prime}0.0261−0.27230.2987

Table 2. Per-channel min-max statistics used for normalisation (Δt+=5\Delta t^{+}=5 snapshot pairs).

CFM Velocity Model Architecture

The CFM velocity model vθ\mathbf{v}_{\theta} is a 3D UNet operating on an input volume of shape [B,7,16,16,64][B,7,16,16,64], formed by concatenating the linearly interpolated state u~s=(1−s)η+su1\tilde{\mathbf{u}}_s=(1-s)\boldsymbol{\eta}+s\mathbf{u}_1 (3 channels), the conditioning snapshot u0\mathbf{u}_0 (3 channels), and the scalar flow-matching time ss broadcast to the spatial volume (1 channel). The output is the predicted velocity field vθ∈R[B,3,16,16,64]\mathbf{v}_{\theta}\in\mathbb{R}^{[B,3,16,16,64]}.

Architecture. The encoder has four levels with base width 64 and doubling channels (64→128→256→51264\rightarrow128\rightarrow256\rightarrow512 at the bottleneck). Each level applies two 333^3 convolution blocks (Conv3d →\rightarrow BatchNorm3d →\rightarrow ELU); spatial downsampling after each of the first three levels uses 232^3 average pooling. The decoder mirrors this structure: each level upsamples transposed-convolution, concatenates the corresponding encoder skip connection, and applies a single convolution block. The output layer uses a linear (identity) activation. The total parameter count is 23.4M.

LevelRoleChannelsSpatial
0Encoder (2 conv blocks)7→647\rightarrow6416×16×6416\times16\times64
1Encoder (2 conv blocks)64→12864\rightarrow1288×8×328\times8\times32
2Encoder (2 conv blocks)128→256128\rightarrow2564×4×164\times4\times16
3Bottleneck (2 conv blocks)256→512256\rightarrow5122×2×82\times2\times8
4Decoder512+256→256512+256\rightarrow2564×4×164\times4\times16
5Decoder256+128→128256+128\rightarrow1288×8×328\times8\times32
6Decoder128+64→64128+64\rightarrow6416×16×6416\times16\times64
7Output64→364\rightarrow316×16×6416\times16\times64

Table 3. CFM UNet encoder–decoder channel widths and spatial resolutions. Input spatial shape: 16×16×6416\times16\times64.

Training objective. The model minimises the conditional flow matching loss

LCFM=Et,η,u1[∥vθ(u~s,u0,t)−(u1−η)∥2],(13)\mathcal{L}_{\mathrm{CFM}}=\mathbb{E}_{t,\eta,\mathbf{u}_1}\left[\left\|\mathbf{v}_{\theta}(\tilde{\mathbf{u}}_s,\mathbf{u}_0,t)-(\mathbf{u}_1-\boldsymbol{\eta})\right\|^2\right], \tag*{(13)}

where η∼N(0,I)\boldsymbol{\eta}\sim\mathcal{N}(0,I), t∼Uniform⁡(0,1)t\sim\operatorname{Uniform}(0,1), and the target velocity (u1−η)(\mathbf{u}_1-\boldsymbol{\eta}) is constant along each straight-line OT path.

Training details. Optimiser: AdamW with learning rate 10−310^{-3}, weight decay 10−210^{-2}, gradient clipping at 1.0, and a ReduceLROnPlateau scheduler (patience 20, factor 0.5). Training runs for up to 500 epochs with early stopping (patience 50). Batch size is 256. ODE integration at inference uses 50 Euler steps.

Reward Predictor Architecture

The reward predictor Rψ(xs,s)R_{\psi}(x_s,s) is a 3D CNN with four convolutional stages and Feature-wise Linear Modulation (FiLM) time conditioning.

Time embedding. The scalar ODE time s∈[0,1]s \in[0,1] is encoded as a 64-dimensional sinusoidal embedding:

emb⁡(s)=[sin⁡(sfk), cos⁡(sfk)]k=031,fk=10,000−k/32,(14)\operatorname{emb}(s)=\left[\sin(s f_k),\ \cos(s f_k)\right]_{k=0}^{31}, \qquad f_k=10{,}000^{-k/32}, \tag*{(14)}

FiLM conditioning. At each stage the embedding is linearly projected to a scale–shift pair (γ,β)∈R2(\gamma,\beta) \in\mathbb{R}^{2} and applied after group normalisation: h←γ⊙GroupNorm⁡(h)+βh \leftarrow\gamma\odot\operatorname{GroupNorm}(h)+\beta.

Network stages. Four convolutional stages with base width h=64h=64 (1.29M parameters total):

StageKernelStrideCh. in → outOutput spatialNorm
0333^313→643 \to6416×16×6416 \times16 \times64GN(8)
1333^3264→6464 \to648×8×328 \times8 \times32GN(8)
2333^3164→12864 \to1288×8×328 \times8 \times32GN(8)
3333^31128→256128 \to2568×8×328 \times8 \times32GN(8)
Head131^3–256→1256 \to18×8×328 \times8 \times32–

Table 4. Reward predictor stage-wise architecture. Input shape: [B,3,16,16,64][B, 3, 16, 16, 64].

After stage 3, a 131^3 convolution produces a spatial score map of shape [B,1,8,8,32][B,1,8,8,32]; summing its 8×8×32=2,0488 \times8 \times32=2{,}048 values yields the scalar prediction Rψ∈RBR_{\psi} \in\mathbb{R}^{B}. The head weights are initialised by dividing by 2,048. All stages use SiLU activations.

Training. RψR_{\psi} is trained with AdamW (learning rate 10−410^{-4}, weight decay 10−210^{-2}, gradient clipping at 1.0) and a cosine annealing schedule over 500 epochs with early stopping (patience 50), batch size 128. Unguided ODE rollouts from the frozen vθv_{\theta} are generated using 50 Euler steps split into T=25T=25 segments; the terminal reward rTr_T is assigned as the regression target to each intermediate state u~si\tilde{u}_{s_i}, minimising

LRP(ψ)=Ei[si⋅(Rψ(u~si,si)−rT)2],(15)\mathcal{L}_{\mathrm{RP}}(\psi)=\mathbb{E}_{i}\left[s_i\cdot\left(R_{\psi}(\tilde{u}_{s_i},s_i)-r_T\right)^2\right], \tag*{(15)}

where the linear ramp w(si)=siw(s_i)=s_i concentrates gradient signal near s=1s=1, where ODE states closely resemble physical velocity snapshots. At inference, the ODE is steered via ∇u~Rψ\nabla_{\tilde{u}}R_{\psi} using the same 50 Euler steps.

Inference. At each ODE step, the gradient ∇u~Rψ(u~,s)\nabla_{\tilde{u}}R_{\psi}(\tilde{u},s) steers the frozen integrator toward high-reward terminal states. With pre-guidance (default), the correction is applied before the flow model:

u~s+=u~s+γ δs ∇u~sRψ(u~s,s),u~s+δs←u~s++vθ(u~s+,u0,s) δs,(16)\tilde{u}_{s}^{+}=\tilde{u}_{s}+\gamma\,\delta s\,\nabla_{\tilde{u}_{s}}R_{\psi}(\tilde{u}_{s},s), \qquad\tilde{u}_{s+\delta s}\leftarrow\tilde{u}_{s}^{+}+v_{\theta}(\tilde{u}_{s}^{+},u_0,s)\,\delta s, \tag*{(16)}

so the flow model acts as a regulariser on the perturbation. Setting γ=0\gamma=0 recovers the unguided base ODE. The guidance scale γ\gamma is a free inference hyperparameter unconstrained by any noise schedule.

TRG Guidance on All Regimes

Figures 5 and 6 extend Figure 2 to opposition-controlled (D2\mathcal{D}_2) and DRL-WSE (D3\mathcal{D}_3) initial conditions. In all regimes, guided rollouts achieve higher terminal reward than unguided rollouts, with the improvement consistent across independent noise initialisations. The energy penalty is fairly noticeable for opposition-controlled and DRL-WSE with guided ODE rollouts resulting in a significant reduction in actuation energy.

Streamwise velocity fluctuations in the x-y plane for three held-out opposition-controlled snapshots

Figure 5. Streamwise velocity fluctuations u′(x,y)u'(x,y) in the xx-yy plane for three held-out opposition-controlled (D2D_2) snapshots. Columns show the conditioning snapshot u0u_0, the unguided CFM terminal state u1CFMu_1^{\mathrm{CFM}}, the TRG-guided terminal state u1TRGu_1^{\mathrm{TRG}}, and the absolute difference u1TRG−u1CFMu_1^{\mathrm{TRG}}-u_1^{\mathrm{CFM}}, with reward values (DR, EactE_{\mathrm{act}}) annotated per panel. Guidance scale γ=5\gamma=5.

DRL Training Details

Training configuration Training is conducted across six distinct initial conditions over 100 episodes. At the start of each episode, an initial condition is selected via a seeded random number generator to ensure reproducibility. Each episode spans a simulation time of t+=1,500t^+=1{,}500, scaled by the viscous time unit (t∗=ν/uτ2t^*=\nu/u_\tau^2, t+=t/t∗t^+=t/t^*), and the policy is updated every Δtupd=180\Delta t_{\mathrm{upd}}=180 using 64 mini-batch gradient steps. The control policy is evaluated at intervals of δt+=0.6\delta t^+=0.6, yielding 2,500 actuation updates per episode. Mini-batches of size 64 are sampled from a replay buffer of capacity 10610^6 containing state-action-reward tuples. Exploration is promoted by adding zero-mean Gaussian noise with standard deviation 0.2 uτ0.2\,u_\tau to the actor output prior to critic evaluation. All remaining hyperparameters are set to the default values of the Stable-Baselines3 implementation [36].

Policy algorithm We adopt the model-free, off-policy twin-delayed deep deterministic policy gradient (TD3) [14] to train the DRL controller. TD3 mitigates the Q-value overestimation bias of DDPG [27] through clipped double Q-learning, and has demonstrated effectiveness in recent turbulent flow-control studies [25, 41, 15, 46, 8, 49]. The off-policy design is particularly suited to wall-bounded turbulence: TD3 accumulates transitions in a replay buffer and reuses them across multiple updates, providing a statistically richer coverage of the chaotic flow dynamics without simulation cost. For comparison, we also report results for soft actor-critic (SAC) [16] under the WSE reward formulation. SAC exhibited high episode variance and divergent behaviour in later training phases, consistent with known instabilities of entropy-regularised methods under sparse reward signals, as reflected in the lower drag reduction reported in Table 1; TD3 is therefore used for all reward formulation comparisons. The configurations are summarized in Table 1.

CategoryMethodDR (%)EactE_{\mathrm{act}} (×10−1\times10^{-1})Rˉ\bar{R}
MeanStdMeanStd
Training BenchmarksOpp-Control [9]35.2310.090.06850.02410.345
TD3-WSE [15]42.1312.713.6920.04310.052
ExperimentsPolicy-DRIFT (Ours)48.9511.700.09900.07110.480
TD3-WEN42.8611.770.06090.04060.423
TD3-WSE42.1312.713.6920.04310.052
SAC-WSE32.312.330.11930.02280.311

Table 1. Drag reduction (DR, %) and normalised actuation energy EactE_{\mathrm{act}} averaged over 6 held-out initial conditions, with standard deviation across trajectories. Rˉ=DR−Eact\bar{R} = \mathrm{DR} - E_{\mathrm{act}} is the cumulative cost-aware objective evaluated from mean quantities. Training benchmarks (top) provide the data on which the CFM is trained; all experimental variants (bottom) are evaluated under identical conditions.

Table 5 summarises all hyperparameters used for DRL training and evaluation. Note that we applied the default parameters by StableBaselines3 [36] for the rest.

Streamwise velocity fluctuations in the x-y plane for three held-out TD3-WSE snapshots

Figure 6. Streamwise velocity fluctuations u′(x,y)u'(x,y) in the xx-yy plane for three held-out TD3-WSE (D3D_3) snapshots. Columns show the conditioning snapshot u0u_0, the unguided CFM terminal state u1CFMu_1^{\mathrm{CFM}}, the TRG-guided terminal state u1TRGu_1^{\mathrm{TRG}}, and the absolute difference u1TRG−u1CFMu_1^{\mathrm{TRG}} - u_1^{\mathrm{CFM}}, with reward values (DR, EactE_{\mathrm{act}}) annotated per panel. The energy penalty is more pronounced than in D2D_2, reflecting the higher baseline actuation of the DRL-WSE regime. Guidance scale γ=5\gamma= 5.

CategoryParameterValueNotes
ArchitectureCritic hidden widths[16, 64, 64]MLP, ReLU
ArchitectureActor hidden width[8]MLP, ReLU
CommonTraining episodes400—
CommonLearning rate10−410^{-4}—
Off-policy (TD3/SAC)Replay buffer size10610^6TD3, SAC
Off-policy (TD3/SAC)Batch size64—
Off-policy (TD3/SAC)Gradient updates64Every 300 env. steps
CheckpointCriterionMean episode rewardHeld-out initial condition

Table 5. Policy architecture and training hyperparameters.

Guidance Scale Ablation

We sweep the TRG guidance scale γ∈{5,10,20,30}\gamma\in\{5, 10, 20, 30\} and train separate DRL policies for each value, evaluating mean DR% and EactE_{\mathrm{act}} over six held-out initial conditions.

As γ\gamma increases, the guidance signal exerts stronger influence over the ODE trajectory, progressively shifting the terminal states toward lower actuation energy at the cost of drag reduction. However, the energy savings do not compensate for the loss in drag reduction: moving from γ=5\gamma= 5 to γ=20\gamma= 20 reduces EactE_{\mathrm{act}} by a factor of approximately 2.4 while sacrificing 12.4 percentage points of drag reduction. We therefore select γ=5\gamma= 5 as it achieves the highest drag reduction while maintaining low actuation energy. Table 6 summarises the results.

Guidance Scale (γ\gamma)DR (%) MeanDR (%) StdEactE_{\mathrm{act}} (×10−1\times10^{-1}) MeanEactE_{\mathrm{act}} (×10−1\times10^{-1}) Std
548.9511.700.09900.0711
1047.2210.610.15490.1104
2036.587.180.04190.0140
3035.666.780.05330.0165

Table 6. Guidance scale sweep for Policy-DRIFT: drag reduction (DR, %) and mean actuation energy EactE_{\mathrm{act}} averaged over 6 held-out initial conditions. The selected value is highlighted.

Computational Resources

All experiments were conducted on an HPC cluster. Two node types were employed:

GPU node (CFM training) One node was used for all CFM training runs: dual-socket x86_64 architecture, 64 physical cores (2×322 \times32-core sockets, 1 thread per core), 1,512 GB RAM, and 4×4 \times NVIDIA H200 GPUs.

CPU node (CFD simulation and DRL training) One node was used for flow simulations: dual-socket x86_64 architecture, 192 physical cores (2×962 \times96-core sockets, 1 thread per core), 373 GB RAM.

Each experimental run used a single node. DRL training was performed on the GPU node, while turbulent flow simulations, DRL training and inference were executed on the CPU node. The conditional flow matching (CFM) model converged in approximately 2.5 hours (500 epochs) on a single NVIDIA H200 GPU. The reward predictor (TRG) converged in approximately 10 hours (100 epochs) using a single NVIDIA H200 GPU.

Licenses

The entire codebase will be open-sourced upon publication in compliance with the licenses of all packages used. The relevant licenses are listed below:

  1. Dedalus — GNU GPL v3.0

  2. PyTorch — BSD 3-Clause License

  3. Stable-Baselines3 — MIT License

References

  1. [1]A. Ajay, Y. Du, A. Gupta, J. B. Tenenbaum, T. S. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision making? In International Conference on Learning Representations, 2023.arxiv.org/abs/2211.15657
  2. [2]M. S. Albergo and E. Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In International Conference on Learning Representations, 2023.arxiv.org/abs/2209.15571
  3. [3]M. Beneitez, A. Cremades, L. Guastoni, and R. Vinuesa. Improving turbulence control through explainable deep learning. arXiv preprint arXiv:2504.02354, 2025.arxiv.org/abs/2504.02354
  4. [4]T. R. Bewley, P. Moin, and R. Temam. Dns-based predictive control of turbulence: an optimal benchmark for feedback algorithms. Journal of Fluid Mechanics, 447:179–225, 2001.DOI
  5. [5]K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations, 2024.arxiv.org/abs/2305.13301
  6. [6]A. J. Bose, T. Akhound-Sadegh, K. Fatras, G. Huguet, J. Rector-Brooks, C.-H. Liu, A. C. Nica, M. Korablyov, M. M. Bronstein, and A. Tong. SE(3)-stochastic flow matching for protein backbone generation. arXiv preprint arXiv:2310.02391, 2023.arxiv.org/abs/2310.02391
  7. [7]K. J. Burns, G. M. Vasil, J. S. Oishi, D. Leocanet, and B. P. Brown. Dedalus: A flexible framework for numerical simulations with spectral methods. Physical Review Research, 2(2):023068, 2020.DOI
  8. [8]G. M. Cavallazzi, L. Guastoni, R. Vinuesa, and A. Pinelli. Deep reinforcement learning for the management of the wall regeneration cycle in wall-bounded turbulent flows. Flow, Turbulence and Combustion, 115(3):1291–1317, 2025.DOI
  9. [9]H. Choi, P. Moin, and J. Kim. Active turbulence control for drag reduction in wall-bounded flows. Journal of Fluid Mechanics, 262:75–110, 1994.DOI
  10. [10]J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414–419, 2022.DOI
  11. [11]P. Dhariwal and A. Q. Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021.
  12. [12]G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9):2419–2468, 2021.
  13. [13]J. Fan, S. Shen, C. Cheng, Y. Chen, C. Liang, and G. Liu. Online reward-weighted fine-tuning of flow matching with wasserstein regularization. In International Conference on Learning Representations, 2025.arxiv.org/abs/2502.06061
  14. [14]S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587–1596, 2018.
  15. [15]L. Guastoni, J. Rabault, P. Schlatter, H. Azizpour, and R. Vinuesa. Deep reinforcement learning for turbulent drag reduction in channel flows. The European Physical Journal E, 46(4):27, 2023.arxiv.org/abs/2301.09889
  16. [16]T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861–1870, 2018.arxiv.org/abs/1801.01290
  17. [17]D. Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan. Inverse reward design. Advances in Neural Information Processing Systems, 30, 2017.
  18. [18]H. Harder, A. Vishwasrao, L. Guastoni, R. Vinuesa, and S. Peitz. Efficient probabilistic surrogate modeling techniques for partially-observed large-scale dynamical systems. arXiv preprint arXiv:2511.04641, 2025.arxiv.org/abs/2511.04641
  19. [19]J. Ho and T. Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021.arxiv.org/abs/2207.12598
  20. [20]M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022.arxiv.org/abs/2205.09991
  21. [22]Y. Kametani, K. Fukagata, R. Örlü, and P. Schlatter. Effect of uniform blowing/suction in a turbulent boundary layer at moderate Reynolds number. International Journal of Heat and Fluid Flow, 55:132–142, 2015.DOI
  22. [23]J. Kim, P. Moin, and R. Moser. Turbulence statistics in fully developed channel flow at low Reynolds number. Journal of Fluid Mechanics, 177:133–166, 1987.DOI
  23. [24]C. Laidlaw, S. Singhal, and A. Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. In International Conference on Learning Representations, 2025.arxiv.org/abs/2403.03185
  24. [25]T. Lee, J. Kim, and C. Lee. Turbulence control for drag reduction through deep reinforcement learning. Physical Review Fluids, 8(2):024604, 2023.DOI
  25. [26]M. Lienen, D. Lüdke, J. Hansen-Palmus, and S. Günnemann. From zero to turbulence: Generative modeling for 3D flow simulation. In International Conference on Learning Representations, 2024.arxiv.org/abs/2306.01776
  26. [27]T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.arxiv.org/abs/1509.02971
  27. [28]Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02747
  28. [29]X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.arxiv.org/abs/2209.03003
  29. [30]J. Luo, Z. Hu, C. Xu, S. Gadipudi, A. Sharma, R. Ahmad, S. Schaal, C. Finn, A. Gupta, and S. Levine. SERL: A software suite for sample-efficient robotic reinforcement learning. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 2023.arxiv.org/abs/2401.16013
  30. [31]B. K. Miller, R. T. Q. Chen, A. Sriram, and B. M. Wood. FlowMM: Generating materials with riemannian flow matching. In International Conference on Machine Learning, 2024.arxiv.org/abs/2406.04713
  31. [32]A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In IEEE International Conference on Robotics and Automation, pages 7559–7566, 2018.arxiv.org/abs/1708.02596
  32. [33]M. Olivecrona, T. Blaschke, O. Engkvist, and H. Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 9(1):48, 2017.
  33. [34]A. Pan, K. Bhatia, and J. Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2201.03544
  34. [35]Y. Qin, Y. Wang, H. Zhou, P. Liu, H. Dong, and Y. Ji. IPD: Boosting sequential policy with imaginary planning distillation in offline reinforcement learning. arXiv preprint arXiv:2603.04289, 2026.arxiv.org/abs/2603.04289
  35. [36]A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-Baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1–8, 2021.
  36. [37]A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz. Diffusion policy policy optimization. In International Conference on Learning Representations, 2025.arxiv.org/abs/2409.00588
  37. [38]A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015.arxiv.org/abs/1511.06295
  38. [39]J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471, 2022.arxiv.org/abs/2209.13085
  39. [40]Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  40. [41]T. Sonoda, Z. Liu, T. Itoh, and Y. Hasegawa. Reinforcement learning of control strategies for reducing skin friction drag in a fully developed turbulent channel flow. Journal of Fluid Mechanics, 960: A30, 2023.DOI
  41. [42]A. Tong, N. Malkin, G. Huguet, Y. Zhang, J. Rector-Brooks, K. Fatrás, G. Wolf, and Y. Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023.arxiv.org/abs/2302.00482
  42. [44]R. Vinuesa, H. Azizpour, I. Leite, M. Balaam, V. Dignum, S. Doinproceedings, A. Felländer, S. D. Langhans, M. Tegmark, and F. Fuso Nerini. The role of artificial intelligence in achieving the sustainable development goals. Nature Communications, 11(1):233, 2020.DOI
  43. [45]A. Vishwasrao, S. B. C. Gutha, A. Cremades, K. Wijk, A. Patil, C. Gorle, B. J. McKeon, H. Azizpour, and R. Vinuesa. Diff-SPORT: Diffusion-based sensor placement optimization and reconstruction of turbulent flows in urban environments. arXiv preprint arXiv:2506.00214, 2025.arxiv.org/abs/2506.00214
  44. [46]D. Wälchli, L. Guastoni, R. Vinuesa, and P. Koumoutsakos. Drag reduction in a minimal channel flow with scientific multi-agent reinforcement learning. Journal of Physics: Conference Series, 2753(1):012024, 2024.DOI
  45. [47]Y. Wang, P. Suarez, M. Bode, and R. Vinuesa. Physics-guided surrogate learning enables zero-shot control of turbulent wings. arXiv preprint arXiv:2604.09434, 2026.arxiv.org/abs/2604.09434
  46. [48]G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic MPC for model-based reinforcement learning. In IEEE International Conference on Robotics and Automation, pages 1714–1721, 2017.DOI
  47. [49]Z. Zhou, M. Zhang, and X. Zhu. Reinforcement-learning-based control of turbulent channel flows at high Reynolds numbers. Journal of Fluid Mechanics, 1006:A12, 2025.DOI

Paper details

vlnc:W_96jv2za7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents