betaFeedback

WorksPhysics and Astronomy

Deep reinforcement learning with spatial and temporal awareness for active boundary control of buoyancy-driven convection

Abstract

Deep reinforcement learning (DRL) applied to thermal convection control consistently produces degenerate actuation: wall-temperature policies whose outputs are saturated, pseudo-random, or spatially incoherent. Two compounding deficiencies are responsible: multilayer-perceptron policies that discard spatial flow structure, and memoryless policies that cannot distinguish self-induced flow changes from background evolution. Together they degrade the actuation into forms whose relation to the convective topology cannot be read off and which are not realisable at the actuator, even when cell coalescence (the merging of convection rolls into fewer, larger structures), which would reduce Nu, is accessible to boundary actuation. The present framework addresses both causes through four targeted design choices: convolutional policy networks, Gated Recurrent Unit (GRU) memory, off-policy training (TD3/MADDPG), and action-smoothness constraints. A systematic 2x2 factorial design isolates the contribution of each component. On Rayleigh-Benard convection at Ra = 10,000, all four configurations achieve cell coalescence and reduce Nu to as low as 1.83 (26% below the uncontrolled baseline) in 350 episodes, without the full-field data augmentation required by prior work. Crucially, coalescence is achieved even by the single-agent configuration, demonstrating that the multi-agent formulation is not a prerequisite once the policy architecture is sufficiently expressive. Applied to double-diffusive convection in the salt-finger regime, the framework spontaneously discovers a travelling-wave actuation whose phase speed adapts to the evolving mixing state of the flow, enhancing heat transfer by 19.1% and reducing salinity variance by 21.0%.

Introduction

Active control of buoyancy-driven flows, where thermal or compositional density differences drive fluid motion, is a problem of broad scientific and engineering relevance. The ability to suppress or enhance convective heat and mass transfer through boundary actuation has direct applications in thermal management, geophysical modelling, and industrial processes involving stratified fluids. The present work targets two such mechanisms — cell coalescence in Rayleigh–Bénard convection and salt-finger disruption in double-diffusive convection — and uses deep reinforcement learning as the means of inducing them at the boundary. Among buoyancy-driven flows, Rayleigh–Bénard convection (RBC), the motion driven by heating a fluid layer from below and cooling it from above, occupies a central and canonical place. At moderate Rayleigh numbers (Ra=gβΔTH3/νκ\mathrm{Ra} = g \beta\Delta T H^3/\nu\kappa, where gg is the gravitational acceleration, β\beta the thermal expansion coefficient, ΔT\Delta T the imposed temperature difference, HH the layer height, ν\nu the kinematic viscosity, and κ\kappa the thermal diffusivity) the flow self-organises into steady convection rolls whose number and spatial arrangement set the global heat transfer rate, quantified by the Nusselt number (Nu=qwH/λΔT\mathrm{Nu} = q_w H/\lambda\Delta T, where qwq_w is the wall heat flux and λ\lambda the thermal conductivity). Reducing Nu through boundary actuation is physically equivalent to reorganising this roll topology: by promoting cell coalescence (i.e. the merging of multiple rolls into fewer, larger structures) the boundary-layer contact area per roll decreases, plume impingement weakens, and heat transport approaches the conductive limit. The roll topology and the coherent structures that organise it — plumes, thermals, and a large-scale circulation — have been characterised in detail in moderately wide cylindrical domains (Sakievich et al., [27]). The simplicity of the control metric (a single scalar, Nu) and the geometric interpretability of the underlying mechanism make RBC a uniquely clean testbed for data-driven control.

However, deriving effective control laws for such flows is fundamentally difficult: the governing equations are nonlinear, thermal perturbations at the wall must first diffuse and then be advected through the bulk before their effect becomes observable, and the optimal strategy is rarely known a priori. Classical feedback approaches, ranging from proportional–derivative (PD) controllers to linear optimal control, require detailed knowledge of the linearised flow dynamics and lose effectiveness as the Rayleigh number increases and nonlinear interactions intensify [18]. Data-driven and machine learning methods offer an alternative route [5, 36], and reinforcement learning in particular is well suited to sequential decision-making in systems where no explicit model is available [12].

The application of deep reinforcement learning (DRL) to active flow control was demonstrated by Rabault et al. [26], who showed that a neural network policy trained through interaction with a high-fidelity numerical solver can discover effective drag-reduction strategies for flow past a cylinder at Re⁡=100\operatorname{Re} = 100. Their work established the paradigm of coupling a DRL agent to a fluid solver through a standardised environment interface, and has since been extended across a wide range of configurations. For turbulent channel flow, where classical active control strategies rely on wall blowing and suction to suppress near-wall turbulence and reduce skin-friction drag, Guastoni et al. [13] demonstrated that off-policy algorithms can learn effective actuation strategies directly from DNS data, without requiring a prescribed control law. In a similar physical setting, Cavallazzi et al. [6] targeted the self-sustaining wall regeneration cycle, achieving drag reduction through stabilisation of velocity streaks. Font et al. [9] demonstrated DRL-based control of a 3D turbulent separation bubble over a flat plate under adverse pressure gradient, achieving a 9% reduction in bubble area; the learned policy remained effective when transferred from the coarse training grid to a finer resolution.

When the flow domain is wide or the actuation is spatially distributed, however, a single agent observing the full field becomes increasingly impractical, both in terms of observation dimensionality and the difficulty of learning coordinated spatial actuation. Building on these single-agent results, multi-agent formulations have been applied to flows past three-dimensional cylinders in the transitional regime with Reynolds numbers up to Re⁡D=400\operatorname{Re}_{D} = 400 [30]. In the fully turbulent regime at Re⁡D=3900\operatorname{Re}_{D} = 3900, Suárez et al. [31] have discovered multi-frequency control laws two orders of magnitude more mass-efficient than conventional periodic forcing. Most recently, García et al. [11] have extended this lineage to airfoil separation control, achieving a 43.9% drag reduction on a NACA 0012 at Re⁡D=3000\operatorname{Re}_D = 3000 via PPO-controlled blowing and suction at the surface.

In parallel with these aerodynamic applications, a separate thread has applied DRL to forced-convection cooling. Wang et al. [38] use a deep Q-network to regulate inlet conditions in an open cavity with discrete heat sources, reaching wall-temperature reductions of approximately 8 K over manually tuned controllers; Wang et al. [39] subsequently show that policies trained on a 2D configuration transfer to 3D at roughly 10% of the direct training cost, while noting that the translational-invariance argument behind multi-agent RBC control does not extend cleanly to forced-convection settings.

A common thread runs through these developments. Given enough interaction and a large enough network, a reinforcement learning agent should in principle be able to learn a control law for any of these problems, so the practical obstacle is not what the policy class can represent but what the optimiser can find: when a system carries strong underlying structure and the learning problem is formulated without reference to it, the region of the action space that has to be explored grows with the number of actuators and the exploration cost becomes prohibitive. Belus et al. [3] made this argument directly for the stabilisation of a falling liquid film, where the combined combinatorial size of the output domain grows as a power of the number of jets, and showed that sharing one policy across local agents and rewarding each of them locally recovers what a monolithic formulation loses. That argument, rather than any property of a particular algorithm, is what the multi-agent formulations of Vignon et al. [35], Font et al. [9] and Suárez et al. [31] exploit, and it admits more than one implementation. Jeon et al. [16] revisit the configuration of Vignon et al. [35] and build the local reflection and rotation symmetries of the flow into group-invariant convolutional networks, retaining the multi-agent re-centring for the global translational invariance and improving the reproducibility of the learnt policies. The same reasoning applies on the temporal side, where Wang et al. [37] show for cylinder wake control with sparse pressure sensing that a policy acting on the instantaneous sensor signal is handicapped by the delay between actuation and response, and recover the missing information by lifting the measurement with its time derivative. Recurrence is an alternative route to that end, and the contribution of the present work is to supply the spatial and the temporal structure together, through a convolutional policy with gated recurrent memory that requires neither re-centring nor a delay. horizon fixed in advance. The two routes to temporal context are compared directly in § 4.

Returning to RBC, Beintema et al. [2] were the first to apply RL to this problem, suppressing convection entirely for Ra≲3×104Ra \lesssim3 \times10^{4} in a unit-aspect-ratio domain and outperforming classical PD control. However, their limited domain constrained the flow to a single-cell configuration. Vignon et al. [35] extended the approach to wider periodic domains using a multi-agent RL (MARL) framework with full-field re-centred observations, where optimising for Nu reduction leads to cell coalescence, reducing Nu from approximately 2.7 to 2.0 at Ra=10,000Ra = 10{,}000. Their approach draws two distinct benefits from the multi-agent decomposition: each environment step yields as many transitions as there are agents, and the re-centring of the full field about each agent is what allows a single shared policy to act differently on different segments. The MARL framework was subsequently extended to three-dimensional RBC by Vasanth et al. [34], achieving convection-intensity reductions of 23.5% and 8.7% at Ra=500Ra = 500 and 750, respectively. More recent work has extended DRL-based RBC control toward higher Rayleigh numbers: Markmann et al. [24] report Nu reductions of up to 33% at moderate RaRa and 10% in the chaotic regime using single-agent PPO with reward shaping; Chen and Constante-Amores [7] couple RL with a reduced-order model derived from proper orthogonal decomposition and autoencoders, achieving 16–23% Nu reduction under both single- and dual-boundary actuation.

Outside the buoyancy-driven setting, Holme et al. [15] address the multi-timescale nature of rotating detonation engine control through a moving reference frame that separates fast wave propagation from slower mode transitions, and through a physically anchored credit-assignment horizon γ=1−Δt/Th\gamma= 1 - \Delta t/T_h with ThT_h chosen on the gain-recovery timescale; the action period Δt\Delta t itself is treated as a swept hyperparameter rather than tied to a particular physical scale.

Despite these advances, the policies reported in the RBC control literature exhibit a recurring and underappreciated pathology: degenerate actuation. The actor outputs typically either saturate at the control bounds or settle into spatially disordered patterns with no clear connection to the convective topology. Such strategies may lower the reward functional by a small margin yet encode no physical mechanism. In turbulent flow control, where the actuation cadence is fast relative to the flow dynamics, this degeneracy manifests most visibly as bang-bang actuation: policies that alternate rapidly between extreme actuator values at each step, producing physically non-realisable switching that is energetically prohibitive and harmful to actuator hardware. In RBC at moderate Rayleigh numbers, where the actuation timescale is comparable to the convective turnover, the same root causes produce a related but distinct failure mode: spatially incoherent or saturated wall-temperature patterns that happen to perturb the flow without driving meaningful roll reorganisation.

We identify two compounding deficiencies as responsible. The first is insufficient network expressivity: all prior DRL approaches to RBC use MLP policies that receive the flow state as a flattened vector, discarding spatial locality and translational structure. An MLP cannot easily learn that adjacent wall segments should be actuated in concert to match the wavelength of the underlying convection rolls, and instead converges to the degenerate patterns described above. The second deficiency is the absence of temporal context. In a multi-agent setting, where each agent observes only a local fraction of the flow domain, the individual observation is no longer a sufficient statistic for the full system state: the problem is formally a decentralised partially observable Markov decision process (Dec-POMDP). A memoryless policy operating on a single snapshot cannot distinguish whether a local temperature change is the result of its own prior actuation or of the natural flow evolution, and this ambiguity drives the optimiser toward saturated or random outputs as a hedging strategy. An alternative remedy, recently applied to bluff-body wake control by Jia and Xu [17], augments the observation with a Kalman-filtered state estimate rather than introducing recurrence; the recurrent route adopted here learns the relevant summary internally and scales more directly to the high-dimensional spatially distributed observations of multi-agent wall control. To the best of our knowledge, both deficiencies are present, to a varying degree, in all existing DRL approaches to thermal convection control, and they have not been addressed together.

Both deficiencies have known remedies in the broader reinforcement learning literature that have not yet been brought to bear on thermal convection control. A recurrent policy, in particular one built around a gated unit such as the GRU, carries a hidden state across decision steps that lets each agent follow the delayed flow response to its own actuation and separate self-induced changes from background evolution; the architecture has a longstanding role in partially observable RL [14, 8, 19] and has been shown to encode genuine flow physics in turbulent wall modelling [1]. A convolutional encoder, in turn, treats the flow field as a structured spatial array and extracts coherent features without the full-field data augmentation of Vignon et al. [35]; graph neural networks extend the same inductive bias to arbitrary geometries [21], and explainability analyses confirm that spatially aware policies recover physically meaningful flow structures [4]. Off-policy training closes the loop: TD3 (Twin Delayed Deep Deterministic policy gradient) for the single-agent case and MADDPG (Multi-Agent Deep Deterministic Policy Gradient; Lowe et al. [23]) for the multi-agent case reuse past transitions through a replay buffer, cut the number of costly solver episodes [13], and accommodate recurrent actors through sequence sampling [19].

The present work addresses these gaps simultaneously through four design choices. Convolutional policy networks operating on local spatial patches replace the global MLP observations of Vignon et al. [35], providing spatial feature extraction through local receptive fields and weight sharing across the domain. Recurrent actors, augmented with a Gated Recurrent Unit (GRU), maintain temporal context across actuation steps, enabling each agent to track the flow response to its own past actions. Off-policy training, using a deterministic policy gradient algorithm for the single-agent and a multi-agent extension thereof (Lowe et al., [23]) for the multi-agent configuration, replaces on-policy methods, storing past transitions in a replay buffer and reusing them across multiple updates to reduce the number of expensive solver episodes required. Finally, action constraints, including a zero-mean projection, an amplitude cap, and spatial and temporal smoothness penalties, explicitly discourage the saturated and incoherent outputs that characterise degenerate policies.

Four configurations are compared: single-agent and multi-agent formulations, each trained with and without GRU recurrence, allowing the contributions of temporal memory and of the multi-agent structure to be assessed independently. The key finding is that, with sufficient architectural expressivity and temporal memory, even a single-agent policy achieves cell coalescence and meaningful Nu\mathrm{Nu} reduction at Ra=10,000Ra = 10{,}000 in a wide periodic domain. The barrier in prior work was not the agent topology but the policy architecture, and the multi-agent formulation of Vignon et al. [35] is not a prerequisite for discovering the correct physical mechanism. The multi-agent configuration provides additional benefits, including better spectral alignment with the dominant convective mode and a larger Nu\mathrm{Nu} reduction, while GRU recurrence accelerates convergence by approximately 100 episodes across all configurations. The framework is validated on two buoyancy-driven configurations of increasing complexity: RBC at Ra=10,000\mathrm{Ra}=10{,}000, where the objective is to reduce Nu\mathrm{Nu} through cell coalescence, and double-diffusive convection in the salt-finger regime, where the objective is to enhance heat transfer and accelerate scalar mixing. In both cases the learned strategies are smooth, spatially structured, and physically interpretable, with no trace of the degenerate outputs that motivated the architectural choices above.

The remainder of the paper is structured as follows. § 2 describes the governing equations and flow configurations. § 2.2 details the numerical solver. § 3 presents the DRL formulation. § 4 and § 5 report training and evaluation results for the RBC case. § 6 presents the double-diffusive extension. § 7 summarises the main findings.

Physical problem and discrete formulation

Governing equations, computational domain and boundary conditions

We consider a two-dimensional rectangular domain of width Lx=2πL_x = 2\pi and height Ly=2HL_y = 2H, with H=1H = 1 the half-height used for non-dimensionalisation. The fluid is incompressible and obeys the Boussinesq approximation: density variations appear only in the buoyancy term.

Using the free-fall scaling, lengths are made non-dimensional by HH, velocities by the free-fall velocity Uf=g β ΔT HU_f = \sqrt{g \, \beta\, \Delta T \, H}, time by H/UfH/U_f, pressure by ρ0Uf2\rho_0 U_f^2, and the temperature by ΔT\Delta T, i.e. the difference between the mean bottom-wall temperature THT_H and the top-wall temperature TCT_C. The resulting dimensionless system reads:

∇⋅u=0,(1)\nabla\cdot\boldsymbol{u} = 0, \tag*{(1)}
∂u∂t+(u⋅∇)u=−∇p+PrRa∇2u+Te^y,(2)\frac{\partial\boldsymbol{u}}{\partial t} + (\boldsymbol{u} \cdot\nabla)\boldsymbol{u} = -\nabla p + \sqrt{\frac{\mathrm{Pr}}{\mathrm{Ra}}}\nabla^2 \boldsymbol{u} + T\hat{\boldsymbol{e}}_y, \tag*{(2)}
∂T∂t+(u⋅∇)T=1Ra Pr∇2T.(3)\frac{\partial T}{\partial t} + (\boldsymbol{u} \cdot\nabla)T = \frac{1}{\sqrt{\mathrm{Ra}\,\mathrm{Pr}}}\nabla^2 T. \tag*{(3)}

The system is governed by the Rayleigh number Ra\mathrm{Ra} and the Prandtl number Pr=ν/κT\mathrm{Pr} = \nu/\kappa_T. No-slip conditions (u=0)(\boldsymbol{u} = 0) are imposed at both horizontal walls. The top wall is held at fixed temperature TC=1T_C = 1, while the bottom wall temperature TH=2T_H = 2 serves as the reference value around which the agents apply zero-mean perturbations δT(x)\delta T(x), so that the effective bottom-wall temperature is TH+δT(x)T_H + \delta T(x) at any given time. The xx-direction is periodic. The non-dimensional temperature is shifted rather than centred about its mean, so the uniform part of the buoyancy term in equation (2) is constant and is absorbed into the pressure gradient.

Throughout the paper the Nusselt number is evaluated from the wall temperature gradients as the mean of the values at the two horizontal walls,

Nu=12(Nubot+Nutop),Nuwall=−HTH−TC⟨∂T∂y⟩x,wall,(4)\mathrm{Nu} = \frac{1}{2}\left(\mathrm{Nu}_{\mathrm{bot}} + \mathrm{Nu}_{\mathrm{top}}\right), \qquad\mathrm{Nu}_{\mathrm{wall}} = -\frac{H}{T_H-T_C}\left\langle\frac{\partial T}{\partial y}\right\rangle_{x,\mathrm{wall}}, \tag*{(4)}

where ⟨⋅⟩x,wall\langle\cdot\rangle_{x,\mathrm{wall}} denotes the average along the wall. At the bottom wall the gradient is evaluated through the ghost cell, so that the instantaneous actuated temperature TH+δT(x)T_H + \delta T(x) enters the estimate, while the normalisation retains the fixed reference difference TH−TCT_H-T_C rather than the instantaneous wall temperature. Equation (4) is the quantity reported in all figures and tables, and it is the one entering the reward of § 3.4.

All the present simulations use Ra=10,000\mathrm{Ra} = 10{,}000, Pr=0.71\mathrm{Pr} = 0.71 (air) and Lx=2πL_x = 2\pi. The uncontrolled steady state exhibits four counter-rotating convection rolls, yielding a baseline Nusselt number Nu0≈2.48\mathrm{Nu}_0 \approx2.48. The DRL control objective is to reduce Nu by promoting cell coalescence, the merging of rolls into fewer, larger structures under wall actuation, which weakens convective mixing and drives Nu toward the conductive limit (Nu = 1), where heat transfer occurs entirely by diffusion.

Numerical method

The governing equations are solved using a staggered-grid pressure correction method ([33]) implemented entirely in PyTorch, enabling execution on both CPU and GPU. To avoid numerical dissipation, all spatial operators are discretised using second-order central differences. Time integration uses an explicit Adams-Bashforth scheme with adaptive time stepping controlled by CFL and Fourier number constraints, maintaining CFL ≈0.1\approx0.1 and Fo <0.25< 0.25 on a 96×6496 \times64 uniform grid. The solver is wrapped in Gymnasium (single-agent) and PettingZoo (multi-agent) environment interfaces that handle observation extraction, action application, reward computation, and episode management.

Deep reinforcement learning formulation

In the reinforcement learning paradigm, a decision-making agent interacts repeatedly with an environment (here the Boussinesq flow solver) by observing its current state, selecting an action, and receiving a scalar reward that reflects how desirable the outcome was. Through trial and error the agent learns a policy, i.e. a mapping from observed states to actions, that maximises the cumulative reward over time. Applied to the present problem, the agent observes the instantaneous temperature and velocity fields, perturbs the bottom-wall temperature over a set of discrete segments, and is rewarded according to the resulting change in heat transfer.

Formally, the control task is cast as a Markov decision process (S,A,P,r,γ)(S, A, P, r, \gamma), where SS is the space of instantaneous flow fields sts_t, at∈A⊆RNa_t \in A \subseteq\mathbb{R}^{N} the wall-temperature perturbations over NN segments, PP the transition kernel P(st+1∣st,at)P(s_{t+1} \mid s_t, a_t) (i.e. the probability of the flow advancing to state st+1s_{t+1} from state sts_t under actuation ata_t) governed by the Boussinesq equations, r(st,at)r(s_t, a_t) the scalar reward, and γ∈(0,1)\gamma\in(0,1) a discount factor that down-weights rewards received further in the future, ensuring the cumulative sum remains bounded. The agent seeks a deterministic policy πθ:S→A\pi_\theta: S \to A that maximises the cumulative discounted reward over a trajectory, starting from the cached uncontrolled steady state s0s_0,

J(θ)=∑t=0∞γtr(st,πθ(st)),st+1=P(st,πθ(st)),(5)J(\theta) = \sum_{t=0}^{\infty} \gamma^t r(s_t, \pi_\theta(s_t)), \qquad s_{t+1} = P(s_t, \pi_\theta(s_t)), \tag*{(5)}

where the flow state st+1s_{t+1} is obtained by advancing the Boussinesq solver one actuation step from sts_t under the wall temperature prescribed by πθ(st)\pi_\theta(s_t). In plain terms, the agent is rewarded at every step for reducing heat transfer, but rewards collected further in the future count for less (through the factor γt\gamma^t); the policy πθ\pi_\theta is then the wall-temperature strategy that accumulates the highest total score over the entire episode.

Two off-policy algorithms are used to optimise J(θ)J(\theta): Twin Delayed Deep Deterministic policy gradient (TD3; [10]) for the single-agent case, and Multi-Agent Deep Deterministic Policy Gradient (MADDPG; [23]) for the multi-agent case. Both TD3 and MADDPG improve the policy by following the gradient of J(θ)J(\theta) with respect to the network parameters θ\theta ([28, 22]),

∇θJ(θ)=E[∇θπθ(s)∇aQπ(s,a)∣a=πθ(s)],(6)\nabla_\theta J(\theta) = \mathbb{E}\left[\nabla_\theta\pi_\theta(s) \left.\nabla_a Q^\pi(s,a)\right|_{a=\pi_\theta(s)}\right], \tag*{(6)}

where Qπ(s,a)Q^{\pi}(s,a) is the action-value function, i.e. the total discounted reward expected from state ss when action aa is applied and the policy πθ\pi_{\theta} is followed thereafter,

Qπ(s,a)=∑t≥0γtr(st,at),s0=s, a0=a.(7)Q^{\pi}(s,a) = \sum_{t \ge0} \gamma^{t} r(s_{t},a_{t}), \qquad s_{0}=s,\ a_{0}=a. \tag*{(7)}

In practice, QπQ^{\pi} is not known and is approximated by a second neural network (the critic), trained concurrently with the policy (the actor). The gradient in equation (6) then tells the actor how to adjust its output so that the critic predicts a higher future reward.

When multiple agents control different portions of the wall simultaneously, each one can only sense its local neighbourhood rather than the full flow domain. This limited observability means the single-agent MDP framework is no longer sufficient: the action of one agent affects the observations of its neighbours, so the agents must learn to coordinate through the shared reward signal alone, without any direct communication between them. When multiple agents control different portions of the wall simultaneously, each one can only sense its local neighbourhood rather than the full flow domain. The action of one agent therefore affects the observations of its neighbours, so the agents must learn to coordinate through the shared reward signal alone, without any direct communication between them. The multi-agent setting is then described by extending the MDP tuple to include a separate observation space Oj\mathcal{O}_{j} for each agent jj,

(S,{Aj}j=1N,P,r,{Oj}j=1N,γ),(8)\left(\mathcal{S},\{\mathcal{A}_{j}\}_{j=1}^{N},P,r,\{\mathcal{O}_{j}\}_{j=1}^{N},\gamma\right), \tag*{(8)}

in which agent jj receives only the local observation oj=Oj(s)o_{j}=\mathcal{O}_{j}(s), the flow state projected onto the patch of three consecutive wall segments visible to that agent (Figure 1), and outputs aj∈Aj=[−1,1]a_{j}\in\mathcal{A}_{j}=[-1,1].

Schematic of the control setup for the multi-agent formulation

Figure 1. Schematic of the control setup for the multi-agent formulation. The bottom wall is divided into N=10N = 10 segments; agent 5 observes the local flow state over a patch of three consecutive segments (shaded orange box) but applies its temperature perturbation δT\delta T to its own central segment only (shaded green). The top wall is held at fixed temperature TCT_C; the xx-direction is periodic.

All agents share the same scalar reward rr, so they are collectively driven to improve the global heat transfer objective rather than optimising their individual segments in isolation. During training, a shared critic is given access to the full flow state and all agents’ actions, allowing it to assess the global consequence of each agent’s local decision. At deployment, however, each agent acts on its local observation alone, without knowledge of what its neighbours are doing. This centralised-training, decentralised-execution strategy (CTDE; [23]) allows the agents to learn coordinated behaviour during training while remaining independently deployable.

Four configurations are compared: single-agent (TD3) and multi-agent (MADDPG), each trained with and without GRU recurrence. The effect of each choice is isolated by comparing all four combinations, as detailed in the subsections below.

Single-agent TD3

The single-agent formulation treats the entire bottom wall as one control surface, observing the full downsampled flow field and outputting temperature perturbations for all N=10N = 10 segments simultaneously. The three physical fields (T,u,v)(T, u, v) are downsampled by a factor of 2 (from 96×6496 \times64 to 48×3248 \times32), and an optional fourth channel encodes the previous action as a spatially constant field, giving an observation o∈RC×32×48o \in\mathbb{R}^{C \times32 \times48} with C=3C = 3 or 4.

The actor processes this observation through four stride-2 convolutional layers with 3×33 \times3 kernels and LeakyReLU activation, reducing the spatial dimensions from 32×4832 \times48 to 2×32 \times3 and yielding a 768-dimensional feature vector after flattening. A fully connected layer with LayerNorm maps this to a 256-dimensional hidden representation, followed by a final linear layer with tanh activation producing a∈[−1,1]10a \in[-1, 1]^{10}.

The off-policy algorithm underlying the single-agent formulation is Twin Delayed DDPG (TD3; [10]), chosen for its sample efficiency and training stability relative to on-policy alternatives such as PPO. The architecture and training hyperparameters follow Fujimoto et al. [10]; the reader is referred there for full algorithmic details.

Multi-agent MADDPG

As in the single-agent formulation, the bottom wall is divided into N=10N = 10 equal segments, but here each segment is controlled by a dedicated agent jj (j=1,…,10j = 1, \ldots, 10). Agent jj controls the temperature over its segment of width Lx/NL_x/N and observes a local patch spanning 3 consecutive segments (its own plus one neighbour on each side, with periodic wrapping — see Figure 1). All agents share the same policy network.

The value N=10N = 10 follows Vignon et al. [35] and is retained here so that the two studies remain directly comparable without the segmentation acting as a confounding difference. Appendix § 9 reports what happens when it is varied.

Each agent uses a compact convolutional network: two convolutional layers extract spatial features from the local observation, followed by a fully connected layer that produces a single temperature perturbation for that agent’s segment.

During training, a centralised critic has access to the full flow state and all agents’ actions, allowing it to assess the global consequence of each local decision. At deployment, only the local actor is used — each agent acts on its own observation alone, without knowledge of what its neighbours are doing, consistent with the strategy described in § 3.

Action constraints

All configurations share the same action-processing pipeline. The raw actor outputs are first shifted to have zero mean,

aj′=aj−1N∑k=0N−1ak,(9)a'_j = a_j - \frac{1}{N}\sum_{k=0}^{N-1} a_k, \tag*{(9)}

so that the actuation adds no net heat to the bottom wall. The corrected actions are then rescaled to prevent any single segment from saturating the actuator. Whenever the largest absolute value exceeds a threshold (here C=0.75C = 0.75), all actions are scaled down proportionally,

a~j={aj′if ∥a′∥∞≤C,aj′⋅C/∥a′∥∞otherwise.(10)\tilde{a}_j = \begin{cases} a'_j & \text{if } \lVert a' \rVert_\infty\le C,\\ a'_j \cdot C/\lVert a' \rVert_\infty& \text{otherwise}. \end{cases} \tag*{(10)}

Finally, cubic Hermite interpolation over 10% of the segment width smooths the temperature profile at segment boundaries, yielding the wall condition Twall(x)=TH+δT(x)T_{\mathrm{wall}}(x) = T_H + \delta T(x).

Reward design

At each actuation step tt, each agent jj receives a scalar measure of how much the instantaneous heat transfer (i.e. the instantaneous Nusselt number) has been reduced relative to the uncontrolled baseline (i.e. a reward),

rjt=m[nshift−(1−β) Nuinstt−β Nuloc,jt].(11)r_j^t = m\left[n_{\mathrm{shift}} - (1-\beta)\,\mathrm{Nu}_{\mathrm{inst}}^t - \beta\,\mathrm{Nu}_{\mathrm{loc},j}^t\right]. \tag*{(11)}

In equation (11), nshift=2.67n_{\mathrm{shift}} = 2.67 is an offset chosen so that the reward is positive when Nu falls below the uncontrolled baseline, and m=1m = 1 is a scaling factor. The small weight β=0.0015\beta= 0.0015 blends a local Nusselt number Nuloc,jt\mathrm{Nu}_{\mathrm{loc},j}^t, computed over agent jj’s own segment, into the global value, giving each agent a localised measure of its individual contribution to the overall heat transfer reduction. For the single-agent case the local term vanishes (β=0\beta= 0) and the reward simplifies to r=m(nshift−Nuinst)r = m(n_{\mathrm{shift}} - \mathrm{Nu}_{\mathrm{inst}}): i.e. the agent is rewarded in direct proportion to the reduction in global heat transfer.

GRU architecture

The flow response to wall actuation is not instantaneous, as convective structures take time to form, merge, and reorganise after a change in boundary temperature. As a result, the instantaneous flow snapshot alone does not carry enough information to determine the best next action: the same snapshot could arise from a flow that is responding to a recent actuation or from one that has been evolving freely, and the two situations call for different responses. Retaining a memory of past observations resolves this ambiguity, allowing the agent to track how the flow is evolving over time and to attribute changes in heat transfer to its own previous actions. To provide this memory, a Gated Recurrent Unit (GRU; [8]) is incorporated into the policy network, between the convolutional encoder (the CNN that processes the flow snapshot) and the output layer that produces the wall-temperature action. At each actuation step, the GRU updates a hidden state vector that summarises the history of past observations, which is then carried forward to the next step. The CNN feature vector is first projected to dimension h=64h = 64 via a linear layer, then passed through the GRU, whose update equations are:

rt=σ(Wrzt+Urht−1+br),(12)r_t = \sigma(W_r z^t + U_r h^{t-1} + b_r), \tag*{(12)}
h~t=tanh⁡(Whzt+Uh(rt⊙ht−1)+bh),(13)\tilde{h}_t = \tanh(W_h z^t + U_h(r_t \odot h^{t-1}) + b_h), \tag*{(13)}
gt=σ(Wgzt+Ught−1+bg),(14)g_t = \sigma(W_g z^t + U_g h^{t-1} + b_g), \tag*{(14)}
ht=(1−gt)⊙ht−1+gt⊙h~t,(15)h^t = (1-g_t) \odot h^{t-1} + g_t \odot\tilde{h}_t, \tag*{(15)}

where ztz^t is the projected CNN feature vector, hth^t the hidden state carrying memory across actuation steps, σ\sigma the sigmoid function, and ⊙\odot elementwise multiplication. The reset gate rtr_t controls how much of the past memory is retained at each step, while the update gate gtg_t balances the contribution of the new flow observation against that memory.

Training with memory requires temporally contiguous data, since the past observations must be propagated forward in time to build up a meaningful memory. Subsequences of length L=16L = 16 are therefore sampled from stored episodes. The first B=4B = 4 steps of each subsequence are used solely to warm up the memory before the policy update is computed, and are not used in the parameter update ([19]).

Training procedure

All four configurations (single-agent TD3 and multi-agent MADDPG, each with and without GRU recurrence) are trained for 350 episodes of 200 actions each (150,000 solver time steps per episode). Before training, the solver is advanced for 400 thermal diffusion times (tκT/H2=400t_{\kappa T}/H^2 = 400) to reach the uncontrolled steady state, which is saved and reloaded at the start of each episode.

Key training parameters (shared across all configurations unless noted) are listed in Table 1.

CategoryParameterValue
PhysicsRayleigh number RaRa10,000
PhysicsPrandtl number PrPr0.71
PhysicsGrid96×6496 \times64
ControlSegments NN10
ControlSubsteps per action750
ControlAmplitude cap CC0.75
ControlSmoothing fraction10%
NetworksGRU hidden size64
NetworksActor / critic LR10−310^{-3} / 10−310^{-3}
NetworksTarget update τ\tau0.005
TrainingMax episodes350
TrainingEpisode length200 actions
TrainingBatch size64 (TD3) / 32 (MADDPG)
TrainingDiscount γ\gamma0.99
TrainingGradient clip0.75
ExplorationInitial noise σ0\sigma_00.1
ExplorationFinal noise σf\sigma_f / decay0.01 / 100 ep

Table 1. Training hyperparameters.

Left unconstrained, the actor tends to produce actuation patterns that are spatially discontinuous, temporally erratic, or saturated at the maximum allowable value, none of which correspond to physically meaningful control strategies. To discourage these degenerate behaviours, four penalty terms are added to the training objective, each targeting a specific undesirable pattern. The first penalises any residual deviation from zero mean after the projection (9),

Lzero=⟨aˉ2⟩,aˉ=1N∑j=0N−1aj.(16)\mathcal{L}_{\mathrm{zero}} = \langle\bar{a}^{2} \rangle, \qquad\bar{a} = \frac{1}{N}\sum_{j=0}^{N-1} a_j. \tag*{(16)}

The second penalty addresses spatial discontinuities: abrupt jumps in wall temperature between adjacent segments can produce localised flow structures at segment boundaries that are consequences of the discretisation rather than genuine control features,

Lsmooth=⟨(aj+1−aj)2⟩,(17)\mathcal{L}_{\mathrm{smooth}} = \left\langle(a_{j+1} - a_j)^2 \right\rangle, \tag*{(17)}

while the temporal smoothness loss penalises rapid changes between consecutive actuation steps,

Ltemporal=⟨(ajt−ajt−1)2⟩.(18)\mathcal{L}_{\mathrm{temporal}} = \left\langle(a_j^t - a_j^{t-1})^2 \right\rangle. \tag*{(18)}

The temporal smoothness penalty follows the CAPS regularisation of Mysore et al. [25], originally introduced for single-agent continuous control; the spatial smoothness term enforces coordination across the segmented action vector and has no direct precedent in that line of work. Finally, since the actor output is bounded to [−1,1][-1, 1] by the tanh activation, a penalty on large action magnitudes discourages the policy from saturating at these limits, where the gradient vanishes and learning stalls,

Lenergy=⟨aj2⟩.(19)\mathcal{L}_{\mathrm{energy}} = \left\langle a_j^2 \right\rangle. \tag*{(19)}

The four penalties are combined with the main training objective through weights λzero\lambda_{\mathrm{zero}}, λsmooth\lambda_{\mathrm{smooth}}, λtemporal\lambda_{\mathrm{temporal}}, and λenergy\lambda_{\mathrm{energy}}, whose values differ between the single-agent and multi-agent configurations. These are listed in Table 2.

LossMADDPGTD3
λzero\lambda_{\mathrm{zero}}0.12.0
λsmooth\lambda_{\mathrm{smooth}}0.050.02
λtemporal\lambda_{\mathrm{temporal}}0.0250.025
λenergy\lambda_{\mathrm{energy}}0.10.1

Table 2. Auxiliary loss weights.

Training results: Rayleigh–Bénard convection

The four configurations compared throughout this section and the next are collected in Table 3, together with the labels and colours used to identify them in every figure. The actor sizes are worth noting, since they show that the recurrent variants are not simply larger networks: in the single-agent case the GRU actor is the smaller of the two, because the projection to the hidden state replaces the wide dense layer that follows the flattened convolutional features.

LabelAlgorithmAgentsObservationMemoryActor
Single-agent, no GRUTD31full fieldnone441,066
Single-agent + GRUTD31full fieldGRU, h=64h = 64315,946
Multi-agent, no GRUMADDPG103 segmentsnone54,609
Multi-agent + GRUMADDPG103 segmentsGRU, h=64h = 6479,569
Multi-agent, stack k=4k = 4MADDPG103 segments4 frames56,337
Multi-agent, wideMADDPG103 segmentsnone81,537

Table 3. The four control configurations and the labels used to identify them in the text and in every figure. Blue denotes the single-agent configurations and orange the multi-agent ones, with dark shades for the GRU-equipped variants and light shades for the memoryless ones, and green the frame-stacked variant. Actor sizes are the number of trainable parameters of the policy network; the multi-agent actor is shared by all ten agents.

Figure 2 shows how the Nusselt number evolves over the course of training for all four configurations (single-agent TD3 and multi-agent MADDPG, each with and without GRU recurrence). Two metrics are reported. The first is the Nusselt number averaged over the full episode, Nuavg\mathrm{Nu}_{\mathrm{avg}}; this includes the initial transient during which the agent is still learning to modify the flow. The second is the Nusselt number averaged over the last 20% portion of the episode, Nulate\mathrm{Nu}_{\mathrm{late}}; this reflects how effectively the agent sustains heat transfer reduction once the flow has fully responded to its actuation.

Figure 2. Training convergence of the Nusselt number for four agent configurations. Top: episode-averaged Nu. Bottom: late-episode Nu (last 20% of each episode). Blue: single-agent (TD3); orange: multi-agent (MADDPG). Dark shades: GRU-equipped; light shades: feedforward. Lines show rolling mean (window = 10 episodes); shaded envelopes indicate the range of raw values within each window.

All four control strategies achieve a similar flow topology characterised by cell coalescence, reducing Nu well below the uncontrolled baseline of 2.48. This confirms that the CNN architecture is sufficient to learn effective control regardless of whether a single or multiple agents are used, and stands in

contrast to Vignon et al. [35], where achieving coalescence with a multi-agent approach required re-centring the full field about each agent, which effectively provides ten times more full-field trajectories for the shared policy.

The multi-agent strategies consistently reach lower NuNu values than the single-agent approach, particularly in the late-episode metric, suggesting that distributing control across local observers better matches the spatially distributed nature of the convective structures. The GRU-equipped variants learn faster, most visibly in the multi-agent case where NulateNu_{\mathrm{late}} drops below 2.0 within the first 50 episodes, reflecting the benefit of temporal memory in tracking the delayed response of the flow to wall temperature changes and anticipating the evolution of the convective structures. Finally, NuavgNu_{\mathrm{avg}} proves more informative than NulateNu_{\mathrm{late}} alone, as it reflects not only the final heat transfer reduction but also how quickly the flow reorganises into the coalesced state within each episode: a lower NuavgNu_{\mathrm{avg}} at comparable NulateNu_{\mathrm{late}} indicates that coalescence is reached earlier in the episode.

A fixed window of past observations and a larger network are the two obvious ways of augmenting the baseline feedforward policy without introducing recurrence, and neither reaches what the recurrent state provides. Both were trained with every other setting held fixed. Stacking the four most recent observations as additional input channels gives a median NulateNu_{\mathrm{late}} over three seeds of 1.970, against 1.865 with the GRU and 1.801 with no temporal input at all, so extending the observation backwards in time does not improve the converged policy and here degrades it. Convergence is also markedly slower, the ten-episode rolling mean of NulateNu_{\mathrm{late}} first reaching 1.9 after 305 episodes. against 147 with no temporal input and 49 with the GRU. Widening the actor instead, to 81,537 trainable parameters against the 79,569 of the recurrent policy, fares no better: that configuration settles at 2.072 and does not reach 1.9 at all within 350 episodes.

Capacity therefore does not account for the difference. The stacked actor already carries more parameters than the unstacked one, 56,337 against 54,609, and the widened actor carries more than the recurrent one while performing worse than either. What distinguishes the recurrent policy is not the size of its input or of its parameter vector but what it does with them. A stack of kk frames fixes the memory horizon in advance and multiplies the observation dimension by kk, whereas the recurrent state learns what to retain and over what horizon. That matters here because the delay between wall actuation and its effect on the bulk is set by the diffusive and advective transport times of the flow rather than by a number available before training.

Since the reward is defined as a decreasing function of the instantaneous Nusselt number, it mirrors the trends described above. Figure 3 shows its evolution over training, confirming that the learning curves are consistent across both metrics.

Figure 3. Episode reward during training for the four configurations. Lines show rolling mean (window = 20); shaded envelopes show raw value range. Higher reward corresponds to lower Nu.

Comparison with previous results

Table 4 compares the present approach with Vignon et al. [35]. The key difference is architectural: Vignon et al. [35] use a flat network that receives the entire flow field as a vector, discarding its spatial structure, while the present work uses convolutional networks that process the field as a two-dimensional array and extract spatially coherent features. In their multi-agent setup, each agent is given the full flow field rotated so that its own segment always appears at the centre, effectively multiplying the number of training trajectories by a factor of ten. The multi-agent setup proposed here is fundamentally different: each agent observes only a local patch of 3 neighbouring segments, with no rotation or data augmentation. The convolutional architecture achieves comparable results without this augmentation, indicating that processing the flow field spatially and restricting each agent to its local neighbourhood are together sufficient to learn effective control.

AspectVignon et al.This work
AlgorithmPPO (on-policy)TD3 / MADDPG (off-policy)
ArchitectureMLP (2 × 512)CNN (stride-2 encoder)
ObservationFull field, recentred per agentFull field (single) / local patch (multi)
MARL designRecentering trick (each agent sees full field rotated above it)Local patches (each agent sees 3 neighbouring segments)
Data multiplication350 ep. × 10 full-field views = 3500 eff. trajectories350 ep. (local transitions, no full-field augmentation)
Cell coalescenceYes (MARL only, after 3500 eff. trajectories)Yes (all 4 configs in 350 episodes)
RecurrenceNoGRU (optional, accelerates training)

Table 4. Comparison with the multi-agent formulation of Vignon et al. [35].

Expected behaviour at higher Rayleigh number

The results above are obtained in a regime where the uncontrolled flow is steady, and it is worth setting out what would change as RaRa is raised towards the chaotic regime, since the auxiliary weights of Table 2 are calibrated against physical scales rather than being free constants.

The temporal weight is referenced to the time the flow takes to respond to a change in wall temperature, which is governed by conduction across the thermal boundary layer. That thickness follows δ/H∼1/(2 Nu)\delta/H \sim1/(2\,\mathrm{Nu}) and so contracts slowly, from 0.20 at Ra=104\mathrm{Ra}=10^{4} to 0.145 at Ra=8×104\mathrm{Ra}=8 \times10^{4}, while the ratio of the diffusive to the free-fall time scale, Ra Pr\sqrt{\mathrm{Ra}\,\mathrm{Pr}}, grows from 84 to 238. In free-fall units the response delay is therefore comparatively insensitive to Ra\mathrm{Ra} and the actuation interval used here would transfer. What does not transfer is the assumption of a steady response: once the flow fluctuates on the timescale of the actuation, a penalty tuned against a steady approach to equilibrium over-regularises.

The spatial weight is referenced to the roll wavelength, and this scale cannot be inferred from Ra\mathrm{Ra} alone in a domain of fixed width. Raising Ra\mathrm{Ra} eightfold in the present box in fact increases the dominant horizontal scale, the uncontrolled state passing from four rolls to two, so the reference length has to be taken from the observed flow. The same caution applies to the observation resolution, which must continue to resolve the near-wall gradient as δ\delta contracts.

Two limitations follow. The controller learnt here exploits a well-defined topology, so where the flow is chaotic the reward becomes a noisy function of the actuation and a smaller reduction should be expected. And the reduction available through coalescence is bounded by how far the uncontrolled state already sits from the largest structure the domain admits, so results at different Ra\mathrm{Ra} in a fixed box are not comparable unless that distance is. We would expect the qualitative conclusion of this work to carry over to the chaotic regime, but establishing it there is a study in its own right rather than a change of one parameter.

Evaluation results: Rayleigh–Bénard convection

The four control strategies are evaluated over episodes of 200 actions (300 free-fall time units H/UfH/U_f, or approximately 3.6 thermal diffusion times H2/κTH^{2}/\kappa_{T}), starting from the same uncontrolled steady state used during training. Figure 4 shows the instantaneous Nusselt number during evaluation. The uncontrolled flow maintains Nu≈2.5\mathrm{Nu} \approx2.5 throughout, while all controlled cases progressively reduce Nu\mathrm{Nu} toward the conductive limit. The multi-agent strategy achieves a deeper and faster reduction than the single-agent one, consistent with the advantage observed during training.

Figure 4. Instantaneous Nusselt number during evaluation for all four configurations. The baseline (grey) maintains Nu≈2.5\mathrm{Nu} \approx2.5, while every controlled case reduces Nu\mathrm{Nu} toward the conductive limit. Blue denotes the single-agent configurations and orange the multi-agent ones, with dark shades for the GRU-equipped variants.

Individual evaluation curves such as those of Figure 4 are single realisations of a stochastic training process, and the approach to the controlled state is not a clean transition. Figure 5 shows the two multi-agent configurations evaluated from three independent training runs each, differing only in the random seed. Within one configuration the realisations follow visibly different paths: one memoryless run reorganises the flow before t=50t = 50, another holds near Nu=2.2\mathrm{Nu} = 2.2 until t≈150t \approx150 and then drops abruptly, and the third descends gradually. All six settle into the same interval, 1.763≤Nu≤1.9381.763 \le\mathrm{Nu} \le1.938, so the level reached by the multi-agent policies is the same to within the scatter between realisations and no ordering can be read from any single pair of curves. The recurrent variant is, however, the more consistent of the two in where it settles: its three plateaux span 1.817 to 1.870, an interval contained entirely within the one spanned by the memoryless runs, and its curves hold their level once reached, drifting by −0.043-0.043, −0.007-0.007 and −0.022-0.022 between mid-episode and the end of the episode where the memoryless runs drift upward by +0.001+0.001, +0.022+0.022 and +0.139+0.139. The lowest single value of all six belongs to a memoryless run, so temporal memory is not what sets the depth of the reduction; what it provides is a policy that reaches a stable configuration and stays there.

Figure 5. Instantaneous Nusselt number during evaluation for the two multi-agent configurations, each trained three times from independent random seeds and evaluated from its best checkpoint. Line style identifies the seed. The grey band spans the plateaux of all six runs, 1.763 to 1.938; the dashed line is the uncontrolled value.

The physical origin of this reduction is visible in Figure 6, which shows the temperature field at several instants during the episode. All controlled cases undergo progressive cell coalescence: the initial four-roll structure merges into fewer, larger rolls, and it is this reorganisation of the convective topology that drives Nu\mathrm{Nu} downward. The multi-agent strategy promotes coalescence more rapidly, with the transition from four to two rolls well under way by t≈75t \approx75. Figure 7 compares the final temperature fields from the multi-agent strategies with and without GRU memory. Both reach the coalesced state, confirming that the convolutional architecture alone provides sufficient spatial expressivity to discover the correct flow reorganisation. The variant with temporal memory (i.e. the GRU-equipped variant) reaches this state earlier in training (cf. Figure 2). This is expected, since convective structures take time to reorganise in response to a change in wall temperature: without temporal context, the same instantaneous flow snapshot can arise from very different actuation histories, making it harder to identify an effective control strategy. The temporal memory resolves this ambiguity by carrying information about the recent evolution of the flow forward in time.

Figure 6. Temperature field at four instants during evaluation. Rows: multi-agent + GRU (top), single-agent + GRU (middle), uncontrolled baseline (bottom). The controlled cases show cell coalescence: merging of convection rolls into larger structures. The mechanism is apparent at the lower boundary, where the policy introduces heat at the edges of the base of each cell rather than at its centre, widening the region held above the mean wall temperature and so broadening the base over which the cell is supported.

Figure 7. Final temperature fields: multi-agent + GRU (top) vs. multi-agent without GRU (bottom). Both reach cell coalescence, but the GRU variant achieves this earlier during training (cf. Figure 2).

The time-averaged Nusselt numbers for all four strategies are reported in Table 5. The wall temperature maps in Figure 8–Figure 11 show the spatiotemporal evolution of the actuation applied by each strategy. In all cases the actuation is smooth and spatially structured, with no trace of the rapid switching or saturated outputs that arise when the policy fails to identify a physical control mechanism. The single-agent maps display broad, slowly evolving patterns spanning the full domain, while the multi-agent maps exhibit finer spatial modulation reflecting the individual agents’ segment-by-segment control. The strategies equipped with temporal memory produce smoother wall-temperature histories than their memoryless counterparts, reflecting the regularising effect of carrying information forward across succes-

ConfigurationNu\mathrm{Nu}Reduction
Baseline (uncontrolled)2.48—
Single-agent, no GRU2.06 ± 0.1717.1%
Single-agent + GRU2.17 ± 0.0812.5%
Multi-agent, no GRU1.83 ± 0.0026.3%
Multi-agent + GRU1.85 ± 0.0725.5%

Table 5. Evaluation Nusselt numbers (time-averaged over the last 90% of the episode) for the four control strategies.

Figure 8. Space–time map of bottom-wall temperature Twall(x,t)T_{\mathrm{wall}}(x,t) for the single-agent + GRU strategy. The reference value TH=2T_H = 2 corresponds to white.

Figure 11. Space–time map of bottom-wall temperature for the multi-agent, no GRU strategy. The same edge-heating mechanism is recovered without memory, and the actuation is steadier than in Figure 9, but only once a good configuration has been reached, at which point the instantaneous field is unambiguous enough for a memoryless policy to act on.

Figure 9. Space–time map of bottom-wall temperature for the multi-agent + GRU strategy. The edge-heating mechanism of Figure 6 is sustained here as a persistently dynamic actuation, the hidden state carrying information about the preceding evolution of the flow.

sive actuation steps.

The zero-mean projection of equation (9) guarantees that the actuation adds no net heat to the bottom wall, but holding a spatially non-uniform and time-varying wall temperature is not thermodynamically free, and the engineering value of the control rests on how its cost compares with the transport it suppresses. A lower bound on that cost follows from the reversible redistribution of heat along the wall: transferring a quantity of heat QQ from a segment held at Tabs−δTT_{\mathrm{abs}}-\delta T to one held at Tabs+δTT_{\mathrm{abs}}+\delta T requires at least Wmin⁡≃2QδT/TabsW_{\min} \simeq2Q\delta T/T_{\mathrm{abs}} of work, where TabsT_{\mathrm{abs}} is the absolute operating temperature. Since the non-dimensional formulation leaves TabsT_{\mathrm{abs}} free, the ratio must be evaluated at a chosen operating point. For an air layer with ΔT=10 K\Delta T = 10\,\mathrm{K} about an ambient temperature of 300 K300\,\mathrm{K}, the largest perturbation admitted by equation (10), C ΔT=7.5 KC\,\Delta T = 7.5\,\mathrm{K}, places Wmin⁡/QW_{\min}/Q at a few per cent, to be set against the 26% reduction in the heat crossing the layer. This estimate counts only the reversible work of redistributing heat along the wall, and excludes the irreversibility of any physical thermal actuator, the power drawn by sensing and by evaluating the policy, and the transient cost of establishing the actuation at the start of an episode, all of which depend on the device rather than on the flow.

Figure 10. Space–time map of bottom-wall temperature for the single-agent, no GRU strategy.

Double-diffusive convection in the salt-finger regime

The DRL framework is now extended to double-diffusive convection, where two scalar fields with different molecular diffusivities, i.e. the temperature TT and the salinity SS, drive buoyancy simultaneously. The governing equations retain the Boussinesq framework of § 2.1, augmenting the momentum equation with a two-component buoyancy term and adding a salinity transport equation [32]:

∂u∂t+(u⋅∇)u=−∇p+PrRa∇2u+(T−NρS)e^y(20)\frac{\partial\mathbf{u}}{\partial t} + (\mathbf{u} \cdot\nabla)\mathbf{u} = -\nabla p + \sqrt{\frac{\mathrm{Pr}}{\mathrm{Ra}}}\nabla^{2}\mathbf{u} + (T - N_{\rho}S)\hat{\mathbf{e}}_{y} \tag*{(20)}
∂S∂t+(u⋅∇)S=1LeRaPr∇2S(21)\frac{\partial S}{\partial t} + (\mathbf{u} \cdot\nabla)S = \frac{1}{\mathrm{Le}\sqrt{\mathrm{Ra}}\mathrm{Pr}}\nabla^{2}S \tag*{(21)}

Two additional non-dimensional parameters appear: the Lewis number Le=κT/κS\mathrm{Le} = \kappa_T/\kappa_S, the ratio of thermal to solutal diffusivity, and the buoyancy ratio Nρ=βSΔS/(βTΔT)N_{\rho} = \beta_S\Delta S/(\beta_T\Delta T), which measures the relative strength of solutal to thermal buoyancy.

These equations admit the salt-finger regime Stern [29], named after the tall, thin fingers of alternately sinking and rising fluid that form through a positive feedback: downward-displaced parcels rapidly lose heat but retain salt, become denser, and continue sinking. The parameters Ra=7×106\mathrm{Ra}=7\times10^{6}, Pr=ν/κT=7\mathrm{Pr}=\nu/\kappa_{T}=7 (water-like), Le=κT/κS=100\mathrm{Le}=\kappa_{T}/\kappa_{S}=100, and Nρ=0.167N_{\rho}=0.167 are selected to promote this regime; together they imply a well-separated hierarchy of boundary-layer thicknesses δν≫δT≫δS\delta_{\nu}\gg\delta_{T}\gg\delta_{S}, as expected in the salt-finger regime. The same Pr/Prc\mathrm{Pr}/\mathrm{Pr}_{c} pair has recently been used in wall-resolved LES of bounded turbulent double-diffusive convection across a wider range of thermal and concentration Rayleigh numbers Kenjeres and Roovers [20]; that work provides a reference point for the uncontrolled flow physics on which the present control task acts. The computational domain is a tall cavity with aspect ratio Lx/Ly=1/3L_{x}/L_{y}=1/3, resolved on a 128×512128\times512 grid. Temperature uses Dirichlet boundary conditions (TH=1T_{H}=1 at the bottom, TC=0T_{C}=0 at the top), while salinity uses no-flux (Neumann) conditions at both walls.

In this configuration, the control task differs from the RBC case in both objective and mechanism: rather than suppressing convection, the goal is to enhance it. Eight agents (N=8N=8) modulate the bottom-wall temperature in zero-mean segments, using the same MADDPG framework with GRU-augmented actors described in § 3.2. The objective is to maximise the Nusselt number; temperature actuation is a natural lever in this regime because, with Le=100\mathrm{Le}=100, heat diffuses two orders of magnitude faster than salt, making wall temperature perturbations an efficient means of modifying the local buoyancy field.

The trained policy is evaluated over a 200-action episode starting from the uncontrolled steady state. The controlled policy achieves a sustained increase in heat transfer: the time-averaged Nusselt number rises from Nubase=10.44±2.54\mathrm{Nu}_{\mathrm{base}}=10.44\pm2.54 to Nuctrl=12.44±2.41\mathrm{Nu}_{\mathrm{ctrl}}=12.44\pm2.41, an improvement of 19.1%. The Sherwood number Sh=m˙SH/(DSΔS)\mathrm{Sh}=\dot{m}_{S}H/(D_{S}\Delta S), (the solutal analogue of Nu, measuring the non-dimensional salt flux) decreases slightly by ∣−3.5∣%\lvert-3.5\rvert\%, while the salinity spatial variance σS2\sigma_{S}^{2} is reduced by 21.0% (from 0.0629 to 0.0497), indicating faster mixing. These quantities are summarised in Table 6.

QuantityBaselineControlledChange
Nusselt Nu10.44±2.5410.44 \pm2.5412.44±2.4112.44 \pm2.41+19.1%+19.1\%
Sherwood Sh198.97±44.16198.97 \pm44.16191.93±45.47191.93 \pm45.47−3.5%-3.5\%
Salinity variance σs2\sigma_s^20.06290.06290.04970.0497−21.0%-21.0\%

Table 6. Double-diffusive control results (time-averaged over the last 90% of the episode).

Figure 12 shows the time evolution of Nu and Sh. The Nusselt increase is established rapidly and sustained throughout the episode, while the Sherwood number remains close to its baseline value, confirming that the policy enhances thermal mixing without significantly altering the salt flux.

Figure 12. Time evolution of NuNu (top) and ShSh (bottom) for the double-diffusive case. The policy increases NuNu by 19.1% while leaving salt flux nearly unchanged.

The evolution of salinity spatial variance σS2\sigma_{S}^{2} (Figure 13) confirms faster homogenisation: the controlled case reduces σS2\sigma_{S}^{2} by 21.0% relative to the

Figure 13. Salinity spatial variance σS2\sigma_S^2: the controlled case achieves 21.0% reduction relative to the baseline, reflecting faster mixing of the salt distribution.

baseline, with the half-life of variance decay shortening from 96.0 to 83.2 time units (a 1.2×\times speedup).

Figure 14 and Figure 15 compare the temperature and salinity fields at five instants. The controlled case shows a progressive disruption of the finger pattern and faster homogenisation of the salt distribution, consistent with the reduced salinity variance reported above.

Figure 14. Double-diffusive case: temperature field at five instants. Top row: controlled; bottom row: baseline.

Figure 15. Double-diffusive case: salinity field at five instants. Top row: controlled; bottom row: baseline. The controlled case shows faster homogenisation of the salt distribution.

Beyond the integral quantities, the wall temperature map (Figure 16) reveals a more subtle and physically revealing feature of the learned policy: a coherent travelling-wave structure in the actuation pattern. Tracking the phase of the fundamental Fourier mode (k=1k = 1) of the wall perturbation yields a piecewise-linear phase displacement (Figure 17), indicating two distinct regimes of constant wave speed. During the first phase (t≲220t \lesssim220) the pattern propagates at c1≈−0.053c_1 \approx-0.053, while in the second phase (t≳220t \gtrsim220) the wave slows to c2≈−0.028c_2 \approx-0.028, a 46% reduction. The slowdown coincides with the salinity field approaching a more mixed state: as the salt fingers weaken, the policy naturally reduces the aggressiveness of its stirring strategy. This adaptive behaviour emerges entirely from the reward signal, without any explicit encoding of wave speed or mixing state in the control formulation.

Figure 16. Space–time map of bottom-wall temperature for the double-diffusive case.

Figure 17. Cumulative phase displacement of the k=1k=1 Fourier mode of the wall temperature perturbation, with piecewise-linear fit.

Summary and conclusions

Degenerate actuation, in the form of saturated, pseudo-random, or spatially incoherent wall-temperature policies, has been identified as a recurring pathology in DRL-based thermal convection control, rooted in two compounding deficiencies: MLP policies that discard spatial flow structure, and

memoryless policies that cannot attribute local flow changes to their own prior actuation. A framework addressing both causes has been presented, combining convolutional policy networks, Gated Recurrent Unit (GRU) memory, off-policy training (TD3 / MADDPG), and explicit action-smoothness constraints. The framework has been tested on two buoyancy-driven configurations of increasing physical complexity.

In the Rayleigh–Bénard convection case at Ra=10,000\mathrm{Ra}=10{,}000, all four agent configurations, single-agent TD3 and multi-agent MADDPG each with and without GRU recurrence, successfully learn cell-coalescence strategies, reducing Nu to as low as 1.83 (26% below the uncontrolled baseline of 2.48) within 350 training episodes. This matches the effective sample efficiency of the 3500-trajectory approach of Vignon et al. [35] without requiring their re-centred full-field observations, demonstrating that convolutional inductive biases and local observations are together sufficient. The multi-agent configuration yields deeper Nu reductions than the single-agent case, reflecting better spectral alignment with the dominant convective mode, while GRU recurrence accelerates convergence by approximately 100 episodes across all configurations. Coalescence is achieved even by the single-agent policy, establishing that the multi-agent formulation is not a prerequisite once the policy architecture is sufficiently expressive. The learned actuation in every case is smooth and spatially structured, with none of the hallmark failures of degenerate control. In double-diffusive convection in the salt-finger regime, the multi-agent recurrent policy enhances heat transfer by 19.1% and reduces salinity variance by 21.0%, accelerating scalar mixing. The learned wall-temperature actuation organises spontaneously into a coherent travelling wave whose phase speed adjusts in response to the evolving state of the flow: the wave propagates at c1≈−0.053c_1 \approx-0.053 during the initial finger-dominated phase and slows to c2≈−0.028c_2 \approx-0.028 as the salinity field approaches a more homogeneous state, a 46% reduction in propagation speed. This state-dependent actuation pattern arises entirely from the scalar reward signal, with no prescribed wave structure and no explicit knowledge of the mixing state beyond the instantaneous Nusselt number. That a boundary-temperature policy should spontaneously select a travelling-wave strategy to disrupt salt fingers is a physically nontrivial outcome, one that would be difficult to anticipate or derive from linear stability arguments alone.

Set against the wider landscape, these results support the view that the progress reported across the active flow control literature comes from building the structure of the problem into the formulation rather than from any particular learning algorithm. Belus et al. [3] made that case for the spatial dimension, and multi-agent decomposition has since been its usual expression; what the present results add is that a convolutional policy reaches the same end without decomposing the domain at all, so the decomposition is one way of supplying the inductive bias rather than the only one. The temporal half of the problem, identified for wake control by Wang et al. [37] and addressed there through an explicit delay feature, is met here by recurrence, which learns the relevant summary of the past instead of committing to a memory horizon chosen in advance. Read together with the group-invariant architectures of Jeon et al. [16], the picture that emerges is one of interchangeable mechanisms for injecting the same spatial and temporal structure, and the practical question for future work is which of them compound and at what training cost.

Higher Rayleigh numbers will shorten the convective timescale relative to the actuation period and bring the flow into a chaotic regime, and it is far from obvious that the recurrent architecture will remain stable under the much noisier observations that follow. Three-dimensional geometries add a spanwise dimension to the convective organisation and a substantial training cost, for which reduced-order surrogate environments offer a plausible route. The double-diffusive case hints at a wider class of buoyancy-driven flows with competing scalar fields, of which thermohaline circulation and certain crystallisation processes are concrete examples, where the present framework could be transferred without major modification. Physical experiments will add sensor noise, actuator inertia and finite-bandwidth boundary conditions, none of which are represented in the idealised simulations used here.

Sensitivity to the auxiliary loss weights

The weights of Table 2 are intended to shape the actuation without reshaping the objective, and it is worth recording how little the outcome depends on their exact values. Each of λsmooth\lambda_{\mathrm{smooth}} and λtemporal\lambda_{\mathrm{temporal}} was varied by a factor of ten either side of its tabulated value, all other settings held fixed, on the multi-agent recurrent configuration. The unmodified configuration was also trained at three random seeds, which sets the scatter against which any effect has to be judged. Table A.1 reports the average over the final 50 training episodes of each run.

Nulate\mathrm{Nu}_{\mathrm{late}}⟨(aj+1−aj)2⟩\langle(a_{j+1}-a_j)^2\rangle⟨(ajt−ajt−1)2⟩\langle(a_j^t-a_j^{t-1})^2\ranglearmsa_{\mathrm{rms}}
three seeds, weights as tabulated1.834–2.1110.568–0.6870.012–0.0670.403–0.449
λsmooth\lambda_{\mathrm{smooth}}0.0051.8790.7290.0670.473
0.051.8650.5800.0670.449
0.51.9040.2450.0220.404
λtemporal\lambda_{\mathrm{temporal}}0.00251.9140.5820.0220.431
0.0251.8650.5800.0670.449
0.251.8510.6550.0240.434

Table A.1. Sensitivity of the trained policy to the smoothness weights. The first row gives the range obtained from three random seeds at the tabulated weights; armsa_{\mathrm{rms}} is the actuation amplitude against a cap of C=0.75C = 0.75.

The control objective is insensitive over the whole range. The late-episode Nusselt number spans 1.851 to 1.914 across two decades of either weight, a band that lies inside the 1.834 to 2.111 obtained from three seeds at the tabulated values, so either weight can be moved by an order of magnitude with less effect on the result than changing the random seed produces. The spatial weight nonetheless acts on the quantity it targets, the roughness falling monotonically from 0.729 to 0.245, although sublinearly, since a hundredfold change in the weight produces a threefold change in the roughness. The temporal weight is not resolved by this design, its variation across two decades moving the realised jitter by less than the seeds move it at fixed weights. No setting drove the policy towards saturation, the actuation amplitude remaining between 0.40 and 0.47 against the cap of 0.75 throughout. The penalties therefore guide the optimiser gently, among the many policies that reduce Nu comparably well, towards the smooth and spatially coherent ones, rather than reshaping the objective around themselves.

Sensitivity to the number of wall segments

The choice N=10N = 10 of § 3.2 was tested by repeating the multi-agent recurrent training at N=5N = 5 and N=20N = 20, a factor of four in segment count, with all other settings held fixed. Table B.1 reports the same quantities as Table A.1 together with the number of episodes taken for the ten-episode rolling mean of Nulate\mathrm{Nu}_{\mathrm{late}} to fall to 2.0, and the range obtained from three random seeds at N=10N = 10 again sets the scale against which the differences are read.

NNwidthNulate\mathrm{Nu}_{\mathrm{late}}⟨(aj+1−aj)2⟩\langle(a_{j+1}-a_j)^2\rangle⟨(ajt−ajt−1)2⟩\langle(a_j^t-a_j^{t-1})^2\ranglearmsa_{\mathrm{rms}}episodes to 2.0
51.26H1.26H1.8350.7350.0360.39644
100.63H0.63H1.8650.5800.0670.44977
200.31H0.31H1.8760.4390.0430.45780
three seeds at N=10N = 101.834–2.1110.568–0.6870.012–0.0670.403–0.44938–77

Table B.1. Sensitivity of the trained policy to the number of wall segments. The final row gives the range obtained from three random seeds at N=10N = 10.

There is no critical segment count within this range. The final Nusselt number spans 1.835 to 1.876 across a fourfold change in NN, against 1.834 to 2.111 from three seeds at fixed NN, and the number of episodes needed to reach Nulate=2.0\mathrm{Nu}_{\mathrm{late}} = 2.0 falls within the 38 to 80 spanned by the same comparison. Coarsening the actuation to five segments, so that a single segment covers four fifths of an uncontrolled roll, still selects the coalesced state, and refining it to twenty brings no measurable gain. The decrease of ⟨(aj+1−aj)2⟩\langle(a_{j+1}-a_j)^2\rangle with NN follows from measuring the difference between neighbouring segments on a finer discretisation rather than from a change in the wall profile, and reflects the actuation approaching a continuous field as the segmentation is refined, the regime in which the smoothness penalty rather than the segmentation sets the scale of the actuation.

Acknowledgements

G.M.C. acknowledges support from EPSRC UKRI. This work was supported by the Horizon Europe MSCA DN project with acronym SCALE, Agreement No. 101120014.

Code and data availability

The PyTorch solver, training scripts and post-processing utilities used to reproduce the results reported here are openly available at https://github.com/gmcavallazzi/RBC-GRU_MARL.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

CRediT author contribution statement

Giorgio Maria Cavallazzi: Conceptualization, Methodology, Software, Investigation, Formal analysis, Visualization, Writing – original draft. Miguel Pérez Cuadrado: Software, Data curation, Formal analysis, Validation, Writing – review & editing. Alfredo Pinelli: Conceptualization, Supervision, Writing – original draft, Writing – review & editing.

Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work the authors used Claude (Anthropic) in order to check grammar and spelling. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

References

  1. [1]Bae, H.J., Koumoutsakos, P., 2022. Scientific multi-agent reinforcement learning for wall-models of turbulent flows. Nature Communications 13, 1443.DOI
  2. [2]Beintema, G., Corbetta, A., Biferale, L., Toschi, F., 2020. Controlling Rayleigh–Bénard convection via reinforcement learning. Journal of Turbulence 21, 585–605.
  3. [3]Belus, V., Rabault, J., Viquerat, J., Che, Z., Hachem, E., Reglade, U., 2019. Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film. AIP Advances 9, 125014.DOI
  4. [4]Beneitez, M., Cremades, A., Guastoni, L., Vinuesa, R., 2025. Improving turbulence control through explainable deep learning. arXiv preprint arXiv:2504.02354 .arxiv.org/abs/2504.02354
  5. [5]Brunton, S.L., Noack, B.R., Koumoutsakos, P., 2020. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics 52, 477–508.
  6. [6]Cavallazzi, G.M., Guastoni, L., Vinuesa, R., Pinelli, A., 2025. Deep reinforcement learning for the management of the wall regeneration cycle in wall-bounded turbulent flows. Flow, Turbulence and Combustion 115, 1291–1317.DOI
  7. [7]Chen, Q., Constante-Amores, C.R., 2025. Stabilizing Rayleigh–Bénard convection with reinforcement learning trained on a reduced-order model. arXiv preprint arXiv:2510.26705 .arxiv.org/abs/2510.26705
  8. [8]Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724–1734.arxiv.org/abs/1406.1078
  9. [9]Font, B., Alcántara-Ávila, F., Rabault, J., Vinuesa, R., Lehmkuhl, O., 2025. Deep reinforcement learning for active flow control in a turbulent separation bubble. Nature Communications 16, 1422.DOI
  10. [10]Fujimoto, S., van Hoof, H., Meger, D., 2018. Addressing function approximation error in actor-critic methods, in: Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 1587–1596.arxiv.org/abs/1802.09477
  11. [11]Garcia, X., Miró, A., Suárez, P., Alcántara-Ávila, F., Rabault, J., Font, B., Lehmkuhl, O., Vinuesa, R., 2025. Deep-reinforcement-learning-based separation control in a two-dimensional airfoil. International Journal of Heat and Fluid Flow 116, 109913.
  12. [12]Garnier, P., Viquerat, J., Rabault, J., Larcher, A., Kuhnle, A., Hachem, E., 2021. A review on deep reinforcement learning for fluid mechanics. Computers & Fluids 225, 104973.DOI
  13. [13]Guastoni, L., Rabault, J., Schlatter, P., Azizpour, H., Vinuesa, R., 2023. Deep reinforcement learning for turbulent drag reduction in channel flows. The European Physical Journal E 46, 27.
  14. [14]Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neural Computation 9, 1735–1780.
  15. [15]Holme, K., Rabault, J., Vinuesa, R., Mortensen, M., 2026. Timescale separation enables deep reinforcement learning control of rotating detonation engine mode transitions. arXiv preprint arXiv:2604.14398.arxiv.org/abs/2604.14398
  16. [16]Jeon, J., Rabault, J., Vasanth, J., Alcántara-Ávila, F., Baral, S., Vinuesa, R., 2025. Inductive biased-deep reinforcement learning methods for flow control: group-invariant and positional-encoding networks improve learning reproducibility and quality. Physics of Fluids 37, 077189.
  17. [17]Jia, W., Xu, H., 2025. State-augmented deep reinforcement learning for active flow control around an elliptical cylinder. International Journal of Heat and Fluid Flow 116, 109950.
  18. [18]Jiren, M., Yousif, M.Z., Song, J., Lim, H.C., 2025. A comprehensive review of control strategies for Rayleigh–Bénard convection from classical feedback methods to artificial intelligence-driven optimization. Physics of Fluids 37, 101304.
  19. [19]Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., Dabney, W., 2019. Recurrent experience replay in distributed reinforcement learning, in: International Conference on Learning Representations.
  20. [20]Kenjeres, S., Roovers, R., 2025. Modulation of the local mass and heat transfer of turbulent double-diffusive convection under stable thermal stratifications. International Journal of Heat and Fluid Flow 111, 109636.DOI
  21. [21]Kurz, M., Kaushik, R., Blind, M., Kopper, P., Schwarz, A., Rodach, F., Beck, A., 2025. Invariant control strategies for active flow control using graph neural networks. Computers and Fluids 303, 106854.
  22. [22]Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D., 2016. Continuous control with deep reinforcement learning. International Conference on Learning Representations (ICLR) .
  23. [23]Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I., 2017. Multi-agent actor-critic for mixed cooperative-competitive environments, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 6382–6393.arxiv.org/abs/1706.02275
  24. [24]Markmann, T., Straat, M., Peitz, S., Hammer, B., 2025. Control of Rayleigh–Bénard convection: effectiveness of reinforcement learning in the turbulent regime. arXiv preprint arXiv:2504.12000 .arxiv.org/abs/2504.12000
  25. [25]Mysore, S., Mabsout, B., Mancuso, R., Saenko, K., 2021. Regularizing action policies for smooth control with reinforcement learning, in: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1810–1816.
  26. [26]Rabault, J., Kuchta, M., Jensen, A., Réglade, U., Cerardi, N., 2019. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. Journal of Fluid Mechanics 865, 281–302.
  27. [27]Sakievich, P.J., Peet, Y.T., Adrian, R.J., 2016. Large-scale thermal motions of turbulent Rayleigh–Bénard convection in a wide aspect-ratio cylindrical domain. International Journal of Heat and Fluid Flow 61, 183–196.DOI
  28. [28]Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M., 2014. Deterministic policy gradient algorithms, in: Proceedings of the 31st International Conference on Machine Learning (ICML), pp. 387–395.
  29. [29]Stern, M.E., 1960. The “salt-fountain” and thermohaline convection. Tellus 12, 172–175.DOI
  30. [30]Suárez, P., Alcántara-Ávila, F., Font, B., Lehmkuhl, O., Vinuésa, R., 2025a. Flow control of three-dimensional cylinders transitioning to turbulence via multi-agent reinforcement learning. Communications Engineering 4, 113.
  31. [31]Suárez, P., Alcántara-Ávila, F., Miro, A., Rabault, J., Font, B., Lehmkuhl, O., Vinuesa, R., 2025b. Active flow control for drag reduction through multi-agent reinforcement learning on a turbulent cylinder at \(Re_D = 3900\). Flow, Turbulence and Combustion 115, 3–27.
  32. [32]Turner, J.S., 1974. Double-diffusive phenomena. Annual Review of Fluid Mechanics 6, 37–56.
  33. [33]Van Kan, J.J., 1986. A second-order accurate pressure-correction scheme for viscous incompressible flow. SIAM journal on scientific and statistical computing 7, 870–891.
  34. [34]Vasanth, J., Rabault, J., Alcántara-Ávila, F., Mortensen, M., Vinuesa, R., 2025. Multi-agent reinforcement learning for the control of three-dimensional Rayleigh–Bénard convection. Flow, Turbulence and Combustion 115, 1319–1355.
  35. [35]Vignon, C., Rabault, J., Vasanth, J., Alcántara-Ávila, F., Mortensen, M., Vinuesa, R., 2023. Effective control of two-dimensional Rayleigh–Bénard convection: invariant multi-agent reinforcement learning is all you need. Physics of Fluids 35, 065146.
  36. [36]Vinuesa, R., Brunton, S.L., 2022. Enhancing computational fluid dynamics with machine learning. Nature Computational Science 2, 358–366.
  37. [37]Wang, Q., Yan, L., Hu, G., Chen, W., Rabault, J., Noack, B.R., 2024. Dynamic feature-based deep reinforcement learning for flow control of circular cylinder with sparse surface pressure sensing. Journal of Fluid Mechanics 988, A4.
  38. [38]Wang, Y.Z., He, X.J., Hua, Y., Chen, Z.H., Wu, W.T., Zhou, Z.F., 2023. Closed-loop forced heat convection control using deep reinforcement learning. International Journal of Heat and Mass Transfer 202, 123655.
  39. [39]Wang, Y.Z., Peng, J.Z., Aubry, N., Li, Y.B., Chen, Z.H., Wu, W.T., 2024. Control policy transfer of deep reinforcement learning based intelligent forced heat convection control. International Journal of Thermal Sciences 195, 108618.

Paper details

vlnc:W_3epua44g
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Claude (Anthropic)

View statement
Looking for related papers…
Contents