WorksPhysics and Astronomy
Deep reinforcement learning with spatial and temporal awareness for active boundary control of buoyancy-driven convection
Abstract
Deep reinforcement learning (DRL) applied to thermal convection control consistently produces degenerate actuation: wall-temperature policies whose outputs are saturated, pseudo-random, or spatially incoherent. Two compounding deficiencies are responsible: multilayer-perceptron policies that discard spatial flow structure, and memoryless policies that cannot distinguish self-induced flow changes from background evolution. Together they degrade the actuation into forms whose relation to the convective topology cannot be read off and which are not realisable at the actuator, even when cell coalescence (the merging of convection rolls into fewer, larger structures), which would reduce Nu, is accessible to boundary actuation. The present framework addresses both causes through four targeted design choices: convolutional policy networks, Gated Recurrent Unit (GRU) memory, off-policy training (TD3/MADDPG), and action-smoothness constraints. A systematic 2x2 factorial design isolates the contribution of each component. On Rayleigh-Benard convection at Ra = 10,000, all four configurations achieve cell coalescence and reduce Nu to as low as 1.83 (26% below the uncontrolled baseline) in 350 episodes, without the full-field data augmentation required by prior work. Crucially, coalescence is achieved even by the single-agent configuration, demonstrating that the multi-agent formulation is not a prerequisite once the policy architecture is sufficiently expressive. Applied to double-diffusive convection in the salt-finger regime, the framework spontaneously discovers a travelling-wave actuation whose phase speed adapts to the evolving mixing state of the flow, enhancing heat transfer by 19.1% and reducing salinity variance by 21.0%.
Introduction
Active control of buoyancy-driven flows, where thermal or compositional density differences drive fluid motion, is a problem of broad scientific and engineering relevance. The ability to suppress or enhance convective heat and mass transfer through boundary actuation has direct applications in thermal management, geophysical modelling, and industrial processes involving stratified fluids. The present work targets two such mechanisms — cell coalescence in Rayleigh–Bénard convection and salt-finger disruption in double-diffusive convection — and uses deep reinforcement learning as the means of inducing them at the boundary. Among buoyancy-driven flows, Rayleigh–Bénard convection (RBC), the motion driven by heating a fluid layer from below and cooling it from above, occupies a central and canonical place. At moderate Rayleigh numbers (, where is the gravitational acceleration, the thermal expansion coefficient, the imposed temperature difference, the layer height, the kinematic viscosity, and the thermal diffusivity) the flow self-organises into steady convection rolls whose number and spatial arrangement set the global heat transfer rate, quantified by the Nusselt number (, where is the wall heat flux and the thermal conductivity). Reducing Nu through boundary actuation is physically equivalent to reorganising this roll topology: by promoting cell coalescence (i.e. the merging of multiple rolls into fewer, larger structures) the boundary-layer contact area per roll decreases, plume impingement weakens, and heat transport approaches the conductive limit. The roll topology and the coherent structures that organise it — plumes, thermals, and a large-scale circulation — have been characterised in detail in moderately wide cylindrical domains (Sakievich et al., [27]). The simplicity of the control metric (a single scalar, Nu) and the geometric interpretability of the underlying mechanism make RBC a uniquely clean testbed for data-driven control.
However, deriving effective control laws for such flows is fundamentally difficult: the governing equations are nonlinear, thermal perturbations at the wall must first diffuse and then be advected through the bulk before their effect becomes observable, and the optimal strategy is rarely known a priori. Classical feedback approaches, ranging from proportional–derivative (PD) controllers to linear optimal control, require detailed knowledge of the linearised flow dynamics and lose effectiveness as the Rayleigh number increases and nonlinear interactions intensify [18]. Data-driven and machine learning methods offer an alternative route [5, 36], and reinforcement learning in particular is well suited to sequential decision-making in systems where no explicit model is available [12].
The application of deep reinforcement learning (DRL) to active flow control was demonstrated by Rabault et al. [26], who showed that a neural network policy trained through interaction with a high-fidelity numerical solver can discover effective drag-reduction strategies for flow past a cylinder at . Their work established the paradigm of coupling a DRL agent to a fluid solver through a standardised environment interface, and has since been extended across a wide range of configurations. For turbulent channel flow, where classical active control strategies rely on wall blowing and suction to suppress near-wall turbulence and reduce skin-friction drag, Guastoni et al. [13] demonstrated that off-policy algorithms can learn effective actuation strategies directly from DNS data, without requiring a prescribed control law. In a similar physical setting, Cavallazzi et al. [6] targeted the self-sustaining wall regeneration cycle, achieving drag reduction through stabilisation of velocity streaks. Font et al. [9] demonstrated DRL-based control of a 3D turbulent separation bubble over a flat plate under adverse pressure gradient, achieving a 9% reduction in bubble area; the learned policy remained effective when transferred from the coarse training grid to a finer resolution.
When the flow domain is wide or the actuation is spatially distributed, however, a single agent observing the full field becomes increasingly impractical, both in terms of observation dimensionality and the difficulty of learning coordinated spatial actuation. Building on these single-agent results, multi-agent formulations have been applied to flows past three-dimensional cylinders in the transitional regime with Reynolds numbers up to [30]. In the fully turbulent regime at , Suárez et al. [31] have discovered multi-frequency control laws two orders of magnitude more mass-efficient than conventional periodic forcing. Most recently, García et al. [11] have extended this lineage to airfoil separation control, achieving a 43.9% drag reduction on a NACA 0012 at via PPO-controlled blowing and suction at the surface.
In parallel with these aerodynamic applications, a separate thread has applied DRL to forced-convection cooling. Wang et al. [38] use a deep Q-network to regulate inlet conditions in an open cavity with discrete heat sources, reaching wall-temperature reductions of approximately 8 K over manually tuned controllers; Wang et al. [39] subsequently show that policies trained on a 2D configuration transfer to 3D at roughly 10% of the direct training cost, while noting that the translational-invariance argument behind multi-agent RBC control does not extend cleanly to forced-convection settings.
A common thread runs through these developments. Given enough interaction and a large enough network, a reinforcement learning agent should in principle be able to learn a control law for any of these problems, so the practical obstacle is not what the policy class can represent but what the optimiser can find: when a system carries strong underlying structure and the learning problem is formulated without reference to it, the region of the action space that has to be explored grows with the number of actuators and the exploration cost becomes prohibitive. Belus et al. [3] made this argument directly for the stabilisation of a falling liquid film, where the combined combinatorial size of the output domain grows as a power of the number of jets, and showed that sharing one policy across local agents and rewarding each of them locally recovers what a monolithic formulation loses. That argument, rather than any property of a particular algorithm, is what the multi-agent formulations of Vignon et al. [35], Font et al. [9] and Suárez et al. [31] exploit, and it admits more than one implementation. Jeon et al. [16] revisit the configuration of Vignon et al. [35] and build the local reflection and rotation symmetries of the flow into group-invariant convolutional networks, retaining the multi-agent re-centring for the global translational invariance and improving the reproducibility of the learnt policies. The same reasoning applies on the temporal side, where Wang et al. [37] show for cylinder wake control with sparse pressure sensing that a policy acting on the instantaneous sensor signal is handicapped by the delay between actuation and response, and recover the missing information by lifting the measurement with its time derivative. Recurrence is an alternative route to that end, and the contribution of the present work is to supply the spatial and the temporal structure together, through a convolutional policy with gated recurrent memory that requires neither re-centring nor a delay. horizon fixed in advance. The two routes to temporal context are compared directly in § 4.
Returning to RBC, Beintema et al. [2] were the first to apply RL to this problem, suppressing convection entirely for in a unit-aspect-ratio domain and outperforming classical PD control. However, their limited domain constrained the flow to a single-cell configuration. Vignon et al. [35] extended the approach to wider periodic domains using a multi-agent RL (MARL) framework with full-field re-centred observations, where optimising for Nu reduction leads to cell coalescence, reducing Nu from approximately 2.7 to 2.0 at . Their approach draws two distinct benefits from the multi-agent decomposition: each environment step yields as many transitions as there are agents, and the re-centring of the full field about each agent is what allows a single shared policy to act differently on different segments. The MARL framework was subsequently extended to three-dimensional RBC by Vasanth et al. [34], achieving convection-intensity reductions of 23.5% and 8.7% at and 750, respectively. More recent work has extended DRL-based RBC control toward higher Rayleigh numbers: Markmann et al. [24] report Nu reductions of up to 33% at moderate and 10% in the chaotic regime using single-agent PPO with reward shaping; Chen and Constante-Amores [7] couple RL with a reduced-order model derived from proper orthogonal decomposition and autoencoders, achieving 16–23% Nu reduction under both single- and dual-boundary actuation.
Outside the buoyancy-driven setting, Holme et al. [15] address the multi-timescale nature of rotating detonation engine control through a moving reference frame that separates fast wave propagation from slower mode transitions, and through a physically anchored credit-assignment horizon with chosen on the gain-recovery timescale; the action period itself is treated as a swept hyperparameter rather than tied to a particular physical scale.
Despite these advances, the policies reported in the RBC control literature exhibit a recurring and underappreciated pathology: degenerate actuation. The actor outputs typically either saturate at the control bounds or settle into spatially disordered patterns with no clear connection to the convective topology. Such strategies may lower the reward functional by a small margin yet encode no physical mechanism. In turbulent flow control, where the actuation cadence is fast relative to the flow dynamics, this degeneracy manifests most visibly as bang-bang actuation: policies that alternate rapidly between extreme actuator values at each step, producing physically non-realisable switching that is energetically prohibitive and harmful to actuator hardware. In RBC at moderate Rayleigh numbers, where the actuation timescale is comparable to the convective turnover, the same root causes produce a related but distinct failure mode: spatially incoherent or saturated wall-temperature patterns that happen to perturb the flow without driving meaningful roll reorganisation.
We identify two compounding deficiencies as responsible. The first is insufficient network expressivity: all prior DRL approaches to RBC use MLP policies that receive the flow state as a flattened vector, discarding spatial locality and translational structure. An MLP cannot easily learn that adjacent wall segments should be actuated in concert to match the wavelength of the underlying convection rolls, and instead converges to the degenerate patterns described above. The second deficiency is the absence of temporal context. In a multi-agent setting, where each agent observes only a local fraction of the flow domain, the individual observation is no longer a sufficient statistic for the full system state: the problem is formally a decentralised partially observable Markov decision process (Dec-POMDP). A memoryless policy operating on a single snapshot cannot distinguish whether a local temperature change is the result of its own prior actuation or of the natural flow evolution, and this ambiguity drives the optimiser toward saturated or random outputs as a hedging strategy. An alternative remedy, recently applied to bluff-body wake control by Jia and Xu [17], augments the observation with a Kalman-filtered state estimate rather than introducing recurrence; the recurrent route adopted here learns the relevant summary internally and scales more directly to the high-dimensional spatially distributed observations of multi-agent wall control. To the best of our knowledge, both deficiencies are present, to a varying degree, in all existing DRL approaches to thermal convection control, and they have not been addressed together.
Both deficiencies have known remedies in the broader reinforcement learning literature that have not yet been brought to bear on thermal convection control. A recurrent policy, in particular one built around a gated unit such as the GRU, carries a hidden state across decision steps that lets each agent follow the delayed flow response to its own actuation and separate self-induced changes from background evolution; the architecture has a longstanding role in partially observable RL [14, 8, 19] and has been shown to encode genuine flow physics in turbulent wall modelling [1]. A convolutional encoder, in turn, treats the flow field as a structured spatial array and extracts coherent features without the full-field data augmentation of Vignon et al. [35]; graph neural networks extend the same inductive bias to arbitrary geometries [21], and explainability analyses confirm that spatially aware policies recover physically meaningful flow structures [4]. Off-policy training closes the loop: TD3 (Twin Delayed Deep Deterministic policy gradient) for the single-agent case and MADDPG (Multi-Agent Deep Deterministic Policy Gradient; Lowe et al. [23]) for the multi-agent case reuse past transitions through a replay buffer, cut the number of costly solver episodes [13], and accommodate recurrent actors through sequence sampling [19].
The present work addresses these gaps simultaneously through four design choices. Convolutional policy networks operating on local spatial patches replace the global MLP observations of Vignon et al. [35], providing spatial feature extraction through local receptive fields and weight sharing across the domain. Recurrent actors, augmented with a Gated Recurrent Unit (GRU), maintain temporal context across actuation steps, enabling each agent to track the flow response to its own past actions. Off-policy training, using a deterministic policy gradient algorithm for the single-agent and a multi-agent extension thereof (Lowe et al., [23]) for the multi-agent configuration, replaces on-policy methods, storing past transitions in a replay buffer and reusing them across multiple updates to reduce the number of expensive solver episodes required. Finally, action constraints, including a zero-mean projection, an amplitude cap, and spatial and temporal smoothness penalties, explicitly discourage the saturated and incoherent outputs that characterise degenerate policies.
Four configurations are compared: single-agent and multi-agent formulations, each trained with and without GRU recurrence, allowing the contributions of temporal memory and of the multi-agent structure to be assessed independently. The key finding is that, with sufficient architectural expressivity and temporal memory, even a single-agent policy achieves cell coalescence and meaningful reduction at in a wide periodic domain. The barrier in prior work was not the agent topology but the policy architecture, and the multi-agent formulation of Vignon et al. [35] is not a prerequisite for discovering the correct physical mechanism. The multi-agent configuration provides additional benefits, including better spectral alignment with the dominant convective mode and a larger reduction, while GRU recurrence accelerates convergence by approximately 100 episodes across all configurations. The framework is validated on two buoyancy-driven configurations of increasing complexity: RBC at , where the objective is to reduce through cell coalescence, and double-diffusive convection in the salt-finger regime, where the objective is to enhance heat transfer and accelerate scalar mixing. In both cases the learned strategies are smooth, spatially structured, and physically interpretable, with no trace of the degenerate outputs that motivated the architectural choices above.
The remainder of the paper is structured as follows. § 2 describes the governing equations and flow configurations. § 2.2 details the numerical solver. § 3 presents the DRL formulation. § 4 and § 5 report training and evaluation results for the RBC case. § 6 presents the double-diffusive extension. § 7 summarises the main findings.
Physical problem and discrete formulation
Governing equations, computational domain and boundary conditions
We consider a two-dimensional rectangular domain of width and height , with the half-height used for non-dimensionalisation. The fluid is incompressible and obeys the Boussinesq approximation: density variations appear only in the buoyancy term.
Using the free-fall scaling, lengths are made non-dimensional by , velocities by the free-fall velocity , time by , pressure by , and the temperature by , i.e. the difference between the mean bottom-wall temperature and the top-wall temperature . The resulting dimensionless system reads:
The system is governed by the Rayleigh number and the Prandtl number . No-slip conditions are imposed at both horizontal walls. The top wall is held at fixed temperature , while the bottom wall temperature serves as the reference value around which the agents apply zero-mean perturbations , so that the effective bottom-wall temperature is at any given time. The -direction is periodic. The non-dimensional temperature is shifted rather than centred about its mean, so the uniform part of the buoyancy term in equation (2) is constant and is absorbed into the pressure gradient.
Throughout the paper the Nusselt number is evaluated from the wall temperature gradients as the mean of the values at the two horizontal walls,
where denotes the average along the wall. At the bottom wall the gradient is evaluated through the ghost cell, so that the instantaneous actuated temperature enters the estimate, while the normalisation retains the fixed reference difference rather than the instantaneous wall temperature. Equation (4) is the quantity reported in all figures and tables, and it is the one entering the reward of § 3.4.
All the present simulations use , (air) and . The uncontrolled steady state exhibits four counter-rotating convection rolls, yielding a baseline Nusselt number . The DRL control objective is to reduce Nu by promoting cell coalescence, the merging of rolls into fewer, larger structures under wall actuation, which weakens convective mixing and drives Nu toward the conductive limit (Nu = 1), where heat transfer occurs entirely by diffusion.
Numerical method
The governing equations are solved using a staggered-grid pressure correction method ([33]) implemented entirely in PyTorch, enabling execution on both CPU and GPU. To avoid numerical dissipation, all spatial operators are discretised using second-order central differences. Time integration uses an explicit Adams-Bashforth scheme with adaptive time stepping controlled by CFL and Fourier number constraints, maintaining CFL and Fo on a uniform grid. The solver is wrapped in Gymnasium (single-agent) and PettingZoo (multi-agent) environment interfaces that handle observation extraction, action application, reward computation, and episode management.
Deep reinforcement learning formulation
In the reinforcement learning paradigm, a decision-making agent interacts repeatedly with an environment (here the Boussinesq flow solver) by observing its current state, selecting an action, and receiving a scalar reward that reflects how desirable the outcome was. Through trial and error the agent learns a policy, i.e. a mapping from observed states to actions, that maximises the cumulative reward over time. Applied to the present problem, the agent observes the instantaneous temperature and velocity fields, perturbs the bottom-wall temperature over a set of discrete segments, and is rewarded according to the resulting change in heat transfer.
Formally, the control task is cast as a Markov decision process , where is the space of instantaneous flow fields , the wall-temperature perturbations over segments, the transition kernel (i.e. the probability of the flow advancing to state from state under actuation ) governed by the Boussinesq equations, the scalar reward, and a discount factor that down-weights rewards received further in the future, ensuring the cumulative sum remains bounded. The agent seeks a deterministic policy that maximises the cumulative discounted reward over a trajectory, starting from the cached uncontrolled steady state ,
where the flow state is obtained by advancing the Boussinesq solver one actuation step from under the wall temperature prescribed by . In plain terms, the agent is rewarded at every step for reducing heat transfer, but rewards collected further in the future count for less (through the factor ); the policy is then the wall-temperature strategy that accumulates the highest total score over the entire episode.
Two off-policy algorithms are used to optimise : Twin Delayed Deep Deterministic policy gradient (TD3; [10]) for the single-agent case, and Multi-Agent Deep Deterministic Policy Gradient (MADDPG; [23]) for the multi-agent case. Both TD3 and MADDPG improve the policy by following the gradient of with respect to the network parameters ([28, 22]),
where is the action-value function, i.e. the total discounted reward expected from state when action is applied and the policy is followed thereafter,
In practice, is not known and is approximated by a second neural network (the critic), trained concurrently with the policy (the actor). The gradient in equation (6) then tells the actor how to adjust its output so that the critic predicts a higher future reward.
When multiple agents control different portions of the wall simultaneously, each one can only sense its local neighbourhood rather than the full flow domain. This limited observability means the single-agent MDP framework is no longer sufficient: the action of one agent affects the observations of its neighbours, so the agents must learn to coordinate through the shared reward signal alone, without any direct communication between them. When multiple agents control different portions of the wall simultaneously, each one can only sense its local neighbourhood rather than the full flow domain. The action of one agent therefore affects the observations of its neighbours, so the agents must learn to coordinate through the shared reward signal alone, without any direct communication between them. The multi-agent setting is then described by extending the MDP tuple to include a separate observation space for each agent ,
in which agent receives only the local observation , the flow state projected onto the patch of three consecutive wall segments visible to that agent (Figure 1), and outputs .

Figure 1. Schematic of the control setup for the multi-agent formulation. The bottom wall is divided into segments; agent 5 observes the local flow state over a patch of three consecutive segments (shaded orange box) but applies its temperature perturbation to its own central segment only (shaded green). The top wall is held at fixed temperature ; the -direction is periodic.
All agents share the same scalar reward , so they are collectively driven to improve the global heat transfer objective rather than optimising their individual segments in isolation. During training, a shared critic is given access to the full flow state and all agents’ actions, allowing it to assess the global consequence of each agent’s local decision. At deployment, however, each agent acts on its local observation alone, without knowledge of what its neighbours are doing. This centralised-training, decentralised-execution strategy (CTDE; [23]) allows the agents to learn coordinated behaviour during training while remaining independently deployable.
Four configurations are compared: single-agent (TD3) and multi-agent (MADDPG), each trained with and without GRU recurrence. The effect of each choice is isolated by comparing all four combinations, as detailed in the subsections below.
Single-agent TD3
The single-agent formulation treats the entire bottom wall as one control surface, observing the full downsampled flow field and outputting temperature perturbations for all segments simultaneously. The three physical fields are downsampled by a factor of 2 (from to ), and an optional fourth channel encodes the previous action as a spatially constant field, giving an observation with or 4.
The actor processes this observation through four stride-2 convolutional layers with kernels and LeakyReLU activation, reducing the spatial dimensions from to and yielding a 768-dimensional feature vector after flattening. A fully connected layer with LayerNorm maps this to a 256-dimensional hidden representation, followed by a final linear layer with tanh activation producing .
The off-policy algorithm underlying the single-agent formulation is Twin Delayed DDPG (TD3; [10]), chosen for its sample efficiency and training stability relative to on-policy alternatives such as PPO. The architecture and training hyperparameters follow Fujimoto et al. [10]; the reader is referred there for full algorithmic details.
Multi-agent MADDPG
As in the single-agent formulation, the bottom wall is divided into equal segments, but here each segment is controlled by a dedicated agent (). Agent controls the temperature over its segment of width and observes a local patch spanning 3 consecutive segments (its own plus one neighbour on each side, with periodic wrapping — see Figure 1). All agents share the same policy network.
The value follows Vignon et al. [35] and is retained here so that the two studies remain directly comparable without the segmentation acting as a confounding difference. Appendix § 9 reports what happens when it is varied.
Each agent uses a compact convolutional network: two convolutional layers extract spatial features from the local observation, followed by a fully connected layer that produces a single temperature perturbation for that agent’s segment.
During training, a centralised critic has access to the full flow state and all agents’ actions, allowing it to assess the global consequence of each local decision. At deployment, only the local actor is used — each agent acts on its own observation alone, without knowledge of what its neighbours are doing, consistent with the strategy described in § 3.
Action constraints
All configurations share the same action-processing pipeline. The raw actor outputs are first shifted to have zero mean,
so that the actuation adds no net heat to the bottom wall. The corrected actions are then rescaled to prevent any single segment from saturating the actuator. Whenever the largest absolute value exceeds a threshold (here ), all actions are scaled down proportionally,
Finally, cubic Hermite interpolation over 10% of the segment width smooths the temperature profile at segment boundaries, yielding the wall condition .
Reward design
At each actuation step , each agent receives a scalar measure of how much the instantaneous heat transfer (i.e. the instantaneous Nusselt number) has been reduced relative to the uncontrolled baseline (i.e. a reward),
In equation (11), is an offset chosen so that the reward is positive when Nu falls below the uncontrolled baseline, and is a scaling factor. The small weight blends a local Nusselt number , computed over agent ’s own segment, into the global value, giving each agent a localised measure of its individual contribution to the overall heat transfer reduction. For the single-agent case the local term vanishes () and the reward simplifies to : i.e. the agent is rewarded in direct proportion to the reduction in global heat transfer.
GRU architecture
The flow response to wall actuation is not instantaneous, as convective structures take time to form, merge, and reorganise after a change in boundary temperature. As a result, the instantaneous flow snapshot alone does not carry enough information to determine the best next action: the same snapshot could arise from a flow that is responding to a recent actuation or from one that has been evolving freely, and the two situations call for different responses. Retaining a memory of past observations resolves this ambiguity, allowing the agent to track how the flow is evolving over time and to attribute changes in heat transfer to its own previous actions. To provide this memory, a Gated Recurrent Unit (GRU; [8]) is incorporated into the policy network, between the convolutional encoder (the CNN that processes the flow snapshot) and the output layer that produces the wall-temperature action. At each actuation step, the GRU updates a hidden state vector that summarises the history of past observations, which is then carried forward to the next step. The CNN feature vector is first projected to dimension via a linear layer, then passed through the GRU, whose update equations are:
where is the projected CNN feature vector, the hidden state carrying memory across actuation steps, the sigmoid function, and elementwise multiplication. The reset gate controls how much of the past memory is retained at each step, while the update gate balances the contribution of the new flow observation against that memory.
Training with memory requires temporally contiguous data, since the past observations must be propagated forward in time to build up a meaningful memory. Subsequences of length are therefore sampled from stored episodes. The first steps of each subsequence are used solely to warm up the memory before the policy update is computed, and are not used in the parameter update ([19]).
Training procedure
All four configurations (single-agent TD3 and multi-agent MADDPG, each with and without GRU recurrence) are trained for 350 episodes of 200 actions each (150,000 solver time steps per episode). Before training, the solver is advanced for 400 thermal diffusion times () to reach the uncontrolled steady state, which is saved and reloaded at the start of each episode.
Key training parameters (shared across all configurations unless noted) are listed in Table 1.
| Category | Parameter | Value |
| Physics | Rayleigh number | 10,000 |
| Physics | Prandtl number | 0.71 |
| Physics | Grid | |
| Control | Segments | 10 |
| Control | Substeps per action | 750 |
| Control | Amplitude cap | 0.75 |
| Control | Smoothing fraction | 10% |
| Networks | GRU hidden size | 64 |
| Networks | Actor / critic LR | / |
| Networks | Target update | 0.005 |
| Training | Max episodes | 350 |
| Training | Episode length | 200 actions |
| Training | Batch size | 64 (TD3) / 32 (MADDPG) |
| Training | Discount | 0.99 |
| Training | Gradient clip | 0.75 |
| Exploration | Initial noise | 0.1 |
| Exploration | Final noise / decay | 0.01 / 100 ep |
Table 1. Training hyperparameters.
Left unconstrained, the actor tends to produce actuation patterns that are spatially discontinuous, temporally erratic, or saturated at the maximum allowable value, none of which correspond to physically meaningful control strategies. To discourage these degenerate behaviours, four penalty terms are added to the training objective, each targeting a specific undesirable pattern. The first penalises any residual deviation from zero mean after the projection (9),
The second penalty addresses spatial discontinuities: abrupt jumps in wall temperature between adjacent segments can produce localised flow structures at segment boundaries that are consequences of the discretisation rather than genuine control features,
while the temporal smoothness loss penalises rapid changes between consecutive actuation steps,
The temporal smoothness penalty follows the CAPS regularisation of Mysore et al. [25], originally introduced for single-agent continuous control; the spatial smoothness term enforces coordination across the segmented action vector and has no direct precedent in that line of work. Finally, since the actor output is bounded to by the tanh activation, a penalty on large action magnitudes discourages the policy from saturating at these limits, where the gradient vanishes and learning stalls,
The four penalties are combined with the main training objective through weights , , , and , whose values differ between the single-agent and multi-agent configurations. These are listed in Table 2.
| Loss | MADDPG | TD3 |
| 0.1 | 2.0 | |
| 0.05 | 0.02 | |
| 0.025 | 0.025 | |
| 0.1 | 0.1 |
Table 2. Auxiliary loss weights.
Training results: Rayleigh–Bénard convection
The four configurations compared throughout this section and the next are collected in Table 3, together with the labels and colours used to identify them in every figure. The actor sizes are worth noting, since they show that the recurrent variants are not simply larger networks: in the single-agent case the GRU actor is the smaller of the two, because the projection to the hidden state replaces the wide dense layer that follows the flattened convolutional features.
| Label | Algorithm | Agents | Observation | Memory | Actor |
| Single-agent, no GRU | TD3 | 1 | full field | none | 441,066 |
| Single-agent + GRU | TD3 | 1 | full field | GRU, | 315,946 |
| Multi-agent, no GRU | MADDPG | 10 | 3 segments | none | 54,609 |
| Multi-agent + GRU | MADDPG | 10 | 3 segments | GRU, | 79,569 |
| Multi-agent, stack | MADDPG | 10 | 3 segments | 4 frames | 56,337 |
| Multi-agent, wide | MADDPG | 10 | 3 segments | none | 81,537 |
Table 3. The four control configurations and the labels used to identify them in the text and in every figure. Blue denotes the single-agent configurations and orange the multi-agent ones, with dark shades for the GRU-equipped variants and light shades for the memoryless ones, and green the frame-stacked variant. Actor sizes are the number of trainable parameters of the policy network; the multi-agent actor is shared by all ten agents.
Figure 2 shows how the Nusselt number evolves over the course of training for all four configurations (single-agent TD3 and multi-agent MADDPG, each with and without GRU recurrence). Two metrics are reported. The first is the Nusselt number averaged over the full episode, ; this includes the initial transient during which the agent is still learning to modify the flow. The second is the Nusselt number averaged over the last 20% portion of the episode, ; this reflects how effectively the agent sustains heat transfer reduction once the flow has fully responded to its actuation.
Figure 2. Training convergence of the Nusselt number for four agent configurations. Top: episode-averaged Nu. Bottom: late-episode Nu (last 20% of each episode). Blue: single-agent (TD3); orange: multi-agent (MADDPG). Dark shades: GRU-equipped; light shades: feedforward. Lines show rolling mean (window = 10 episodes); shaded envelopes indicate the range of raw values within each window.
All four control strategies achieve a similar flow topology characterised by cell coalescence, reducing Nu well below the uncontrolled baseline of 2.48. This confirms that the CNN architecture is sufficient to learn effective control regardless of whether a single or multiple agents are used, and stands in
contrast to Vignon et al. [35], where achieving coalescence with a multi-agent approach required re-centring the full field about each agent, which effectively provides ten times more full-field trajectories for the shared policy.
The multi-agent strategies consistently reach lower values than the single-agent approach, particularly in the late-episode metric, suggesting that distributing control across local observers better matches the spatially distributed nature of the convective structures. The GRU-equipped variants learn faster, most visibly in the multi-agent case where drops below 2.0 within the first 50 episodes, reflecting the benefit of temporal memory in tracking the delayed response of the flow to wall temperature changes and anticipating the evolution of the convective structures. Finally, proves more informative than alone, as it reflects not only the final heat transfer reduction but also how quickly the flow reorganises into the coalesced state within each episode: a lower at comparable indicates that coalescence is reached earlier in the episode.
A fixed window of past observations and a larger network are the two obvious ways of augmenting the baseline feedforward policy without introducing recurrence, and neither reaches what the recurrent state provides. Both were trained with every other setting held fixed. Stacking the four most recent observations as additional input channels gives a median over three seeds of 1.970, against 1.865 with the GRU and 1.801 with no temporal input at all, so extending the observation backwards in time does not improve the converged policy and here degrades it. Convergence is also markedly slower, the ten-episode rolling mean of first reaching 1.9 after 305 episodes. against 147 with no temporal input and 49 with the GRU. Widening the actor instead, to 81,537 trainable parameters against the 79,569 of the recurrent policy, fares no better: that configuration settles at 2.072 and does not reach 1.9 at all within 350 episodes.
Capacity therefore does not account for the difference. The stacked actor already carries more parameters than the unstacked one, 56,337 against 54,609, and the widened actor carries more than the recurrent one while performing worse than either. What distinguishes the recurrent policy is not the size of its input or of its parameter vector but what it does with them. A stack of frames fixes the memory horizon in advance and multiplies the observation dimension by , whereas the recurrent state learns what to retain and over what horizon. That matters here because the delay between wall actuation and its effect on the bulk is set by the diffusive and advective transport times of the flow rather than by a number available before training.
Since the reward is defined as a decreasing function of the instantaneous Nusselt number, it mirrors the trends described above. Figure 3 shows its evolution over training, confirming that the learning curves are consistent across both metrics.
Figure 3. Episode reward during training for the four configurations. Lines show rolling mean (window = 20); shaded envelopes show raw value range. Higher reward corresponds to lower Nu.
Comparison with previous results
Table 4 compares the present approach with Vignon et al. [35]. The key difference is architectural: Vignon et al. [35] use a flat network that receives the entire flow field as a vector, discarding its spatial structure, while the present work uses convolutional networks that process the field as a two-dimensional array and extract spatially coherent features. In their multi-agent setup, each agent is given the full flow field rotated so that its own segment always appears at the centre, effectively multiplying the number of training trajectories by a factor of ten. The multi-agent setup proposed here is fundamentally different: each agent observes only a local patch of 3 neighbouring segments, with no rotation or data augmentation. The convolutional architecture achieves comparable results without this augmentation, indicating that processing the flow field spatially and restricting each agent to its local neighbourhood are together sufficient to learn effective control.
| Aspect | Vignon et al. | This work |
| Algorithm | PPO (on-policy) | TD3 / MADDPG (off-policy) |
| Architecture | MLP (2 × 512) | CNN (stride-2 encoder) |
| Observation | Full field, recentred per agent | Full field (single) / local patch (multi) |
| MARL design | Recentering trick (each agent sees full field rotated above it) | Local patches (each agent sees 3 neighbouring segments) |
| Data multiplication | 350 ep. × 10 full-field views = 3500 eff. trajectories | 350 ep. (local transitions, no full-field augmentation) |
| Cell coalescence | Yes (MARL only, after 3500 eff. trajectories) | Yes (all 4 configs in 350 episodes) |
| Recurrence | No | GRU (optional, accelerates training) |
Table 4. Comparison with the multi-agent formulation of Vignon et al. [35].
Expected behaviour at higher Rayleigh number
The results above are obtained in a regime where the uncontrolled flow is steady, and it is worth setting out what would change as is raised towards the chaotic regime, since the auxiliary weights of Table 2 are calibrated against physical scales rather than being free constants.
The temporal weight is referenced to the time the flow takes to respond to a change in wall temperature, which is governed by conduction across the thermal boundary layer. That thickness follows and so contracts slowly, from 0.20 at to 0.145 at , while the ratio of the diffusive to the free-fall time scale, , grows from 84 to 238. In free-fall units the response delay is therefore comparatively insensitive to and the actuation interval used here would transfer. What does not transfer is the assumption of a steady response: once the flow fluctuates on the timescale of the actuation, a penalty tuned against a steady approach to equilibrium over-regularises.
The spatial weight is referenced to the roll wavelength, and this scale cannot be inferred from alone in a domain of fixed width. Raising eightfold in the present box in fact increases the dominant horizontal scale, the uncontrolled state passing from four rolls to two, so the reference length has to be taken from the observed flow. The same caution applies to the observation resolution, which must continue to resolve the near-wall gradient as contracts.
Two limitations follow. The controller learnt here exploits a well-defined topology, so where the flow is chaotic the reward becomes a noisy function of the actuation and a smaller reduction should be expected. And the reduction available through coalescence is bounded by how far the uncontrolled state already sits from the largest structure the domain admits, so results at different in a fixed box are not comparable unless that distance is. We would expect the qualitative conclusion of this work to carry over to the chaotic regime, but establishing it there is a study in its own right rather than a change of one parameter.
Evaluation results: Rayleigh–Bénard convection
The four control strategies are evaluated over episodes of 200 actions (300 free-fall time units , or approximately 3.6 thermal diffusion times ), starting from the same uncontrolled steady state used during training. Figure 4 shows the instantaneous Nusselt number during evaluation. The uncontrolled flow maintains throughout, while all controlled cases progressively reduce toward the conductive limit. The multi-agent strategy achieves a deeper and faster reduction than the single-agent one, consistent with the advantage observed during training.
Figure 4. Instantaneous Nusselt number during evaluation for all four configurations. The baseline (grey) maintains , while every controlled case reduces toward the conductive limit. Blue denotes the single-agent configurations and orange the multi-agent ones, with dark shades for the GRU-equipped variants.
Individual evaluation curves such as those of Figure 4 are single realisations of a stochastic training process, and the approach to the controlled state is not a clean transition. Figure 5 shows the two multi-agent configurations evaluated from three independent training runs each, differing only in the random seed. Within one configuration the realisations follow visibly different paths: one memoryless run reorganises the flow before , another holds near until and then drops abruptly, and the third descends gradually. All six settle into the same interval, , so the level reached by the multi-agent policies is the same to within the scatter between realisations and no ordering can be read from any single pair of curves. The recurrent variant is, however, the more consistent of the two in where it settles: its three plateaux span 1.817 to 1.870, an interval contained entirely within the one spanned by the memoryless runs, and its curves hold their level once reached, drifting by , and between mid-episode and the end of the episode where the memoryless runs drift upward by , and . The lowest single value of all six belongs to a memoryless run, so temporal memory is not what sets the depth of the reduction; what it provides is a policy that reaches a stable configuration and stays there.
Figure 5. Instantaneous Nusselt number during evaluation for the two multi-agent configurations, each trained three times from independent random seeds and evaluated from its best checkpoint. Line style identifies the seed. The grey band spans the plateaux of all six runs, 1.763 to 1.938; the dashed line is the uncontrolled value.
The physical origin of this reduction is visible in Figure 6, which shows the temperature field at several instants during the episode. All controlled cases undergo progressive cell coalescence: the initial four-roll structure merges into fewer, larger rolls, and it is this reorganisation of the convective topology that drives downward. The multi-agent strategy promotes coalescence more rapidly, with the transition from four to two rolls well under way by . Figure 7 compares the final temperature fields from the multi-agent strategies with and without GRU memory. Both reach the coalesced state, confirming that the convolutional architecture alone provides sufficient spatial expressivity to discover the correct flow reorganisation. The variant with temporal memory (i.e. the GRU-equipped variant) reaches this state earlier in training (cf. Figure 2). This is expected, since convective structures take time to reorganise in response to a change in wall temperature: without temporal context, the same instantaneous flow snapshot can arise from very different actuation histories, making it harder to identify an effective control strategy. The temporal memory resolves this ambiguity by carrying information about the recent evolution of the flow forward in time.
Figure 6. Temperature field at four instants during evaluation. Rows: multi-agent + GRU (top), single-agent + GRU (middle), uncontrolled baseline (bottom). The controlled cases show cell coalescence: merging of convection rolls into larger structures. The mechanism is apparent at the lower boundary, where the policy introduces heat at the edges of the base of each cell rather than at its centre, widening the region held above the mean wall temperature and so broadening the base over which the cell is supported.
Figure 7. Final temperature fields: multi-agent + GRU (top) vs. multi-agent without GRU (bottom). Both reach cell coalescence, but the GRU variant achieves this earlier during training (cf. Figure 2).
The time-averaged Nusselt numbers for all four strategies are reported in Table 5. The wall temperature maps in Figure 8–Figure 11 show the spatiotemporal evolution of the actuation applied by each strategy. In all cases the actuation is smooth and spatially structured, with no trace of the rapid switching or saturated outputs that arise when the policy fails to identify a physical control mechanism. The single-agent maps display broad, slowly evolving patterns spanning the full domain, while the multi-agent maps exhibit finer spatial modulation reflecting the individual agents’ segment-by-segment control. The strategies equipped with temporal memory produce smoother wall-temperature histories than their memoryless counterparts, reflecting the regularising effect of carrying information forward across succes-
| Configuration | Reduction | |
| Baseline (uncontrolled) | 2.48 | — |
| Single-agent, no GRU | 2.06 ± 0.17 | 17.1% |
| Single-agent + GRU | 2.17 ± 0.08 | 12.5% |
| Multi-agent, no GRU | 1.83 ± 0.00 | 26.3% |
| Multi-agent + GRU | 1.85 ± 0.07 | 25.5% |
Table 5. Evaluation Nusselt numbers (time-averaged over the last 90% of the episode) for the four control strategies.
Figure 8. Space–time map of bottom-wall temperature for the single-agent + GRU strategy. The reference value corresponds to white.
Figure 11. Space–time map of bottom-wall temperature for the multi-agent, no GRU strategy. The same edge-heating mechanism is recovered without memory, and the actuation is steadier than in Figure 9, but only once a good configuration has been reached, at which point the instantaneous field is unambiguous enough for a memoryless policy to act on.
Figure 9. Space–time map of bottom-wall temperature for the multi-agent + GRU strategy. The edge-heating mechanism of Figure 6 is sustained here as a persistently dynamic actuation, the hidden state carrying information about the preceding evolution of the flow.
sive actuation steps.
The zero-mean projection of equation (9) guarantees that the actuation adds no net heat to the bottom wall, but holding a spatially non-uniform and time-varying wall temperature is not thermodynamically free, and the engineering value of the control rests on how its cost compares with the transport it suppresses. A lower bound on that cost follows from the reversible redistribution of heat along the wall: transferring a quantity of heat from a segment held at to one held at requires at least of work, where is the absolute operating temperature. Since the non-dimensional formulation leaves free, the ratio must be evaluated at a chosen operating point. For an air layer with about an ambient temperature of , the largest perturbation admitted by equation (10), , places at a few per cent, to be set against the 26% reduction in the heat crossing the layer. This estimate counts only the reversible work of redistributing heat along the wall, and excludes the irreversibility of any physical thermal actuator, the power drawn by sensing and by evaluating the policy, and the transient cost of establishing the actuation at the start of an episode, all of which depend on the device rather than on the flow.
Figure 10. Space–time map of bottom-wall temperature for the single-agent, no GRU strategy.
Double-diffusive convection in the salt-finger regime
The DRL framework is now extended to double-diffusive convection, where two scalar fields with different molecular diffusivities, i.e. the temperature and the salinity , drive buoyancy simultaneously. The governing equations retain the Boussinesq framework of § 2.1, augmenting the momentum equation with a two-component buoyancy term and adding a salinity transport equation [32]:
Two additional non-dimensional parameters appear: the Lewis number , the ratio of thermal to solutal diffusivity, and the buoyancy ratio , which measures the relative strength of solutal to thermal buoyancy.
These equations admit the salt-finger regime Stern [29], named after the tall, thin fingers of alternately sinking and rising fluid that form through a positive feedback: downward-displaced parcels rapidly lose heat but retain salt, become denser, and continue sinking. The parameters , (water-like), , and are selected to promote this regime; together they imply a well-separated hierarchy of boundary-layer thicknesses , as expected in the salt-finger regime. The same pair has recently been used in wall-resolved LES of bounded turbulent double-diffusive convection across a wider range of thermal and concentration Rayleigh numbers Kenjeres and Roovers [20]; that work provides a reference point for the uncontrolled flow physics on which the present control task acts. The computational domain is a tall cavity with aspect ratio , resolved on a grid. Temperature uses Dirichlet boundary conditions ( at the bottom, at the top), while salinity uses no-flux (Neumann) conditions at both walls.
In this configuration, the control task differs from the RBC case in both objective and mechanism: rather than suppressing convection, the goal is to enhance it. Eight agents () modulate the bottom-wall temperature in zero-mean segments, using the same MADDPG framework with GRU-augmented actors described in § 3.2. The objective is to maximise the Nusselt number; temperature actuation is a natural lever in this regime because, with , heat diffuses two orders of magnitude faster than salt, making wall temperature perturbations an efficient means of modifying the local buoyancy field.
The trained policy is evaluated over a 200-action episode starting from the uncontrolled steady state. The controlled policy achieves a sustained increase in heat transfer: the time-averaged Nusselt number rises from to , an improvement of 19.1%. The Sherwood number , (the solutal analogue of Nu, measuring the non-dimensional salt flux) decreases slightly by , while the salinity spatial variance is reduced by 21.0% (from 0.0629 to 0.0497), indicating faster mixing. These quantities are summarised in Table 6.
| Quantity | Baseline | Controlled | Change |
| Nusselt Nu | |||
| Sherwood Sh | |||
| Salinity variance |
Table 6. Double-diffusive control results (time-averaged over the last 90% of the episode).
Figure 12 shows the time evolution of Nu and Sh. The Nusselt increase is established rapidly and sustained throughout the episode, while the Sherwood number remains close to its baseline value, confirming that the policy enhances thermal mixing without significantly altering the salt flux.
Figure 12. Time evolution of (top) and (bottom) for the double-diffusive case. The policy increases by 19.1% while leaving salt flux nearly unchanged.
The evolution of salinity spatial variance (Figure 13) confirms faster homogenisation: the controlled case reduces by 21.0% relative to the
Figure 13. Salinity spatial variance : the controlled case achieves 21.0% reduction relative to the baseline, reflecting faster mixing of the salt distribution.
baseline, with the half-life of variance decay shortening from 96.0 to 83.2 time units (a 1.2 speedup).
Figure 14 and Figure 15 compare the temperature and salinity fields at five instants. The controlled case shows a progressive disruption of the finger pattern and faster homogenisation of the salt distribution, consistent with the reduced salinity variance reported above.
Figure 14. Double-diffusive case: temperature field at five instants. Top row: controlled; bottom row: baseline.
Figure 15. Double-diffusive case: salinity field at five instants. Top row: controlled; bottom row: baseline. The controlled case shows faster homogenisation of the salt distribution.
Beyond the integral quantities, the wall temperature map (Figure 16) reveals a more subtle and physically revealing feature of the learned policy: a coherent travelling-wave structure in the actuation pattern. Tracking the phase of the fundamental Fourier mode () of the wall perturbation yields a piecewise-linear phase displacement (Figure 17), indicating two distinct regimes of constant wave speed. During the first phase () the pattern propagates at , while in the second phase () the wave slows to , a 46% reduction. The slowdown coincides with the salinity field approaching a more mixed state: as the salt fingers weaken, the policy naturally reduces the aggressiveness of its stirring strategy. This adaptive behaviour emerges entirely from the reward signal, without any explicit encoding of wave speed or mixing state in the control formulation.
Figure 16. Space–time map of bottom-wall temperature for the double-diffusive case.
Figure 17. Cumulative phase displacement of the Fourier mode of the wall temperature perturbation, with piecewise-linear fit.
Summary and conclusions
Degenerate actuation, in the form of saturated, pseudo-random, or spatially incoherent wall-temperature policies, has been identified as a recurring pathology in DRL-based thermal convection control, rooted in two compounding deficiencies: MLP policies that discard spatial flow structure, and
memoryless policies that cannot attribute local flow changes to their own prior actuation. A framework addressing both causes has been presented, combining convolutional policy networks, Gated Recurrent Unit (GRU) memory, off-policy training (TD3 / MADDPG), and explicit action-smoothness constraints. The framework has been tested on two buoyancy-driven configurations of increasing physical complexity.
In the Rayleigh–Bénard convection case at , all four agent configurations, single-agent TD3 and multi-agent MADDPG each with and without GRU recurrence, successfully learn cell-coalescence strategies, reducing Nu to as low as 1.83 (26% below the uncontrolled baseline of 2.48) within 350 training episodes. This matches the effective sample efficiency of the 3500-trajectory approach of Vignon et al. [35] without requiring their re-centred full-field observations, demonstrating that convolutional inductive biases and local observations are together sufficient. The multi-agent configuration yields deeper Nu reductions than the single-agent case, reflecting better spectral alignment with the dominant convective mode, while GRU recurrence accelerates convergence by approximately 100 episodes across all configurations. Coalescence is achieved even by the single-agent policy, establishing that the multi-agent formulation is not a prerequisite once the policy architecture is sufficiently expressive. The learned actuation in every case is smooth and spatially structured, with none of the hallmark failures of degenerate control. In double-diffusive convection in the salt-finger regime, the multi-agent recurrent policy enhances heat transfer by 19.1% and reduces salinity variance by 21.0%, accelerating scalar mixing. The learned wall-temperature actuation organises spontaneously into a coherent travelling wave whose phase speed adjusts in response to the evolving state of the flow: the wave propagates at during the initial finger-dominated phase and slows to as the salinity field approaches a more homogeneous state, a 46% reduction in propagation speed. This state-dependent actuation pattern arises entirely from the scalar reward signal, with no prescribed wave structure and no explicit knowledge of the mixing state beyond the instantaneous Nusselt number. That a boundary-temperature policy should spontaneously select a travelling-wave strategy to disrupt salt fingers is a physically nontrivial outcome, one that would be difficult to anticipate or derive from linear stability arguments alone.
Set against the wider landscape, these results support the view that the progress reported across the active flow control literature comes from building the structure of the problem into the formulation rather than from any particular learning algorithm. Belus et al. [3] made that case for the spatial dimension, and multi-agent decomposition has since been its usual expression; what the present results add is that a convolutional policy reaches the same end without decomposing the domain at all, so the decomposition is one way of supplying the inductive bias rather than the only one. The temporal half of the problem, identified for wake control by Wang et al. [37] and addressed there through an explicit delay feature, is met here by recurrence, which learns the relevant summary of the past instead of committing to a memory horizon chosen in advance. Read together with the group-invariant architectures of Jeon et al. [16], the picture that emerges is one of interchangeable mechanisms for injecting the same spatial and temporal structure, and the practical question for future work is which of them compound and at what training cost.
Higher Rayleigh numbers will shorten the convective timescale relative to the actuation period and bring the flow into a chaotic regime, and it is far from obvious that the recurrent architecture will remain stable under the much noisier observations that follow. Three-dimensional geometries add a spanwise dimension to the convective organisation and a substantial training cost, for which reduced-order surrogate environments offer a plausible route. The double-diffusive case hints at a wider class of buoyancy-driven flows with competing scalar fields, of which thermohaline circulation and certain crystallisation processes are concrete examples, where the present framework could be transferred without major modification. Physical experiments will add sensor noise, actuator inertia and finite-bandwidth boundary conditions, none of which are represented in the idealised simulations used here.
Sensitivity to the auxiliary loss weights
The weights of Table 2 are intended to shape the actuation without reshaping the objective, and it is worth recording how little the outcome depends on their exact values. Each of and was varied by a factor of ten either side of its tabulated value, all other settings held fixed, on the multi-agent recurrent configuration. The unmodified configuration was also trained at three random seeds, which sets the scatter against which any effect has to be judged. Table A.1 reports the average over the final 50 training episodes of each run.
| three seeds, weights as tabulated | 1.834–2.111 | 0.568–0.687 | 0.012–0.067 | 0.403–0.449 | |
| 0.005 | 1.879 | 0.729 | 0.067 | 0.473 | |
| 0.05 | 1.865 | 0.580 | 0.067 | 0.449 | |
| 0.5 | 1.904 | 0.245 | 0.022 | 0.404 | |
| 0.0025 | 1.914 | 0.582 | 0.022 | 0.431 | |
| 0.025 | 1.865 | 0.580 | 0.067 | 0.449 | |
| 0.25 | 1.851 | 0.655 | 0.024 | 0.434 |
Table A.1. Sensitivity of the trained policy to the smoothness weights. The first row gives the range obtained from three random seeds at the tabulated weights; is the actuation amplitude against a cap of .
The control objective is insensitive over the whole range. The late-episode Nusselt number spans 1.851 to 1.914 across two decades of either weight, a band that lies inside the 1.834 to 2.111 obtained from three seeds at the tabulated values, so either weight can be moved by an order of magnitude with less effect on the result than changing the random seed produces. The spatial weight nonetheless acts on the quantity it targets, the roughness falling monotonically from 0.729 to 0.245, although sublinearly, since a hundredfold change in the weight produces a threefold change in the roughness. The temporal weight is not resolved by this design, its variation across two decades moving the realised jitter by less than the seeds move it at fixed weights. No setting drove the policy towards saturation, the actuation amplitude remaining between 0.40 and 0.47 against the cap of 0.75 throughout. The penalties therefore guide the optimiser gently, among the many policies that reduce Nu comparably well, towards the smooth and spatially coherent ones, rather than reshaping the objective around themselves.
Sensitivity to the number of wall segments
The choice of § 3.2 was tested by repeating the multi-agent recurrent training at and , a factor of four in segment count, with all other settings held fixed. Table B.1 reports the same quantities as Table A.1 together with the number of episodes taken for the ten-episode rolling mean of to fall to 2.0, and the range obtained from three random seeds at again sets the scale against which the differences are read.
| width | episodes to 2.0 | |||||
| 5 | 1.835 | 0.735 | 0.036 | 0.396 | 44 | |
| 10 | 1.865 | 0.580 | 0.067 | 0.449 | 77 | |
| 20 | 1.876 | 0.439 | 0.043 | 0.457 | 80 | |
| three seeds at | 1.834–2.111 | 0.568–0.687 | 0.012–0.067 | 0.403–0.449 | 38–77 |
Table B.1. Sensitivity of the trained policy to the number of wall segments. The final row gives the range obtained from three random seeds at .
There is no critical segment count within this range. The final Nusselt number spans 1.835 to 1.876 across a fourfold change in , against 1.834 to 2.111 from three seeds at fixed , and the number of episodes needed to reach falls within the 38 to 80 spanned by the same comparison. Coarsening the actuation to five segments, so that a single segment covers four fifths of an uncontrolled roll, still selects the coalesced state, and refining it to twenty brings no measurable gain. The decrease of with follows from measuring the difference between neighbouring segments on a finer discretisation rather than from a change in the wall profile, and reflects the actuation approaching a continuous field as the segmentation is refined, the regime in which the smoothness penalty rather than the segmentation sets the scale of the actuation.
Acknowledgements
G.M.C. acknowledges support from EPSRC UKRI. This work was supported by the Horizon Europe MSCA DN project with acronym SCALE, Agreement No. 101120014.
Code and data availability
The PyTorch solver, training scripts and post-processing utilities used to reproduce the results reported here are openly available at https://github.com/gmcavallazzi/RBC-GRU_MARL.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
CRediT author contribution statement
Giorgio Maria Cavallazzi: Conceptualization, Methodology, Software, Investigation, Formal analysis, Visualization, Writing – original draft. Miguel Pérez Cuadrado: Software, Data curation, Formal analysis, Validation, Writing – review & editing. Alfredo Pinelli: Conceptualization, Supervision, Writing – original draft, Writing – review & editing.
Declaration of generative AI and AI-assisted technologies in the writing process
During the preparation of this work the authors used Claude (Anthropic) in order to check grammar and spelling. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.
References
- [1]Bae, H.J., Koumoutsakos, P., 2022. Scientific multi-agent reinforcement learning for wall-models of turbulent flows. Nature Communications 13, 1443.DOI
- [2]Beintema, G., Corbetta, A., Biferale, L., Toschi, F., 2020. Controlling Rayleigh–Bénard convection via reinforcement learning. Journal of Turbulence 21, 585–605.
- [3]Belus, V., Rabault, J., Viquerat, J., Che, Z., Hachem, E., Reglade, U., 2019. Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film. AIP Advances 9, 125014.DOI
- [4]Beneitez, M., Cremades, A., Guastoni, L., Vinuesa, R., 2025. Improving turbulence control through explainable deep learning. arXiv preprint arXiv:2504.02354 .arxiv.org/abs/2504.02354
- [5]Brunton, S.L., Noack, B.R., Koumoutsakos, P., 2020. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics 52, 477–508.
- [6]Cavallazzi, G.M., Guastoni, L., Vinuesa, R., Pinelli, A., 2025. Deep reinforcement learning for the management of the wall regeneration cycle in wall-bounded turbulent flows. Flow, Turbulence and Combustion 115, 1291–1317.DOI
- [7]Chen, Q., Constante-Amores, C.R., 2025. Stabilizing Rayleigh–Bénard convection with reinforcement learning trained on a reduced-order model. arXiv preprint arXiv:2510.26705 .arxiv.org/abs/2510.26705
- [8]Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724–1734.arxiv.org/abs/1406.1078
- [9]Font, B., Alcántara-Ávila, F., Rabault, J., Vinuesa, R., Lehmkuhl, O., 2025. Deep reinforcement learning for active flow control in a turbulent separation bubble. Nature Communications 16, 1422.DOI
- [10]Fujimoto, S., van Hoof, H., Meger, D., 2018. Addressing function approximation error in actor-critic methods, in: Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 1587–1596.arxiv.org/abs/1802.09477
- [11]Garcia, X., Miró, A., Suárez, P., Alcántara-Ávila, F., Rabault, J., Font, B., Lehmkuhl, O., Vinuesa, R., 2025. Deep-reinforcement-learning-based separation control in a two-dimensional airfoil. International Journal of Heat and Fluid Flow 116, 109913.
- [12]Garnier, P., Viquerat, J., Rabault, J., Larcher, A., Kuhnle, A., Hachem, E., 2021. A review on deep reinforcement learning for fluid mechanics. Computers & Fluids 225, 104973.DOI
- [13]Guastoni, L., Rabault, J., Schlatter, P., Azizpour, H., Vinuesa, R., 2023. Deep reinforcement learning for turbulent drag reduction in channel flows. The European Physical Journal E 46, 27.
- [14]Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neural Computation 9, 1735–1780.
- [15]Holme, K., Rabault, J., Vinuesa, R., Mortensen, M., 2026. Timescale separation enables deep reinforcement learning control of rotating detonation engine mode transitions. arXiv preprint arXiv:2604.14398.arxiv.org/abs/2604.14398
- [16]Jeon, J., Rabault, J., Vasanth, J., Alcántara-Ávila, F., Baral, S., Vinuesa, R., 2025. Inductive biased-deep reinforcement learning methods for flow control: group-invariant and positional-encoding networks improve learning reproducibility and quality. Physics of Fluids 37, 077189.
- [17]Jia, W., Xu, H., 2025. State-augmented deep reinforcement learning for active flow control around an elliptical cylinder. International Journal of Heat and Fluid Flow 116, 109950.
- [18]Jiren, M., Yousif, M.Z., Song, J., Lim, H.C., 2025. A comprehensive review of control strategies for Rayleigh–Bénard convection from classical feedback methods to artificial intelligence-driven optimization. Physics of Fluids 37, 101304.
- [19]Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., Dabney, W., 2019. Recurrent experience replay in distributed reinforcement learning, in: International Conference on Learning Representations.
- [20]Kenjeres, S., Roovers, R., 2025. Modulation of the local mass and heat transfer of turbulent double-diffusive convection under stable thermal stratifications. International Journal of Heat and Fluid Flow 111, 109636.DOI
- [21]Kurz, M., Kaushik, R., Blind, M., Kopper, P., Schwarz, A., Rodach, F., Beck, A., 2025. Invariant control strategies for active flow control using graph neural networks. Computers and Fluids 303, 106854.
- [22]Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D., 2016. Continuous control with deep reinforcement learning. International Conference on Learning Representations (ICLR) .
- [23]Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I., 2017. Multi-agent actor-critic for mixed cooperative-competitive environments, in: Advances in Neural Information Processing Systems (NeurIPS), pp. 6382–6393.arxiv.org/abs/1706.02275
- [24]Markmann, T., Straat, M., Peitz, S., Hammer, B., 2025. Control of Rayleigh–Bénard convection: effectiveness of reinforcement learning in the turbulent regime. arXiv preprint arXiv:2504.12000 .arxiv.org/abs/2504.12000
- [25]Mysore, S., Mabsout, B., Mancuso, R., Saenko, K., 2021. Regularizing action policies for smooth control with reinforcement learning, in: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1810–1816.
- [26]Rabault, J., Kuchta, M., Jensen, A., Réglade, U., Cerardi, N., 2019. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. Journal of Fluid Mechanics 865, 281–302.
- [27]Sakievich, P.J., Peet, Y.T., Adrian, R.J., 2016. Large-scale thermal motions of turbulent Rayleigh–Bénard convection in a wide aspect-ratio cylindrical domain. International Journal of Heat and Fluid Flow 61, 183–196.DOI
- [28]Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M., 2014. Deterministic policy gradient algorithms, in: Proceedings of the 31st International Conference on Machine Learning (ICML), pp. 387–395.
- [29]Stern, M.E., 1960. The “salt-fountain” and thermohaline convection. Tellus 12, 172–175.DOI
- [30]Suárez, P., Alcántara-Ávila, F., Font, B., Lehmkuhl, O., Vinuésa, R., 2025a. Flow control of three-dimensional cylinders transitioning to turbulence via multi-agent reinforcement learning. Communications Engineering 4, 113.
- [31]Suárez, P., Alcántara-Ávila, F., Miro, A., Rabault, J., Font, B., Lehmkuhl, O., Vinuesa, R., 2025b. Active flow control for drag reduction through multi-agent reinforcement learning on a turbulent cylinder at \(Re_D = 3900\). Flow, Turbulence and Combustion 115, 3–27.
- [32]Turner, J.S., 1974. Double-diffusive phenomena. Annual Review of Fluid Mechanics 6, 37–56.
- [33]Van Kan, J.J., 1986. A second-order accurate pressure-correction scheme for viscous incompressible flow. SIAM journal on scientific and statistical computing 7, 870–891.
- [34]Vasanth, J., Rabault, J., Alcántara-Ávila, F., Mortensen, M., Vinuesa, R., 2025. Multi-agent reinforcement learning for the control of three-dimensional Rayleigh–Bénard convection. Flow, Turbulence and Combustion 115, 1319–1355.
- [35]Vignon, C., Rabault, J., Vasanth, J., Alcántara-Ávila, F., Mortensen, M., Vinuesa, R., 2023. Effective control of two-dimensional Rayleigh–Bénard convection: invariant multi-agent reinforcement learning is all you need. Physics of Fluids 35, 065146.
- [36]Vinuesa, R., Brunton, S.L., 2022. Enhancing computational fluid dynamics with machine learning. Nature Computational Science 2, 358–366.
- [37]Wang, Q., Yan, L., Hu, G., Chen, W., Rabault, J., Noack, B.R., 2024. Dynamic feature-based deep reinforcement learning for flow control of circular cylinder with sparse surface pressure sensing. Journal of Fluid Mechanics 988, A4.
- [38]Wang, Y.Z., He, X.J., Hua, Y., Chen, Z.H., Wu, W.T., Zhou, Z.F., 2023. Closed-loop forced heat convection control using deep reinforcement learning. International Journal of Heat and Mass Transfer 202, 123655.
- [39]Wang, Y.Z., Peng, J.Z., Aubry, N., Li, Y.B., Chen, Z.H., Wu, W.T., 2024. Control policy transfer of deep reinforcement learning based intelligent forced heat convection control. International Journal of Thermal Sciences 195, 108618.















