betaFeedback

WorksComputer Science

Energy-Efficient Reinforcement Learning for Large Language Models

Abstract

Reinforcement learning for language models alternates between generating responses and training the policy. These stages place different demands on GPUs, creating opportunities to reduce clock frequency with little loss in throughput. Selecting frequencies online is difficult because response lengths change and frequency trials themselves consume time and energy. We present EE-RL (energy-efficient RL), an online controller that selects separate frequencies for generation and training during RL execution. It compares candidate settings with a prediction of the default GPU governor on the same token workload, and keeps the most energy-efficient setting that meets a throughput threshold. The controller then monitors throughput and energy and adjusts frequencies as the workload changes. Fixed-frequency sweeps on H100 and GB200 show GPU energy savings of up to 28.2% per valid training token across synchronous and asynchronous execution, while largely preserving throughput. EE-RL turns this headroom into online savings, reducing GPU energy per valid training token by up to 16.9% at a modest throughput cost.

Introduction

Reinforcement learning (RL) post-training improves a language model by repeatedly generating responses, evaluating them, and updating the model [19] (Fu et al. 2025). This process can consume substantial energy. Post-training Olmo 3 32B as a reasoning model, for example, used 17× more datacenter energy than its instruction-tuned counterpart, largely due to rollout generation [14]. The growing electricity demand of AI data centers makes energy efficiency important for operating costs and infrastructure expansion [9]. Reducing energy consumption without substantially extending GPU allocation time can lower electricity costs while preserving resource availability. Because GPUs consume energy while computing, moving data, and waiting, improving energy efficiency requires identifying when higher clock frequencies accelerate post-training and when they merely increase power consumption.

An RL update includes response generation, log-probability computation, policy training, and weight transfer to the generation engine. These phases differ in arithmetic intensity [23] and therefore place different demands on the hardware. Autoregressive generation produces responses one token at a time and can be limited by memory bandwidth, while training processes batches of complete responses and typically uses more parallel computation. Weight transfers, long responses that leave the rest of a batch waiting [6], and dependencies

Preprint. between phases introduce more waiting. Default GPU management, which we call the governor, nevertheless runs all of this work near peak clock frequency, so memory-bound work and idle waiting can consume additional power without a corresponding throughput gain.

This mismatch creates headroom, an opportunity to lower GPU frequency, and with it power, while losing little end-to-end throughput relative to the governor. The benefit depends on whether lower power outweighs any increase in execution time. Figure 1 illustrates the opportunity by comparing power under the governor with a lower-frequency setting. Power is lower in all phases, while their durations remain similar. The available headroom varies with the workload and how generation and training are distributed across GPUs.

Figure 1. Phase-average GPU power in Qwen3-30B-A3B on 32 H100s under the governor and at fixed 1650/1350 MHz generation/training frequencies. The time axis omits inter-phase gaps, which the annotated changes include.

Finding an efficient setting in advance is expensive. It requires repeated RL runs that consume valuable GPU hours, and the result may change with the model, batch size, or hardware allocation. A setting chosen at startup can also become less suitable as the model produces longer or shorter responses [6, 20]. Selecting frequencies online avoids a separate search for every workload, but the controller must distinguish these changes in work from the effect of changing frequency. It must also save enough energy to offset the cost of testing candidate settings.

We present EE-RL (energy-efficient RL), a controller that selects separate generation and training frequencies while the RL workload runs. It measures GPU energy per valid training token and throughput in valid training tokens per second, rather than per step, because response lengths vary and steps do not represent a fixed amount of work. Valid training tokens are response tokens consumed by training that contribute to the loss. The controller tests frequency settings in short trials against the governor. It keeps the most energy-efficient setting within its throughput limit, then monitors energy and throughput to adjust frequencies as needed.

The paper makes three contributions:

  • Evidence of energy-saving headroom. Measurements on H100 and GB200 show where lower phase frequencies save energy and where longer execution offsets the power reduction.

  • Online per-phase frequency control. EE-RL accounts for changing token workloads and selects the most energy-efficient tested setting that satisfies its throughput bound relative to the governor.

  • Controller evaluation. Matched comparisons measure energy savings and throughput over the full measured workload in synchronous and asynchronous RL, including the cost of testing and changing frequencies.

Background: How RL Uses GPU Resources

Higher GPU clock frequencies can speed up compute-intensive work, but offer less benefit when performance is limited by memory movement or waiting. RL includes phases with different demands, and the GPU allocation also influences their execution time and compute demands. These differences create opportunities to lower frequencies—and energy consumption—with little loss in overall performance. The available headroom varies with the workload and how generation and training are distributed across GPUs.

Generation, training, and useful work

Our workloads use Group Relative Policy Optimization (GRPO), which generates several responses to each prompt and compares their rewards within the group to update the model [18]. We call this repeated process an update. Generation produces the responses, and training uses them to change the model’s parameters. The updated parameters are then made available to the generation engine.

Each update also includes log-probability computation and weight refit. Log-probability computation scores response tokens for the training objective, while weight refit transfers updated policy parameters to the generation engine. These stages can have different power demands, but their shorter durations make separate frequency settings less worthwhile. To limit clock-switching overhead, we set log-probability computation to use the training frequency, while weight refit runs under both. Energy measurements include all allocated GPUs, covering generation, training, weight transfer, and waiting.

How generation and training share GPUs

We consider three execution modes. In colocated synchronous execution, generation and training take turns on the same GPUs. Both contribute directly to the time required for an update, and frequency changes follow their phase boundaries.

In disaggregated synchronous execution, generation and training use separate GPU pools but remain synchronized at each update. Training waits for the generated batch, and the next generation round waits for updated policy weights. Pool sizes can differ, and the waiting pool continues to consume power.

In both synchronous modes, frequency changes preserve the intended update dependencies. Each training update consumes the completed batch before the next generation round uses the updated policy. Unlike asynchronous execution, changing a phase’s duration does not change which batch is scheduled for that update, so frequency changes should not affect the model’s final accuracy.

In asynchronous execution, generation produces responses while training consumes previously generated work. A faster pool may therefore tolerate a lower frequency without slowing overall post-training progress. Timing can also affect which responses are available when training samples a batch. We define a response’s policy age as the difference between the policy version used for generation and the version against which it is consumed for training. Bounding this age limits the staleness of consumed responses.

We evaluate two weight-update schedules. With drained updates, in-progress responses finish before new weights are loaded into the generation engine. With in-flight updates, generation pauses with responses still in progress, loads new weights, recomputes their KV caches, and resumes. The policy-age limits used by each schedule are reported with the results.

Power, time, and energy

The governor serves as our reference, with observed peak clocks of 1980 MHz on H100 and 2062 MHz on GB200. EE-RL requests GPU core frequencies while hardware manages voltage. Dynamic power scales approximately as Pdyn∝VddfP_{\mathrm{dyn}} \propto V_{\mathrm{dd}}^{f} [2]. Reducing frequency lowers switching activity and can also permit lower voltage. Energy savings depend on execution time as well as power. Slowing one phase can keep both active and waiting GPUs drawing power for longer. Over a measurement interval of duration TT, we define energy as E=∫0T∑g∈GPg(t) dtE = \int_{0}^{T} \sum_{g \in G} P_g(t)\,\mathrm{d}t, throughput as q=N/Tq = N/T, and energy per valid training token as e=E/Ne = E/N, where GG is the allocated GPU set, Pg(t)P_g(t) is the power of GPU gg, and NN counts valid training tokens. Energy includes every allocated GPU, whether active or waiting, but excludes CPU, network, and cooling energy.

Characterizing energy-saving headroom

We now quantify this headroom. We sweep reduced generation and training frequencies and compare their energy per valid training token and end-to-end throughput with the governor across workloads, GPU allocations, and execution modes.

Where power can be reduced

Figure 1 compares the same workload and seed under the governor and a lower-frequency setting. Power falls in every phase, most during log-probability computation and training, while phase durations remain close. This example illustrates a setting where lower power outweighs the added

ModelSmall (S)Medium (M)Large (L)
Qwen(32, 4, 128)(64, 8, 512)(128, 16, 2048)
Nemotron(16, 4, 64)(16, 8, 128)(32, 8, 256)
Sequence limitsQwen 4096; Nemotron 8192.
DeploymentGPUsSML
Qwen3-30B-A3B
Colocated sync.32, 64✓✓✓
Disaggregated sync.8/32, 16/32, 8/24✓✓✓
Async, drained16/32, 8/24✓✓
Async, in-flight16/32, 8/24✓✓✓
Nemotron-Super-49B
Colocated sync.64✓✓✓
Disaggregated sync.16/32✓✓—
16/64✓✓✓
Async, drained16/32, 16/64✓✓—
Async, in-flight16/32✓✓—
16/64✓✓✓
DeploymentGPUsSML
Qwen3-30B-A3B
Colocated sync.16✓✓✓
Disaggregated sync.8/16✓✓✓
12/24✓✓✓
Async, drained4/16—✓—
Async, in-flight4/16—✓—
Nemotron-Super-49B
Colocated sync.32✓✓✓

Table 1. GRPO headroom workloads and GPU configurations for updates 5–25. Tuples give prompts per update PP, responses per prompt KK, and training batch size BB. GPU splits denote generation/training allocations. Checkmarks indicate tested sizes and dashes indicate sizes not included.

execution time. The multi-seed sweeps below examine how consistently this trade-off holds across frequency pairs and workloads.

How the opportunity varies across workloads

We vary two frequency settings, referred to as generation and training frequencies. In colocated execution, these settings follow phase boundaries on shared GPUs. In disaggregated and asynchronous execution, each applies to its corresponding GPU pool. Each setting is compared with the governor using the same workload and GPU allocation size. Table 1 summarizes these configurations.

Table 2 reports representative settings for every execution mode, with energy and throughput changes relative to the governor.

Among the synchronous settings, colocated execution saves 5.08–11.53% energy per valid training token across both platforms, with throughput reductions below 2%. Disaggregated execution offers more headroom, because each pool waits while the other is busy and that waiting time can run at a lower clock. On H100, Qwen and Nemotron save 23.75–28.23% with at most 1.31% lower throughput. On GB200, disaggregated Qwen saves 14.31% and 13.27% on the small and medium workloads, with less than 1% lower throughput.

When generation overlaps training

Asynchronous execution also makes the relative completion times of the generation and training pools important. A pool that repeatedly finishes before the next dependency may tolerate a lower frequency without reducing end-to-end throughput because its additional computation time can replace time otherwise spent waiting for the other pool.

The asynchronous rows of Table 2 cover drained updates with policy age 1 and in-flight updates with policy age 2. The selected drained and in-flight settings save up to 21.85%

ModelGPUsWorkloadGen/train (MHz)Energy saving (%)Throughput change (%)
Qwen32 H100Large1800/15008.29−0.99
Qwen16 GB200Medium1875/102011.53−0.69
Nemotron32 GB200Medium1800/15005.08−1.79
Qwen8/32 H100Large1500/120028.23−0.36
Nemotron16/64 H100Medium1350/135026.14−1.16
Nemotron16/64 H100Small1350/150023.75−1.31
Qwen12/24 GB200Small1800/140014.31−0.44
Qwen12/24 GB200Medium1800/140013.27−0.47
ModelGPUsWorkloadGen/train (MHz)Energy saving (%)Throughput change (%)Δ\Delta training reward
Nemotron16/64 H100Medium1725/81021.85−1.83+0.0015
Nemotron16/64 H100Medium1830/120021.13−0.26−0.0038
Qwen8/24 H100Medium1830/81019.89−1.37−0.0006
Qwen4/16 GB200Medium1900/17007.84−1.90+0.0005
Nemotron16/64 H100Small1725/120021.69+1.48−0.0047
Nemotron16/64 H100Medium1725/120021.27−1.37+0.0052
Qwen8/24 H100Large1830/81019.91−0.78−0.0009
Qwen4/16 GB200Medium1900/14008.91−1.24+0.0015

Table 2. Representative headroom relative to the governor, with four matched seeds per row. Energy saving measures the reduction in all-GPU energy per valid training token, and throughput measures valid training tokens per second. Synchronous execution; Colocated; Disaggregated; Asynchronous execution; Drained updates (policy age 1); In-flight updates (policy age 2). Pool splits give generation/training GPU counts. Each row includes its study’s idle clock setting. All rows cover updates 5–25. Δ\Delta training reward is the signed difference in mean training reward from the governor and serves as a learning-behavior diagnostic.

and 21.69% energy per valid training token, respectively, with measured throughput reductions below 2%. Drained results also lower the clock of idle pools, which the governor does not do, so their savings combine active and idle clock control. For these rows, the table also reports training-reward differences as a learning-behavior diagnostic rather than evidence of held-out accuracy equivalence.

Across all modes, the most efficient pair depends on the workload and deployment, so a pair selected for one configuration should not be assumed suitable for another.

Using Headroom to Reduce Energy

The headroom measurements show that lower frequencies can save energy, but further reductions can eventually reverse the benefit. A useful frequency choice must balance lower power against longer execution, including the energy consumed by waiting GPUs.

Balancing power reduction against slower progress

When a phase is limited by memory movement or communication, additional core speed may do little to improve throughput. Reducing frequency can therefore lower power with little effect on completion time, consistent with the distinction between compute and bandwidth limits in the Roofline model ([23]). At sufficiently low frequencies, however, slower computation can extend the phase and reduce end-to-end throughput.

The energy cost follows directly from the metrics in § 2. With Pˉ\bar{P} denoting average power summed across the allocated GPUs,

e=EN=PˉTN=Pˉq.(1)e = \frac{E}{N} = \frac{\bar{P}T}{N} = \frac{\bar{P}}{q}. \tag*{(1)}

Energy per valid training token falls when power decreases proportionally more than throughput. A further frequency reduction becomes counterproductive when the additional execution time outweighs the power saving. The lowest available frequency is therefore not necessarily the most efficient choice.

Figure 2 separates throughput loss, power reduction, and the resulting energy change. For disaggregated Qwen, the lowest-frequency setting reduces power further but slows execution enough to give back much of the saving. In contrast, the asynchronous Nemotron results show substantial energy savings with little throughput loss. These examples illustrate why frequency selection must consider both power and end-to-end throughput.

Figure 2. Throughput, power, and energy-per-token changes relative to the governor for representative generation/training frequency pairs, as geometric means over four matched seeds.

Choosing a setting under a throughput limit

We express the performance requirement as an allowed throughput loss relative to the governor. Among tested settings that meet this requirement and save energy, we select the one with the lowest measured energy per valid training token. A setting that saves more energy but exceeds the throughput-loss allowance is not eligible, and the governor remains available if no tested setting qualifies. Table 3 lists the throughput-loss allowance.

SettingSync.DrainedIn-flight
Allowed loss2%2%2%
Calibration15 updates15 updates45 updates
Trial3 updates3 updates3, 6, or 12 updates
Confirmation6 fresh updates6 fresh updates1–2 trials
Search grid90 MHz50/100 MHz50/100 MHz
Feedback step45/90 MHz50 MHz—
Monitoring20 updates, every 1020 updates, every 101 governor per 2 candidate blocks
Idle clock345 MHz345 MHz—

Table 3. Controller configuration.

EE-RL: RUNTIME GPU FREQUENCY CONTROL

Choosing frequencies at runtime exploits energy-saving headroom without extensive profiling in advance. EE-RL tests lower generation and training clocks in short trials against the governor and selects the most energy-efficient setting that retains at least 98% of governor throughput. It then monitors energy and throughput to guide adjustments as the workload changes.

Two questions shape the design. The first is how to judge a trial fairly when the work changes from update to update (§ 5.1), and the second is how to search when the stages interact differently in each execution mode (§ 5.2 and § 5.3).

Comparing at equal work

An update with longer responses takes longer at any clock, so elapsed time alone would confuse a change in work with the effect of a frequency. EE-RL instead asks what the governor would have done with the same work. Each run begins with a calibration phase, in which the first updates run at governor clocks and the controller only observes. From governor updates only, starting with these, it fits

log⁡y^A=β0+∑kβkzk,(2)\log\hat{y}_A = \beta_0 + \sum_k \beta_k z_k, \tag*{(2)}

where y^A\hat{y}_A is a predicted governor time and each zkz_k is the standardized log⁡(1+x)\log(1+x) of a token count xx, which is the number of generated tokens GG, valid training tokens NN, or tokens in the longest response LL. Generation time uses GG and LL, since generation cannot finish before its longest response does. Training time uses NN. When the stages run in sequence, update time is their sum plus remaining work such as refit and reward. When they overlap, it is the longer of the two. Figure 3 tests this adjustment by predicting later governor updates from earlier ones. Token counts lower median prediction error in every measured configuration, most for H100 Nemotron. The gain is smallest for GB200 Qwen, particularly for energy.

Figure 3. Median absolute error in predicting governor throughput and energy for later updates, with and without token counts. Lower is better.

Choosing a frequency pair

In synchronous execution and with drained asynchronous updates, every update runs under a single clock setting, so a trial of a few updates can be judged directly against the governor prediction. The governor observations around each trial are added to the references, so the prediction draws on recent governor behavior. A candidate passes if it retains 98% of predicted throughput and lowers energy per valid training token (Algorithm 1). Search follows the execution schedule: synchronous execution searches both phase clocks together, while drained execution searches the pools sequentially (§ 6).

A setting that passes a short trial may become less suitable as the workload changes. In synchronous and drained execution, EE-RL checks a trailing window of 20 usable updates every 10 updates, comparing throughput and energy with the governor reference for the same work. A throughput shortfall prompts a clock increase, while sufficient margin permits a further reduction. Each clock change starts a fresh lookback window. Workload drift or repeated coverage failures trigger new governor measurements to refresh the reference.

When synchronous or drained execution places generation

Algorithm 1 Choosing GPU frequencies in synchronous and drained execution

Require: Supported frequencies, governor observations

  1. Fit the governor predictor on token counts and longest response

  2. for each search round do

  3. {Synchronous: both clocks. Drained: first pool, then second pool.}

  4. repeat

  5. Propose a candidate for this round

  6. Run it between governor observations

  7. Compare throughput and energy with the governor prediction

  8. until the round’s search ends

  9. Confirm the passing setting with the lowest energy per token on fresh observations

  10. If confirmation fails, try the next passing setting or continue the search

  11. Keep the confirmed setting fixed in later rounds

  12. end for

  13. Lock the confirmed setting

and training on separate GPU pools, one pool often waits for the other before its next phase can start. The controller drops the waiting pool to an idle clock until that phase begins. No computation runs in these gaps, so this saves energy that the governor spends on idle GPUs, without slowing the update.

Choosing frequencies with in-flight updates

With in-flight updates, generation continues across updates, pausing only for each weight refit. Requests started under one clock can finish under the next, so the in-flight controller, unlike the drained one, cannot fit generation time at a given clock. EE-RL therefore treats selection as an experiment. Each trial places a block of candidate updates between governor blocks and compares their throughput and energy at matched work. Block length is set from the noise measured during calibration. As with drained updates, pools are searched one at a time, beginning with the less busy one, and the best setting is confirmed before it is locked.

Because a single confirmation cannot rule out a small loss, the controller keeps one governor block for every two candidate blocks after locking, so the comparison with the governor continues.

Control across execution modes

A slower phase does not always mean a slower RL update. In synchronous execution, added computation time extends the update. With drained updates, it may replace time spent waiting for the other pool. In-flight responses can cross a clock change, making their cost hard to attribute to either setting. These dependencies determine how EE-RL measures trials and searches frequencies. Table 3 gives the configuration.

Synchronous execution

Generation and training run in sequence, so their added execution times share the same throughput allowance. EE-

@references♀♀♀♀♀♀

Allowed loss is relative to governor throughput. Asynchronous search uses 50 MHz spacing for the first pool and 100 MHz for the second. Synchronous feedback uses 45 MHz steps, increasing to 90 MHz above 4% throughput loss. GPUs support only discrete clock frequencies, so applied frequency steps can differ slightly from these nominal sizes. The idle clock applies only to separate GPU pools.

RL fits a governor predictor for each phase and adds the median remaining time, including refit and reward. During a trial, it measures each phase’s time and energy relative to its governor prediction at the observed token workload. Applying these ratios to the governor reference updates estimates the candidate’s throughput and energy over a common workload, reducing sensitivity to the few updates in a trial.

The phase timings identify which clock needs to rise when a pair is too slow. Search starts with both clocks reduced and raises the phase responsible for most of the added time. Once a pair passes, it explores lower settings for further energy savings.

Asynchronous execution with drained updates

Generation and training overlap but must rendezvous before each weight update, and clocks change at this rendezvous. The later arrival determines update time, so slower computation in the earlier pool may simply shorten its wait. EE-RL records both arrivals and searches the pool that usually arrives first, leaving the other on the governor. Its confirmed setting then provides the reference for lowering the second pool. The combined pair must meet the throughput requirement and use no more energy than the first setting alone, so a second lowered clock is kept only if it benefits the whole allocation.

Response-length variation can obscure this waiting interval. When only training is slowed, EE-RL applies the measured slowdown to the governor references and recomputes update time from their recorded arrivals. Holding generation fixed estimates how much training can slow before delaying the rendezvous. Other candidates use the whole-update governor prediction.

Asynchronous execution with in-flight updates

A response may begin under one clock and finish under another. Measuring the responses completed within an update would therefore mix work from different settings. The runtime instead counts the valid tokens consumed by training, changes clocks during the weight-refit pause, and leaves updates carrying requests from the preceding setting out of trial decisions. The longest consumed response remains a predictor feature because it improves held-out estimates when response lengths vary, even though it no longer determines an update boundary.

Neighboring updates also share work, so a short trial may contain less independent evidence than its length suggests. Calibration estimates this correlation and selects the shortest trial block meeting a 2% standard-error target for comparison with the governor. Each candidate block is bracketed by governor blocks. Without a drained rendezvous to measure waiting directly, search starts with the less busy pool, then tests the second with the first setting held. Governor blocks continue after locking to check the throughput requirement.

Runtime measurement

The energy accounting must include waiting GPUs: slowing one pool can extend the time the other remains powered. The runtime samples each GPU every 200 ms and sums interpolated NVML energy counters across the allocation ([15]). Acceptance uses whole-update time and energy, including waits and clock changes.

Evaluation

EE-RL saves energy in all eight controller comparisons after including selection and adaptation costs. We examine how the execution schedule determines the useful control target and why the savings after locking differ from those over the complete interval.

Experimental setup

We evaluate Qwen3-30B-A3B and Mistral-NeMo 12B on H100 in colocated and disaggregated synchronous execution, and with drained or in-flight asynchronous updates. Mistral-NeMo was not part of the headroom study, so it tests the controller on a model without a prior sweep. Nemotron-Super-49B’s longer updates left too few steps within the allocated runtime for controller evaluation.

Each comparison matches the seed, workload, and GPU topology, using separate physical allocations. Rates use summed valid training tokens, time, and all-GPU energy over the same updates, including search, confirmation, correction, waiting, and synchronization. Table 4 reports every configuration. RL workloads also vary from run to run, even with matched seeds, because sampling and the policy drift apart and response lengths change. We therefore also compare each controller run with a governor model fitted on two pooled governor runs of the same workload, and report these workload-normalized results in Appendix § 18.

Model / executionGPUs(P,K,B)(P, K, B)Token limitUpdatesEnergy/token saving (%)Throughput change (%)Accuracy change (pp)
Qwen, colocated sync.32 H10064/8/51240962–1077.00−1.25—
Qwen, disaggregated sync.8/32 H10064/8/51240962–9316.94−1.39—
Mistral, colocated sync.16 H10064/8/51240962–2654.14+1.62—
Mistral, disaggregated sync.8/16 H10064/8/51240962–3158.89−1.52—
Qwen, async drained8/24 H10032/4/12840962–17514.51−1.970.00
Mistral, async drained8/16 H10064/8/51240962–30013.56+1.46+0.90
Qwen, async in-flight16/32 H10032/4/12840962–3357.24−1.08+0.18
Mistral, async in-flight8/16 H10064/8/51240962–5606.52−1.61−0.14

Table 4. Controller performance relative to matched governor runs over the listed updates, including search and adaptation costs. Energy savings and throughput changes use valid training tokens. Accuracy change is the difference in final validation accuracy from the governor, in percentage points. Dashes indicate unreported accuracy. GPU splits give generation/training counts. Drained and in-flight runs use policy ages 1 and 2, respectively.

Savings survive online selection

Disaggregated synchronous Qwen achieves the largest saving: 16.9% with 1.4% lower throughput, compared with 7.0% energy saving on colocated. Mistral-NeMo shows the same ordering. Separate pools expose idle intervals in addition to headroom during computation, and the controller reduces clocks in both pools.

The Mistral-NeMo results show that useful settings can be found without a prior headroom sweep. Their savings still depend on the balance between the pools and their execution schedule.

Overlap makes training the useful control target

In all four asynchronous runs, the controller tests lower generation clocks but ultimately slows only training. These choices suggest that training can run at a lower frequency while generation prepares the next batch, whereas slowing generation does not yield an accepted energy–throughput trade-off. Arrival order or utilization determines which GPU pool is searched first, but the controller’s decisions depend on how each setting affects end-to-end progress.

In drained execution, this spare training capacity translates into energy savings of about 14% over the full interval and about 20% after locking. Slower training uses time that would otherwise be spent waiting at the rendezvous. Lowering clocks during idle periods adds to the savings for Qwen, but rarely triggers for Mistral-NeMo because its idle gaps are short. The two workloads thus achieve similar savings through different combinations of active and idle frequency control, with the benefit measured across the whole GPU allocation.

Selection and monitoring have different costs

Every run saves more energy after locking than before it (Table 5). Disaggregated synchronous Qwen saves 22.4% after locking, versus 16.9% over the complete interval. Figure 4 shows this selection process in a separate colocated run, where candidate trials alternate with governor observations before confirmation. Reporting only the after-lock interval would miss these selection costs.

Figure 4. Measured phase clocks in a separate 90-update colocated Qwen run on 32 H100s. Confirmation after update 53 selects different generation and training frequencies. This trace illustrates search and is separate from the scored runs.

After locking, monitoring both recovers throughput and finds further reductions. For disaggregated synchronous Mistral-NeMo, three clock increases are followed by lower throughput losses: 3.30% to 1.65%, 2.35% to 1.20%, and 2.70% to 0.34% in the next 20-update windows. These windows retain 9.06–9.66% energy savings in the controller’s work-normalized estimates. For drained Qwen, monitoring identifies sufficient throughput margin to lower the training clock from 1065 to 1005 MHz. Following confirmation, this setting achieves 19.50% energy savings with 0.56% throughput loss against the paired governor over updates 142–175.

In-flight execution also pays for measurement after selection. Its full-interval savings are roughly half those of drained execution. Noisier measurements delay locking, and about a third of after-lock updates still use the governor for continued comparison. In-flight runs also omit idle dropping. The different schedules and allocations prevent attributing the gap to any one factor, but the persistent governor blocks explain why a longer run alone would not recover all of the available saving. Longer stable runs can amortize a completed search, but not a fixed fraction of time spent rechecking the choice.

Model / executionIntervalUpdatesGen/train (MHz)Energy/token saving (%)Throughput change (%)
Qwen, colocated sync.Before lock2–77Varies6.59−1.13
Qwen, colocated sync.After lock78–1071830/14258.04−1.55
Qwen, disaggregated sync.Before lock2–61Varies14.02−1.27
Qwen, disaggregated sync.After lock62–931695/142522.44−1.60
Mistral, colocated sync.Before lock2–172Varies3.95−0.18
Mistral, colocated sync.After lock173–2651785/16954.56+4.99
Mistral, disaggregated sync.Before lock2–59Varies8.04−1.01
Mistral, disaggregated sync.After lock60–315Varies9.08−1.64
Qwen, async drained, 8/24Before lock2–89Varies9.91−2.96
Qwen, async drained, 8/24After lock90–175Gov/Varies19.23−0.94
Mistral, async drained, 8/16Before lock2–190Varies9.32−0.10
Mistral, async drained, 8/16After lock191–300Gov/118520.87+4.19
Qwen, async in-flight, 16/32Before lock2–257Varies6.45−0.89
Qwen, async in-flight, 16/32After lock258–335Gov/11259.82−1.70
Mistral, async in-flight, 8/16Before lock2–313Varies5.32−2.45
Mistral, async in-flight, 8/16After lock314–560Gov/14258.03−0.52

Table 5. Controller performance before and after locking, relative to the paired governor over the same updates. Energy savings are measured per valid training token. All runs use H100 GPUs. GPU splits give generation/training counts. Gov denotes native governor control, and Varies denotes changing frequency settings. After-lock intervals include subsequent adaptation and, for in-flight runs, governor comparison blocks.

Interpreting throughput and learning quality

All eight full-interval comparisons lose less than 2%2\% throughput, although two asynchronous runs exceed that loss before locking (Table 5). A locally accepted setting therefore does not guarantee a run-level bound that includes unsuccessful trials and corrections.

Workload differences also matter. Synchronous Mistral-NeMo controller runs consume more valid tokens, and drained Mistral-NeMo has shorter longest responses after locking. These changes contribute to throughput gains and energy-per-token savings, so they should not be read as clock-induced speedups. The asynchronous validation-accuracy changes in Table 4 are all below one percentage point in magnitude, but one matched seed per configuration

cannot establish equal learning quality.

Discussion and limitations

Controlling the final throughput cost. The difference between trial-level acceptance and complete-run throughput suggests managing performance as an accumulated budget. A controller could track estimated accumulated slowdown and use the remaining allowance to limit the duration of new trials. This would connect exploration cost to the job’s remaining horizon. Early in a run, more exploration may be economical. For production runs spanning thousands of updates, the longer horizon could justify a more extensive search: its upfront cost can be amortized over many subsequent updates, potentially allowing the controller to find more energy-efficient settings and achieve larger net savings. Near completion, retaining an accepted setting may contribute more than another search step.

Workload and allocation dependence. A frequency choice depends on token counts, response lengths, the work within each stage, and the GPU allocation. The stages themselves depend on the RL algorithm. We evaluate only GRPO, while methods that train a value model, such as PPO ([17]), add stages with their own frequency demands. Changes in pool size or parallelism can shift the balance between stages, and we also observed governor-performance differences across physical allocations. Resuming a checkpoint on a different allocation should therefore trigger a fresh frequency search. Our controller evaluation uses one matched seed per configuration. Repeated seeds and component ablations would quantify variability and separate the contributions of prediction, search, confirmation, and feedback. Replaying a fixed pair over the same horizon would directly compare adaptation with static control.

Learning quality. Across the four asynchronous controller runs, final validation accuracy differs from the paired governor by between −0.14 and +0.90 percentage points. These differences are small, but each comparison uses one matched seed. Asynchronous timing can change which responses training consumes and their policy ages. Repeated runs across seeds would better characterize the effect on learning quality, and comparisons at a common held-out quality target would test whether the energy savings persist when matching achieved quality.

Pool allocation and idle power. The benefit of frequency control depends on the balance between GPU pools. Slowing one pool can reduce its active power while extending the other pool’s wait. Idle control can reduce the energy consumed during that wait, but its additional benefit depends on the pool’s existing idle power. Whole-allocation energy captures both effects. Results that combine active-frequency control with idle dropping therefore measure both mechanisms together. They do not isolate the contribution of either mechanism.

Related Work

The balance between power and progress also arises in conventional model training and in serving, where prior systems adjust GPU power, frequency, or scheduling to improve efficiency. These efforts complement work on RL systems that improves resource use through placement, parallelism, and scheduling. Together, they provide the context for EE-RL’s approach to frequency control during RL post-training.

Energy-aware training. Zeus jointly tunes batch size and GPU power limit for recurring training jobs through online exploration [24]. EnvPipe and Perseus lower the frequency of computation off the critical path of pipeline-parallel training [3, 4], and kernel-level DVFS selects clock frequencies for individual kernels [21]. Zeus uses short iterative profiles to estimate power and throughput, while Perseus constructs frequency schedules over a static per-iteration computation graph. In RL post-training, generation and training respond differently to frequency. Response-length changes can alter execution time independently of frequency, and slowing one stage can extend the other stage’s waiting time. EE-RL therefore accounts for response-length changes when comparing settings, measures the energy of the whole allocation, including waiting GPUs, and includes exploration costs in its savings.

Energy-aware LLM serving. DynamoLLM reconfigures instance count, parallelism, and GPU frequency for pools of similar requests under latency targets [22], and throttLLM sets frequency from projected KV-cache usage and batch size [10]. With prefill and decode disaggregated onto separate GPU pools (Zhong et al., 2024), GreenLLM, VoltanaLLM, and DualScale set a separate frequency for each phase [12, 1] (Yu et al. 2025), and AFlex further separates attention and feed-forward frequencies (Hu et al., 2026). These systems optimize inference energy under serving latency constraints. EE-RL instead controls the coupled generation and training stages of RL post-training, targeting whole-allocation energy per valid training token under an end-to-end throughput allowance. Serving-engine optimizations such as iteration-level scheduling (Yu et al., 2022) and paged KV-cache management [11] are complementary. They improve scheduling and memory efficiency and can change the frequency settings that are beneficial.

RL post-training systems. HybridFlow supports flexible placement and execution of RL dataflows [19], ReaL reallocates parameters to adapt parallelization across the stages of an RL step [13], and HetRL schedules RL across heterogeneous GPUs (He et al., 2026). RollPacker mitigates long-tail rollouts in synchronous RL post-training [6]. AReaL decouples generation from training while controlling data staleness (Fu et al., 2025), and PipelineRL and Laminar also run the two concurrently [16, 20]. These systems primarily improve throughput or learning speed through placement, parallelism, and scheduling rather than GPU frequency control. Measurements of Olmo 3 32B Think attribute most of its Think-specific post-training energy to rollout generation [14]. EE-RL instead targets energy by controlling clocks within a given placement, using phase boundaries and pool balance to identify headroom. Changes to placement or parallelism can alter phase balance and therefore warrant re-evaluating the selected frequencies.

Conclusion

Generation and training expose different opportunities for GPU frequency reduction during RL post-training. Measurements of GRPO workloads on H100 and GB200 show that these opportunities can reduce GPU energy per valid training token while largely preserving governor throughput. EE-RL selects frequencies during execution, accounts for response-length variation, and revisits its choices as the workload changes. Phase-aware frequency control complements RL scheduling and placement optimizations by adapting GPU speed to the work being performed.

Acknowledgments

We thank Trivikram Krishnamurthy, Sridutt Bhalachandra, Terry Kong, and Bhargava Gopireddy for their support of this work.

AI Usage Disclosure

AI-based tools assisted with manuscript writing and editing, experimental and analysis scripts, and result consistency checks. The authors take responsibility for the final content and findings.

Appendix

This appendix expands the headroom, token-work prediction, and controller experiments with measurement details, the complete valid headroom inventory, and workload-normalized controller results.

Estimation and Selection

For seed ss and setting pp, form qs,p=∑N/∑Tq_{s,p} = \sum N / \sum T and es,p=∑E/∑Ne_{s,p} = \sum E / \sum N, where NN counts valid training tokens. For each matched seed let xs=log⁡(qs,p/qs,A)x_s = \log(q_{s,p}/q_{s,A}). The tables report

R^q=exp⁡(xˉ).\widehat{R}_q = \exp(\bar{x}).

The energy ratio uses the same construction. Energy saving is 100(1−Re)100(1-R_e), and throughput change is 100(Rq−1)100(R_q-1). Selection targets lower energy per token while retaining at least 98% of governor throughput.

Accounting for Token Work and Response Length

A fixed number of RL updates does not imply a fixed amount of work. Longer responses increase the tokens processed by generation and training, while a particularly long response can delay generation even when total token counts are similar. Figure A1 shows that work, time, and energy vary even under unchanged governor control. Without accounting for this variation, a frequency trial can appear faster or slower simply because its workload differs.

Figure A1. Workload variation under governor-only Nemotron-Super-49B execution on 64 H100s. Faint lines show individual updates, and solid lines show trailing five-update means. Values are normalized to their mean over the first ten measured updates. Elapsed time sums measured update durations.

EE-RL therefore predicts governor performance for the work observed during each candidate trial. The synchronous and drained references both use log-transformed generated and valid training token counts, together with the maximum response length, which governs when generation finishes. The in-flight reference uses the valid training tokens and longest response of the batch that training consumes. These models fit only governor observations, keeping candidate measurements from changing their own reference.

To evaluate token-count normalization, we use nine governor-only colocated runs: three seeds each for Qwen on 32 H100s, Nemotron on 64 H100s, and Qwen on 16 GB200s. Each forward test fits 15 past observations, predicts the next three, and advances by three, producing 88, 69, and 80 blocks, respectively. Compared with a constant reference, normalization reduces median absolute throughput prediction error from 1.57% to 0.59%, 5.28% to 0.31%, and 1.29% to 1.01%. Update-energy prediction errors also decrease.

ConfigurationBlocksThroughput error (%) ConstantThroughput error (%) Token-workEnergy error (%) ConstantEnergy error (%) Token-work
Qwen, 32 H100881.570.590.510.30
Nemotron, 64 H100695.280.313.670.56
Qwen, 16 GB200801.291.010.700.69

Table 1 (PDF p. 13).

A separate governor-only Mistral-NeMo 12B experiment tests the contribution of response length. Across 24 held-out three-update blocks, adding a longest-response proxy to the token-count model reduces update-time prediction RMS log error from 0.085 to 0.044. Work normalization improves the reference but does not eliminate the need for repeated measurements and continued monitoring.

Complete Synchronous Headroom Results

Note on GB200 measurements. We observed substantial run-to-run variation on the GB200 cluster. The GB200 results in the synchronous and asynchronous tables below, as well as the GB200 rows of Table 2, should be read with this variation in mind.

The tables contain 208 H100 and 85 GB200 non-governor comparisons, grouped by model and execution mode with hardware identified in each heading. Each uses a matched native-governor reference. Clock pairs are requested frequencies. Observed clocks may be slightly lower, and “Gov” leaves the phase governed. All synchronous comparisons aggregate four matched seeds. Row suffixes specify measured updates when they differ from updates 5–25.

Colocated synchronous: Qwen, 32 H100

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1200/1200-12.1020.914
Small1500/1500-4.849.564
Small1800/1800-0.80-0.134
Small1980/1980+0.39-0.704
Small1650/1200-4.8511.854
Small1650/1350-3.2410.034
Small1800/1200-2.279.974
Small1800/1350-2.547.454
Small1800/1500-1.684.694
Medium1200/1200-10.1724.054
Medium1500/1500-4.6710.884
Medium1800/1800-1.500.154
Medium1980/1980-1.21-1.234
Medium1650/1200-5.4314.424
Medium1650/1350-3.8912.484
Medium1800/1200-4.2812.004
Medium1800/1350-3.549.444
Medium1800/1500-2.285.504
Large1200/1200-5.8226.594
Large1500/1500-1.9512.744
Large1800/1800-0.140.254
Large1980/1980-0.10-0.344
Large1650/1200-3.1718.884
Large1650/1350-2.3115.604
Large1800/1200-2.9916.574
Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Large1800/1350-2.0313.514
Large1800/1500-0.998.294

Table 6.

Colocated synchronous: Qwen, 64 H100

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1200/1200-14.1618.264
Small1500/1500-5.578.524
Small1800/1800-1.84-0.514
Small1980/1980-1.84-1.714
Small1650/1200-6.5910.054
Small1650/1350-5.298.334
Small1800/1200-4.767.634
Small1800/1350-3.836.204
Small1800/1500-2.933.354
Medium1200/1200-11.4422.154
Medium1500/1500-4.5310.144
Medium1800/1800-1.390.064
Medium1980/1980-1.50-1.374
Medium1650/1200-5.2412.674
Medium1650/1350-4.1510.754
Medium1800/1200-4.479.964
Medium1800/1350-2.868.344
Medium1800/1500-2.344.374
Large1200/1200-7.1025.704
Large1500/1500-2.7412.154
Large1800/1800-0.360.624
Large1980/1980-0.48-0.734
Large1650/1200-3.4517.354
Large1650/1350-2.5014.314
Large1800/1200-3.0214.764
Large1800/1350-2.1611.714
Large1800/1500-1.437.134

Table 7.

Colocated synchronous: Qwen, 16 GB200

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1875/1020+1.6212.614
Small1875/930+1.5311.984
Small1950/1020+4.3512.624
Small1950/930+3.4312.024
SmallGov/1020+2.519.884
SmallGov/840+2.0210.404
SmallGov/885+4.6712.124
SmallGov/930-0.989.124
SmallGov/975-3.496.954
Medium1875/1020-0.6911.534
Medium1875/930+1.0512.824
Medium1950/1020-2.1910.364
Medium1950/930+3.1014.554
MediumGov/1020+11.4416.994
MediumGov/840+2.3813.714
MediumGov/885+0.0310.414
MediumGov/930+6.1914.644
MediumGov/975+0.8011.904
Large1875/1020-1.0813.444
Large1875/930+0.4614.464
Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Large1950/1020+0.6313.624
Large1950/930-7.658.784
LargeGov/1020-3.3010.034
LargeGov/840-3.9512.194
LargeGov/885-5.4212.054
LargeGov/930-9.006.654
LargeGov/975+1.3513.654

Table 8.

Colocated synchronous: Nemotron, 64 H100

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1200/1200-11.8721.334
Small1500/1500-3.268.324
Small1800/1800-0.410.034
Small1980/1980-1.80-1.414
Small1650/1200-4.387.514
Small1650/1350-3.646.404
Small1800/1200-3.644.544
Small1800/1350-2.033.754
Small1800/1500-1.121.464
SmallGov/1500-0.731.884
SmallGov/1350-2.523.194
Small1800/Gov-0.800.044
Small1650/Gov-2.983.034
SmallGov/1200-2.244.774
Small1800/1650-0.630.784
Medium1200/1200-10.9222.484
Medium1500/1500-3.828.974
Medium1800/1800-0.75-0.104
Medium1980/1980-0.77-0.514
Medium1650/1200-4.899.424
Medium1650/1350-4.047.934
Medium1800/1200-4.496.644
Medium1800/1350-2.215.004
Medium1800/1500-2.222.504
MediumGov/1500-1.802.094
MediumGov/1350-2.324.714
Medium1800/Gov-0.800.414
Medium1650/Gov-2.683.014
MediumGov/1200-3.276.364
Medium1800/1650-2.010.684
Large1200/1200-11.3823.024
Large1500/1500-2.648.724
Large1800/1800-0.88-0.444
Large1980/1980-0.95-1.024
Large1650/1200-6.9510.504
Large1650/1350-5.528.314
Large1800/1200-6.037.704
Large1800/1350-4.545.794
Large1800/1500-2.072.524
LargeGov/1500-2.112.374
LargeGov/1350-4.355.254
Large1800/Gov-1.16-0.194
Large1650/Gov-1.902.664
LargeGov/1200-5.377.674
Large1800/1650-1.520.174

Table 9.

Colocated synchronous: Nemotron, 32 GB200

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1200/1200-12.997.494
Small1500/1500-8.595.784
Small1800/1800-3.012.954
Small1980/1980-0.840.434
Small1650/1200-5.006.614
Small1650/1350-5.144.854
Small1800/1200-2.536.164
Small1800/1350-3.274.244
Small1800/1500-3.524.074
Medium1200/1200-13.617.524
Medium1500/1500-4.687.584
Medium1800/1800-1.672.714
Medium1980/1980+0.32-0.194
Medium1650/1200-6.295.814
Medium1650/1350-5.086.444
Medium1800/1200-4.944.734
Medium1800/1350-3.914.504
Medium1800/1500-1.795.084
Large1200/1200-14.527.324
Large1500/1500-6.956.524
Large1800/1800-3.993.344
Large1980/1980-1.33-0.274
Large1650/1200-8.687.004
Large1650/1350-6.475.274
Large1800/1200-8.803.304
Large1800/1350-6.793.884
Large1800/1500-8.862.764

Table 10.

Disaggregated synchronous: Qwen, H100 Gen8/Train32

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1980/1980-1.3611.254
Small1800/1500-1.5616.194
Small1650/1500-3.1217.404
Small1500/1500-4.8818.444
Small1500/1350-5.6821.594
Small1500/1200-5.9523.384
Small1200/1500-13.0918.574
Small1200/1350-12.5622.254
Small1200/1200-12.8724.164
Small810/810-34.628.194
Medium1980/1980-0.4411.764
Medium1800/1350-1.3621.274
Medium1650/1350-3.0922.144
Medium1500/1500-4.7819.534
Medium1500/1350-3.5323.724
Medium1500/1200-5.2425.444
Medium1200/1500-10.6320.794
Medium1200/1350-10.7124.244
Medium1200/1200-10.6126.804
Medium810/810-30.7115.104
Large1980/1980+0.5813.234
Large1800/1200-0.8224.734
Large1650/1200-1.4525.824
Large1500/1500-1.4121.744
Large1500/1350-2.7925.014
Large1500/1200-0.3628.234

Table 11.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Large1200/1500-4.0524.724
Large1200/1350-3.8428.174
Large1200/1200-4.4530.264

caption:

:::

Disaggregated synchronous: Qwen, H100 Gen16/Train32

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1980/1980-1.738.314
Small1800/1200-2.1618.114
Small1650/1200-5.0818.634
Small1500/1500-6.9415.254
Small1500/1350-6.6518.244
Small1500/1200-6.5420.554
Small1200/1500-14.3217.364
Small1200/1350-14.3620.504
Small1200/1200-14.3122.314
Small810/810-36.066.324
Medium1980/1980-0.749.004
Medium1800/1200-1.3220.554
Medium1650/1200-3.3121.884
Medium1500/1500-3.4618.224
Medium1500/1350-4.5821.104
Medium1500/1200-4.2923.714
Medium1200/1500-9.5720.734
Medium1200/1350-9.3224.414
Medium1200/1200-10.6526.124
Medium810/810-30.6114.964
Large1980/1980-0.008.994
Large1800/1200-1.8522.154
Large1650/1200-2.1524.054
Large1500/1500-2.9018.704
Large1500/1350-3.4822.684
Large1500/1200-3.3225.594
Large1200/1500-6.5021.664
Large1200/1350-6.8525.754
Large1200/1200-8.0227.694
Large810/810-24.6420.854

Table C.7.

Disaggregated synchronous: Qwen, H100 Gen8/Train24

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1800/1650-1.8412.174
Small1800/1500-1.8615.634
Small1650/1650-2.2614.334
Small1650/1500-3.5316.304
Medium1800/1500-1.2715.674
Medium1800/1350-1.1420.004
Medium1650/1500-2.8117.004
Medium1650/1350-3.5220.634
Large1800/1500-0.9616.384
Large1800/1425-2.0818.034
Large1800/1350-2.0620.164
Large1800/1200-3.1622.054
Large1650/1350-3.3621.144
Large1650/1200-2.2324.214

Table C.8.

Disaggregated synchronous: Qwen, GB200

Four matched seeds per comparison. Rows marked 5–15 cover updates 5–15 because at least one seed did not reach update 25 within the job’s time limit. All other rows cover updates 5–25.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
S 8/161800/1800-2.689.414
S 8/161400/1800-10.474.824
S 8/161800/1400-3.9111.674
S 8/161400/1400-9.418.284
M 8/161800/1800-0.6210.514
M 8/161400/1800-5.888.224
M 8/161800/1400+1.5516.264
M 8/161400/1400-4.2711.534
L 8/16; 5–151800/1800-3.989.184
L 8/16; 5–151400/1800-3.6712.814
L 8/16; 5–151800/1400-7.0610.254
L 8/16; 5–151900/1900+0.0213.144
S 12/241800/1800-1.1510.444
S 12/241400/1800-8.067.514
S 12/241800/1400-0.4414.314
S 12/241400/1400-8.5410.254
M 12/241800/1800-0.7310.624
M 12/241400/1800-4.899.614
M 12/241800/1400-0.4713.274
M 12/241400/1400-3.1814.674
L 12/241800/1800+1.4314.104
L 12/241400/1800+0.5314.964
L 12/241800/1400+1.4118.284
L 12/24; 5–151400/1400-1.4917.644

Table 12.

Disaggregated synchronous: Nemotron, H100 Gen16/Train64

Four matched seeds per comparison.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1350/1350-2.4426.294
Small1500/1500-1.4620.964
Small1350/1500-1.3123.754
Small1500/1350-2.0723.814
Small1500/1650-3.7417.204
Small1650/1500-3.1518.304
Medium1350/1350-1.1626.144
Medium1500/1500-2.5218.924
Medium1350/1500-1.7721.714
Medium1500/1350-2.4823.334
Medium1500/1650-0.0117.114
Medium1650/1500-1.8318.204
Large1350/1350-3.9523.944
Large1500/1500-1.9617.844
Large1350/1500-4.1018.514
Large1500/1350-4.4522.024
Large1500/1650-1.0214.204
Large1650/1500-2.4916.284

Table 13.

Disaggregated synchronous: Nemotron, H100 Gen16/Train32

Four matched seeds per comparison. Rows marked 5–15 cover updates 5–15 because at least one seed did not reach update 25 within the job’s time limit. All other rows cover updates 5–25.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
Small1350/1350-1.7324.984
Small1500/1500-2.7217.894
Small1350/1500-1.2021.534
Small1500/1350-3.2621.204
Small1500/1650-1.9015.504
Small1650/1500-2.6416.014
Medium1350/1350-4.2223.504
Medium1500/1500-3.3117.084
Medium1350/1500-1.2820.224
Medium1500/1350-2.2022.264
Medium1500/1650-2.6414.254
Medium1650/1500-1.4616.634
Large; 5–151350/1350-0.0625.204
Large; 5–151500/1500+1.1718.874
Large; 5–151350/1500+1.6620.574
Large; 5–151500/1350+0.9924.314
Large; 5–151500/1650+0.2414.364
Large; 5–151650/1500+1.0617.764

Table C.11.

Disaggregated synchronous: Nemotron, GB200

Four matched seeds per comparison. Rows marked 5–15 cover updates 5–15 because at least one seed did not reach update 25 within the job’s time limit.

Workload / splitGen/train MHzΔq\Delta q (%)Energy saving (%)nn
S 8/16; 5–151400/1400-4.9913.314
S 8/16; 5–151900/1800+1.4015.724
M 8/16; 5–151400/1400-0.8816.274
M 8/16; 5–151800/1800-0.739.424
M 8/16; 5–151400/1800+1.9816.084
M 8/16; 5–151800/1400-2.4613.664
M 8/16; 5–151800/1900+0.6410.554

Table C.12.

Asynchronous headroom and training reward

The tables report 67 comparisons over updates 5–25, each with four matched seeds and a native-governor reference over the same interval: 32 drained and 31 in-flight H100 comparisons, plus two Qwen comparisons of each mode on GB200. Drained and in-flight execution use policy ages 1 and 2, respectively. No shorter-window estimates or comparisons with fewer than four paired seeds are included. Incomplete or invalid comparisons are omitted, including large Nemotron in-flight at 16/32 H100s and large Qwen drained runs.

Energy saving is the reduction in whole-allocation GPU energy per valid training token, and Δq\Delta q is the change in valid training tokens per second. Splits give generation/training GPU counts, S/M/L denote workload sizes, and clock pairs give requested generation/training MHz. Each comparison retains its study’s idle-control policy.

Reward differences are candidate minus governor in absolute reward units. Training reward is a diagnostic, not final validation accuracy.

SplitSizeGen/train MHzEnergy saving (%)Δq\Delta q (%)Δ\Delta reward
8/24S1725/810+19.36-3.29-0.0046
8/24S1830/345+15.75-3.93-0.0023
8/24S1830/525+17.91-2.29-0.0017
8/24S1830/810+18.73-1.67+0.0015
SplitSizeGen/train MHzEnergy saving (%)Δq\Delta q (%)Δ\Delta reward
8/24S1830/1200+17.53-0.84+0.0007
8/24M1725/810+20.95-2.66+0.0009
8/24M1830/345+17.28-2.06+0.0011
8/24M1830/525+19.10-1.42-0.0001
8/24M1830/810+19.89-1.37-0.0006
8/24M1830/1200+19.12-0.85+0.0004
16/32S1725/810+17.40-2.88+0.0028
16/32S1830/345+14.39-2.64+0.0045
16/32S1830/525+15.69-2.13+0.0020
16/32S1830/810+16.30-1.51+0.0035
16/32S1830/1200+15.33-1.09+0.0007
16/32M1725/810+18.50-2.26-0.0003
16/32M1830/345+8.42-15.72-0.0010
16/32M1830/525+16.32-1.73-0.0003
16/32M1830/810+16.74-2.38-0.0003
16/32M1830/1200+15.60-2.12+0.0000

Table D1. Qwen, H100, asynchronous drained; updates 5–25. Four matched seeds per row.

SplitSizeGen/train MHzEnergy saving (%)Δq\Delta q (%)Δ\Delta reward
16/32S1725/810+19.07-1.68-0.0019
16/32S1830/810+17.83+0.72-0.0017
16/32S1830/1200+17.14+0.95+0.0000
16/32S1830/1500+9.11+0.14+0.0045
16/32M1830/1200+17.41-2.94-0.0025
16/64S1725/810+21.46-2.92+0.0000
16/64S1830/810+20.80-1.84-0.0007
16/64S1830/1200+20.16-2.12-0.0045
16/64S1830/1500+13.70-3.07+0.0028
16/64M1725/810+21.85-1.83+0.0015
16/64M1830/1200+21.13-0.26-0.0038
16/64M1830/1500+13.91+2.41-0.0025

Table D2. Nemotron, H100, asynchronous drained; updates 5–25. Four matched seeds per row.

SplitSizeGen/train MHzEnergy saving (%)Δq\Delta q (%)Δ\Delta reward
4/16M1900/1400+7.60-3.10+0.0006
4/16M1900/1700+7.84-1.90+0.0005

Table D3. Qwen, GB200, asynchronous drained; updates 5–25. Four matched seeds per row.

SplitSizeGen/train MHzEnergy saving (%)Δq\Delta q (%)Δ\Delta reward
8/24S1830/810+18.22-2.00+0.0060
8/24S1830/1200+17.06-0.80-0.0008
8/24M1725/1200+18.60-2.02+0.0013
8/24M1830/810+19.11-1.47-0.0007
8/24M1830/1200+17.68-0.89+0.0033
8/24L1725/1200+19.00-2.27-0.0002
8/24L1830/810+19.91-0.78-0.0009
8/24L1830/1200+17.93-0.93+0.0001
16/32S1725/1200+16.02-2.12+0.0112
16/32S1830/810+15.39-2.20+0.0086
16/32S1830/1200+14.49-0.96+0.0060
16/32M1725/1200+17.78-1.01+0.0009
16/32M1830/810+16.89-1.16+0.0014

Table D4. Qwen, H100, asynchronous in-flight; updates 5–25. Four matched seeds per row.

Qwen, H100, asynchronous in-flight; updates 5–25. (continued)

SplitSizeGen/train MHzEnergy saving (%)Δq\Delta q (%)Δ\Delta reward
16/32M1830/1200+16.07-0.24-0.0004
16/32L1725/1200+17.37-1.92-0.0011
16/32L1830/810+17.43-0.42-0.0003
16/32L1830/1200+16.36-0.09+0.0001

Table 14.

SplitSizeGen/train MHzEnergy saving (%)Δq\Delta q (%)Δ\Delta reward
16/32S1725/1200+17.67-1.78-0.0041
16/32S1830/810+10.05-13.26-0.0024
16/32S1830/1200+15.92-2.14-0.0058
16/32M1725/1200+18.12-4.05-0.0035
16/32M1830/810+13.00-15.68-0.0020
16/32M1830/1200+16.38-4.21+0.0012
16/64S1725/1200+21.69+1.48-0.0047
16/64S1830/810+18.64-5.18-0.0037
16/64M1725/1200+21.27-1.37+0.0052
16/64M1830/810+15.70-13.57-0.0093
16/64M1830/1200+20.83+0.11-0.0058
16/64L1725/1200+19.23-7.33+0.0009
16/64L1830/810+12.32-25.93-0.0009
16/64L1830/1200+18.52-7.35+0.0012

Table D5. Nemotron, H100, asynchronous in-flight; updates 5–25. Four matched seeds per row.

SplitSizeGen/train MHzEnergy saving (%)Δq\Delta q (%)Δ\Delta reward
4/16M1900/1400+8.91-1.24+0.0015
4/16M1900/1700+4.85+0.67+0.0004

Table D6. Qwen, GB200, asynchronous in-flight; updates 5–25. Four matched seeds per row.

Workload-Normalized Controller Results

Update time and energy depend on the work each update receives. Response lengths and token counts differ between runs even with the same code and seed, because sampling, node timing, and the policy itself drift apart. These differences often reach 2–3% of throughput, the same size as the clock effects we measure. A direct comparison with one governor run therefore cannot fully separate the cost of the clocks from a difference in work. We repeat each controller comparison against a governor model fitted to the work, using the controller run and two governor runs of the same configuration.

This model is fitted offline and is separate from the controller’s online predictor. For each model and execution mode, we pool every update from both governor runs, starting at update 2, and fit update time and GPU energy by ordinary least squares on each update’s workload features. Only governor updates enter the fit, so the model describes only what governor clocks do. The features follow what sets update time in each mode. In synchronous execution, generation is fitted on generated tokens GG and longest-response tokens LL, training on valid training tokens NN, and the remaining time is the governor median. With drained updates, Mistral-NeMo uses LL and GG and Qwen uses GG and NN, since nearly every Qwen update reaches the 4096-token limit and LL barely varies. With in-flight updates, the fit uses the valid training tokens and longest response of the batch each update consumes, together with the previous update’s valid training tokens, because in-flight work spills into the next update.

For each interval, a run’s ratio is its measured total divided by the total the model predicts for the same updates. A ratio of one means the run matched the governor on its own work. The governor reference is the mean of the two governor ratios. Throughput change is the governor ratio divided by the controller ratio, minus one, and energy saving is one minus the controller ratio divided by the governor ratio. Ratios of sums weight each update by its length, as the totals in Table 4 do. All runs are scored on the same updates, and search, transition, and clock-switch updates all count.

Pooling two governor runs widens the range of work the model covers, so it predicts the controller’s updates without extrapolating. It also reduces the weight of any one governor run, each of which has its own nodes and small slowdowns. Because one model scores both governors, the spread of their ratios shows how much variation remains after accounting for work. Over each full run, every governor lies within 0.7% of the pooled model in time.

RunWhole run: UpdatesWhole run: Energy savingWhole run: ThroughputAfter lock: UpdatesAfter lock: Energy savingAfter lock: Throughput
Qwen, colocated sync.2–1077.1%−1.17%78–1078.3%−1.21%
Qwen, disaggregated sync.2–9316.4%−1.67%62–9322.0%−1.71%
Mistral, colocated sync.2–2653.1%−0.42%173–2652.5%+0.08%
Mistral, disaggregated sync.2–3159.3%−1.65%60–3159.7%−1.53%
Qwen, async drained2–17513.9%−2.19%90–17518.8%−0.95%
Mistral, async drained2–30011.7%−1.84%191–30018.7%−1.02%
Qwen, async in-flight2–3357.5%−1.25%258–33510.3%−1.57%
Mistral, async in-flight2–5606.6%−1.18%314–5607.7%−0.60%

Table 5 (PDF p. 23).

Normalization keeps every energy saving within 2 points of Table 4, and it moves most throughput changes by less than half a point. The exceptions are the Mistral-NeMo runs with throughput gains in Table 4. After accounting for differences in work, both show small losses. Drained Qwen loses 2.19% over its full run, slightly beyond the 2% limit.

References

  1. [1]Basit, O., Liu, Y., Kong, Z. J., and Hu, Y. C. DualScale: Energy-efficient disaggregated LLM serving via phase-aware placement and DVFS. arXiv preprint arXiv:2602.18755, 2026. URL https://arxiv.org/abs/2602.18755.
  2. [2]Chandrakasan, A. P., Sheng, S., and Brodersen, R. W. Low-power CMOS digital design. IEEE Journal of Solid-State Circuits, 27(4):473–484, 1992. doi: 10.1109/4.126534.
  3. [3]Choi, S., Koo, I., Ahn, J., Jeon, M., and Kwon, Y. En- ergy Performance-preserving DNN training framework for saving energy. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pp. 851–864. USENIX Association, 2023. URL https://www.usenix.org/conference/atc23/presentation/choi.
  4. [4]Chung, J.-W., Gu, Y., Jang, I., Meng, L., Bansal, N., and Chowdhury, M. Reducing energy bloat in large model training. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (SOSP ’24), pp. 144–159, 2024. doi: 10.1145/3694715.3695970.
  5. [6]Gao, W., Zhao, Y., An, D., Wu, T., Cao, L., Xiong, S., Huang, J., Wang, W., Yang, S., Su, W., Wang, J., Qu, L., Zheng, B., and Wang, W. RollPacker: Taming long-tail rollouts for RL post-training with tail batching. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pp. 849–866. USENIX Association, 2026.
  6. [9]International Energy Agency. Energy and AI. Technical report, International Energy Agency, 2025. URL https://www.iea.org/reports/energy-and-ai.
  7. [10]Kakolyris, A. K., Masouros, D., Vavaroutsos, P., Xydis, S., and Soudris, D. throttLL’eM: Predictive GPU throttling for energy efficient LLM inference serving. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 1363–1378, 2025. doi: 10.1109/HPCA61900.2025.00103.
  8. [11]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023. doi: 10.1145/3600006.3613165.
  9. [12]Liu, Q., Huang, D., Zapater, M., and Atienza, D. Green-LLM: SLO-aware dynamic frequency scaling for energy-efficient LLM serving. In Proceedings of the 63rd ACM/IEEE Design Automation Conference (DAC ’26), 2026. arXiv:2508.16449.
  10. [13]Mei, Z., Fu, W., Li, K., Wang, G., Zhang, H., and Wu, Y. ReaL: Efficient RLHF training of large language models with parameter reallocation. In Proceedings of Machine Learning and Systems, volume 7, 2025.
  11. [14]Morrison, J., Smith, N. A., and Strubell, E. The hidden cost of thinking: Energy use and environmental impact of LMs beyond pretraining. arXiv preprint arXiv:2605.01158, 2026. URL https://arxiv.org/abs/2605.01158.
  12. [15]NVIDIA. NVML API Reference Guide: Device Queries (nvmlDeviceGetTotalEnergyConsumption), 2026. URL https://docs.nvidia.com/deploy/nvml-api/api/group__nvmlDeviceQueries.html. Accessed October 30, 2026.
  13. [16]Piché, A., Kamalloo, E., Pardinas, R., Chen, X., and Bahdanau, D. PipelineRL: Faster on-policy reinforcement learning for long sequence generation. Transactions on Machine Learning Research, 2026. arXiv:2509.19128.
  14. [17]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347.
  15. [18]Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeek-Math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300.
  16. [19]Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. HybridFlow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25), 2025. doi: 10.1145/3689031.3696075.
  17. [20]Sheng, G., Tong, Y., Wan, B., Zhang, W., Jia, C., Wu, X., Wu, Y., Li, X., Zhang, C., Peng, Y., Lin, H., Liu, X., and Wu, C. Lamin ar: A scalable asynchronous RL post-training framework. In Proceedings of the Twenty-First European Conference on Computer Systems (EuroSys ’26), 2026. arXiv:2510.12633.arxiv.org/abs/2510.12633
  18. [21]Spaan, J., Chen, K.-H., and Varbanescu, A.-L. How much energy is wasted in LLM operations? Evidence from kernel-level DVFS. In Proceedings of the 23rd ACM International Conference on Computing Frontiers (CF ’26), 2026. doi: 10.1145/3801487.3801822.DOI
  19. [22]Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., and Choukse, E. DynamicLoLM: Designing LLM inference clusters for performance and energy efficiency. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 1348–1362, 2025. doi: 10.1109/HPCA61900.2025.00102.
  20. [23]Williams, S., Waterman, A., and Patterson, D. Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009. doi: 10.1145/1498765.1498785.DOI
  21. [24]You, J., Chung, J.-W., and Chowdhury, M. Zeus: Understanding and optimizing GPU energy consumption of DNN training. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pp. 119–139. USENIX Association, 2023. URL https://www.usenix.org/conference/nsdi23/presentation/you.

Paper details

vlnc:W_rchw3pe4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI-based tools

View statement
Looking for related papers…
Contents