WorksEngineering
LLM-PowerTrace: A Dataset of Time-Resolved GPU Power Traces for AI Compute
Abstract
AI compute is becoming large and fast-changing electrical loads. Its impacts on data center infrastructure and power grids depend not only on total energy consumption, but also on how their power draw moves over sub-second to minute timescales, including peaks, ramps, and synchronized fluctuations across accelerators. Existing energy benchmarks for large language models (LLMs) report aggregate quantities such as joules per token and rarely release the underlying time-series measurements, leaving these power dynamics poorly characterized and difficult to model. We introduce LLM-PowerTrace, an open dataset of time-resolved, per-GPU power traces for LLM fine-tuning and inference, sampled every 100,ms on varying NVIDIA platforms paired with complete workload and system metadata. For fine-tuning, LLM-PowerTrace profiles open model families spanning a wide range of model sizes and fine-tuning methods, together with controlled sweeps over batch size, sequence length, precision, parallelism, and checkpointing. For inference, it covers variations in concurrency, GPU frequency, KV-cache and weight quantization, prompt and output lengths, and request-arrival patterns. LLM-PowerTrace profiles a set of open model families across a wide range of model sizes, and controlled sweeps over batch size, sequence length, precision, parallelism, and checkpointing for fine-tuning, and it sweeps concurrency, GPU frequency, KV cache and weight quantization, prompt and output lengths, and request arrival patterns for inference. Built upon power traces, we define power-grid-facing metrics as well as AI-compute power&energy scaling law for extrapolating beyond the measured settings. Together, LLM-PowerTrace provides a dataset foundation for modeling, forecasting, and managing the fast-changing electrical behavior of AI compute.
Introduction
AI data centers are among the fastest-growing sources of electricity demand [28, 13]. Beyond their substantial energy consumption, their electrical behavior is shaped by the tightly coordinated execution of AI workloads. Modern AI clusters can synchronously shift thousands of accelerators among compute-bound, memory-bound, communication-bound, and idle states, producing rapid and possibly structured variations in power demand. These dynamics are already becoming operationally significant, where large-scale training jobs can generate synchronized power swings that require mitigation through software, hardware, and energy-storage mechanisms [3], while LLM workloads can induce irregular power transients impacting power grids [16]. For instance, in July 2024, approximately 1,500 MW of data center load in Northern Virginia was abruptly disconnected almost simultaneously following a single transmission-system fault [20]. As power availability increasingly constrains the expansion and operation of AI data centers, understanding the magnitude, variability, and temporal structure of AI power demand is essential for both data center operation and energy systems planning. Existing characterizations of AI power and energy use remain predominantly energy-centric ([5, 21]). However, grid operators, data center owners, and AI servers are concerned not only about overall bulk energy or tokens per joule, but also about when power is drawn and how fast it changes. A workload with low total energy consumption can still create operational challenges through high power peaks, steep ramps, or unregulated fluctuations. Such power dynamics can impose additional operational requirements both on data center electrical infrastructure such as feeders, transformers, power distribution and backup units, as well as on power systems such as frequency regulation, ramping capability, and other balancing resources ([2]).
Yet the data needed to study AI compute’s power and energy dynamics remain limited. Benchmarks commonly report total energy consumption, energy per token, query, or run ([27, 17, 5, 9, 32]). LLM serving datasets capture request arrivals and workload statistics but not corresponding power measurements ([33]), while cluster telemetry datasets are not organized around controlled, configuration-labeled LLM workloads ([26]). As a result, researchers seeking to characterize ((Maliakel et al., 2026)), model and scale ([11]), forecast, or control AI power dynamics ([6]) lack open data that directly links workload configurations and execution events to high-resolution power trajectories.
To address this gap, we introduce LLM-POWERTRACE to connect time-resolved GPU power measurements with fine-tuning and inference workload characteristics, execution configurations, and hardware properties. It contains 1788 fine-tuning runs and 405 full serving runs on variations of NVIDIA GPU platforms and datasets, each with 100 ms per-GPU telemetry, the full run configuration, and timestamped execution events. Our contributions are threefold. First, LLM-POWERTRACE provides a systematic, configuration-labeled collection of high-resolution power traces for both LLM fine-tuning and inference, enabling characterization of rich power behaviors beyond aggregate or average energy metrics. Second, building on these traces, we define grid-relevant metrics and demonstrate both model-based analysis and empirical scaling laws for identifying key factors and extrapolating power and energy behaviors beyond the measured configurations. Third, we release the data, metric-computation code, and an interactive explorer, together with validation pipeline that supports reviewed community extensions of the dataset.
Related Work
Energy of LLM training and inference. As AI models’ parameters increase sharply over the past decade, AI training and inference’s energy and emission impacts are drawing attention ([31, 24]). For inference, Samsi et al. [27] benchmark LLaMA across GPU generations and sharding choices. Luccioni et al. [17] compare tasks and architectures, Fernandez et al. [9] study efficiency optimizations, ML.ENERGY automates inference’s energy measurement ([5]), and MLPerf Power unifies system-level reporting ([32]). Note these works mostly report aggregate energy with varying benchmarks and system-level control knobs; while LLM-POWERTRACE targets the power waveform and releases raw time series.
Grid impact of AI data centers. Industry reports have documented synchronized training power swings and their mitigation ([3, 15]); Li et al. [16] analyzes LLM-induced transients from a power systems perspective, and reliability organizations now treat large computational loads as a distinct and urgent reliability risk ([20]). Together this calls for large AI load modeling and configuration-resolved data.
Power management for LLM systems. System work identifies GPU frequency (via DVFS), power caps, KV cache, and instance scheduling to save energy in training [36, 4] and serving [22, 30, 29, 10], and separates or interleaves compute-bound prefill and memory-bound decode [23, 37, 1]. Traces behind these studies are usually isolated or unreleased; LLM-POWERTRACE recovers the same knobs with the power trajectory as one of the targets.
AI Workload and energy datasets. Serving traces such as BurstGPT [33] capture requests but not power, and the MIT Supercloud dataset [26] releases telemetry from heterogeneous user jobs rather than controlled LLM sweeps. BuildingsBench [8] showed how an open load dataset with standard tasks can mobilize ML research on building energy, which motivates us to adapt design principles for profiling AI power demands (Table 1).
Table 1. Comparison with related resources. “Power series”: time-resolved power traces are released. “Controlled”: runs form controlled configuration sweeps.
The LLM-PowerTrace Dataset
LLM-POWERTRACE records GPU power together with workload configuration and execution metadata. The main fine-tuning collection includes Gemma 1–4, Llama 2 and 3.x, Qwen2 and Qwen3, DeepSeek LLM, and Mistral/Mixtral. The collection includes both dense and mixture-of-experts (MoE) models, with full fine-tuning, LoRA [12], and QLoRA [7]. Both data parallelism (DP) and layer-wise model parallelism (MP) are involved. In DP, each worker keeps a model replica. In MP, consecutive model layers are placed across multiple GPUs. The main models, fine-tuning/inference configuration, hardware specifications are summarized in Table 4.
| L40S | A100 | H100 | H200 | |
| Architecture | Ada Lovelace | Ampere | Hopper | Hopper |
| Memory | 48 GB GDDR6 | 80 GB HBM2e | 80 GB HBM3 | 141 GB HBM3e |
| Memory bandwidth | 864 GB/s | 2.04 TB/s | 3.35 TB/s | 4.8 TB/s |
| BF16 Tensor Core | 362 TFLOPS | 312 TFLOPS | 989 TFLOPS | 989 TFLOPS |
| Max. power limit | 350 W | 400 W | 700 W | 700 W |
| Inter-GPU link | PCIe (no NVLink) | NVLink 600 GB/s | NVLink 900 GB/s | NVLink 900 GB/s |
| Subsets | Train | Train, Infer | Train, Infer | Train, Infer |
Table 4. GPU platforms in LLM-PowerTrace.
Inference traces are collected with vLLM [14] on A100, H100, and H200 GPUs, serving dense Qwen3 (0.6B–32B) and Llama-3.1 (8B, 70B) models and the MoE Qwen3-30B-A3B at tensor-parallel degrees 1–4. Each sweep varies one factor at a time: maximum concurrency, SM clock, KV-cache and weight quantization, input/output length, and the arrival pattern. Arrival patterns are among closed-loop (a fixed number of requests in flight), Poisson, or replayed BurstGPT traffic. Runs also include a request timeline aligned to the power trace. The main configurations are summarized in Table 2, and the full sweep values are listed in Appendix § 8.
| Dimension | Fine-tuning subset | Inference subset |
| Observation unit | A fine-tuning execution with per-GPU traces and aggregate GPU power. | A serving execution with device telemetry and available request records. |
| GPU deployment | L40S, A100, H100, H200; four homogeneous GPUs per main-collection run. | A100, H100, H200; TP 1, 2, and 4 in designated configurations. |
| Models | Gemma, Llama, Qwen, DeepSeek, Mistral/Mixtral; dense and MoE models. Model inventory in Table 5. | Qwen3 dense models (0.6B–32B), Qwen3-30B-A3B MoE, and Llama-3.1-8B/70B. |
| Execution stack | Transformers and Accelerate; full fine-tuning, LoRA, QLoRA; DP or layer-wise MP. | vLLM 0.28.0; version 0.21.0 for the concurrency sweep. |
| Input workload | Alpaca instruction data, up to 52K examples, with common preprocessing in the main collection. | Controlled input/output lengths; chat, code, and math prompts; Poisson arrivals and BurstGPT replay. |
| Workload sweeps | Micro-batch {1, 2, 4, 8, 16, 32}; sequence length {128, 256, 512, 1024, 2048}; model size and type. | Concurrency {1, 2, 4, . . . , 256}; input/output lengths 128/128, 512/512, 2048/128, 512/2048, 2048/2048; thinking on/off in free generation. |
| Execution sweeps | Accumulation and checkpoint interval {1, 2, 4, 8, 16}; DP/MP and GPU platform. Main save intervals are in optimizer steps. | TP degree; H100 SM clocks {345, 600, 900, 1200, 1605, 1980} MHz; Poisson rates {0.25, 0.5, 1, 2, 4, 8} requests/s. |
| Precision and quantization | Recorded bf16, fp16, and no settings; QLoRA uses 4-bit NF4 and double quantization in the main implementation. | KV cache: BF16 or FP8 (E4M3). Weights: BF16, FP8, or INT4 (GPTQ). |
| Power measurement | Nominal 100-ms nvidia-smi polling; recorded timestamps, device power, SM clock, and available utilization, memory, and temperature fields. | Nominal 100-ms DCGM polling; device power (field 155), cumulative energy, clocks, thermal/throttle and activity counters (no activity counters in the concurrency sweep or on A100). |
Table 2. Overview of the fine-tuning and inference data in LLM-PowerTrace.
Hosting and Maintenance. The datasets together with an interactive explorer that filters by model family, GPU, workload type, and precision are available in our online website repository. The explorer also accepts community traces in the same format: submissions are validated automatically, stay private until reviewed, and are published only with the contributor’s explicit consent. The complete dataset will be released upon acceptance. The code used to generate the dataset is open sourced4.
Notes: DP-data parallelism; MP-layer-wise model parallelism; TP-tensor parallelism. Details of the inference patterns and designated multi-GPU configurations are in Appendix § 8. The inference column summarizes the main vLLM collection; the SGLang prefill–decode runs used in § 5.3 form a separate calibration collection.
Metrics and Empirical Characterization
Grid-facing Metrics
Let denote the power of GPU , and let denote the aggregate GPU power of the participating GPUs. We use to denote either a per-GPU or aggregate GPU power trace over an interval . Let denote the power at sample time . We characterize power dynamics using the following metrics. Energy: and energy per token , where is the number of training tokens for fine-tuning or generated tokens for inference over . We also report energy per iteration for fine-tuning and per request for inference. Power level: mean and quantiles , , where denotes the empirical -quantile operator. Upper quantiles capture peaks while resisting single outliers. Ramps: with , where is the time lag, we report the 99th-percentile absolute ramp rate at . Regularity: for the mean-centered power sequence , we compute the FFT power spectrum , where denotes the discrete Fourier transform, exclude the DC component, and normalize the retained nonzero-frequency bins as . The normalized spectral entropy is ; lower values indicate power concentrated at fewer frequencies and hence a more regular periodic pattern. Synchronization: for the per-device traces of one node, is the node’s ramp as a fraction of the summed per-device ramps; values near one indicate that device ramps coincide and add fully, lower values that they partly cancel.
Characteristics of Fine-tuning and Serving Power Traces
Fine-tuning power trace. In this work, we use fine-tuning workloads to represent the training-side workloads in our dataset. Fine-tuning starts from a pretrained model and retains the standard forward–backward–optimization structure of model training Narayanan et al. [19]; Hu et al. [12], while the set of parameters being updated depends on the fine-tuning method Hu et al. [12]; Dettmers et al. [7]. Detailed method-specific computation is discussed in Appendix § 7.§ 2. LLM fine-tuning is an iterative workload with a structured execution pattern. Each run consists of a startup phase followed by a fine-tuning loop (Figure 2a). The startup phase exhibits two relatively stable power plateaus, corresponding to process startup and runtime initialization. The fine-tuning loop then follows a nested periodic structure determined by gradient accumulation and checkpoint frequency. We refer to one optimizer step as one fine-tuning iteration. Each iteration consists of several gradient-accumulation micro-steps (a forward and backward pass on one micro-batch each), followed by gradient processing and the parameter update. During checkpoint saving, computation largely pauses and power drops to a plateau near the active-GPU baseline, producing a regular power swing at each checkpoint. Within each micro-step, the forward pass produces a short high-power interval, whereas the backward pass is longer and exhibits greater power variation. This difference reflects the additional computation required for backpropagation: for a typical linear layer, the forward pass requires one matrix multiplication to compute the output, while the backward pass requires two to compute the input and weight gradients.
Figure 2. Representative LLM-POWERTRACE examples for fine-tuning and inference. (a) Fine-tuning power trace for QLoRA fine-tuning of Llama-2-13B on four H100 GPUs with data parallelism. (b) Inference power trace for Qwen3-32B served by vLLM on one H100 GPU measured using field 157 on DCGM with 20ms polling.
Inference-serving power trace. LLM inference is also iterative, but its execution structure is organized around requests. Every run splits into startup and a serving loop (Figure 2b). Startup again shows two stable power plateaus, corresponding to process startup and runtime initialization, with brief bursts from graph compilation and CUDA-graph capture. The serving loop consists of repeated request cycles, each divided into prefill and decode. Prefill processes prompt tokens in parallel, producing a short compute-bound interval with high tensor-core activity and power. Decode generates one token per sequence per step while repeatedly accessing model weights and the growing KV cache, resulting in a longer memory-bound interval with lower tensor-core activity and power. Under closed-loop load, the requests in flight start together, reach their output length at similar times, and are replaced together, so power repeats with a batch-turnover period: the time between these replacements, and the inference counterpart of the fine-tuning iterations. With a fixed number of requests in flight, this period is close to the mean request latency, so it is set largely by the output length and by how fast each decode step runs. Unlike checkpoint saving in fine-tuning, inference cycles do not return power to the GPU idle level for continuous serving, so steady serving appears as repeated prefill–decode power swings rather than large periodic drops.
Factors Shaping Power Traces
Despite sharing the same hierarchical structure, the traces exhibit substantial differences in power level, time scale, waveform characteristics, and cross-GPU power distribution. Fine-tuning factors fall into three layers (Figure 1, Table 7): the workload (model, method, batch, sequence length, precision) determines the computation and data movement per iteration. The execution configuration (parallelism, gradient accumulation, checkpointing) determines how computation, communication, and I/O events are ordered, synchronized, and repeated. Finally, the hardware (compute throughput, data-transfer capability, power envelope) determines the dynamic response characteristics of the power trace.

Figure 1. Schematic of power and energy data collection based on fine-tuning and inference profiling. To characterize power consumption, we identify levers including workload and execution configuration both determine hardware-dependent resource activities ; the hardware configuration and measurement process produce the observed time series trace , which supports scaling law analysis of LLM’s power demand.
| Layer | Factor | Primary trace manifestation |
| Fine-tuning method | Overall power profile and phase characteristics | |
| Model scale | Power level and iteration duration | |
| Workload | Model architecture | Waveform morphology and temporal pattern |
| Batch size | Micro-step duration and power level | |
| Sequence length | Forward/backward duration | |
| Numerical precision | Forward/backward power contrast and duration | |
| Execution | Parallelism strategy | Inter-GPU power distribution; communication-related variation |
| Gradient accumulation | Number of micro-steps per optimizer step | |
| Checkpoint interval | Spacing of checkpoint-saving segments | |
| Hardware | GPU platform | Power range and execution timescale |
Table 7. Primary effects of the investigated factors on fine-tuning power traces.
Inference factors fall into the same three layers (Table 10): the workload (model, concurrency, input and output length, weight and KV-cache precision, arrival process) determines the computation and data movement per decode step. The execution configuration (tensor parallelism) determines how computation and communication are split and synchronized across GPUs. Finally, the hardware (GPU platform, SM clock) determines the dynamic response characteristics of the power trace.
| Layer | Factor | Primary trace manifestation |
| Workload | Arrival process | Idle–busy switching; ramp magnitude |
| Workload | Concurrency limit | Power level until saturation; ramps peak mid-load |
| Workload | Model scale | Power level and batch-turnover period |
| Workload | Model architecture | Power level and cross-GPU balance |
| Workload | Input length | Prefill share and proximity to the power limit |
| Workload | Output length | Spacing of batch-turnover events |
| Workload | Weight precision | Execution speed (turnover period, energy per token) |
| Workload | KV-cache precision | Small power reduction; little change in waveform |
| Execution | Tensor parallelism | Inter-GPU distribution; node power; turnover period |
| Hardware | GPU platform | Power range and execution timescale |
| Hardware | SM clock | Power level, energy per token, and ramp magnitude |
Table 10. Primary effects of the investigated factors on inference power traces.
Figure 3 reveals several consistent patterns for LLM fine-tuning. We treat the fine-tuning method as a workload characteristic because it changes the set of trainable parameters and the associated gradient and optimizer states. First, increasing model size, batch size, or sequence length generally increases power, but the increase becomes smaller as power approaches a hardware-dependent envelope. Beyond this point, additional workload is reflected more in longer execution time than in higher power. This limited power headroom also helps explain why ramping does not necessarily increase monotonically with workload size. Second, different workload configurations affect energy efficiency in distinct ways. Larger batch sizes reduce energy per token by amortizing fixed overheads, while MoE models require less effective computation than dense counterparts. Despite lower memory footprint of QLoRA, its fine-tuning is slower and consumes more energy than Full and LoRA. Finally, hardware changes both magnitude and temporal structure of power traces. Compared with earlier GPU models, H100 and H200 achieve lower energy per token and shorter, more regular fine-tuning iterations, but operate at higher power levels with steeper ramps. See more comparisons in Appendix § 9.§ 1.
Figure 3. Effects of workload, fine-tuning configuration, and hardware on LLM fine-tuning power. Energy and ramp metrics are shown on logarithmic scales.
Figure 4 shows several consistent patterns for LLM serving. First, heavier AI load, whether more requests in flight, longer prompts, or larger models, raises power only until the GPU reaches its power limit; beyond it, additional work appears as longer request latency rather than higher power. As in fine-tuning, this saturation compresses the power envelope, so ramps peak at intermediate concurrency and are smallest for long prompts that hold the GPU at its limit. Second, serving configurations change energy efficiency in distinct ways. Batching more requests amortizes each decode step and cuts energy per output token by 17–72× from 1 to 256 requests in flight; FP8 and INT4 weights cut it by a median 30% and 39% by shortening each step, whereas an FP8 KV cache saves only about 3%. Third, the arrival process shapes the waveform more than any other factor: sparse or bursty arrivals let the GPU drain to idle between requests, widening the power band from about 20 to 1,100 W and raising the 99th-percentile ramp about twentyfold, and because tensor-parallel GPUs ramp together, their ramps add at the node. Finally, hardware sets the power range and the timescale. Newer GPU models lower energy per token and turn batches over faster and, unlike in fine-tuning, also run smoother; on a single GPU, the SM clock that minimizes energy per token (1200 MHz) is not the one that minimizes ramps. Detailed comparisons are provided in Appendices § 10.§ 2 and § 10.§ 3.
Figure 4. Effects of main serving knobs on energy per output token, aggregate node GPU power, and the 99th-percentile 1-s ramp for Qwen3-8B. Each shaded band changes one knob from the default serving configuration (dashed line; H200, 32 requests in flight, BF16, TP 1); the SM-clock band is on H100, and the concurrency band comes from the separate concurrency sweep (vLLM 0.21.0). Energy and ramp axes are logarithmic.
Models and Benchmark Results
To help identify the key levers shaping power and energy behaviors, as well as generalizing beyond measured configurations to unseen AI model sizes and fine-tuning/inference settings, we propose to demonstrate the utilization of LLM-POWERTRACE for both model-based analysis and empirical scaling laws. For fine-tuning, optimizer-step energy closely follows analytical FLOPs, while model size and sequence length increase computational work and larger micro-batches reduce energy per token by amortizing step-level overhead. For inference, node power follows a hardware-dependent saturation relationship driven by model size, concurrency, and context length, while request-arrival patterns have the strongest effect on short-term power ramps.
Fine-tuning Power and Energy Scaling Law
We first validate a FLOPs-based scaling law for fine-tuning energy, summarized here with additional details provided in Appendix § 7.§ 2. In the scaling-law notation below, the subscript “step” refers to the same iteration-level interval defined in § 4.1. The computational workload of an optimizer step can be characterized as , where , , and are the gradient accumulation, micro-batch size, and sequence length; and , , and are the active parameter count, layer count, and hidden dimension respectively; and is an architecture-dependent attention factor. We then relate the analytical computation to measured optimizer-step energy and normalize by the number of processed tokens (approximated by ), yielding a compact model characterizing the energy per token
where the coefficients absorb hardware, execution, and architecture-dependent factors. Architecture- and method-specific FLOP calculations, the derivation of the energy model, and additional validation details are provided in Appendix § 7.§ 2.
The measured data closely follow the predicted scaling forms, with fitted coefficients and goodness-of-fit statistics reported in Appendix § 7.§ 3. To evaluate generalization beyond the calibration data, we further perform level-held-out validation: each distinct batch size, sequence length, or model size is excluded in turn, and its energy is predicted using a model fitted to the remaining levels. Figure 5 compares these out-of-sample predictions with measurements. The close agreement indicates that the analytical structure Eq. (1) provides a compact surrogate for how fine-tuning energy scales with key workload dimensions, rather than simply interpolating individual measured configurations.

Figure 5. Out-of-sample validation of the fine-tuning-energy scaling law. Measured and leave-one-level-out predictions as batch size, sequence length, and model size vary. Model-size points are subsampled for visualization only; all samples are retained for fitting and evaluation.
Inference Power and Energy Scaling Law
We next demonstrate how the dataset can be used to fit an empirical scaling law for LLM inference, which is strongly related to input sequence length (ISL) and output sequence length (OSL). An inference server continuously processes the requests currently in flight. We therefore model instantaneous node power as a function of the number of active requests obtained from the serving log. The law has three kinds of inputs (Table 3). Discrete settings select constants: GPU config sets loaded idle power , power scale and the intercept ; the tensor-parallel (TP) degree multiplies the per-GPU terms; quantized weights shift the load by ; an FP8 KV cache scales the power by ; and a locked SM clock frequency scales it by . Three continuous inputs enter the load index linearly on a log scale: the active parameters per GPU (), the running requests () and the mean context held during decoding (). The identified nonlinear parts are a quadratic term in , the clock factor , and the logistic function , which maps the load index to the fraction of the power scale in use as the following relationship:
| Type | Input | Enters inference scaling law Eq. (2) as |
| Discrete | platform | (loaded idle power per GPU, measured), (power scale per GPU), (intercept) |
| TP | multiplies the per-GPU terms; divides in | |
| weight precision | shift in (FP8, INT4; 0 for BF16 and on A100) | |
| KV-cache precision | factor on (FP8; 1 for BF16) | |
| Linear in | model size | , in billions |
| running requests | , from the serving log | |
| request length | , realized tokens (mean context during decoding) | |
| Nonlinear | load | in |
| locked SM clock | on , ; when unlocked | |
| saturation | , fraction of in use; no incremental power when |
Table 3. Inputs of the inference scaling law (Eq. (2)). Logarithms are taken relative to the reference values shown. Fitted values are in Table 13.
Every workload factor moves the GPU along the same saturating curve: larger models per GPU, more running requests and longer requests push it toward the envelope, where further work changes power little. Note mean power, peak power and ramps can all be treated as statistics of the predicted trace, so the arrival pattern enters only through requests . The 13 continuous parameters are fitted jointly to measured mean power, with the precision terms fixed from paired runs. Fitted values, the evidence for each term and the limitations are in Appendix § 8.§ 1.
Beyond mean power, power and energy scaling law we propose is generalizable to modeling the ramp rates (Appendix § 8.§ 1). Figure 6 summarizes the model fit and validation. Across the 253 fitted runs the median error in mean node power is 2.7%: 1.7–4.2% for every sweep and GPU except tensor parallelism, which the law overpredicts (median error 5.2–7.9%). On sets never used to fit a coefficient or to choose a term it is 1.3–6.1% (Table 14). Over 27 BurstGPT replays, we are able to fit a short-term power load prediction, where the predicted 1-s trace strongly correlates with the measured one at a median of 0.83. Eq. (2) is fitted directly to measured traces, illustrating effective use of LLM-PowerTrace.

Figure 6. Fit of the inference scaling law (Eq. (2)). (a) Predicted versus measured mean node power for the 162 concurrency-sweep runs (six dense Qwen3 models on A100, H100, and H200), shaded by the maximum concurrency from 1 (light) to 256 (dark); median error 3.0%, 67% of runs within ±5%. (b) Predicted versus measured mean node power per run. Grey points are the 253 fitted runs; colored points were not used to fit the coefficients. The shaded band marks ±5%. (c) Measured and predicted total GPU power at 1-s resolution for a replayed bursty BurstGPT segment never used in fitting (Qwen3-32B, 2×H200, predicted from per-request log).
| Set | Median |err| (%) | ≤ 5% | Bias (%) | |
| Mean power, fitted runs | 253 | 2.7 | 68% | +0.1 |
| A100 FP8/INT4 weights (no term) | 8 | 4.3 | 62% | −2.9 |
| Free-length generation | 9 | 4.6 | 78% | −3.5 |
| Throttled clock (about 1830 MHz) | 3 | 1.3 | 67% | +1.6 |
| Llama-3.1-70B, closed loop | 7 | 6.0 | 43% | +10.4 |
| Poisson, 2-min segments, per-request log | 120 | 2.3 | 100% | – |
| Poisson, 2-min segments, 10-s count | 180 | 3.2 | 62% | – |
| BurstGPT, H200, mean power | 25 | 2.3 | 68% | +2.3 |
| BurstGPT, H100 locked clock, mean power | 23 | 6.1 | 39% | −2.2 |
| BurstGPT, Llama-3.1-70B (H200), mean power | 10 | 2.4 | 100% | +2.3 |
| BurstGPT, per second (per-request log) | 27 | 2.8 | – | – |
| BurstGPT, P95 power (per-request log) | 27 | 2.3 | 78% | – |
Table 14. Table 14: Accuracy of the frozen law: median absolute error, share within 5%, and bias (positive is overprediction). Only the first row was fitted, and the A100 runs informed Step 5; every other set was held out. “10-s count” is vLLM’s running count; “per-request log” rebuilds from request times on a 50-ms grid.
Model Calibration and Validation for Disaggregated-Serving Inference
Finally, we demonstrate how the dataset can be used to calibrate and validate the analytical model for inference serving under prefill–decode disaggregation (Yan et al., [34]), where each request is processed first by a pool of prefill instances and then by a pool of decode instances. The model maps workload characteristics, hardware limits, and the numbers of prefill and decode instances to serving capacity and total GPU power. For this evaluation, each coefficient set is fitted once from its designated traces and then applied to every deployment without deployment-specific adjustment. The traces were collected while serving Qwen3-32B with SGLang on H200 GPUs, with one GPU assigned to each prefill or decode instance, under a fixed-length workload and under the inference request length distributions of the Mooncake (Qin et al., [25]) and Azure (Patel et al., [23]) conversation traces. We report MAPE for serving capacity and total GPU power, with the model definition, the calibration protocol and the per-deployment values in Appendix B.2. Deployment-level comparisons of modeled and measured serving capacity and total GPU power are provided in Figure 7. This shows that the model-based characterization of power and serving capacity closely matches the measured data and generalizes across different inference request distributions and workload types.
Figure 7. Serving capacity and total GPU power for the evaluated prefill–decode deployments. Labels follow the form pd$, where p denotes prefill, d denotes decode, and and give the corresponding numbers of instances.
Conclusion and Discussion
In this work, we present LLM-PowerTrace, an open dataset and benchmark of time-resolved GPU power traces for LLM fine-tuning and inference, with grid-facing metrics, and power and energy scaling laws. Beyond aggregated energy and joules per token metrics, our profiled traces show that the power waveform of AI workloads exhibit structured relationships with choices of AI model, hardware, execution, and AI serving parameters. LLM-PowerTrace provides a novel and necessary connection of fine-grained AI compute power modeling with data center and energy system tasks. In the future work, we will maintain and update the dataset through the community pipeline, and add multi-node training, server- and facility-level measurements, practical serving policies (e.g., disabling or tuning chunked prefill, speculative decoding), and incoming generation of accelerators.
Fine-Tuning Subset Details
Hardware, models, and fine-tuning configurations
Hardware. The LLM fine-tuning power traces were collected on a university High Performance Computing (HPC) cluster. We selected NVIDIA L40S, A100, H100, and H200 to cover a broad range of accelerator generations and hardware capabilities, including differences in compute performance, memory capacity, and memory bandwidth. Detailed specifications of the GPU platforms evaluated are summarized in Table 4.
Notes: BF16 throughput is dense (no structured sparsity). The power limit is the configurable per-GPU limit, not the measured power. A100, H100, and H200 are SXM variants.
Logger. We use nvidia-smi for fine-tuning because it provides a common lightweight interface across the evaluated GPU platforms for collecting the per-GPU power, clock, and utilization fields required by our measurements. At each 100 ms poll the logger queries nvidia-smi for timestamp, index, power.draw.instant, clocks.sm, utilization.gpu, and utilization.memory. Node-level power is the sum of the participating GPUs’ power at each aligned timestamp. The observed median and 95th-percentile sampling intervals of every run are stored in its metadata. For serving runs, the logger queries DCGM every 100 ms for power (field 155), the cumulative energy counter (field 156), SM and memory clocks, temperatures, utilization, memory use, and throttle reasons, plus profiling counters for SM, tensor-core, and DRAM activity and PCIe/NVLink traffic on H100 and H200. We use DCGM for serving because it samples every GPU of a node together and exposes the energy counter and these activity counters, which nvidia-smi does not report. Field 155 is a smoothed average that attenuates sub-second peaks; future releases will log the instantaneous power (field 157), as in Figure 2b, to capture them.
Caveats. On-board power sensors on A100 and H100 sample only part of the runtime and may average over a window [35]. Sub-100 ms dynamics are therefore not resolved, and short spikes may be attenuated.
Models. Table 5 shows the LLMs that we characterize. Our selection represents some of the most representative open-source LLM families. Default configuration for the fine-tuning experiments is shown in Table 6.
| Family | Generation | Model sizes (B) |
| Main fine-tuning collection | ||
| Gemma | Gemma | 2, 7 |
| Gemma 2 | 2, 9, 27 | |
| Gemma 3 | 0.27, 1, 4, 12, 27 | |
| Gemma 4 | E2, E4, 26(4), 31 | |
| Llama | Llama 2 | 7, 13, 70 |
| Llama 3/3.1/3.3 | 8, 70 | |
| Qwen | Qwen2 | 0.5, 1.5, 7, 57(14), 72 |
| Qwen3 | 0.6, 1.7, 4, 8, 14, 32, 30(3), 235(22) | |
| DeepSeek | DeepSeek LLM | 7, 67 |
| DeepSeek-MoE | 16(2.8) | |
| Mistral/Mixtral | Mistral 7B | 7 |
| Mistral Small | 22, 24 | |
| Ministral | 8 | |
| Mistral 3 | 3, 8, 14 | |
| Mixtral | 8 × 7B: 47(13); 8 × 22B: 141(39) | |
| Additional fine-tuning runs | ||
| GPT-J | GPT-J | 6 |
| GPT-NeoX | GPT-NeoX | 20 |
| Pythia | Pythia | 6.9 |
Table 5. Models included in the main fine-tuning collection and additional fine-tuning runs. Notes: Model sizes are in billions of parameters. For MoE models, denotes B total and B activated parameters per token. Mixtral names give nominal expert sizes. Gemma 4 E2/E4 are effective-size labels for dense models with 5.1B/8B total parameters, as recorded in the manuscript inventory.
| Parameter | Default | Parameter | Default |
| Node | 1 node, 4 identical GPUs | Learning rate | |
| Framework | Transformers + Accelerate | Warmup | 100 optimizer steps |
| Dataset | Alpaca, ≤52K samples | Gradient clipping | max norm 1.0 |
| Sequence length | 1,024 tokens | Attention | eager |
| Micro-batch size | 8 per process/sharded run | LoRA rank/scale/drop. | 64 / 16 / 0.05 |
| Grad. accumulation | 4 micro-steps | QLoRA quantization | 4-bit NF4, double |
| Optimizer | AdamW, , wd 0 | Checkpoint interval | every 4 opt. steps |
| Precision | BF16 | Power sampling | 100 ms |
Table 6. Default configuration for the controlled LLM fine-tuning experiments.
Fine-tuning configuration and sweeps. All models in the main fine-tuning collection are fine-tuned on the same Alpaca subset using a common preprocessing pipeline. Fine-tuning is implemented with Hugging Face Transformers and Accelerate, with full fine-tuning, LoRA, and QLoRA, and with either data parallelism (DP) or layer-wise model parallelism (MP). Table 6 summarizes the default fine-tuning configuration, while Table 7 summarizes how the investigated workload, execution, and hardware factors manifest in the power trace. The controlled experiments further sweep model size and architecture, micro-batch size, sequence length, gradient accumulation, checkpoint interval, parallelism strategy, and GPU platform, with the main sweep ranges summarized in Table 2.
Fine-tuning Power and Energy Scaling Law derivations
FLOPs-based computational workload estimation
To obtain a hardware-independent representation of the computational workload, we analytically estimate the matrix-multiplication FLOPs of each optimizer step from the model architecture and fine-tuning configuration. We count one fused multiply–add (FMA) as two FLOPs. As shown in § 4.2, dominant power variation during fine-tuning comes from the forward and backward compute. We therefore use analytical GEMM FLOPs as the primary workload measure, treating normalization, activation, softmax, and quantization/dequantization as secondary contributions.
Let denote the microbatch size, the padded sequence length, the number of Transformer layers, the hidden dimension, the MLP intermediate dimension, the number of query heads, the number of key/value heads, the head dimension, and the vocabulary size. For a standard dense Transformer, we decompose the base-model forward computation as
where contains the attention projections, MLP layers, and LM head, while contains the two attention matrix multiplications.
For the attention projections, the query projection maps to , the key and value projections map to , and the output projection maps back to . Their total cost is
Because the query and key/value projections use and separately, the expression also covers grouped-query attention (GQA), where .
For a gated MLP with gate, up, and down projections, the MLP cost is
where for gated MLPs and 4 for conventional two-projection feed-forward blocks (GPT-J, GPT-NeoX, and Pythia). The final language-model head contributes
Therefore,
The attention-score and value-aggregation matrix multiplications are evaluated layer by layer. Let denote the effective attention span of layer . Their total cost is
where for full attention and for a sliding-window layer with window . When all layers use full attention,
Hence, the dominant workload contains both approximately linear and quadratic dependence on sequence length.
Fine-tuning method. For full fine-tuning, all base-model weights participate in gradient computation. We use
The backward approximation reflects the two matrix multiplications required to obtain input and weight gradients for a trainable linear operation.
For LoRA and QLoRA, the complete frozen backbone is still executed during the forward pass. Let denote the number of trainable adapter parameters. Since each trainable adapter parameter participates in one matrix-multiplication path per token, the adapter forward cost is
leading to the following
The backward cost is lower than for full fine-tuning because frozen base-model linear layers require the activation gradient but not a weight gradient, whereas the attention matrix multiplications and trainable adapter branches require both backward matrix-multiplication paths. For QLoRA, we use the same analytical GEMM graph as for LoRA and do not assign additional FLOPs to quantization or dequantization kernels.
The total computational workload of one microstep is then
For gradient-accumulation microsteps per optimizer step,
Optimizer updates and checkpoint I/O are excluded from this GEMM count and are modeled separately.
Architecture-specific extensions. For architectures that deviate from the standard dense Transformer, the same accounting is applied using the actual active matrix dimensions. In particular, MoE models count only the routed experts activated for each token, together with shared-expert and router computation. Grouped-query attention, sliding-window attention, KV sharing, and architecture-specific projections are likewise evaluated from their layer-specific dimensions.
For interpretation, the detailed architecture-aware workload can be summarized by the dominant scaling form
where is the active parameter count and absorbs architecture-dependent attention constants. The reported FLOP values use the detailed calculation above, while Eq. (17) provides a compact representation of the dominant workload scaling for parameter-level analysis and for characterizing the relationship between fine-tuning parameters and measured power and energy metrics.
Fine-tuning Scaling Law
Compute–energy scaling. We next relate the computational workload to measured fine-tuning energy of the same optimizer step. In general, the effective workload includes both the analytical computation and additional non-GEMM operations, which can be written conceptually as
where denotes memory-related and runtime work not represented by the analytical GEMM FLOP count. The latter includes memory-related and runtime operations and is particularly relevant to QLoRA because of its additional quantization/dequantization operations. Since GEMM computation dominates the training workload, we use as the primary workload proxy in the following analysis.
Let denote the aggregate GPU energy over an optimizer step, and let denote the corresponding aggregate analytical FLOPs across the participating GPUs (Appendix § 7.2.1). For a fixed hardware and execution realization , we observe an approximately linear relationship
where captures the empirical energy cost per analytical FLOP, while is a fitted energy offset that captures effects not explicitly represented by the analytical FLOP count, and denotes the residual error.
Energy-per-token scaling. Let be the number of fine-tuning tokens processed within the same optimizer step. Dividing Eq. (18) by and substituting the workload scaling in Eq. (17) gives
Thus, energy per token separates naturally into computational work per token and an amortized step-level overhead. For parameter-level analysis, the above relation can be reduced to
where the coefficients absorb the remaining architecture- and hardware-dependent factors.
Power–duration decomposition. Measured step energy satisfies , where and denote the mean aggregate GPU power and duration, respectively, over the same optimizer-step interval. Combining this identity with Eq. (18) yields
At low workload, § 4 shows that both and may increase with . The available pre-saturation measurements, however, are not sufficient to justify a universal parametric form for , so we characterize this regime empirically rather than imposing a specific saturation function.
Once mean power approaches the hardware-dependent sustained envelope , Eq. (21) reduces to
so additional computational work appears primarily as longer execution rather than higher power.
Empirical Fine-tuning Energy Scaling Law
Fitting. We calibrate Eq. (20) using controlled experiments under a fixed hardware and fine-tuning configuration (H200 with QLoRA). For each sweep, we retain only configurations available across all values of the varied parameter and aggregate repeated measurements at the configuration level. The resulting fits are summarized in Table 8 and Figure 8. As an independent check of the underlying assumption, checkpoint intervals are removed and optimizer-step energy is regressed against analytical FLOPs; the relation is approximately affine ().
| Relation | Fitted law | |||
| Compute–energy | 0.992 | |||
| Batch size | 0.997 | |||
| Sequence length | 1.000 | |||
| Model size | 0.940 |
Table 8. Calibration results for the energy scaling law for QLoRA fine-tuning on H200.
Figure 8. Calibration of the energy scaling law for QLoRA fine-tuning on H200. (a) Optimizer-step energy versus analytical FLOPs, excluding checkpoint intervals. (b–d) Energy per token as a function of batch size, sequence length, and model size. Lines show the fitted relations used to motivate Eq. (20).
Notes: denotes analytical FLOPs per optimizer step, expressed in FLOPs, and denotes total model size in billions of parameters, distinct from the active parameter count used in the analytical FLOP model.
Level-holdout validation. Prediction is evaluated without using the target parameter level during calibration. For each distinct value of , , or , all configurations at that value are removed, the corresponding law is refitted on the remaining levels, and the excluded level is predicted. This procedure tests whether the scaling structure transfers across parameter values rather than only fitting the observations used for calibration. Complete held-out results are shown in Figure 5.
Inference Subset Details
Table 9 lists the default serving configuration, which we use as the baseline setting. Each sweep in Table 2 changes one of these settings. Table 10 summarizes where each factor shows up in the power trace.
| Parameter | Default | Parameter | Default |
| Node | 1 node, 1 GPU (TP 1) | Request pattern | closed loop (b32) |
| Engine | vLLM 0.28.0 | Max Batch Size | 32 |
| Batching | continuous | Output length | 512 tokens (EOS ignored) |
| Chunked prefill | on | Maximum context | 2,048 tokens (closed loop) |
| Prefix caching | on | Sampling | temperature 0.7, top-p 0.9{sup}* |
| Weights / KV cache | BF16 / BF16 | Warm-up | min before the window |
| SM clock | unlocked (default boost) | Measurement window | 600 s (closed loop) |
| Power sampling | DCGM, 100 ms | Energy | hardware energy counter |
Table 9. Default configuration for the controlled inference experiments.
Request patterns. b32 is the closed-loop default of Table 9. In the concurrency sweep the client caps requests in flight at 1–256. In the other closed-loop experiments, the server’s maximum batch is 32 and the client keeps 32 in flight, so the server queue stays essentially empty. The concurrency sweep is closed loop with the client capping requests in flight at 1–256, short prompts (median 212 tokens), 512 max output tokens, and 20-minute runs on vLLM 0.21.0. {sup}*This sweep used the models’ default sampling (temperature 0.0, top-k 20, top-p 0.95) and drew prompts in a loop from a pool of 2,000, with a median prefix-cache hit rate of 6%; its DCGM capture has no activity counters. work and burst replay two 20-minute windows of the BurstGPT trace [33] at native speed, taking arrival times and prompt lengths from the trace and letting generation end naturally (at most 2,048 output tokens, maximum context 16,384): work carries steady traffic (6,171 requests, 5.1 req/s, inter-arrival coefficient of variation 2.1), and burst is sparser but spikier (1,921 requests, 1.6 req/s, CV 5.4). The replay sweep adds a window at the trace’s peak arrival rate (13,998 requests) and replays of work rescaled to 0.25, 0.5, 1, and 2 times its native rate. Poisson runs draw arrivals at a fixed rate with free-length generation. Open-loop windows are about 1,200 s at native rate (up to about 4,800 s for slowed replays), longer when a backlogged server drains after the trace ends, and use a maximum context of 16,384 tokens; the long input/output pairs use 4,096 or 8,192. Free generation uses chat (ShareGPT), code (InstructCoder), and math (competition mathematics: AIME 1983–2024 and the NuminaMath-CoT competition subset) prompts.
Coverage. The concurrency sweep covers the six dense Qwen3 models on all three GPUs (162 runs). The TP, weight-precision, and KV-cache sweeps cover Qwen3-8B, 14B, 32B, and 30B-A3B on all three GPUs, plus Llama-3.1-70B at TP 4 on all three GPUs in the TP sweep and at TP 2 on H200 in the precision sweeps (36, 39, and 26 runs). The input/output-length (30 runs), Poisson (21), and trace-replay (37, of which 35 are complete) sweeps run on H200. The SM-clock sweep (54 runs, 52 with complete artifacts, of which 15 did not hold their target clock) runs Qwen3-8B, Qwen3-30B-A3B, and Llama-3.1-8B on rented H100 GPUs, each model’s closed-loop sweep on a single GPU, under b32, work, and burst; clocks are locked with nvidia-smi --lock-gpu-clocks and the achieved clock is read from the trace. Of the 15 runs that missed their target, nine are at 1980 MHz, which throttles to a median of 1830 MHz under load, five at 1605 MHz, and one at 345 MHz.
Repeats and quality flags. No cell is repeated by design, but the closed-loop default appears on the weight-precision, KV-cache, and TP sheets, giving up to three runs of the same configuration per model and GPU; we use these to estimate run-to-run variation: for Qwen3-8B on H200 the three runs agree within 0.7% in energy per token and mean power, while their 99th-percentile ramps span 52–61 W/s (17%). Some rented H100 hosts ran with the GPU power limit lowered to 525–599 W, and the only two-A100 host (all four A100 TP 2 runs) held its application clock at 1155 MHz instead of 1410 MHz; these runs are released with a flag and excluded from the scaling-law fit and the pooled comparisons.
Empirical AI Serving Power and Energy Scaling Law
Overview. Eq. (2) was built one factor at a time. Steps 1–5 each add one factor, fitted on the sweep that varies it, with the parameters of earlier steps held fixed. Step 6 refits every continuous parameter jointly, Step 7 tests the remaining settings for a term, and Step 8 validates the frozen law on runs never used before. The selection rules were fixed before any fitting.
Fitting objective. Each run contributes one data point. Its load is sampled by vLLM’s running-request count, logged every 10 s ( snapshots , 60–484 per run). The predicted power above loaded idle is the mean of the law over these snapshots, with no excess power when . Parameters minimize the squared log error against measured mean power above idle:
is the mean of every power sample in the run’s measurement window, including any idle moments. is measured, not fitted: it is the median power between “server ready” and the first request (A100 72.9, H100 126.3, H200 120.9, and 77.8 W on the locked-clock H100 host).
Run roles. Every run’s role was declared before fitting. We exclude 21 runs on hosts whose GPU power limit was lowered (525–599 W on H100), the four runs on the 2×A100 host whose application clock was held at 1155 MHz, 15 runs that failed or lack a summary, and 3 open-loop thinking runs. The following sets were frozen until Step 8 and informed neither a coefficient nor the choice of a term: free-length generation, the three locked-clock runs that targeted 1980 MHz but throttled to about 1830 MHz, every Llama-3.1-70B run, and every Poisson and BurstGPT run.
Selection rules. Held-out error is the RMS of predicted minus measured on held-out runs. A candidate’s primary score is its mean held-out error over the tests every candidate can be scored on, and the lowest score wins, unless a candidate with fewer parameters lies within of it (the run-to-run spread of repeated baseline runs on one host). A categorical term (precision, thinking mode, architecture) is kept only if it lowers held-out error and a paired -test on the held-out absolute errors gives . A kept term goes inside (a shift along the curve) or outside (a factor on ), whichever has the lower held-out error.
Derivation. Table 11 lists every candidate and its held-out scores. Step 1 fits the 162 concurrency runs: six dense Qwen3 models at 1–256 requests on A100, H100 and H200. A power law in model size fails, because its size slope falls toward zero as every model approaches the power limit. The saturating form , which reduces to the power law at light load, wins with a quadratic load term. Step 2 fits 25 tensor-parallel runs, refitting only . It keeps . Each GPU behaves like one holding a model times smaller. This is the weakest choice, since a free exponent is steeper (). Step 3 fits 15 runs on the locked-clock H100 host and places the clock as a curved factor on the power scale. Step 4 fits 15 H200 runs at five forced ISL/OSL pairs. The mean context held during decoding, ISL OSL/2, ties with total length, and the lower score is taken. Step 5 estimates precision from BF16/quantized pairs on the same host (Table 12). On Hopper, quantized weights act inside , because their saving shrinks near the ceiling as a lighter workload’s would. An FP8 KV cache is a flat factor on . A100 behaves differently, because it has no FP8 tensor cores: vLLM serves FP8 weights there weight-only through the Marlin kernel and converts them to 16-bit inside every matrix multiply, while H200 runs them natively. INT4 weights also use a different kernel on each GPU (Marlin on A100, Machete on Hopper). On A100, quantized weights draw up to 15% more power than BF16, neither term passes, and they get no term. Step 6 refits the 13 continuous parameters jointly on 253 runs, with the precision terms fixed (Table 13). Every step-wise value lies inside its 95% bootstrap interval. Step 7 finds no term for thinking mode or MoE, because a term raised held-out error. Step 8 evaluates the frozen law on the held-out sets (Table 14). On BurstGPT, the predicted 1-s trace correlates with the measured one at a median of 0.83.
| Candidate | Form | T1 | T2 | T3 | T4 | Primary | |
| Step 1, saturation (162 runs). T1: leave one model out. T2: 1–64 128–256 requests. | |||||||
| Power law | 5 | 0.222 | 0.256 | 0.239 | |||
| Power law, quadratic load | 6 | 0.216 | 0.301 | 0.259 | |||
| Saturating | 8 | 0.164 | 0.228 | 0.196 | |||
| Saturating, quadratic load | 9 | 0.107 | 0.114 | 0.110 | |||
| Floor plus rise | , two logistics | 12 | 0.110 | 0.120 | 0.115 | ||
| Step 2, TP (25 runs). T1: TP 1–2 4. T2: TP 2–4 1. T3: leave one model out. | |||||||
| in | 3 | 0.155 | 0.107 | 0.170 | 0.170 | ||
| Free | in | 4 | 0.329 | 0.069 | 0.178 | 0.178 | |
| Free per GPU | in | 6 | – | 0.154 | 0.177 | 0.177 | |
| , TP step | in | 4 | 0.166 | – | 0.171 | 0.171 | |
| One scale per node | in , node scale | 4 | 0.853 | 0.176 | 0.543 | 0.543 | |
| Step 3, clock (15 runs). T1: leave one clock out. T2: leave one model out. T3: 600–1200 1605 MHz. | |||||||
| In , linear | in | 2 | 0.409 | 0.296 | 0.289 | 0.332 | |
| In , quadratic | in | 3 | 0.344 | 0.251 | 0.072 | 0.222 | |
| On , linear | 2 | 0.265 | 0.150 | 0.136 | 0.184 | ||
| On , quadratic | 3 | 0.051 | 0.047 | 0.072 | 0.057 | ||
| Step 4, length (15 runs). T1: leave one pair out. T2, T3: ISL, OSL . T4: leave one model out. | |||||||
| No length term | – | 3 | 0.074 | 0.109 | 0.043 | 0.117 | 0.086 |
| ISL and OSL | 5 | 0.086 | 0.123 | 0.086 | 0.035 | 0.083 | |
| Total length | 4 | 0.080 | 0.072 | 0.035 | 0.046 | 0.0583 | |
| Decode context | 4 | 0.100 | 0.063 | 0.026 | 0.042 | 0.0577 |
Table 11. Candidate forms and held-out error at each step.
| Level | Pairs | No term | Inside | On | Term kept | |
| KV cache FP8, all GPUs | 10 | 0.037 | 0.022 | 0.017 | 0.021 | |
| FP8 weights, H200 | 4 | 0.197 | 0.032 | 0.082 | 0.012 | |
| INT4 weights, H200 | 4 | 0.232 | 0.025 | 0.089 | 0.015 | |
| FP8 weights, A100 | 4 | 0.070 | 0.089 | 0.056 | 0.96 | none |
| INT4 weights, A100 | 4 | 0.097 | 0.074 | 0.068 | 0.60 | none |
Table 12. Precision terms from paired runs.
Table 13: Fitted values of the inference scaling law (Eq. (2); inputs in Table 3) from the joint fit on 253 runs, with 95% bootstrap intervals in brackets. is measured; the locked-clock host shares . TP enters through . shifts (0 on A100) and multiplies (1 for BF16); both come from paired runs and are held fixed. on the locked-clock host is weakly identified: there , and carries the clock effect.
| Symbol | Meaning | A100 | H100 | H200 | H100, locked clock |
| idle power per GPU (W) | 72.9 | 126.3 | 120.9 | 77.8 | |
| power scale per GPU (W) | 329 [318, 346] | 537 [509, 593] | 604 [583, 639] | shared with H100 | |
| intercept of | 1.99 [1.55, 2.21] | 1.32 [0.88, 1.50] | 1.47 [1.20, 1.68] | 5.53 [2.96, 238] | |
| Shared across all platforms | |||||
| model size per GPU | 0.598 [0.465, 0.666] | ||||
| load, linear | 0.749 [0.649, 0.823] | ||||
| load, quadratic | 0.137 [0.117, 0.157] | ||||
| request length | 0.440 [0.016, 0.551] | ||||
| clock, linear | 1.665 [1.550, 1.836] | ||||
| clock, quadratic | 0.430 [0.380, 0.507] | ||||
| quantized weights (Hopper) | FP8 −0.814, INT4 −0.918 | ||||
| FP8 KV cache | 0.967 |
Table 13.
Table 11: Candidate forms and held-out error at each step. Errors are held-out RMS in . T1–T4 are the step’s tests, listed in its header row; means fit on and predict . The primary score averages the tests every candidate can be scored on; – marks a test on which a parameter is not identified. is the number of free parameters, and the chosen form is in bold. Here , and .
Table 12: Precision terms from paired runs: held-out RMS (leave one pair out) with no term, a shift inside , or a factor on . is the paired -test of the better placement against no term.
Ramps and limitations. Ramps are read off the predicted trace and are not modeled separately. The 99th-percentile 1-s ramp is overpredicted by a median factor of 1.25 on Poisson traffic, consistent with the sensor averaging each step over about a second, and underpredicted by 1.31 on BurstGPT. On always-full servers, ramps come from batch turnover, which does not capture. Tensor parallelism is the weakest term: TP runs are overpredicted by 4.8–8.7%. Llama-3.1-70B, outside the fitted model family, is overpredicted by 10.4% in closed loop. Because is fitted rather than tied to the power limit, the H200 ceiling is 725 W, above the 700 W limit. and the locked-clock are weakly identified, and the clock terms come from one H100 host and three models. absorbs host effects: it is 77.8 W on the locked-clock H100 host and 126.3 W on the other H100 hosts. The coefficients are specific to vLLM and the measured platforms.
Disaggregated-Serving Model Calibration and Validation
The calibration traces provide the measurements needed to identify both the capacity and power models. Fixed-length prefill-limited runs provide service rates across input lengths, which fit the achieved fraction of peak compute and an effective attention coefficient. Decode runs provide mean iteration times across batch sizes and context lengths, which fit the achieved fraction of peak memory bandwidth together with a fixed per-step cost and a per-request cost. Power calibration uses per-instance mean GPU power and serving throughput from 3 fixed-length input-length settings, the request-rate sweep, and the Mooncake and Azure workloads, with each throughput normalized by the capacity limit of the same role under the same workload.
Least-squares fits estimate the capacity coefficients from these timing relationships. The power measurements are pooled across workloads, and bounded nonlinear least squares fits one capped ramp per role, using 905 per-GPU observations for prefill and 781 for decode. Only the request-rate sweep spans the rising part of the decode ramp and identifies its slope and knee; the other workloads contribute one operating point per deployment. The fitted coefficients and the 2 ramps are applied to all evaluated deployments without deployment-specific adjustment, so the comparison below measures agreement under one shared calibration rather than performance on measurements excluded from every fit.
Cross-deployment comparison. Table 15 reports errors across the evaluated deployments for the 3 workloads. Modeled serving capacity and total GPU power remain close to measurement in each case. Figure 7 shows the corresponding deployment-level comparisons.
| Workload | Capacity | Power |
| Fixed length | 1.25 | 2.28 |
| Mooncake | 3.15 | 2.35 |
| Azure | 1.68 | 2.23 |
Table 15. MAPE (%) across the 24 evaluated deployments of each workload under the shared calibration.
Additional Fine-tuning Results
This section provides additional fine-tuning results complementing the analysis in § 4. We first present representative power traces across a broader range of fine-tuning configurations, and then provide complete quantitative comparisons of the major factors considered in our experiments.
Additional Fine-tuning Traces
We provide additional representative fine-tuning power traces to illustrate how workload characteristics, execution configurations, and hardware platforms reshape the LLM fine-tuning power profile. These examples complement the representative cases in the main text and provide broader coverage of the configurations included in our dataset.
| Figure | Analysis | Factor / comparison | Content |
| Figure 8 | Scaling-law calibration | FLOPs, batch size, sequence length, model size | Calibration of the fine-tuning-energy scaling law on H200 with QLoRA. |
| Figure 9 | Scaling-law validation | Held-out parameter levels | Predicted versus measured energy per token under level-held-out validation. |
| Figure 10 | Representative traces | MP vs. DP | Per-GPU and aggregate power under layer-wise model and data parallelism. |
| Figure 11 | Representative traces | Batch size; sequence length | Changes in fine-tuning waveform and power level as workload dimensions increase. |
| Figure 12 | Representative traces | Model size; dense vs. MoE | Effects of model scale and architecture on cycle duration and power profile. |
| Figure 13 | Representative traces | Full, LoRA, QLoRA | Power-profile differences across fine-tuning methods. |
| Figure 14 | Representative traces | GPU platform | Comparison across L40S, A100, H100, and H200. |
| Figure 15 | Quantitative comparison | Model size | Duration, energy, power distribution, ramping rate, and spectral entropy. |
| Figure 16 | Quantitative comparison | Batch size | Six fine-tuning metrics across batch-size levels. |
| Figure 17 | Quantitative comparison | Dense vs. MoE | Six fine-tuning metrics across model architectures. |
| Figure 18 | Quantitative comparison | Fine-tuning method | Six fine-tuning metrics for Full, LoRA, and QLoRA. |
| Figure 19 | Quantitative comparison | GPU platform | Six fine-tuning metrics across L40S, A100, H100, and H200. |
| Figure 20 | Quantitative comparison | Sequence length | Six fine-tuning metrics across sequence-length settings. |
Table 16. Guide to the figures in the appendix.
Figure 9. Predicted versus measured energy per token for the held-out experiments in Figure 5. The dashed diagonal denotes perfect prediction, and the shaded region marks ±5% relative error. Model-size points are subsampled for visualization only; all samples are retained for fitting and evaluation.

(a) Model parallelism

(b) Data parallelism
Figure 10. [Fine-tuning] Parallelism strategy. Comparison of layer-wise model parallelism (MP) and data parallelism (DP) for Llama 2 13B under QLoRA on four H100 GPUs. MP exhibits a staggered and cyclic cross-GPU pattern, with recurring forward- and backward-related peak trains distributed across GPUs. The overlap between adjacent GPU activities produces pronounced surges in aggregate power. In contrast, DP executes replicated computation across GPUs, resulting in closely aligned per-GPU traces with higher per-GPU and aggregate power. The aggregate DP trace therefore largely preserves the individual waveform, while the iteration periods remain comparable between MP and DP.
(a) Batch size.
(b) Sequence length.
Figure 11. [Fine-tuning] Batch size and sequence length. Comparison of fine-tuning power traces across batch sizes and sequence lengths for Llama 2 13B under QLoRA on four H100 GPUs. Increasing either workload dimension increases the computation within each fine-tuning iteration, leading to longer execution intervals and generally higher power levels. These comparisons show that workload scale affects both the temporal structure and power level of fine-tuning.
Figure 12. [Fine-tuning] Model size. Comparison of Gemma 3 4B and 27B shows that increasing model size increases the computational demand of each fine-tuning iteration, resulting in longer execution intervals and generally higher power levels. Model architecture. Gemma 4 26B-A4B and Gemma 4 31B have comparable total parameter counts, but the MoE model activates substantially fewer parameters per token. The resulting lower effective computation leads to shorter fine-tuning cycles and lower sustained power, while its temporal pattern is more irregular than that of the dense model.
Figure 13. [Fine-tuning] Fine-tuning method. Comparison of per-GPU and aggregate power of full fine-tuning, LoRA, and QLoRA power traces for Gemma 3 4B on L40S GPUs. This figure illustrates the power profile under different fine-tuning methods. In our measured traces, the checkpoint-saving duration progressively decreases from full fine-tuning to LoRA and QLoRA. But as QLoRA further reduces the memory footprint, the resulting cross-GPU workload redistribution concentrates more computation on a single GPU in our experiments. As a result, the heavily utilized GPU exhibits a higher power level and longer execution intervals.
Figure 14. [Fine-tuning] GPU platform. Comparison of Gemma 3 27B QLoRA fine-tuning power traces across A100, H100, H200, and L40S GPUs. The L40S and A100 traces exhibit weaker temporal regularity, whereas H100 and H200 show more clearly repeated fine-tuning patterns. The fine-tuning intervals are longest on L40S, followed by A100, while H100 and H200 exhibit similar and substantially shorter durations. In contrast, the sustained power level increases progressively from L40S to A100, H100, and H200.
Figure 15. [Fine-tuning] Model size. Complete quantitative comparison across different model sizes for fine-tuning tasks. The panels report iteration duration, energy per iteration, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes how the six fine-tuning metrics vary with active parameter count. Mean iteration duration and energy per iteration increase with model size, with a particularly pronounced increase for the largest models (Figure 15a, Figure 15b). Energy per token exhibits a similar upward trend at larger model sizes, while remaining non-monotonic across intermediate configurations (Figure 15c). Mean and upper-tail power increase initially before approaching a broad plateau, whereas lower-tail power varies more modestly (Figure 15d). Ramping rate increases toward medium–large models but remains non-monotonic across configurations, while spectral entropy exhibits no consistent relationship with model size (Figure 15e, Figure 15f).
Figure 16. [Fine-tuning] Batch size. Complete quantitative comparison across different batch sizes. The panels report iteration duration, energy per iteration, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes the effects of batch size on the six fine-tuning metrics. Iteration duration and energy per iteration increase with batch size as more samples are processed per optimizer step (Figure 16a, Figure 16b). In contrast, energy per token generally decreases, indicating that larger batches amortize fixed computational and synchronization overheads over more tokens (Figure 16c). Mean and 95th-percentile power increase only moderately, while 5th-percentile power rises sharply at small batch sizes before leveling off (Figure 16d). Ramping rate also increases with batch size, whereas spectral entropy varies non-monotonically and peaks at intermediate batch sizes (Figure 16e, Figure 16f).
Figure 17. [Fine-tuning] Model architecture. Complete quantitative comparison between mixture-of-experts (MoE) and dense model architectures. The panels report iteration duration, energy per iteration, energy per token, power distribution, ramping rate, and spectral entropy. It compares dense and mixture-of-experts (MoE) architectures across the six metrics. Compared with dense models, MoE models have shorter iteration duration and lower energy per iteration and per token (Figure 17a, Figure 17b, Figure 17c). Mean and 95th-percentile power are also lower, while 5th-percentile power remains similar (Figure 17d). MoE models show slightly lower ramping rate and slightly higher spectral entropy, although both exhibit substantial overlap in the error bars (Figure 17e, Figure 17f).
Figure 18. [Fine-tuning] Fine-tuning method. Complete quantitative comparison across different fine-tuning methods. The panels report iteration duration, energy per iteration, energy per token, power distribution, ramping rate, and spectral entropy. It compares full fine-tuning, LoRA, and QLoRA across the six metrics. Full fine-tuning and LoRA have similar mean iteration durations, while QLoRA is substantially slower and more variable (Figure 18a). The same pattern appears in energy per iteration and per token, with QLoRA showing the highest energy consumption (Figure 18b, Figure 18c). In contrast, power distributions are broadly similar across the three methods, with only modest differences in mean and percentile values (Figure 18d). Full fine-tuning and LoRA also have similar ramping rates, while QLoRA is lower; spectral entropy varies only slightly across methods (Figure 18e, Figure 18f).
Figure 19. [Fine-tuning] GPU platform. Complete quantitative comparison across different GPU platforms. The panels report iteration duration, energy per iteration, energy per token, power distribution, ramping rate, and spectral entropy. It compares A100, H100, H200, and L40S GPUs across the six metrics. L40S has the longest mean iteration duration, followed by A100, while H100 and H200 complete iterations substantially faster (Figure 19a). Energy per iteration and per token follow the same broad ordering, with H100 and H200 consuming less energy than A100 and L40S (Figure 19b, Figure 19c). In contrast, H200 and H100 operate at higher mean and upper-tail power, with H200 showing the highest values (Figure 19d). H200 also has the highest ramping rate, whereas L40S has the lowest (Figure 19e). Spectral entropy is lower on H100 and H200 than on A100 and L40S (Figure 19f), indicating more regular power traces.
Figure 20. [Fine-tuning] Sequence length. Quantitative comparison across different sequence lengths. The panels report iteration duration, energy per iteration, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes the effects of sequence length on the six fine-tuning metrics. Iteration duration and energy per iteration increase sharply beyond 256 tokens (Figure 20a, Figure 20b), while energy per token decreases from 16 to 256 tokens before approaching a plateau (Figure 20c). Mean, 5th-percentile, and 95th-percentile power increase with sequence length, although upper-tail power begins to saturate at the longest sequences (Figure 20d). Ramping rate shows no clear monotonic trend, while spectral entropy decreases toward 256 tokens before increasing again at longer sequences (Figure 20e, Figure 20f).
Notes: Representative-trace figures illustrate waveform structure under controlled configurations, while quantitative figures summarize how major fine-tuning factors affect energy, power, ramping, and temporal regularity.
Complete fine-tuning comparisons
We next provide the complete quantitative comparisons across the major fine-tuning factors considered in our experiments. These results complement the representative traces above by systematically quantifying how each factor affects iteration duration, energy per iteration, energy per token, power distribution, ramping behavior, and temporal regularity. We characterize these dimensions using the metrics defined in § 4.1, including power percentiles, ramping rate percentiles, and spectral entropy.
Additional Inference Results
Additional Inference Serving Traces
We provide additional representative serving power traces to illustrate how the arrival pattern, workload, execution configuration, and hardware reshape the LLM serving’s power profile. Unless noted, runs are closed loop with 32 requests in flight and at most 512 output tokens, shown over 60 s taken 120 s into the measurement window.
| Figure | Analysis | Factor / comparison | Content |
| Figure 21 | Representative traces | Tensor-parallel degree | Per-GPU and node power of Qwen3-14B at TP 1, 2, and 4 on H100. |
| Figure 22 | Representative traces | GPU platform | Qwen3-14B at TP 4 on A100, H100, and H200. |
| Figure 23 | Representative traces | Arrival pattern | Steady and bursty BurstGPT replays of Qwen3-32B on two H200s. |
| Figure 24 | Representative traces | Concurrency; input/output length | Segments across concurrency limits 1–256 (A100) and input/output pairs (H200). |
| Figure 25 | Representative traces | Model size; dense vs. MoE | Qwen3-8B, 14B, 32B, and 30B-A3B on two H200s at TP 2. |
| Figure 26 | Representative traces | Weight and KV-cache precision | BF16, FP8, and INT4 weights and an FP8 KV cache for Qwen3-14B on H100. |
| Tab. 18 | Turnover period | Tensor-parallel degree | Batch-turnover period for every dense model and GPU at TP 1, 2, and 4. |
| Figure 27 | Quantitative comparison | Model size | Latency, energy, power distribution, ramping rate, and spectral entropy. |
| Figure 28 | Quantitative comparison | Dense vs. MoE | Six inference metrics across model architectures. |
| Figure 29 | Quantitative comparison | GPU platform | Six inference metrics across A100, H100, and H200. |
| Figure 30 | Quantitative comparison | Concurrency limit | Six inference metrics across 1–256 requests in flight. |
| Figure 31 | Quantitative comparison | Poisson arrival rate | Six inference metrics across 0.25–8 requests/s. |
| Figure 32 | Quantitative comparison | Input/output length | Six inference metrics across forced input/output pairs. |
| Figure 33 | Quantitative comparison | Weight precision | Six inference metrics for BF16, FP8, and INT4 weights. |
| Figure 34 | Quantitative comparison | KV-cache precision | Six inference metrics for BF16 and FP8 KV caches. |
| Figure 35 | Quantitative comparison | Tensor-parallel degree | Six inference metrics at TP 1, 2, and 4. |
| Figure 36 | Quantitative comparison | SM clock | Six inference metrics across locked clocks on H100. |
Table 17. Guide to the inference figures in the appendix.
Notes: Representative-trace figures illustrate waveform structure under controlled serving configurations, while quantitative figures summarize how each serving factor affects request latency, energy, power, ramping, and temporal regularity.
Arrival pattern. Figure 23 replays a steady and a bursty segment of the BurstGPT trace on Qwen3-32B served on two H200 GPUs. Under steady arrivals, both GPUs stay near their power limit for the whole window, with only a few brief dips. Under bursty arrivals, the node switches repeatedly between an idle floor of about 280 W and a serving plateau of about 1.1 kW, with one sustained stretch at the limit during the heaviest burst. Both GPUs switch together, so the node swings by the full sum of their changes, and the 99th-percentile 1-s ramp rises from 35 to 738 W/s.
Figure 23. [Inference] Representative power traces of Qwen3-32B served on two H200 GPUs with tensor parallelism, replaying (a) a steady and (b) a bursty segment of the BurstGPT trace. Each row shows per-GPU power (left) and their sum (right) over the full measurement window. Model, hardware, and serving settings are identical; only the arrival pattern differs.
Concurrency and input/output length. Figure 24 places 30-s segments of separate runs side by side. For Qwen3-32B on one A100, power rises only from about 350 to 400 W as the concurrency limit grows from 1 to 256; at low concurrency the trace is flat with isolated spikes at request turnover, and from 64 requests on it becomes a dense fluctuation near the power limit. For Qwen3-14B on one H200, 2048-token prompts hold power within a few percent of the 700 W limit with small ramps, whereas the 512/512 and 512/2048 workloads show regular turnover dips spaced several seconds apart, which produce the largest ramps.
Figure 24. [Inference] Workload sweeps: (a) maximum concurrency from 1 to 256 for Qwen3-32B on one A100; (b) input/output length from 128/128 to 2048/2048 tokens for Qwen3-14B on one H200 with 32 requests in flight. Each setting is a separate run; 30 s of each run are shown side by side, separated by grey lines, so the segments are not contiguous in time.
Model size and architecture. Figure 25 compares four Qwen3 models on two H200 GPUs at TP 2. From Qwen3-8B to 14B to 32B, node power rises from 0.91 to 1.06 to 1.17 kW and the batch-turnover period lengthens from 2.6 to 3.4 to 6.5 s, with the turnover spikes becoming more distinct. The MoE Qwen3-30B-A3B draws less power than the dense Qwen3-32B (0.98 kW) and turns over faster (5.1 s), but its dips appear mainly on one of its two GPUs.
Figure 25. [Inference] Model-size and architecture examples on two H200 GPUs at TP 2: (a) Qwen3-8B, (b) Qwen3-14B, (c) Qwen3-32B (dense), and (d) Qwen3-30B-A3B (MoE). Each row shows per-GPU power (left) and their sum (right) over 60 s taken 120 s into the measurement window. The dense/MoE pair differs in both total and active parameters.
Numerical precision. Figure 26 serves Qwen3-14B on one H100. FP8 and GPTQ-INT4 weights lower power (554, 532, and 498 W) and shorten the turnover period from 6.9 to 4.8 and 4.6 s, so the dips become more frequent and, for INT4, nearly vanish. An FP8 KV cache leaves the period unchanged (6.9 s) and changes power little.
Figure 26. [Inference] Numerical-precision examples for Qwen3-14B on one H100: (a) BF16, (b) FP8 weights, (c) GPTQ-INT4 weights, and (d) FP8 KV cache with BF16 weights, over 60 s taken 120 s into the measurement window. Runs (a)–(c) ran on a host with the GPU power limit lowered to 575 W.
Tensor parallelism. Figure 21 serves Qwen3-14B on one, two, and four H100 GPUs. Per-GPU power falls from 551 to 527 and 439 W while node power rises from 0.55 to 1.05 and 1.75 kW, and the GPUs of each group rise and fall together. The batch-turnover period shortens from 6.9 to 4.2 and 2.8 s; Table 18 gives the period for every dense model and GPU.
Figure 21. [Inference] Tensor-parallel degree: Qwen3-14B on (a) one, (b) two, and (c) four H100 GPUs. Each row shows per-GPU power (left) and their sum (right) over 60 s taken 120 s into the measurement window. The TP 1 run ran on a host with the GPU power limit lowered to 575 W; TP 2 and TP 4 ran at 700 W.
| GPU | Model | TP1 | TP2 | TP4 |
| A100 | Qwen3-8B | 7.4 | 5.3† | – |
| A100 | Qwen3-14B | 12.9 | 8.2† | 5.3 |
| A100 | Qwen3-32B | 26.4 | 16.2† | 9.7 |
| H100 | Qwen3-14B | 6.9‡ | 4.2 | 2.8 |
| H100 | Qwen3-32B | 14.3‡ | 8.3 | 5.0 |
| H200 | Qwen3-8B | 3.3 | 2.6 | – |
| H200 | Qwen3-14B | 5.2 | 3.4 | 2.5 |
| H200 | Qwen3-32B | 11.1 | 6.5 | 4.2 |
Table 18. Batch-turnover period (s) versus tensor-parallel degree, closed-loop load with 32 requests in flight. †Host with its application clock held at 1155 MHz (maximum 1410 MHz). ‡Host with the GPU power limit lowered to 575 W. Both are excluded from the scaling-law fit and the pooled comparisons.
GPU platform. Figure 22 serves Qwen3-14B at TP 4 on A100, H100, and H200. The A100 node holds a flat floor of about 1.15 kW and spikes by about 400 W at each turnover, every 5.3 s, as each GPU briefly reaches its 400 W limit. The H100 and H200 nodes draw about 1.75 and 1.70 kW with only small ripples and turn over every 2.8 and 2.5 s. The turnover spikes give the A100 node a 99th-percentile 1-s ramp of 432 W/s, against 99 and 111 W/s on H100 and H200. We compare at TP 4 because the only two-A100 host held its application clock at 1155 MHz (App. § 8).
Figure 22. [Inference] GPU platform: Qwen3-14B at TP 4 on (a) A100, (b) H100, and (c) H200. Each row shows per-GPU power (left) and their sum (right) over 60 s taken 120 s into the measurement window.
Factors Shaping Serving Power Traces
Arrival process. How requests arrive changes the trace more than any other knob (Figure 23). A steady replay of the BurstGPT trace holds the GPU near its power limit, whereas a bursty segment on the same model and GPU switches between an idle floor and a serving plateau, widening the 5th-95th-percentile band from about 20 to 1100 W and raising the 99th-percentile 1-s ramp about twentyfold. Poisson arrivals show the same split (Figure 31): up to 0.5 requests per second the GPU drains to near its idle power between requests, and up to one per second the 99th-percentile ramp exceeds 330 W/s; once arrivals keep the GPU busy (from two per second for Qwen3-14B and 32B, from four for Qwen3-8B), ramps fall about tenfold, to 29–73 W/s.
(a) Request latency
(b) Energy per request
(c) Energy per token
(d) Power distribution
(e) Ramping rate
(f) Spectral entropy
Figure 31. [Inference] Complete quantitative comparison across different Poisson arrival rates. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes the effects of the arrival rate on the six inference metrics (Qwen3-8B, 14B, and 32B on H200, free-length generation; Qwen3-32B at TP 2). Request latency rises only slightly with rate, from 3.1 to 4.1 s, while energy per request falls from 1.9 kJ to 109 J (Figure 31a, Figure 31b). Energy per token falls from 4.9 to 0.28 J as idle time is amortized over more requests (Figure 31c). The 5th-percentile power stays near idle up to 0.5 requests/s (117–128 W on one GPU and 261–270 W for the two-GPU Qwen3-32B node) and rises to 577–638 W on one GPU (1.28 kW for Qwen3-32B) at 8 requests/s, as the GPUs stop draining between requests (Figure 31d). The ramping rate stays at 410–490 W/s up to 1 request/s and drops to 42–48 W/s from 4 requests/s, while spectral entropy peaks at 1 request/s (Figure 31e, Figure 31f).
Parallelism. Tensor parallelism splits every layer, so all GPUs run each decode step together and rise and fall at the same time (Figure 21): the 1-s synchronization factor (§ 4.1) is – (median 0.93 over 19 closed-loop TP runs), so the node’s 99th-percentile ramp is close to the sum of the per-GPU ones. More GPUs lower per-GPU power but raise node power, and they shorten the batch-turnover period, each doubling of the TP degree cutting it to 0.58–0.76 of its previous value for the dense models (Table 18, Figure 35).
Figure 35. [Inference] Complete quantitative comparison across different tensor-parallel degrees. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes the effects of the tensor-parallel degree on the six inference metrics (five models on A100, H100, and H200; node power and energy summed over the GPUs of the group; Llama-3.1-70B runs only at TP 4; with capped hosts excluded, TP 1 pools A100 and H200 runs, TP 2 pools H100 and H200 runs, and TP 4 pools all three GPUs, so the pooled trend mixes the TP degree with the platform). Request latency falls from 11.1 s at TP 1 to 4.8 s at TP 2 and is 6.4 s at TP 4, whose group adds the A100 and Llama-3.1-70B, while energy per request and per token stay flat from TP 1 to 2 and double at TP 4, partly because the TP 4 group includes Llama-3.1-70B (Figure 35a, Figure 35b, Figure 35c). Node power rises from 489 to 1,013 and 1,594 W (Figure 35d). The ramping rate grows with the degree, from 67 to 119 and 316 W/s, as more GPUs ramp together, while spectral entropy falls from 0.65 to 0.54–0.57 (Figure 35e, Figure 35f).
Concurrency. Raising the concurrency limit from 1 to 256 cuts energy per output token by 17–72×, while mean power rises until it reaches the device limit (about 400 W on A100 and up to 690 W on H100 and H200) (Figure 24, Figure 30). In 12 of the 18 GPU–model series, ramps peak at intermediate concurrency (8–64 requests) and fall once the device is saturated: saturation compresses the power envelope and turns additional work into longer execution rather than higher power.
Figure 30. [Inference] Complete quantitative comparison across different maximum concurrency levels. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes the effects of the concurrency limit on the six inference metrics (six dense Qwen3 models from 0.6B to 32B on A100, H100, and H200). Request latency stays near 5–6 s up to 32 concurrent requests and then rises to 21 s at 256, while energy per request falls from 2.0 kJ to 45 J (Figure 30a, Figure 30b). Energy per token falls about 45-fold, from 3.9 to 0.088 J, as more requests share each decode step (Figure 30c). Mean power rises from 373 to 573 W, and the pooled band spans the three GPUs: its 95th percentile (near 694 W from 64 requests on) reflects the H100 and H200 runs and its 5th percentile (198 to 396 W) the A100 runs, which reach their 400 W limit at high concurrency (Figure 30d). The ramping rate peaks at 32 requests (55 W/s) and falls once the GPUs saturate, while spectral entropy rises at high concurrency (Figure 30e, Figure 30f).
Workload characteristics. Larger models raise power and lengthen the turnover period (2.6, 3.4, and 6.5 s for Qwen3-8B, 14B, and 32B on two H200s). How ramps respond depends on the headroom: on two H200s, where each GPU stays well below its limit, the 99th-percentile ramp rises with size (62, 78, and 157 W/s; Figure 25), whereas on a single GPU, which the larger models push closer to its limit, it falls (from 96 to 50 W/s on A100 and from 55 to 47 W/s on H200; Figure 27); the MoE Qwen3-30B-A3B draws less power than the dense Qwen3-32B but ramps about 1.6 times as fast Figure 28. Long prompts hold the GPU within a few percent of its limit with small ramps, while shorter prompts leave headroom and widely spaced turnover events produce the largest ramps (Figure 24, Figure 32).
Figure 27. [Inference] Complete quantitative comparison across different model sizes. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes how the six inference metrics vary with active parameter count (BF16, TP 1, closed loop, pooled over A100, H100, and H200). Request latency and energy per request rise from 5.2 s and 72 J for Qwen3-8B to 18.8 s and 269 J for Qwen3-32B, while the MoE Qwen3-30B-A3B (3B active) is slower and costlier than the dense 8B model (Figure 27a, Figure 27b). Energy per token follows the same order, from 0.14 J for Qwen3-8B to 0.52 J for Qwen3-32B (Figure 27c). Mean and 95th-percentile power rise only modestly with size, from 493 to 516 W on average (Figure 27d). The ramping rate falls from 78 to 48 W/s between Qwen3-8B and Qwen3-32B, with the MoE model (76 W/s) close to Qwen3-8B, while spectral entropy is highest for the MoE model (0.74 against 0.58–0.60) (Figure 27e, Figure 27f).
Figure 28. [Inference] Complete quantitative comparison across different model architectures. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It compares the dense Qwen3-32B and the MoE Qwen3-30B-A3B (BF16, TP 1, closed loop, pooled over A100, H100, and H200). The MoE model serves requests faster (11.5 versus 18.8 s) at lower energy per request (147 versus 269 J) and per token (0.29 versus 0.52 J) (Figure 28a, Figure 28b, Figure 28c). It also draws less power, 458 versus 516 W on average and 599 versus 674 W at the 95th percentile (Figure 28d). Its ramping rate is about 1.6 times that of the dense model (76 versus 48 W/s), and its spectral entropy is higher (0.74 versus 0.58) (Figure 28e, Figure 28f).
Figure 32. [Inference] Complete quantitative comparison across different input/output lengths. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes the effects of the forced input/output length on the six inference metrics (Qwen3-8B, 14B, and 32B on one H200 with 32 requests in flight). Request latency and energy per request grow mainly with the output length, from 1.7 s and 33 J at 128/128 to 35 s and 753 J at 2048/2048 (Figure 32a, Figure 32b). Energy per output token is highest for long prompts with short outputs (0.89 J at 2048/128) and 0.26–0.37 J otherwise (Figure 32c). The 2048-token prompts hold power within a few percent of the 700 W limit, with the 5th percentile at 681 W (Figure 32d). The same workloads have the smallest ramps (20 W/s against 39–56 W/s) and the highest spectral entropy (Figure 32e, Figure 32f).
Weight precision. Weight precision is the inference-side analogue of the fine-tuning method: it changes how much data each decode step moves. It shortens the batch-turnover period in every model and GPU pair and cuts energy per token by a median 30% (FP8) and 39% (GPTQ-INT4). Its effect on power depends on the platform: mean power falls on H100 and H200 (by up to 20%), and ramps change inconsistently (Figure 26, Figure 33). Weight quantization therefore shows up mainly as faster, cheaper execution rather than as a different waveform. An FP8 KV cache changes even less: energy per token falls by a median 2.7% (at most 5%) and the ramp changes by less than its run-to-run spread (Figure 34).
Figure 33. [Inference] Complete quantitative comparison across different weight precisions. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It compares BF16, FP8, and GPTQ-INT4 weights across the six metrics (five models on A100 and H200; all H100 weight-precision runs ran with a lowered power limit and are excluded). Quantized weights shorten request latency from 11.2 to 8.1 and 7.5 s and cut energy per request and per token by about 30% (Figure 33a, Figure 33b, Figure 33c). Mean power falls from 576 to 533 and 505 W, and 95th-percentile power falls more (Figure 33d). The ramping rate rises from 71 to 88 and 101 W/s, while spectral entropy rises slightly, from 0.59 to 0.65 and 0.64 (Figure 33e, Figure 33f).
Figure 34. [Inference] Complete quantitative comparison across different KV-cache precisions. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It compares BF16 and FP8 (E4M3) KV caches across the six metrics (five models on A100, H100, and H200, BF16 weights). Request latency changes by less than 1%, and energy per request and per token fall by about 2% on average (a median 2.7% and at most 5% across the 13 same-host pairs) (Figure 34a, Figure 34b, Figure 34c). Mean power falls slightly, from 574 to 559 W (Figure 34d). The ramping rate rises from 80 to 89 W/s, less than its run-to-run spread, and spectral entropy falls from 0.60 to 0.58 (Figure 34e, Figure 34f).
Hardware. The GPU platform sets the power range and the timescale (Figure 22, Figure 29): the same configuration (Qwen3-14B at TP 4) turns over every 5.3 s on A100 and every 2.8 and 2.5 s on H100 and H200, at about 1.5 times the node power. Within the H100 platform, the SM clock exposes a two-objective trade-off (Figure 36): energy per token is lowest near 1200 MHz, whereas the 99th-percentile ramp keeps growing with clock. The clock that minimizes energy is not the one that minimizes grid impact.
Figure 29. [Inference] Complete quantitative comparison across different GPU platforms. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It compares A100, H100, and H200 (BF16, TP 1, closed loop, pooled over Qwen3-8B and 14B, the only models with an H100 run at the standard power limit; 6, 2, and 6 runs). Request latency falls from 10.1 s on A100 to 5.5 and 4.2 s on H100 and H200, and energy per request and per token fall in the same order (111, 99, and 83 J; 0.22, 0.19, and 0.16 J) (Figure 29a, Figure 29b, Figure 29c). Mean power rises from 353 W on A100 to 565 and 625 W on H100 and H200, each within a narrow band (Figure 29d). The ramping rate falls from 94 to 83 and 55 W/s, and spectral entropy from 0.79 to 0.53 and 0.41 (Figure 29e, Figure 29f).
Figure 36. [Inference] Complete quantitative comparison across different SM clock frequencies. The panels report request latency, energy per request, energy per token, power distribution, ramping rate, and spectral entropy. It summarizes the effects of the locked SM clock on the six inference metrics (Qwen3-8B, Qwen3-30B-A3B, and Llama-3.1-8B on one H100, closed loop; the 1980 MHz setting throttles to a median of 1830 MHz). Request latency falls from 14.4 s at 345 MHz to 5.8 and 5.7 s at 1605 and 1980 MHz, while energy per request and per token are lowest at 1200 MHz (66 J and 0.13 J) and rise again at higher clocks (Figure 36a, Figure 36b, Figure 36c). Mean power rises from 184 to 535 W, with a narrow band at every clock (Figure 36d). The ramping rate grows at every clock step, from 8 to 68 W/s, while spectral entropy falls from 0.48 to 0.43–0.45 (Figure 36e, Figure 36f).
Complete Inference Comparisons
We next provide the complete quantitative comparisons across the major inference factors considered in our experiments, in the same format as Appendix § 9.§ 2. For each factor, runs at the same factor level are pooled over the models and GPU platforms represented in that comparison, and error bars show one standard deviation. These results quantify how each factor affects request latency, energy per request, energy per token, power distribution, ramping behavior, and temporal regularity. Runs on hosts with a lowered GPU power limit or a capped application clock are excluded.
References
- [1]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024. Preprint. Under Review.
- [2]Xin Chen, Xiaoyang Wang, Ana Colacelli, Matt Lee, and Le Xie. Electricity demand and grid impacts of ai data centers: Challenges and prospects. arXiv preprint arXiv:2509.07218, 2025.arxiv.org/abs/2509.07218
- [3]Esha Choukse, Brijesh Warrier, Scot Heath, Luz Belmont, April Zhao, Hassan Ali Khan, et al. Power stabilization for AI training datacenters. arXiv preprint arXiv:2508.14318, 2025.arxiv.org/abs/2508.14318
- [4]Jae-Won Chung, Yile Gu, Insu Jang, Luoxi Meng, Nikhil Bansal, and Mosharaf Chowdhury. Perseus: Reducing energy bloat in large model training. In ACM Symposium on Operating Systems Principles (SOSP), 2024.DOI
- [5]Jae-Won Chung, Jeff J. Ma, Ruofan Wu, Jiachen Liu, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, and Mosharaf Chowdhury. The ML.ENERGY benchmark: Toward automated inference energy measurement and optimization. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. arXiv:2505.06371.arxiv.org/abs/2505.06371
- [6]Philip Colangelo, Ayse K Coskun, Jack Megrue, Ciaran Roberts, Shayan Sengupta, Varun Sivaram, Ethan Tiao, Aroon Vijaykakar, Chris Williams, Daniel C Wilson, et al. Ai data centres as grid-interactive assets. Nature Energy, 11(2):254–261, 2026.DOI
- [7]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2305.14314
- [8]Patrick Emami, Abhijeet Sahu, and Peter Graf. BuildingsBench: A large-scale dataset of 900K buildings and benchmark for short-term load forecasting. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.
- [9]Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, and Emma Strubell. Energy considerations of large language model inference and efficiency optimizations. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 32556–32569, 2025. doi: 10.18653/v1/2025.acl-long.1563.DOI
- [10]Xuan He, Zequan Fang, Jinzhao Lian, Danny HK Tsang, Baosen Zhang, and Yize Chen. Freesh: Fair, resource-and energy-efficient scheduling for llm serving on heterogeneous gpus. arXiv preprint arXiv:2511.00807, 2025.arxiv.org/abs/2511.00807
- [11]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.arxiv.org/abs/2203.15556
- [12]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2106.09685
- [13]International Energy Agency. Energy and AI. Technical report, IEA, Paris, 2025.
- [14]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In ACM Symposium on Operating Systems Principles (SOSP), 2023.arxiv.org/abs/2309.06180
- [15]Yueying Li, Jiayang Chen, Yuanfan Chen, Leo Han, Haoran Qiu, Esha Choukse, Rodrigo Fonseca, and Udit Gupta. Powerslider: Exploiting phase asymmetry for llm serving under demand response. arXiv preprint arXiv:2608.21719, 2026.arxiv.org/abs/2608.21719
- [16]Yuzhuo Li, Mariam Mughees, Yize Chen, and Yunwei Ryan Li. The unseen AI disruptions for power grids: LLM-induced transients. arXiv preprint arXiv:2409.11416, 2024.arxiv.org/abs/2409.11416
- [17]Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of AI deployment? In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2024.DOI
- [19]Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on GPU clusters using Megatron-LM. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2021.arxiv.org/abs/2104.04473
- [20]North American Electric Reliability Corporation. Incident review: Considering simultaneous voltage-sensitive load reductions. Technical report, NERC, 2025. Published January 2025.
- [21]Felipe Oviedo, Fiodar Kazhamiaka, Esha Choukse, Allen Kim, Amy Luers, Melanie Nakagawa, Ricardo Bianchini, and Juan M Lavista Ferres. Energy use of ai inference, efficiency pathways, and test-time scaling. Joule, 2026.DOI
- [22]Pratyush Patel, Esha Choukse, Chaოjie Zhang, Íñigo Goiri, Brijesh Warrier, Nitish Mahalingam, and Ricardo Bianchini. Characterizing power management opportunities for LLMs in the cloud. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024a. doi: 10.1145/3620666.3651329.DOI
- [23]Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In ACM/IEEE International Symposium on Computer Architecture (ISCA), 2024b.
- [24]David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021.arxiv.org/abs/2104.10350
- [25]Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinxuan Xu. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In USENIX Conference on File and Storage Technologies (FAST), 2025.
- [26]Siddharth Samsi, Matthew L. Weiss, David Bestor, Baolin Li, Michael Jones, Albert Reuther, Daniel Edelman, et al. The MIT supercloud dataset. In IEEE High Performance Extreme Computing Conference (HPEC), 2021.
- [27]Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaeleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmarking the energy costs of large language model inference. In IEEE High Performance Extreme Computing Conference (HPEC), 2023. doi: 10.1109/HPEC58863.2023.10363447.DOI
- [28]Arman Shehabi, Sarah J. Smith, Alex Hubbard, Alex Newkirk, Nuoa Lei, Md Abu Bakar Siddik, Billie Holecek, Jonathan Koomey, Eric Masanet, and Dale Sartor. 2024 united states data center energy usage report. Technical Report LBNL-2001637, Lawrence Berkeley National Laboratory, 2024.
- [29]Jovan Stojkovic, Esha Choukse, Chaojie Zhang, Inigo Goiri, and Josep Torrellas. Towards greener llms: Bringing energy-efficiency to the forefront of llm inference. arXiv preprint arXiv:2403.20306, 2024.arxiv.org/abs/2403.20306
- [30]Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. DynamoLLM: Designing LLM inference clusters for performance and energy efficiency. In IEEE International Symposium on High Performance Computer Architecture (HPCA), 2025.DOI
- [31]Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 3645–3650, 2019.arxiv.org/abs/1906.02243
- [32]Arya Tschand, Arun Tejusve Raghunath Rajan, Sachin Idgunji, Anirban Ghosh, Jeremy Holleman, Csaba Kiraly, et al. MLPerf power: Benchmarking the energy efficiency of machine learning systems from microwatts to megawatts for sustainable AI. In IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 1201–1216, 2025. Preprint. Under Review.arxiv.org/abs/2410.12032
- [33]Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, et al. BurstGPT: A real-world workload dataset to optimize LLM serving systems. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2025. doi: 10.1145/3711896.3737413.DOI
- [34]Mingyuan Yan, Haiyu Wang, Linxuan Biao, H. Jonathan Chao, Sai Qian Zhang, and Wenqi Cui. Analytical power-aware provisioning for prefill-decode disaggregated ai inference. arXiv preprint arXiv:2609.24639, 2026. URL https://arxiv.org/abs/2609.24639.
- [35]Zeyu Yang, Karel Adamek, and Wesley Armour. Part-time power measurements: nvidia-smi’s lack of attention. arXiv preprint arXiv:2312.02741, 2023. Published in Proceedings of SC24, doi:10.1109/SC41406.2024.00028.DOI
- [36]Jie You, Jae-Won Chung, and Mosharaf Chowdhury. Zeus: Understanding and optimizing GPU energy consumption of DNN training. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2023.arxiv.org/abs/2208.06102
- [37]Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.arxiv.org/abs/2401.09670












































