betaFeedback

WorksComputer Science

Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters

Abstract

Two NVIDIA DGX Spark GB10 systems connect directly over 200 GbE to serve larger language models and increase inference capacity, but the benefits depend on how they are deployed. We benchmark 17 open-weight checkpoints spanning dense and mixture-of-experts architectures, 8B to 235B parameters and four weight precisions on one and two RDMA-connected systems. Using vLLM, we compare tensor parallelism (TP=2), pipeline parallelism (PP=2) and a two-replica baseline at concurrency 1 to 64, with three independent launches per configuration. TP=2 delivers a median 1.59x peak-throughput gain over one Spark, versus 1.39x for PP=2 and has higher throughput in 93 of 96 matched comparisons with separated 95% Welch intervals. It reduces unloaded inter-token latency to 0.52x to 0.73x of one Spark, whereas PP=2 provides little reduction. TP=2 also achieves the highest goodput under a strict 100 ms inter-token latency objective wherever both strategies are supported and the objective is met. For models fitting on one Spark, TP=2 generally favors low-concurrency serving, while replication offers higher throughput at high concurrency. One Spark uses less GPU energy per output token at matched concurrency. Across three precision ladders, NVFP4 reduces unloaded latency more than adding a second Spark at BF16. NCCL microbenchmarks and comparisons with an earlier TCP campaign suggest that collective communication overhead contributes to the advantage of TP=2. These results provide deployment guidance for desktop-scale inference.

Introduction

Compact local GPU systems are becoming an attractive platform for organizations seeking to serve large language models (LLMs) on premises, particularly when data privacy, infrastructure costs and control over deployment are important considerations. The NVIDIA DGX Spark integrates a GB10 system-on-chip, comprising a Blackwell GPU and a 20-core Arm CPU sharing 128 GB of LPDDR5x memory, into a desktop-scale system. Systems of this size enable local inference with open-weight language models without requiring datacenter-scale hardware. Two DGX Sparks can be connected directly through their ConnectX-7 network ports, doubling the memory capacity. This enables the deployment of models that cannot fit on a single system and provides additional resources for serving concurrent requests. However, the performance benefits depend on how those resources are used. This paper investigates the tradeoffs between distributing a model across two Sparks and deploying independent replicas, evaluating throughput, latency, goodput and GPU energy efficiency.

A model can be distributed across two GPUs using two standard approaches. Tensor parallelism (TP=2) partitions the computation within each layer across both GPUs and requires collective communication during every decode step. Pipeline parallelism (PP=2) assigns different layers to each GPU and transfers activations between pipeline stages. Conventional multi-GPU serving guidance favors tensor parallelism within a server, where high-bandwidth interconnects such as NVLink support frequent collective operations and pipeline parallelism across servers to minimize communication overhead [1]. DGX Spark presents a different hardware configuration. The two GPUs communicate through a pair of 200 GbE connections rather than NVLink, the platform does not support GPUDirect RDMA [2] and each GPU accesses unified LPDDR5x memory instead of dedicated high-bandwidth memory (HBM). These characteristics make the relative benefits of tensor and pipeline parallelism difficult to predict from conventional datacenter experience. We evaluate both strategies using NCCL over RDMA over Converged Ethernet version 2 (RoCEv2), with all four network rails configured according to NVIDIA’s recommended setup [3]. We also use an earlier TCP-based campaign as supporting evidence to examine the role of inter-node communication overhead.

The study covers 17 open-weight checkpoints spanning dense and mixture-of-experts (MoE) architectures, four weight precisions and 8B to 235B parameters. Of these, three require two Sparks for the evaluated configurations. We evaluate up to three deployments per checkpoint: one Spark, two Sparks with TP=2 and two Sparks with PP=2. Each deployment is measured at six concurrency levels, with three independent server launches per measurement point. The resulting campaign comprises 846 measurement windows and 145 measurement hours across 141 launches, without a failed request in the analyzed windows. For models that fit on one Spark, we additionally derive the performance of two independent replicas from single-Spark measurements to determine when splitting a model is preferable to replication. Figure 1 previews the performance gains from distributing a model across two Sparks.

Figure 1. Performance gains from distributing a model across two DGX Sparks. Each marker represents one checkpoint under one parallel deployment, with lines connecting TP=2 and PP=2 results for the same checkpoint. The horizontal axis shows unloaded decode speedup, defined as the ratio of one-Spark to two-Spark inter-token latency at concurrency 1. The vertical axis shows peak throughput speedup, defined as the ratio of two-Spark to one-Spark peak throughput over concurrency levels up to 64. TP=2 improves both decode latency and peak throughput, whereas PP=2 improves peak throughput with little change in unloaded decode latency.

The paper makes four contributions.

  • A reproducible benchmark of LLM inference on DGX Spark (Sections III–V). We evaluate 17 open-weight

checkpoints spanning multiple architectures, model sizes and weight precisions across single- and dual-node deployments. Every measurement point is replicated across three independent server launches, NCCL transport is verified from the logs of every dual-node launch and all reported results are generated from sealed measurements (Appendix C).

  • A systematic comparison of tensor and pipeline parallelism over RDMA (§ 5-A–V-E). We show that TP=2 outperforms PP=2 in peak throughput, unloaded decode latency and goodput under a strict inter-token latency objective. Welch confidence intervals place TP=2 above PP=2 in 93 of 96 throughput comparisons, with none favoring PP=2. We also identify the conditions under which PP=2 is beneficial, particularly during prefill of prompts spanning multiple chunks.

  • An evaluation of model splitting versus independent replication (§ 5-F). For models that fit on one Spark, we compare distributed inference with two independent replicas derived from single-node measurements. We find that TP=2 provides higher throughput at low concurrency, while replication generally performs better at high concurrency. PP=2 does not significantly outperform replication and a single Spark consistently provides the lowest GPU energy per generated token at matched concurrency. These results inform deployment choices based on workload and performance objectives.

  • An analysis of the hardware and communication factors governing inference performance (§ 5-H and VI). We relate unloaded decode latency to memory bandwidth, weight precision, model architecture and inter-node communication. Dense-model decoding operates at 1.12–1.27× of the theoretical weight-read bound and reducing precision from BF16 to NVFP4 improves unloaded latency more than adding a second Spark under either parallel strategy. NCCL microbenchmarks and comparisons with an earlier TCP-based campaign provide supporting evidence that lower collective communication overhead contributes to the observed advantage of TP=2 over PP=2.

Background

The platform

The GB10 system-on-chip integrates a 20-core Arm CPU and a Blackwell GPU sharing 128 GB of unified LPDDR5x memory with a nominal bandwidth of 273 GB/s [4]. Unlike datacenter GPUs with dedicated high-bandwidth memory (HBM), the GB10 GPU accesses shared system memory. At small batch sizes, dense-model decoding is primarily memory-bandwidth-bound, with a theoretical lower bound on decode step time given by the model’s weight footprint divided by available memory bandwidth [5, 6, 7].

Two DGX Sparks can connect directly through their ConnectX-7 ports without a network switch [3, 8]. Each Ethernet interface has a corresponding RDMA interface [8], allowing NVIDIA’s Collective Communications Library (NCCL) [9] to communicate over RDMA over Converged Ethernet version 2 (RoCEv2) [10] instead of TCP sockets. When RDMA devices are unavailable inside the container, NCCL can silently fall back to TCP. We therefore verify the transport used by every dual-node launch (Section IV).

Serving pipeline

We serve models using vLLM [11], which supports continuous batching [12] and paged key-value (KV) cache management. Inference consists of two phases: prefill, which processes the input prompt and generates the first token and decode, which generates subsequent tokens autoregressively. Prefill is typically compute-bound for long prompts and largely determines time to first token (TTFT). With chunked prefill [13], vLLM divides long prompts into chunks based on a token budget and schedules them alongside decoding requests. Decode is typically memory-bandwidth-bound at small batch sizes, as each step accesses model weights and the active KV cache and determines inter-token latency (ITL). Since maximum throughput can occur at loads with unacceptable interactive latency, we also evaluate goodput and GPU energy per generated token. Following the concept introduced by DistServe [14], we define goodput at each load as the rate of completed requests that satisfy both TTFT and ITL objectives.

Two ways to split one model

Figure 2 illustrates the three deployments. TP=2 shards the weight matrices of each layer across both GPUs [1, 15]. Each node reads approximately half the weights per decode step, with an all-reduce after the attention and feed-forward blocks. A dense model with LL layers therefore requires approximately 2L2L all-reduces per step. Each collective operates on an activation tensor proportional to the batch size and hidden dimension. For BF16 activations and a hidden size of 4096, as in Llama-3.1-8B, the payload is 8 KiB per decode token, or 512 KiB for a batch of 64 tokens. A prefill chunk of 2,048 tokens produces a 16 MiB payload. At low concurrency, collective latency dominates communication overhead, while larger batches increase the importance of message size (§ 6).

Figure 2. Three serving deployments. (a) One Spark executes all layers and reads the model weights WW from local memory. (b) TP=2 distributes each layer across both Sparks, requiring two all-reduces per layer. (c) PP=2 assigns half the layers to each Spark and transfers activations between stages.

PP=2 assigns the first half of the layers to one GPU and the remaining layers to the other [16]. Each node reads only its stage’s weights and transfers intermediate activations across the stage boundary. For Llama-style models in vLLM, the transferred hidden state and residual together require twice the payload of one TP all-reduce. At concurrency 1, the stages execute sequentially, so pipeline parallelism cannot reduce decode latency through stage overlap. With multiple requests in flight or long prompts divided into prefill chunks, the stages can process different batches concurrently, improving throughput. Both strategies distribute KV-cache storage across the two nodes, with TP=2 partitioning attention heads and PP=2 partitioning layers.

Precision and sparsity

Weight quantization reduces memory traffic during inference. FP8 (E4M3) approximately halves the BF16 weight footprint [17], while NVFP4 uses 4-bit weights with two-level block scaling, further reducing memory requirements and supporting native execution on Blackwell GPUs [18]. MXFP4 is a separate 4-bit microscaling format used by the gpt-oss mixture-of-experts (MoE) models [19]. Several NVIDIA ModelOpt checkpoints [20] also use an FP8 key-value (KV) cache (Table II), so comparisons across precision variants may reflect changes in both weight and KV-cache precision. MoE architectures [21], [22] activate only a subset of experts for each token, reducing weight memory traffic relative to dense models with similar total parameter counts.

CheckpointTypeAttentionWtsKVGiBRunsKV capacity (req.)
Llama-3.1-8B BF16densefullBF16BF1615.0✓✓✓72163161
Llama-3.1-8B FP8densefullFP8FP88.5✓✓✓158337337
Llama-3.1-8B NVFP4densefullNVFP4FP85.7✓✓✓164343344
Qwen3.6-27B BF16denselinear hybridBF16BF1650.2✓✓✓56187181
Qwen3.6-27B FP8denselinear hybridFP8BF1627.6✓✓✓85218212
Qwen3.6-27B NVFP4denselinear hybridNVFP4†FP819.1✓✓✓161381368
Gemma-4-31B NVFP4densesliding windowNVFP4FP830.1✓✓✓256748
Mistral-Medium-3.5-128B NVFP4densefullNVFP4FP884.1–✓✓–6663
Qwen3.6-35B-A3B BF16MoE 3B act.linear hybridBF16BF1664.7✓✓✓120492494
Qwen3.6-35B-A3B FP8MoE 3B act.linear hybridFP8BF1633.6✓✓✓244621627
Qwen3.6-35B-A3B NVFP4MoE 3B act.linear hybridNVFP4†BF1619.6✓✓✓301683685
Gemma-4-26B-A4B NVFP4MoE 4B act.sliding windowNVFP4FP817.1✓–✓129–179
Nemotron-3.5-Lightning-30B-A3B NVFP4MoE 3B act.Mamba hybridNVFP4†FP817.9✓✓✓89920042002
Qwen3-Next-80B-A3B NVFP4MoE 3B act.linear hybridNVFP4FP844.2✓✓✓320944951
Qwen3-Next-80B-A3B BF16MoE 3B act.linear hybridBF16BF16148.6–✓✓–178169
gpt-oss-120b MXFP4MoE 5.1B act.sliding windowMXFP4BF1666.1✓✓✓59258225
Qwen3-235B-A22B NVFP4MoE 22B act.fullNVFP4FP8130.0–✓✓–6468

Table II. Model checkpoints and deployment configurations, grouped by dense and mixture-of-experts (MoE) architectures. Type indicates active parameters per token for MoE models. Wts and KV denote weight and KV-cache precision. GiB reports the total resident weight footprint across participating nodes. Runs indicates the evaluated deployments: one Spark, TP=2 and PP=2. KV capacity is the number of concurrent 8,704-token sequences supported by vLLM’s allocated KV cache. The workload averages 2,926 tokens per request, allowing greater effective concurrency. † Indicates NVFP4 weights served using vLLM’s weight-only Marlin kernel (§ 5-H).

Methodology

Table I defines the workload variables and the metrics and the reasons for the choices.

Workload

A measurement point consists of one checkpoint, one deployment and one concurrency level c∈{1,4,8,16,32,64}c \in\{1, 4, 8, 16, 32, 64\}. We use four (input, output) token buckets: (128, 128), (1024, 256), (4096, 256) and (8192, 512), with weights 1, 1, 1 and 0.5, respectively. Smooth weighted round robin ensures that the deployments receive the same deterministic bucket sequence at a given concurrency. The workload averages 2,670 input tokens and 256 output tokens per request, yielding approximately 10 prefill tokens per generated token. Reported output throughput counts only generated tokens.

Prompts consist of tokenizer-exact synthetic token sequences with unique eight-token prefixes to prevent prefix reuse. Prefix caching is disabled. Generation uses greedy decoding with fixed output lengths and ignores end-of-sequence tokens. A single asynchronous client on the head node maintains exactly cc concurrent requests against vLLM’s OpenAI-compatible streaming endpoint, submitting a new request whenever one completes. Output tokens are counted from the returned token identifiers.

Windows and accounting

Each measurement consists of three phases. Warmup lasts at least 120 s and continues until every client connection completes one request. The measurement window WW lasts at least 600 s and extends in 60 s increments until at least 20 requests have started, with a maximum duration of 3,600 s. Cooldown lasts at least 30 s, maintaining the workload until all requests admitted during WW complete.

Throughput counts output tokens received during WW from all in-flight requests, while latency and goodput are calculated from requests admitted during WW and tracked to completion. GPU power is sampled using nvidia-smi and integrated over WW, summing both nodes for two-Spark deployments.

Of the 846 measurement windows, 30 required extensions beyond 600 s, all at c≤4c \le4 on the slowest checkpoints, reaching at most 2,280 s. These windows contain only 20 or 21 admitted requests, with as few as 3 from the 8,192-token bucket.

Replicates and intervals

Each replicate consists of a fresh server launch followed by a complete ascending concurrency sweep. Every measurement point has three replicates, with deployment order rotated across replicates to mitigate temporal drift. We replicate server launches to capture variability in memory profiling, KV-cache capacity, compilation state and cluster initialization. This variability is small: the median throughput coefficient of variation (CV) across replicates is 0.4 % (§ 5-I).

QuantityUnitDefinition
Measurement window WWsInterval over which throughput, goodput, and energy are measured: at least 600 s, extended in 60 s steps until at least 20 requests have started inside it (3,600 s cap).
Concurrency ccrequestsRequests the client keeps in flight. A new request is issued the moment one completes (closed loop).
ReplicatelaunchOne fresh server launch followed by a complete ascending sweep over cc. Every point has three replicates; the order of the three deployments rotates across replicates.
Output throughputtok/sOutput tokens received by the client during WW, from all in-flight requests, divided by the length of WW.
Peak throughputtok/sLargest replicate-mean output throughput of a deployment over c∈{1,…,64}c \in\{1,\ldots,64\}; on 43 of the 47 curves it is the value at c=64c=64.
TTFTsTime to first token: request submission to arrival of the first output token at the client.
ITLmsInter-token latency: the mean gap between consecutive output tokens of one request. The paper reports the median over requests started in WW.
Prefill ratetok/sReciprocal slope of a linear fit of median TTFT against prompt length over the four workload buckets at c=1c=1.
Goodputreq/sRequests started in WW with TTFT ≤3\le3 s and mean ITL ≤100\le100 ms (strict tier) or ≤200\le200 ms (loose tier), divided by the length of WW.
Energy per tokenJ/tokGPU energy integrated from sampled device power over WW, summed over both nodes for two-Spark deployments, divided by the output tokens received in WW.
KV-cache capacityrequestsNumber of concurrent 8,704-token requests whose KV cache fits in the memory vLLM reserves at launch, as reported by the engine.
Scaling ratio×\timesA two-Spark deployment’s value divided by the one-Spark value of the same checkpoint and metric. Above 1 is better for throughput, goodput, and capacity; below 1 is better for latency and energy.
Welch intervalas metric95 % confidence interval for the difference between two deployments’ replicate means (Welch’s tt, n=3n=3 each). A margin is called separated when the interval excludes zero.

Table I. Workload variables and reported metrics. Every value in the paper is the mean over three replicate launches unless stated otherwise.

Figures report the mean across three launches, with bands showing the minimum and maximum replicate values. We consider a difference between deployments separated when its 95 % Welch confidence interval excludes zero [23]. These intervals are not adjusted for multiple comparisons. Given the small number of replicates, we interpret separation as evidence of a directional difference rather than definitive statistical significance. Conversely, an undecided comparison does not establish equivalent performance.

Objectives

We evaluate goodput under two service-level objectives (SLOs). Both require time to first token (TTFT) of at most 3 s, with mean inter-token latency (ITL) limited to 100 ms for the strict tier or 200 ms for the loose tier. Goodput is the rate of requests satisfying both objectives. Unless otherwise stated, reported ITL is the median across requests of their individual mean inter-token latencies. The pooled median of individual token gaps is used only for the memory-bandwidth analysis in § 5-H.

Experimental Setup

Hardware and interconnect

Our cluster consists of two ASUS Ascent GX10 systems based on the NVIDIA DGX Spark GB10 design. The head node hosts the benchmark client and the server for single-node experiments, while the second node serves as the worker. The head node identifies as an ASUSTeK GX10. Both nodes have matching GPU models, drivers, container images and GPU clocks. Two direct cables connect their ConnectX-7 ports without a switch. Each node exposes four RoCEv2 interfaces, two per physical port, forming four independent rails on separate IPv4 subnets with an MTU of 4096.

We configured NCCL 2.31.2 to use all four RDMA rails and enabled debug logging to verify the communication transport. We audited all 148 dual-node launches, including canary and failed launches and the excluded checkpoints described in Section IV-C. All 148 used NET/IB across four RoCEv2 devices, with no NCCL channels using TCP sockets or GPUDirect RDMA.

DGX Spark does not support GPUDirect RDMA because GPU-allocated memory cannot be accessed coherently by PCIe devices [2]. Consistent with this limitation, NCCL reported cuMemGdrSupport 0 in every launch and staged transfers through host-registered buffers. Since CPU and GPU share LPDDR5x memory, staging introduces additional memory copies rather than transfers between separate memory pools. § 6 examines communication performance using NCCL microbenchmarks collected on the same network configuration.

Software

We used NVIDIA’s NGC vLLM 26.09 container (vLLM 0.29.0, PyTorch 2.14, CUDA 13.4 and FlashInfer 0.6.18) with driver 580.178.04 and Ray 2.56.1 for multi-node coordination [24]. All checkpoints used the same engine settings: a maximum model length of 8,704 tokens, GPU memory utilization of 0.80, chunked prefill, CUDA graphs, torch.compile and disabled prefix caching.

In vLLM 0.29.0, the default batched-token budget exceeds 2,048 only when NVML reports at least 70 GiB of GPU memory. Because NVML does not report a memory total on GB10, the engine used a 2,048-token budget. All 208 archived server logs confirm this value through compile_ranges_endpoints, which reflects the configured token budget in this version.

Under TP=2, collective operations used NCCL. We disabled FlashInfer kernel autotuning across all deployments to avoid multi-node startup deadlocks associated with differing kernel configurations [25], [26]. Dual-node deployments used --tensor-parallel-size 2 or --pipeline-parallel-size 2, with vLLM’s default pipeline schedule. Appendix B provides the complete configuration.

Checkpoints

We evaluate 17 model checkpoints, including multiple precision variants of the same underlying models. We refer to each distinct set of model weights and quantization settings as a checkpoint. Table II summarizes the 17 checkpoints, spanning dense, mixture-of-experts (MoE) and hybrid architectures. Three model families are evaluated at BF16, FP8 and NVFP4: Llama-3.1-8B [27], Qwen3.6-27B and Qwen3.6-35B-A3B [28]. Gemma-4 contributes a dense 31B and an MoE 26B-A4B model at NVFP4 [29], [30]. Nemotron-3.5-Lightning-30B-A3B uses a hybrid Mamba-MoE architecture [31], while Qwen3-Next-80B-A3B uses hybrid linear attention [32]. The latter is evaluated at NVFP4 on one and two Sparks and at BF16 on two Sparks only. Additional checkpoints include gpt-oss-120b at MXFP4 [19], Mistral-Medium-3.5-128B [33] and Qwen3-235B-A22B [34], [35]. The latter two are evaluated at NVFP4 on two Sparks only (§ 5-G). All Hugging Face revisions were pinned and recorded for reproducibility.

One planned configuration could not be evaluated: TP=2 for Gemma-4-26B-A4B NVFP4, because the FlashInfer CUTLASS MoE implementation in this vLLM version lacks the weight-padding support required for tensor parallelism. The resulting matrix contains 47 deployment curves: 13 checkpoints under all three deployments, referred to as complete checkpoints, Gemma-4-26B-A4B under one Spark and PP=2, and three checkpoints under both two-Spark deployments.

Two additional Nemotron-3-Super-120B-A12B checkpoints were measured but excluded from the analysis. NVIDIA’s container release notes limit the NVFP4 variant to four sequences on DGX Spark [36, 37, 38]. Consequently, measurements above c=4c=4 primarily reflect scheduler queuing rather than deployment performance. The FP8 variant requires two Sparks, but all three PP=2 launches failed before measurement, leaving no comparable PP=2 results. The 72 excluded windows remain available in the released index of 918 windows.

Campaign

The campaign ran unattended from 2026-10-31 to 2026-11-10, spanning 251 hours from the first to the last measurement window. Checkpoints were downloaded, benchmarked and removed sequentially. The analyzed windows contain 272,630 completed requests, with no request failures or scheduler pre-emptions. GPU power measurements were collected from both nodes for every two-Spark window. Following the campaign, a SHA-256 manifest sealed 3,952 artifacts. All 972 index, summary, serving-policy and microbenchmark files used in the analysis were verified against the manifest.

An earlier TCP-based campaign, used for comparison in § 6, ran in July 2026. NCCL used its default TCP socket transport on one interface. This campaign used the NGC 26.04 container (vLLM 0.19), 1,500 s measurement windows and concurrency levels up to 128.

Results

Figure 3 and Table III summarize the campaign and the first five subsections follow the columns of Figure 3. Figure 4 shows the load sweeps of four representative checkpoints and Appendix A shows all 17.

Checkpoint1×TPPPTP/1×PP/1×PP/TP1×TPPP1×TPPP
Llama-3.1-8B BF161312182041.661.550.937238730.180.730.32
Llama-3.1-8B FP83214664391.451.370.944122420.791.560.85
Llama-3.1-8B NVFP43905295221.361.340.992615261.331.851.62
Qwen3.6-27B BF1664.210698.21.651.450.882291232330.000.000.00
Qwen3.6-27B FP881.61371351.681.650.98130701310.000.130.00
Qwen3.6-27B NVFP481.11321081.631.330.828448860.030.220.04
Gemma-4-31B NVFP493.71581331.681.410.84150841540.000.080.00
Mistral-Medium-3.5-128B NVFP4–65.151.5––0.79–215423–0.000.00
Qwen3.6-35B-A3B BF161542542141.651.390.843420330.310.860.42
Qwen3.6-35B-A3B FP81932992801.551.450.942014200.450.860.60
Qwen3.6-35B-A3B NVFP42443683741.511.531.02149.9140.841.431.04
Gemma-4-26B-A4B NVFP4381–476–1.25–35–350.88–1.35
Nemotron-3.5-Lightning-30B-A3B NVFP43444954101.441.190.832213220.801.610.86
Qwen3-Next-80B-A3B NVFP42843863631.361.280.942718260.691.250.76
Qwen3-Next-80B-A3B BF16–215177––0.82–2439–0.430.27
gpt-oss-120b MXFP41862962171.591.170.742818280.330.690.44
Qwen3-235B-A22B NVFP4–124107––0.86–6091–0.070.04
Median (13 complete)1.591.390.93

Table III. Per-checkpoint inference performance across deployments. Peak throughput is the maximum replicate-mean output throughput over c≤64c \le64. Peak ratios compare two-Spark deployments against one Spark and PP=2 against TP=2. ITL is the median request-mean inter-token latency at c=1c=1, while strict-tier goodput is the maximum across tested concurrency levels. Summary medians are calculated over the 13 complete checkpoints.

Figure 3. Performance comparison of one- and two-Spark deployments across model checkpoints. The first five columns show two-Spark performance relative to one Spark: peak throughput over c≤64c \le64, unloaded decode speedup at c=1c=1 (one-Spark ITL divided by two-Spark ITL), prefill rate at c=1c=1, maximum strict-tier goodput and minimum GPU energy per token. The sixth column compares PP=2PP=2 and TP=2TP=2 peak throughput and includes the three checkpoints evaluated only on two Sparks (asterisks). Dashed reference lines indicate parity, except for throughput, where the line marks ideal 2×2\times scaling. Shading indicates favorable performance regions. Goodput uses a logarithmic scale, with zero-baseline cases labeled. Whiskers in the first two columns show conservative replicate bounds, using the smallest numerator over the largest denominator and vice versa. Column headings report medians over the 13 complete checkpoints. The dotted horizontal line separates dense and MoE models.

Figure 4. Inference performance across concurrency levels for four representative checkpoints spanning dense and MoE architectures, four weight precisions and the extremes of the PP=2/TP=2 peak throughput ratio. Rows show output throughput, median request-mean ITL on a logarithmic scale with strict (100 ms) and loose (200 ms) thresholds and goodput under the strict (solid) and loose (dashed) tiers. Lines show means across three independent launches, with bands indicating replicate ranges. Grey dashed lines with open markers represent two independent one-Spark replicas, derived from single-Spark measurements at half the total concurrency (§ 5-F). Annotations report peak throughput ratios relative to one Spark. Results for all checkpoints appear in Appendix A.

Throughput

Two Sparks under TP=2 increase peak throughput across all complete checkpoints by 1.36–1.68× (median 1.59×) relative to one Spark, compared with 1.17–1.65× (median 1.39×) under PP=2. TP=2 achieves the higher peak on 12 of the 13 checkpoints. The exception is Qwen3.6-35B-A3B NVFP4, where PP=2 reaches 1.02× the TP=2 peak, although their difference at c=64c=64 is undecided. Across 96 matched checkpoint and concurrency comparisons, Welch intervals place TP=2 above PP=2 in 93 cases, with none favoring PP=2. The remaining three comparisons are undecided, all at c=64c=64. The peak throughput ratio of PP=2 to TP=2 ranges from 0.74–1.02× (median 0.93×), with the largest gap on gpt-oss-120b (0.74×).

The difference between parallel strategies is most pronounced at low concurrency (Figure 4, top row). For Llama-3.1-8B BF16 at c=1c=1, TP=2 achieves 25.0 tok/s compared with 13.2 tok/s on one Spark, as both GPUs read their respective weight partitions concurrently. In contrast, PP=2 delivers 13.1 tok/s, approximately matching one Spark because its stages execute sequentially for a single request. As concurrency increases, pipeline stages can overlap execution across requests, improving throughput. However, one Spark remains separated above PP=2 in six of 78 comparisons, all at c≤4c \le4.

Throughput continues increasing through c=64c=64 for 43 of 47 deployment curves. The exceptions are three one-Spark curves (Llama-3.1-8B BF16, Qwen3.6-27B FP8, and Qwen3.6-35B-A3B NVFP4) and one TP=2 curve (Llama-3.1-8B BF16), which peak at c=32c=32. These declines are not attributable to KV-cache exhaustion: no requests were preempted and maximum observed cache utilization was 68 % (Qwen3.6-27B BF16 on one Spark at c=64c=64). Neither parallel strategy achieves twice the one-Spark peak throughput within the tested range, with the largest observed gain reaching 1.68×.

The narrowing throughput gap between TP=2TP=2 and PP=2PP=2 at higher concurrency is consistent with reduced pipeline bubbles. At low concurrency, PP=2PP=2 leaves one stage idle while the other executes, limiting GPU utilization. As more requests enter the pipeline, the stages can process different batches concurrently, reducing idle time and improving throughput. In contrast, TP=2TP=2 distributes each layer across both GPUs even at c=1c=1, providing an advantage at low load. Although pipeline utilization improves with concurrency, TP=2TP=2 retains higher throughput in most measured configurations.

Finding 1. Over RDMA, TP=2TP=2 increases peak throughput across all complete checkpoints and is never separated below PP=2PP=2 at any tested concurrency. PP=2PP=2 provides smaller gains, primarily as concurrency increases enough to overlap pipeline stages.

Decode latency

At c=1c=1, inter-token latency (ITL) approximates the time of a single decode step without batching. TP=2TP=2 reduces ITL across all 13 complete checkpoints to 0.52–0.73× of the one-Spark baseline (median 0.58×), with every difference separated. The improvement depends on model architecture. Dense checkpoints achieve 0.52–0.58×, approaching the expected benefit of halving per-node weight traffic. MoE checkpoints achieve 0.60–0.73×, as their smaller active weight footprint makes communication and other overheads relatively more significant (§ 5-H). Similarly, the benefit decreases as Llama-3.1-8B weights are quantized, with ITL ratios of 0.52, 0.55 and 0.58× at BF16, FP8 and NVFP4, respectively.

In contrast, PP=2PP=2 leaves unloaded ITL nearly unchanged at 0.99–1.03× of one Spark, with a maximum penalty of 3.8 ms. This ordering persists under load (Figure 4 and Figure 13). Across 96 matched checkpoint and concurrency comparisons, TP=2TP=2 has separated lower ITL in 90, while PP=2PP=2 leads in none. The remaining six are undecided. At higher concurrency, PP=2 can nevertheless reduce ITL relative to one Spark by distributing requests across pipeline stages. For example, at c=64c=64, Llama-3.1-8B BF16 achieves 311 ms under PP=2 compared with 521 ms on one Spark, despite no improvement in single-request decode latency.

Figure 13. Median request-mean inter-token latency (ITL) versus concurrency on a shared logarithmic scale. Horizontal reference lines indicate the strict (100 ms) and loose (200 ms) latency thresholds. Where measured, TP=2 consistently reduces ITL relative to one Spark, while PP=2 provides little improvement at low concurrency but benefits from increasing load.

Unloaded ITL on one Spark ranges from 14 ms (Qwen3.6-35B-A3B NVFP4) to 229 ms (Qwen3.6-27B BF16). Three dense checkpoints exceed the 100 ms strict-tier threshold: Qwen3.6-27B at BF16 and FP8 and Gemma-4-31B. TP=2 brings Qwen3.6-27B FP8 (70 ms) and Gemma-4-31B (84 ms) below this threshold. Qwen3.6-27B BF16 remains above it under all deployments, reaching 123 ms under TP=2.

Finding 2. TP=2 reduces unloaded ITL to 0.52–0.58× of one Spark on dense checkpoints and 0.60–0.73× on MoE checkpoints. It consistently outperforms PP=2, which provides negligible unloaded latency improvement.

Prefill

At c=1c=1, TTFT scales approximately linearly with prompt length across the four workload buckets, with R2≥0.986R^2 \ge0.986 for

every curve. We estimate prefill rate as the reciprocal of the fitted slope. Relative to one Spark, TP=2 increases prefill rate by 1.29–1.70× (median 1.49×), while PP=2 achieves 1.48–1.68× (median 1.62×). Prefill is the only metric for which PP=2 achieves a larger median gain.

The two strategies improve prefill through different mechanisms (Figure 5). Under PP=2, TTFT remains near the one-Spark baseline for short prompts, at 0.95–1.01× for 128 tokens and 1.00–1.10× for 1,024 tokens. For longer prompts, it falls to 0.76–0.84× at 4,096 tokens and 0.63–0.69× at 8,192 tokens, with little variation across checkpoints. This behavior is consistent with pipeline stages overlapping work on different prefill chunks, as described in SARATHI [13].

Figure 5. Median time to first token (TTFT) versus prompt length at c=1c=1 for two dense checkpoints. Points show means across three launches, with whiskers indicating replicate ranges. PP=2 reduces TTFT for prompts spanning multiple prefill chunks, while TP=2 accelerates long prompts but introduces overhead for short prompts on Llama-3.1-8B.

Under an idealized two-stage pipeline, where each stage takes half the single-Spark processing time per chunk, a prompt divided into kk chunks completes in (k+1)/(2k)(k+1)/(2k) of the single-Spark time. This predicts ratios of 1, 0.75 and 0.625 for one, two and four chunks, respectively. With vLLM’s 2,048-token budget (Section IV), these correspond closely to the observed prompt-length dependence.

Under TP=2, both GPUs process each prefill chunk concurrently, reducing TTFT for 8,192-token prompts to 0.60–0.82× of the one-Spark baseline. However, collective communication can outweigh the computational savings for short prompts. On Llama-3.1-8B, TP=2 adds 24–47 ms for 128-token prompts. At NVFP4, TTFT increases from 37 ms on one Spark to 83 ms under TP=2. NCCL microbenchmarks provide a possible explanation: a 128-token prompt with hidden size 4096 produces 1 MiB all-reduce payloads, requiring 593–658 μ\mus per collective across the measured 1–8 MiB range (Figure 10). For Llama-3.1-8B, 64 such collectives imply approximately 41 ms of communication, comparable to its entire one-Spark 128-token prefill time (37–86 ms across precisions).

Figure 10. NCCL communication performance over RDMA (solid) and TCP sockets (dashed). Top: minimum operation latency across message sizes. Bottom: effective bus bandwidth. Orange curves show all-reduce, the primary TP=2 communication primitive, and green curves show send/recv, used by PP=2. Bands indicate variation across runs. The shaded region marks decode-size payloads from 8 KiB to 1 MiB. RDMA substantially reduces all-reduce overhead in this range. TCP measurements used one cable and a different worker node.

Across the full workload mix at cc=1, TP=2 reduces median TTFT to 0.60–0.99× of one Spark, while PP=2 remains at 0.99–1.10×. The median request contains a 1,024-token prompt, which does not benefit from pipeline overlap. At cc=64, both strategies reduce median TTFT to approximately two-thirds of the one-Spark value (median 0.65× for TP=2 and 0.66× for PP=2, Figure 15).

Figure 15. Median time to first token (TTFT) versus concurrency on a shared logarithmic scale. The horizontal reference line indicates the 3 s TTFT objective.

Finding 3. PP=2 improves prefill primarily for prompts spanning multiple chunks, where pipeline stages can overlap execution. TP=2 accelerates long-prompt prefill but can increase TTFT for short prompts because of collective communication overhead.

Goodput

Goodput captures the tradeoff between throughput and latency (Figure 4, bottom row and Figure 14). Every deployment that reaches the strict tier exhibits a maximum followed by a decline to zero at higher concurrency. At cc=64, none of the 47 curves achieves nonzero strict-tier goodput. At the first concurrency level where goodput vanishes, the median ITL ranges from 107–173 ms across the 38 qualifying curves, while median TTFT exceeds 3 s in only two cases. Thus, the goodput collapse is primarily associated with ITL violations.

Figure 14. Goodput versus concurrency under strict (100 ms, solid) and loose (200 ms, dashed) ITL objectives, with TTFT limited to 3 s. Solid curves include replicate ranges. Zero goodput indicates that no requests satisfied both latency objectives, not that inference stopped. Panels use independent vertical scales.

TP=2 achieves the highest maximum strict-tier goodput on all 12 complete checkpoints that reach the objective. Relative to one Spark, its gain ranges from 1.39–6.62× (median 2.00×), compared with 1.08–1.77× (median 1.23×) for PP=2, considering checkpoints with nonzero one-Spark goodput. Two checkpoints (Qwen3.6-27B FP8 and Gemma-4-31B NVFP4) reach the strict tier only under TP=2. This advantage follows from its lower decode latency (§ 5-B), which allows the median ITL to remain below 100 ms for at least one additional concurrency level relative to PP=2 on 7 of 13 checkpoints and relative to one Spark on 11. TP=2 also achieves the highest loose-tier goodput on 12 of 13 checkpoints.

At the concurrency where TP=2 reaches its maximum goodput, its advantage over PP=2 is separated on 10 of 12 checkpoints. The remaining two (Llama-3.1-8B FP8 and Nemotron-3.5-Lightning-30B-A3B NVFP4) exhibit high variability because PP=2 replicates straddle the 100 ms ITL threshold. Comparing each deployment at its own measured optimum yields separated margins on all 12 checkpoints.

For models that fit on one Spark, we also compare against two derived independent replicas. Under ideal load balancing, their maximum goodput is twice the one-Spark maximum, since all single-node maxima occur at c≤32c \le32. Among checkpoints with nonzero one-Spark goodput, TP=2 is separated above replication on four of ten checkpoints (Llama-3.1-8B BF16, Qwen3.6-27B NVFP4, Qwen3.6-35B-A3B BF16, and gpt-oss-120b MXFP4), below on four and undecided on two (Llama-3.1-8B FP8 and Nemotron-3.5-Lightning-30B-A3B NVFP4). TP=2 additionally enables nonzero strict-tier goodput on two checkpoints where replication cannot meet the objective. Mistral-Medium-3.5-128B meets neither tier under either split, with unloaded ITL of 215 ms under TP=2 and 423 ms under PP=2.

Finding 4. TP=2 achieves the highest strict-tier goodput wherever both splits are supported and the objective is attainable. It also enables two dense checkpoints to meet the strict tier. Compared with two derived replicas, TP=2 leads on six of twelve qualifying checkpoints, trails on four and remains undecided on two.

Energy and power

GPU energy per generated token generally decreases with concurrency as throughput grows faster than power consumption (Figure 6). From cc=1 to cc=64, single-Spark throughput increases by 3.7–15×, while GPU power increases by only 1.2–1.8×. Energy per token decreases through cc=32 on every curve, although six curves show a slight increase at cc=64. At matched concurrency, one Spark consumes less GPU energy per token than either two-Spark deployment across all 78 comparisons with TP=2 and 78 comparisons with PP=2, with every margin separated.

Figure 6. GPU energy per generated token versus output throughput for the checkpoints in Figure 4. Each point represents one concurrency level, connected in order from c=1c=1 to c=64c=64, with whiskers showing replicate ranges. Grey dashed curves represent two derived one-Spark replicas, providing twice the single-Spark throughput at the same per-node concurrency and energy per token. Vertical dashed lines mark single-Spark peak throughput. One Spark provides the lowest GPU energy per token wherever its throughput is sufficient.

At cc=64, TP=2 consumes a median 1.18× the single-Spark energy per token (range 1.09–1.31×), compared with 1.19× under PP=2 (range 1.05–1.33×). The difference is more pronounced at cc=1: TP=2 consumes a median 1.15× the single-Spark energy per token, compared with 1.51× for PP=2. Tensor parallelism offsets some of the additional GPU power through faster decoding, whereas pipeline parallelism provides little unloaded latency improvement. Across 96 matched comparisons, TP=2 has separated lower energy per token in 89, while PP=2 leads in four, all at c=64c=64. The remaining three comparisons are undecided. Nevertheless, PP=2 achieves a lower minimum energy per token on three checkpoints (Table V).

CheckpointMin. J/token 1×Min. J/token TPMin. J/token PPRatio TP/1×Ratio PP/1×GPU power at c=64c=64 (W) 1×GPU power at c=64c=64 (W) TPGPU power at c=64c=64 (W) PP
Llama-3.1-8B BF160.470.530.551.121.1663115111
Llama-3.1-8B FP80.230.270.271.181.2073125119
Llama-3.1-8B NVFP40.170.200.211.181.1967107107
Qwen3.6-27B BF161.111.291.231.151.1071136114
Qwen3.6-27B FP80.770.870.831.121.0762121111
Qwen3.6-27B NVFP40.911.001.061.101.1773132114
Gemma-4-31B NVFP40.740.860.871.151.1770136115
Mistral-Medium-3.5-128B NVFP4–2.182.22–––142115
Qwen3.6-35B-A3B BF160.290.340.371.181.28458779
Qwen3.6-35B-A3B FP80.250.300.311.221.24489086
Qwen3.6-35B-A3B NVFP40.220.260.251.191.13539792
Gemma-4-26B-A4B NVFP40.17–0.23–1.3265–107
Nemotron-3.5-Lightning-30B-A3B NVFP40.190.220.251.201.3264110101
Qwen3-Next-80B-A3B NVFP40.200.260.261.311.33569995
Qwen3-Next-80B-A3B BF16–0.420.46–––9181
gpt-oss-120b MXFP40.360.410.471.151.3267122103
Qwen3-235B-A22B NVFP4–0.921.01–––115108

Table V. Minimum GPU energy per generated token across tested concurrency levels, relative energy consumption compared with one Spark and mean GPU power at c=64c=64. Power measurements are summed across both nodes for two-Spark deployments.

Minimum energy per token varies substantially across checkpoints, from 0.17 J for Gemma-4-26B-A4B NVFP4 on one Spark to 2.22 J for Mistral-Medium-3.5-128B NVFP4 under PP=2, a 13× difference (Figure 16 and Table V). These measurements include GPU energy reported by nvidia-smi, summed across both nodes for distributed deployments. CPU, networking and power-supply energy are excluded.

Figure 16. GPU energy per generated token versus concurrency on a shared logarithmic scale. Two-Spark measurements include both GPUs. Open markers indicate the minimum energy per token for each deployment curve.

Finding 5. One Spark minimizes GPU energy per generated token at matched concurrency, while two-Spark deployments provide higher throughput at an energy cost. At c=64c=64, TP=2 and PP=2 consume a median 1.18× and 1.19× the single-Spark energy per token, respectively.

Splitting against replicating

Models that fit on one Spark can alternatively be deployed as two independent replicas, with requests distributed between them. We derive this baseline from single-Spark measurements, assuming perfect load balancing without routing overhead, independent nodes and identical performance on both Sparks. At total concurrency cc, the replica pair therefore achieves twice the one-Spark throughput at c/2c/2, while retaining the corresponding latency and GPU energy per token. This comparison is available for total concurrency from 8 to 64. Figure 4 and Figure 6 show the derived baseline alongside the measured parallel deployments.

The preferred strategy changes with concurrency. At c=8c=8, TP=2 achieves higher throughput than replication on 10 of 13 complete checkpoints, delivering 0.95–1.55× of the replica pair’s throughput across all checkpoints. It also achieves lower ITL on 10 checkpoints and lower GPU energy per token on 10. At c=64c=64, replication delivers higher throughput on 11 checkpoints, with TP=2 achieving 0.75–1.05× of the replica baseline. Replication also achieves lower ITL on 12 checkpoints and lower energy per token on 12. TP=2 remains ahead in throughput only on gpt-oss-120b MXFP4, while Gemma-4-31B NVFP4 is undecided.

PP=2 is never separated above replication in throughput, latency, or energy efficiency at any evaluated load, reaching at most 0.92× of the replica pair’s throughput. These results reflect the tradeoff between accelerating individual requests and serving independent batches. At low concurrency, TP=2 reduces decode latency by distributing each layer across both GPUs. At high concurrency, replication processes separate batches without the per-layer synchronization required by TP=2.

Finding 6. For models that fit on one Spark, TP=2 generally favors low-concurrency serving, while independent replication offers higher throughput and lower latency at high concurrency. PP=2 provides no separated advantage over replication in the evaluated metrics.

Checkpoints run only on two Sparks

Three checkpoints were evaluated only on two Sparks. Qwen3-Next-80B-A3B BF16 and Qwen3-235B-A22B NVFP4 exceed the memory capacity of one Spark, while Mistral-Medium-3.5-128B NVFP4 requires 84 GiB for weights, leaving limited KV-cache capacity within the configured 0.80 memory fraction.

Across these checkpoints, PP=2 achieves 0.79–0.86× of the TP=2 peak throughput, with 1.52–1.97× its unloaded ITL (Table III). The largest throughput gap occurs on Mistral-Medium-3.5-128B, where TP=2 delivers 65.1 tok/s compared with 51.5 tok/s under PP=2, with unloaded ITL of 215 and 423 ms, respectively. Under TP=2, both GPUs process their weight partitions concurrently, whereas PP=2 executes its stages sequentially at c=1c=1. Of the two checkpoints reaching the strict-tier objective, TP=2 achieves higher maximum goodput, with a separated margin on Qwen3-235B-A22B NVFP4 and an undecided margin on Qwen3-Next-80B-A3B BF16. Thus, TP=2 is the preferred split for all three checkpoints based on throughput and decode latency.

Beyond supporting larger models, two Sparks increase KV-cache capacity by 2.09–4.38× under TP=2 and 1.89–4.11× under PP=2 across the 13 complete checkpoints (Table II). Most gains exceed twofold because model weights are partitioned across nodes, leaving more aggregate memory for the KV cache. This additional capacity was not a limiting factor in our workload, where maximum cache utilization reached only 68% (§ 5-A). Workloads with longer contexts or higher concurrency could benefit more substantially.

Model-side levers

Precision. Figure 7 compares three model families at BF16, FP8 and NVFP4. Across all three, reducing precision from BF16 to NVFP4 on one Spark lowers unloaded ITL to 0.35–0.40× of the BF16 value, a larger improvement than adding a second Spark under TP=2 at BF16 (0.52–0.60×). The impact on peak throughput varies by model. Quantization improves throughput by 2.97× for Llama-3.1-8B, 1.59× for Qwen3.6-35B-A3B and 1.26× for Qwen3.6-27B, compared with respective BF16 TP=2 gains of 1.66, 1.65 and 1.65×. The benefits are complementary: at NVFP4, TP=2 provides additional throughput gains of 1.36, 1.51 and 1.63×, respectively.

Figure 7. Effect of weight precision across three model families at BF16, FP8 and NVFP4. Top: peak output throughput. Bottom: unloaded median request-mean ITL at c=1c=1. Tick labels indicate the served KV-cache precision, which changes across some ModelOpt checkpoints. The Qwen3.6 dense and MoE models also provide an architectural comparison at matched weight precision.

Qwen3.6-27B is an exception to the expected throughput improvement from lower precision. Its NVFP4 checkpoint achieves 81.1 tok/s, comparable to FP8 (81.6 tok/s), despite reducing unloaded ITL from 130 to 84 ms. Its prefill rate also falls to 1,129 input tokens/s, compared with 1,975 for FP8 and 1,248 for BF16. The checkpoint uses weight-only NVFP4 (W4A16_NVFP4), executed through vLLM’s Marlin kernel with 16-bit activations (Appendix B). This execution path is consistent with improved memory-bound decoding but slower compute-intensive prefill. However, the MoE checkpoint using the same kernel path benefits from NVFP4, so kernel selection alone does not establish the cause.

Some ModelOpt checkpoints also change KV-cache precision from BF16 to FP8 (Figure 7). Consequently, these comparisons reflect the released checkpoints as served rather than isolated changes in weight precision.

Sparsity. At matched weight precision, the MoE Qwen3.6-35B-A3B achieves 2.4–3.0× the peak throughput of dense Qwen3.6-27B on one Spark, with 6.2–6.8× shorter unloaded ITL. Similarly, Gemma-4-26B-A4B achieves 381 tok/s compared with 93.7 tok/s for Gemma-4-31B. These comparisons illustrate the serving advantages associated with sparse expert activation, although differences in model size and training prevent attributing the gains solely to sparsity.

Figure 8 relates unloaded decode latency to memory traffic. Dense checkpoints on one Spark require 1.12–1.27× the theoretical time to read their weights once at 273 GB/s, corresponding to an effective weight-read bandwidth of 216–244 GB/s. Additional latency includes KV-cache accesses, computation and other overheads. Under TP=2, each GPU reads approximately half the weights concurrently, yielding 1.24–1.41× the corresponding weight-read bound. Under PP=2, stages execute sequentially at c=1c=1, producing 2.08–2.54× the per-node weight-read reference. MoE checkpoints require only 0.11–0.52× of the full-weight reference time because only active experts are accessed. Their reduced weight traffic also explains why TP=2 provides smaller unloaded latency gains for MoE than dense checkpoints (§ 5-B).

Figure 8. Unloaded decode latency (pooled median token gap) versus resident weight footprint per node. The dashed line represents the time to read the weights once at the nominal 273 GB/s memory bandwidth. Dotted and dash-dotted lines scale this reference by the median factors observed for dense models under one Spark and TP=2 (1.25×1.25 \times) and PP=2 (2.40×2.40 \times), respectively. MoE checkpoints (open markers) fall below the full-weight reference because only active experts are accessed during decoding.

Finding 7. Reducing precision from BF16 to NVFP4 improves unloaded latency more than adding a second Spark under TP=2 across all three precision ladders and the benefits combine. Peak throughput gains range from 1.26× to 2.97×, with the smallest gain associated with a weight-only NVFP4 checkpoint served through the Marlin kernel.

Run-to-run variation

Figure 9 and Table IV quantify run-to-run variability across three independent launches. Median CVs are 0.4% for throughput, 0.4% for ITL, 0.5% for TTFT and 0.5% for energy per token. The 90th percentile throughput CV is 2.1%, with only seven points exceeding 5%. In each outlier, two launches agree within 0.4%, while the third differs by 9–19%. Two Qwen3-235B-A22B outliers originate from the same slow launch. Two others, Gemma-4-31B NVFP4 and Qwen3.6-27B FP8 at c=64c=64 on one Spark, exhibit one faster launch despite identical logged kernel selections. The slower launches consumed less power without higher temperatures, suggesting that thermal throttling was not responsible.

MetricMedian90th pct.Maxn>5%n > 5\%
Output throughput0.382.1310.27
Median ITL0.392.0410.49
Median TTFT0.543.3310.615
Strict goodput0.503.6786.612
Loose goodput0.553.86173.217
Energy per token0.511.445.31
GPU power0.511.618.21

Table IV. Replicate coefficient of variation (CV), defined as standard deviation divided by mean, across 282 measurement points. Values report the median, 90th percentile, maximum and number of points exceeding 5% CV.

Figure 9. Variability across 282 measurement points. Left: distribution of the coefficient of variation (CV) across three independent launches for each metric, with strict goodput restricted to points where it is nonzero. Right: throughput CV by concurrency and deployment, with a 5 % reference line.

Strict-tier goodput exhibits greater variability (maximum CV 86.6%) because small latency differences near the ITL threshold can cause goodput to drop to zero. We therefore report confidence intervals for goodput comparisons (§ 5-D). Despite this variability, the main throughput ordering remains consistent: TP=2 is separated above one Spark in all 78 of 78 matched comparisons, while PP=2 is separated above it in 60 and below it in six, with the latter occurring only at c≤4c \le4.

The Interconnect: From TCP to RDMA

Our initial DGX Spark experiments used NCCL over TCP sockets (NET/Socket). At the time, the out-of-box software configuration did not automatically provision the ConnectX-7 RoCEv2 network for our workload and NCCL consequently fell back to TCP. NVIDIA subsequently introduced updated NCCL support and the NVIDIA Sync Cluster Assistant to simplify multi-node network configuration. In the present campaign, we configured all four RoCEv2 rails and verified that every dual-node launch used NCCL’s RDMA transport (NET/IB, Section IV).

The earlier TCP campaign favored PP=2 over TP=2, consistent with conventional multi-node guidance. In contrast, TP=2 outperforms PP=2 in the present RDMA campaign. We compare the two campaigns and use NCCL microbenchmarks to investigate how reduced communication overhead contributes to this reversal.

Communication overhead

Figure 10 compares the communication primitives used by both parallel strategies. Within the 8 KiB to 1 MiB decode payload range, RDMA provides 2.4–9.0× the all-reduce bandwidth of TCP. For an 8 KiB message, the mean minimum all-reduce latency falls from 143 μ\mus over TCP to 27 μ\mus over RDMA.

This reduction is particularly important for TP=2, which requires approximately two all-reduces per transformer layer during decoding. Llama-3.1-8B performs 64 such operations per step, corresponding to an estimated collective cost of 9.1 ms over TCP versus 1.7 ms over RDMA. The estimated saving of 7.4 ms per token is comparable to the observed reduction of 11, 7 and 10 ms across its BF16, FP8 and NVFP4 checkpoints.

Although send/recv remains more communication-efficient than all-reduce, TP=2 benefits from both GPUs reading their weight partitions concurrently, whereas PP=2 processes its stages sequentially during single-request decoding (§ 5-H). With lower collective overhead, this parallel memory access advantage becomes more pronounced.

Cross-campaign comparison

Figure 11 compares the three Llama-3.1-8B checkpoints measured in both campaigns. Relative to TCP, TP=2 peak throughput increases by 1.68× at BF16, 2.37× at FP8 and 2.58× at NVFP4. In contrast, PP=2 improves by only 1.04–1.07×, while one-Spark throughput remains within approximately 1% of its earlier value.

Figure 11. Performance comparison of the July 2026 TCP campaign (open markers) and the present RDMA campaign (filled markers) across three Llama-3.1-8B precision variants. Columns show peak throughput, unloaded ITL and TTFT at c=1c=1. Annotations give RDMA-to-TCP ratios. One-Spark results provide a control unaffected by inter-node communication. The campaigns also differ in software versions, worker node, cabling and measurement methodology.

The largest relative improvement occurs in TP=2 time to first token. At cc=1, TTFT decreases by 231–246 ms, reaching 0.28–0.40× of its TCP value. For the median 1,024-token prompt, the NCCL microbenchmarks predict approximately 207 ms of collective communication savings across the 64 all-reduces, accounting for most of the observed reduction. These results are consistent with communication overhead being a major factor in the reversal between TP=2 and PP=2.

Finding 8. Between the TCP and RDMA campaigns, TP=2 peak throughput increases by 1.68–2.58×, substantially more than PP=2. NCCL microbenchmarks show that reduced collective latency can explain much of the improvement in TP=2 inference performance, supporting the interconnect as a major contributor to the reversal in parallelism performance.

Discussion

Choosing a deployment

Our measurements suggest the following deployment guidelines for RDMA-connected DGX Spark systems running the evaluated vLLM configuration.

Models requiring two Sparks: Prefer TP=2. For the three checkpoints that could not be served on one Spark under our configuration, TP=2 delivers higher peak throughput and lower unloaded latency. Across these checkpoints, PP=2 achieves only 0.79–0.86× of TP=2’s peak throughput, with 1.52–1.97× its unloaded ITL (§ 5-G).

Latency-sensitive or low-concurrency workloads: Prefer TP=2. For models that fit on one Spark, TP=2 reduces unloaded ITL to 0.52–0.73× of the single-Spark baseline. Unlike replication, which increases aggregate serving capacity without accelerating individual requests, TP=2 improves single-request decoding. At low concurrency, it also outperforms two derived replicas in throughput on most checkpoints (§ 5-F). For Qwen3.6-27B FP8 and Gemma-4-31B NVFP4, this latency improvement enables strict-tier goodput that neither one Spark nor PP=2 achieves.

High-concurrency workloads: Prefer replication. When the model fits on one Spark, independent replicas generally provide better throughput at high concurrency. At cc=64, the derived replica pair outperforms TP=2 on eleven of the thirteen complete checkpoints, generally with lower latency and GPU energy per token. Replication also provides higher throughput than PP=2 across the evaluated checkpoints. These comparisons assume ideal load balancing and identical node performance.

Energy-constrained workloads: Consider one Spark. A single Spark consistently consumes less GPU energy per generated token at matched concurrency than either distributed deployment (§ 5-E). When its throughput and latency satisfy the workload requirements, adding a second Spark offers no GPU-energy-efficiency advantage. Total system energy, however, was not measured.

When to consider PP=2. Pipeline parallelism offers limited advantages in our workload. For models fitting on one Spark, it never achieves a separated advantage over the derived replica pair in throughput, latency, or energy efficiency. For larger models, TP=2 generally performs better. However, PP=2 provides faster prefill for sufficiently long prompts: at 8,192 tokens and c=1c=1, it achieves lower TTFT than TP=2 on ten of the thirteen complete checkpoints. These measurements contain relatively few long-prompt requests. Thus, workloads dominated by long prompts at low concurrency, or models lacking TP=2 kernel support, may favor PP=2.

Quantize first, when model quality permits. Across the three precision ladders, moving from BF16 to NVFP4 reduces unloaded latency more than adding a second Spark under TP=2 and the improvements are complementary (§ 5-H). However, quantization does not guarantee higher throughput. Qwen3.6-27B NVFP4 illustrates how weight-only execution through the Marlin kernel can improve decode latency while limiting prefill and throughput gains. Precision variants should therefore be benchmarked under the intended workload before deployment.

Limitations

Scope. The study evaluates two GB10 systems, pipeline depth two and a single vLLM version. The synthetic workload uses fixed-length greedy decoding, closed-loop concurrency, disabled prefix caching and approximately 10 prompt tokens per generated token. Different workload mixes may change the relative benefits of prefill and decode optimizations. Model quality under quantization was not evaluated.

Derived replication baseline. Two independent replicas were modeled using single-Spark measurements, assuming identical node performance, perfect load balancing and no routing overhead. Actual replica performance may differ.

Measurement and generalizability. Concurrency was limited to c=64c=64, where most throughput curves were still increasing. Reported peaks therefore represent maxima within the tested range rather than saturation throughput. Statistical comparisons use three independent launches per deployment without correction for multiple testing. GPU energy measurements exclude CPU, networking and other system components. NCCL transfers were staged through host memory because GPUDirect RDMA is unsupported on GB10.

Checkpoint coverage. Two measured checkpoints were excluded from the analysis (Section IV-C) and TP=2 could not be evaluated for Gemma-4-26B-A4B due to unsupported kernels in the tested vLLM version.

Serving systems and parallelism. vLLM introduced Page-dAttention for efficient KV-cache management [11], building on Orca’s iteration-level scheduling [12]. SARATHI introduced chunked prefill [13], while DistServe formalized goodput as a serving objective [14]. Megatron-LM [15] and GPipe [16] established tensor and pipeline parallelism, with subsequent work recommending tensor parallelism within servers and pipeline parallelism across servers [1]. Pope et al. examine latency-throughput tradeoffs in inference partitioning [6], while Xu et al. characterize distributed inference communication patterns [39]. Our work extends these investigations to desktop-scale GPU systems with unified memory and RDMA-connected nodes, where inter-node communication and memory bandwidth affect the relative benefits of parallelism strategies.

Inference benchmarking. MLPerf Inference establishes standardized performance scenarios and accuracy requirements [40]. LLM-Inference-Bench evaluates inference performance across accelerators and serving frameworks [41], while Etalon highlights the limitations of aggregate latency metrics for interactive serving [42]. Samsi et al. investigate LLM inference energy consumption on datacenter GPUs [43]. We complement these studies by evaluating throughput, latency, SLO-based goodput and GPU energy efficiency across replicated single- and dual-node experiments.

DGX Spark evaluations. Existing evaluations include single-node performance studies [44], [45] and a two-Spark cluster review [46]. The cluster review compares TP=2TP=2 and PP=2PP=2 on Llama-3.1-8B and gpt-oss-120b using a single 200 GbE connection. For gpt-oss-120b, it reports PP=2PP=2 achieving 2.20×2.20\times the TP=2TP=2 throughput at batch size 128 with equal input and output lengths, whereas TP=2TP=2 leads at lower batch sizes on Llama-3.1-8B. These findings differ from our RDMA results, where TP=2TP=2 leads on gpt-oss-120b throughout the evaluated concurrency range. However, differences in workloads, concurrency, software configurations and unspecified network transport prevent direct attribution of this discrepancy. Our study extends prior evaluations through a broader checkpoint matrix, three independent server launches per configuration, verified NCCL transport and comparisons of model splitting with derived independent replication, including goodput and GPU energy measurements.

Conclusion

We systematically evaluated LLM inference on one and two RDMA-connected DGX Spark GB10 systems across 17 model checkpoints. Contrary to conventional multi-node guidance, tensor parallelism (TP=2TP=2) consistently outperforms pipeline parallelism (PP=2PP=2) in throughput, decode latency and goodput. Across the 13 complete checkpoints, TP=2TP=2 increases median peak throughput to 1.59×1.59\times that of one Spark and reduces median unloaded inter-token latency to 0.58×0.58\times of the single-node baseline. In contrast, PP=2PP=2 achieves a median throughput gain of 1.39×1.39\times without improving unloaded decode latency, although it accelerates prefill for sufficiently long prompts.

The preferred deployment depends on workload characteristics. For models requiring two Sparks or workloads sensitive to decode latency, TP=2TP=2 provides the strongest overall performance. For models that fit on one Spark, TP=2TP=2 often outperforms two derived independent replicas at low concurrency, while replication delivers higher throughput on eleven of the thirteen complete checkpoints at c=64c=64. A single Spark remains the most GPU-energy-efficient configuration at matched concurrency. Weight quantization provides an additional, complementary means of improving inference performance.

Comparisons with an earlier TCP-based campaign and NCCL microbenchmarks suggest that reduced collective communication overhead contributes substantially to TP=2TP=2’s advantage over PP=2PP=2. Although other configuration changes prevent isolating RDMA’s contribution, the results demonstrate that parallelism strategies should be evaluated against the memory architecture, interconnect and workload rather than selected solely from conventional datacenter guidance.

Acknowledgments

This work was conducted at the First State AI Institute (FSAI) at the University of Delaware. This material is based upon work supported by the National Science Foundation under Grant No. 2417814 through the Democratizing Access to Research Software Engineers (DARSE) project. The authors thank ASUS for generously providing two DGX Spark systems used in this research.

Anthropic’s Claude was used to assist with code development and manuscript editing, including language polishing and improvements to clarity. All scientific claims, experimental results and manuscript content were thoroughly reviewed and verified by the authors, who retain full responsibility for the final work.

References

Additional Results

Figure 12–Figure 16 present the complete per-checkpoint results supporting § 5, using a consistent layout across metrics. The first three rows show the BF16, FP8 and NVFP4 precision ladders, with checkpoints requiring two Sparks placed last. Table V and Table VI provide additional numerical results for the energy and interconnect analyses.

CheckpointDeploymentPeak tok/s (c≤64c \le64) TCPPeak tok/s (c≤64c \le64) RDMAPeak tok/s (c≤64c \le64) ratioITL at c=1c=1 (ms) TCPITL at c=1c=1 (ms) RDMAITL at c=1c=1 (ms) ratio
Llama-3.1-8B BF161×1301311.0172721.01
TP=21302181.6848380.78
PP=21972041.0475730.97
Llama-3.1-8B FP81×3233210.9939411.04
TP=21974662.3729220.77
PP=24104391.0742420.99
Llama-3.1-8B NVFP41×3903901.0025261.01
TP=22055292.5825150.60
PP=24875221.0729260.90

Table VI. Performance comparison of the Llama-3.1-8B precision variants between the July 2026 TCP campaign and the present RDMA campaign (Figure 11). Ratios represent RDMA values divided by TCP values.

Figure 12. Output throughput versus concurrency across all checkpoints. Lines show means over three independent launches, with bands indicating replicate ranges. Annotations report peak throughput relative to one Spark, or the PP=2/TP=2 peak ratio for checkpoints evaluated only on two Sparks.

Two observations warrant attention. First, median TTFT decreases from c=32c=32 to c=64c=64 on 12 of 33 two-Spark curves (Figure 15), indicating non-monotonic latency behavior under increasing load. Second, at c=1c=1, the 8,192-token bucket contains fewer than ten requests per measurement window on 26 of 47 curves. These limited samples contribute to the wider replicate ranges observed for long-prompt measurements in Figure 5.

Configuration

Engine and cluster

Table VII summarizes the serving configuration shared across launches. vLLM used its V1 engine with torch.compile and the Inductor backend, CUDA graphs in FULL_AND_PIECEWISE mode and chunked prefill. KV-cache precision was selected automatically: fp8_e4m3 when specified by the checkpoint and the model data type otherwise. Both --max-num-seqs and --max-num-batched-tokens retained their engine defaults. Table VIII reports the quantization formats and kernel backends recorded during engine initialization.

Quantization kernels. The ModelOpt configurations of four checkpoints declare weight-only NVFP4 layers (W4A16_NVFP4). For three checkpoints (Qwen3.6-27B, Qwen3.6-35B-A3B, and Nemotron-3.5-Lightning-30B-A3B), vLLM logged Marlin as the only selected 4-bit kernel, including expert layers in MoE models. This kernel stores weights in 4-bit format while using 16-bit activations, rather than native FP4 computation. In contrast, five other NVFP4 checkpoints used the native FlashInfer CUTLASS implementation. One checkpoint (Mistral-Medium-3.5-128B NVFP4) declared weight-only layers but logged only the native kernel. These observations are based on kernel-selection logs rather than direct kernel profiling.

Marlin also emitted a warning that native FP4 computation was unsupported and that weight-only execution could reduce compute-intensive performance. This warning is generated by the kernel implementation and does not establish that the Blackwell GPU lacks native FP4 support.

Distributed execution and transport. Dual-node deployments used Ray’s distributed executor, with one container per node. NCCL debug logs identified four RoCEv2 interfaces:

NET/IB : Using [0]rocep1s0f0:1/RoCE
[1]rocep1s0f1:1/RoCE [2]roceP2p1s0f0:1/RoCE
[3]roceP2p1s0f1:1/RoCE

Per-channel logs reported NET/IB/<n> without the GDRDMA suffix. The transport audit verified 25,040 channel records across 148 dual-node launches. NCCL microbenchmark logs further confirmed that communication used all four RoCEv2 interfaces across both physical cables (§ 6).

Before each launch, the page cache was cleared on both nodes. Checkpoints were downloaded to the head node, copied to the worker, benchmarked and removed before proceeding to the next checkpoint.

Sampling and windows

GPU power and temperature were sampled using nvidia-smi at 10 samples/s on the head node and 1.0 samples/s on the worker. An in-container Ray actor handled worker sampling at a lower frequency to limit overhead. Maximum sampling gaps were 0.18 s and 1.04 s, respectively. GPU energy was calculated by interpolating power at measurement-window boundaries and integrating using the trapezoidal rule.

Maximum observed GPU temperatures were 88 °C on the head and 87 °C on the worker. The highest 99th-percentile end-to-end request latency across measurement windows was 717 s (Mistral-Medium-3.5-128B NVFP4, PP=2, cc=64) and the longest warmup lasted 846 s.

Provenance

Each measurement window records the engine configuration, pinned model and tokenizer revisions, container digest, driver and CUDA versions, GPU clocks, client Python environment and serving-policy fingerprint. SHA-256 digests link each summary to its raw request, power and metrics logs. The harness source is also fingerprinted and the campaign index rejects records with mismatched fingerprints.

A campaign manifest contains SHA-256 digests for 3,952 artifacts. Before generating results, the analysis scripts verify all 972 required index, summary, serving-policy and microbenchmark files against the manifest, terminating if any mismatch is detected.

Artifact

The benchmark harness, campaign index, per-window summaries, serving configurations, NCCL microbenchmarks and transport audit are publicly available at https://github.com/kevinhbhimani/dgx-spark-llm-serving-benchmark. The RDMA campaign is stored under results/RDMA-Betelgeuse-Suite-benchmark-2026-10-30-001. Raw request-level and GPU power logs are archived separately due to their size and are available from the corresponding author upon request. Reproducing the complete campaign requires exclusive access to two GB10 systems for approximately eleven days.

ItemValue
Container imagenvcr.io/nvidia/vllm (NGC release 26.09) on both nodes, digest sha256:557747337846eecf34486866f3d992383c6f8af869fdb0e365b5093fcaee02c2
EnginevLLM 0.29.0 (build 0.29.0+5013de39.nv26.9), PyTorch 2.14.0a0 (build nv26.9), CUDA runtime 13.4, Python 3.12.3
Kernel librariesFlashInfer 0.6.18.post1, flash-attn 2.8.3.post1
Cluster softwareRay 2.56.1, NCCL 2.31.2
GPU and driverNVIDIA GB10, driver 580.178.04, compute capability 12.1, application clock 2,418 MHz on both nodes
Head operating systemUbuntu 24.04.5 LTS, kernel 7.0.0-1019-nvidia (aarch64); the container runs Ubuntu 24.04.4 LTS
Engine flags, all launches--dtype auto, --max-model-len 8704, --gpu-memory-utilization 0.8, --generation-config vllm, --language-model-only, --no-enable-log-requests, --no-enable-prefix-caching, --no-enable-flashinfer-autotune, and the pinned --revision and --tokenizer-revision of the checkpoint
Two-Spark flagsTP=2: --tensor-parallel-size 2 --pipeline-parallel-size 1; PP=2: --tensor-parallel-size 1 --pipeline-parallel-size 2; both with --distributed-executor-backend ray
Per-checkpoint flag--trust-remote-code for the two Qwen3-Next checkpoints
Networkfour RoCEv2 rails per node, two on each physical ConnectX-7 port (enp1s0f0np0 and enP2p1s0f0np0 on port 0, enp1s0f1np1 and enP2p1s0f1np1 on port 1); one IPv4 /24 subnet per rail; MTU 4096
NCCL environmentNCCL_IB_DISABLE=0; NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME set to the four rails; NCCL_SOCKET_NTHREADS=1; NCCL_NSOCKS_PERTHREAD=1; NCCL_DEBUG=INFO; NCCL_DEBUG_SUBSYS=INIT,NET; NCCL_IB_HCA, NCCL_IB_GID_INDEX, and NCCL_NET_PLUGIN empty
Other environmentTORCHINDUCTOR_AUTOGRAD_CACHE=0
Workload(input, output) token buckets (128, 128), (1024, 256), (4096, 256), (8192, 512) with weights 1, 1, 1, 0.5; closed loop at c∈{1,4,8,16,32,64}c \in\{1, 4, 8, 16, 32, 64\}
Windowwarmup of at least 120 s; measurement of at least 600 s, extended in 60 s steps to at least 20 started requests (3,600 s cap); cooldown of at least 30 s
Harnessmethodology steady-exact-v4, source commit 9c14fab

Table VII. Serving configuration shared across launches, obtained from serving-policy records, cluster topology and benchmark launch scripts.

CheckpointQuantization4-bit linear kernelFP8 linear kernelMoE backendAttention
Llama-3.1-8B BF16none–––FlashAttention
Llama-3.1-8B FP8FP8–FlashInfer scaled MM–FlashInfer
Llama-3.1-8B NVFP4NVFP4FlashInfer CUTLASS––FlashInfer
Qwen3.6-27B BF16none–––FlashAttention
Qwen3.6-27B FP8block FP8–CUTLASS block-scaled–FlashAttention
Qwen3.6-27B NVFP4NVFP4+W4A16+FP8Marlin, weight-onlyFlashInfer scaled MM–FlashInfer
Gemma-4-31B NVFP4NVFP4FlashInfer CUTLASS–––
Mistral-Medium-3.5-128B NVFP4NVFP4+W4A16+FP8FlashInfer CUTLASSFlashInfer scaled MM–FlashInfer
Qwen3.6-35B-A3B BF16none–––FlashAttention
Qwen3.6-35B-A3B FP8block FP8–CUTLASS block-scaled–FlashAttention
Qwen3.6-35B-A3B NVFP4NVFP4+W4A16+FP8Marlin, weight-onlyFlashInfer scaled MMMarlinFlashAttention
Gemma-4-26B-A4B NVFP4NVFP4––FlashInfer CUTLASS–
Nemotron-3.5-Lithium-30B-A3B NVFP4NVFP4+W4A16+FP8Marlin, weight-onlyFlashInfer scaled MMMarlinFlashInfer
Qwen3-Next-80B-A3B NVFP4NVFP4FlashInfer CUTLASS–FlashInfer CUTLASSFlashInfer
Qwen3-Next-80B-A3B BF16none–––FlashAttention
gpt-oss-120b MXFP4MXFP4––MarlinFlashInfer
Qwen3-235B-A22B NVFP4NVFP4FlashInfer CUTLASS–FlashInfer CUTLASSFlashInfer

Table VIII. Checkpoint quantization formats and kernel backends reported during vLLM initialization. ModelOpt formats include NVFP4, FP8 and weight-only NVFP4 (W4A16), with + indicating mixed formats. Other checkpoints use block FP8 or MXFP4. A dash indicates that no corresponding kernel selection was logged.

References

  1. [1]D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on GPU clusters using Megatron-LM,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), ser. SC ’21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3458817.3476209
  2. [2]NVIDIA Corporation, “NVIDIA DGX Spark porting guide: CUDA,” https://docs.nvidia.com/dgx/dgx-spark-porting-guide/latest/porting/cuda.html, 2026, accessed: 2026-11-13.
  3. [3]——, “DGX Spark playbooks: Connect two Sparks,” https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/connect-two-sparks, 2026, accessed: 2026-11-13.
  4. [4]——, “DGX Spark user guide: Hardware overview,” https://docs.nvidia.com/dgx-spark/dgx-spark/hardware.html, 2025, accessed: 2026-11-13.
  5. [5]S. Williams, A. Waterman, and D. Patterson, “Roofline: An insightful visual performance model for multicore architectures,” Communications of the ACM, vol. 52, no. 4, pp. 65–76, Apr. 2009. [Online]. Available: https://doi.org/10.1145/1498765.1498785
  6. [6]R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” in Proceedings of Machine Learning and Systems 5 (MLSys), 2023, pp. 606–624.arxiv.org/abs/2211.05102
  7. [7]Z. Yuan, Y. Shang, Y. Zhang, Y. Zhou, Z. Dong, Z. Zhou, C. Xue, B. Wu, Z. Li, Q. Gu, Y. J. Lee, Y. Yan, B. Chen, G. Sun, and K. Keutzer, “LLM inference unveiled: Survey and roofline model insights,” 2024. [Online]. Available: https://arxiv.org/abs/2402.16363
  8. [8]NVIDIA Corporation, “DGX Spark user guide: ConnectX-7 networking,” https://docs.nvidia.com/dgx/dgx-spark/spark-clustering.html, 2025, accessed: 2026-11-13.
  9. [9]——, “NVIDIA Deep Learning NCCL documentation,” https://docs.nvidia.com/deeplearning/nccl/, 2026, accessed: 2026-11-13.
  10. [10]InfiniBand Trade Association, “Supplement to InfiniBand Architecture Specification Volume 1 Release 1.2.1, Annex A17: RoCEv2,” https://www.infinibandta.org/ibta-specification/, Sep. 2014.
  11. [11]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023, pp. 611–626.arxiv.org/abs/2309.06180
  12. [12]G.-I. Yu, Y. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022, pp. 521–538.
  13. [13]A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, “SARATHI: Efficient LLM inference by piggybacking decoded pages with chunked prefills,” 2023. [Online]. Available: https://arxiv.org/abs/2308.16369
  14. [14]Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024, pp. 193–210.arxiv.org/abs/2401.09670
  15. [15]M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan-zaro, “Megatron-LM: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019.arxiv.org/abs/1909.08053
  16. [16]Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, “GPipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems 32 (NeurIPS), 2019.arxiv.org/abs/1811.06965
  17. [17]P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu, “FP8 formats for deep learning,” 2022. [Online]. Available: https://arxiv.org/abs/2209.05433
  18. [18]E. Alvarez, O. Almog, E. Chung, S. Layton, D. Stosic, R. Krashinsky, and K. Aubrey, “Introducing NVFP4 for efficient and accurate low-precision inference,” NVIDIA Technical Blog, https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/, Jun. 2025, accessed: 2026-11-13.
  19. [19]OpenAI, “gpt-oss-120b & gpt-oss-20b model card,” 2025. [Online]. Available: https://arxiv.org/abs/2508.10925
  20. [20]NVIDIA Corporation, “NVIDIA Model Optimizer (ModelOpt),” https://github.com/NVIDIA/Model-Optimizer, 2025, formerly TensorRT Model Optimizer. Accessed: 2026-11-13.
  21. [21]N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1701.06538
  22. [22]W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022.arxiv.org/abs/2101.03961
  23. [23]B. L. Welch, “The generalization of ‘Student’s problem when several different population variances are involved,” Biometrika, vol. 34, no. 1–2, pp. 28–35, 1947.DOI
  24. [24]P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. Stoica, “Ray: A distributed framework for emerging AI applications,” in 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2018, pp. 561–577.
  25. [25]vLLM contributors, ““Bugfix][PPI][Spec Decode] MiniMax-M3 EA-GLE3 aux-state relay at PP > 1 and per-stage FlashInfer autotune,” vLLM pull request #57197, https://github.com/vllm-project/vllm/pull/57197, 2026, merged 2026-09-30, after the vLLM 0.29.0 release; first released in vLLM 0.32.0. Accessed: 2026-11-13.
  26. [26]——, “Bug: Synchronized FlashInfer autotune deadlocks multi-node TP startup on hosts without GPUDirect RDMA (GB10/DGX Spark),” vLLM issue #52291, https://github.com/vllm-project/vllm/issues/52291, 2026, opened 2026-08-14. Accessed: 2026-11-13.
  27. [27]A. Graffanori et al., “The Llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783
  28. [28]Qwen Team, “Qwen3.6-27B and Qwen3.6-35B-A3B model cards,” Hugging Face, https://huggingface.co/Qwen/Qwen3.6-27B and https://huggingface.co/Qwen/Qwen3.6-35B-A3B, 2026, accessed: 2026-11-13.
  29. [29]Gemma Team, “Gemma 4 technical report,” 2026. [Online]. Available: https://arxiv.org/abs/2607.02770
  30. [30]Google DeepMind, “Gemma 4 model cards (Gemma-4-31B-IT and Gemma-4-26B-A4B-IT),” Hugging Face, https://huggingface.com/google/gemma-4-31B-it and https://huggingface.com/google/gemma-4-26B-A4B-it, 2026, served from the NVIDIA ModelOpt NVFP4 checkpoints https://huggingface.com/nvidia/Gemma-4-31B-IT-NVFP4 and https://huggingface.com/nvidia/Gemma-4-26B-A4B-NVFP4. Accessed: 2026-11-13.
  31. [31]NVIDIA, “NVIDIA Nemotron-3.5-Lightning-30B-A3B-NVFP4 model card,” Hugging Face, https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, 2026, accessed: 2026-11-13.
  32. [32]Qwen Team, “Qwen3-Next-80B-A3B-Instruct model card,” Hugging Face, https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct, 2025, accessed: 2026-11-13.
  33. [33]Mistral AI, “Mistral Medium 3.5 128B model card,” Hugging Face, https://huggingface.co/mistralai/Mistral-Medium-3.5-128B; served from the NVIDIA ModelOpt NVFP4 checkpoint https://huggingface.co/nvidia/Mistral-Medium-3.5-128B-NVFP4, 2026, accessed: 2026-11-13.
  34. [34]A. Yang et al., “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388
  35. [35]NVIDIA, “Qwen3-235B-A22B-Instruct-2507-NVFP4 model card,” Hugging Face, https://huggingface.co/nvidia/Qwen3-235B-A22B-Instruct-2507-NVFP4, 2026, accessed: 2026-11-13.
  36. [36]——, “NVIDIA Nemotron 3: Efficient and open intelligence,” 2025. [Online]. Available: https://arxiv.org/abs/2512.20856
  37. [37]——, “Nemotron 3 Super: Efficient mixture-of-experts hybrid Mamba-Transformer model for agentic reasoning,” 2026. [Online]. Available: https://arxiv.org/abs/2604.12374
  38. [38]NVIDIA Corporation, “vLLM release notes, release 26.09,” https://docs.vllm.ai/en/latest/serving/frameworks/vllm-release-26-09.html, 2026, accessed: 2026-11-13.
  39. [39]L. Xu, K. K. Suresh, Q. Anthony, N. Alnaasan, and D. K. Panda, “Characterizing communication patterns in distributed large language model inference,” in 2025 IEEE Symposium on High-Performance Interconnects (HOTI), 2025, pp. 1–11.
  40. [40]V. J. Reddi, C. Cheng, D. Kanter, P. Mattson et al., “MLPerf inference benchmark,” in ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020, pp. 446–459.
  41. [41]K. T. Chitty-Venkata, S. Raskar, B. Kale, F. Ferdaus, A. Tanikanti, K. Raffinetti, V. Taylor, M. Emani, and V. Vishwanath, “LLM-Inference-Bench: Inference benchmarking of large language models on AI accelerators,” in SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (PMBS), IEEE, 2024, pp. 1362–1379.arxiv.org/abs/2411.00136
  42. [42]A. Agrawal, A. Agarwal, N. Kedia, J. Mohan, S. Kundu, N. Kwatra, R. Ramjee, and A. Tumanov, “Etalon: Holistic performance evaluation framework for LLM inference systems,” 2024. [Online]. Available: https://arxiv.org/abs/2407.07000
  43. [43]S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaelas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V. Gadepally, “From words to watts: Benchmarking the energy costs of large language model inference,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC), 2023, pp. 1–9.arxiv.org/abs/2310.03003
  44. [44]J. Zhou and R. Chen, “NVIDIA DGX Spark in-depth review: A new standard for local AI inference,” LMSYS Org Blog, https://lmsys.org/blog/2025-10-13-nvidia-dgx-spark/, Oct. 2025, accessed: 2026-11-13.
  45. [45]J. Frank, “vLLM on the DGX Spark: Architecture, configuration, and local evaluation,” vLLM Blog, https://vllm-project.github.io/2026/06/01/vllm-dgx-spark.html, Jun. 2026, accessed: 2026-11-13.
  46. [46]D. Dougherty and D. Jain, “NVIDIA DGX Spark cluster review: Distributed inference on Dell, GIGABYTE, and HP,” StorageReview, https://www.storagereview.com/review/nvidia-dgx-spark-cluster-review-distributed-inference-on-dell-gigabyte-and-hp, May 2026, accessed: 2026-11-13.

Paper details

vlnc:W_9z3fb6e3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Anthropic’s Claude

View statement
Looking for related papers…
Contents