Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

Introduction

Large language model inference increasingly stresses not only accelerator throughput, but also latency, energy efficiency, and system cost. While training and prefill are dominated by large matrix operations and comparatively large communication transfers, autoregressive decode generates one token or verifies only a few at a time and repeatedly executes fine-grained communication whose latency directly affects user-facing responsiveness [29, 4] (Yu et al. 2022; Zhong et al. 2024).

At the same time, growing model size, mixture-of-experts architectures, and long-context inference increase parameter, activation, and KV-cache footprints [29, 5, 18]. As these requirements outgrow per-accelerator memory capacity, individual model instances increasingly span many accelerators [5, 1] (Yang et al. 2025), creating pressure to extend tightly coupled scale-up fabrics to larger domains and beyond conventional rack boundaries [8] (Shou et al. 2025; Zuo et al. 2025).

The consequence is a tension between scale and latency: decode benefits from low token-to-token latency and therefore low round-trip time within the scale-up fabric (Singhania et al., 2026). However, extending electronic-packet-switched scale-up networks beyond a single rack typically requires additional switching levels, longer physical paths, or both, increasing end-to-end communication latency.

Optical links are attractive for spanning larger accelerator domains, while optical circuit switching (OCS) can avoid repeated electronic packet switching (EPS) at intermediate levels [27]. Figure 1 contrasts conventional packet-switched fabrics with our workload-static OCS approach: circuits are configured for the model deployment and reused throughout decode rather than reconfigured within the token-generation loop. Efficient use of such circuits requires physical topology, routing, collective schedules, device placement, and model parallelization to be co-designed [27] (Wang et al. 2023; Wang et al. 2024).

Prior work has addressed distributed training [36, 26], workload and network simulation [3] (Won et al. 2023), inference serving and disaggregation [29, 4] (Yu et al. 2022; Zhong et al. 2024; Patel et al. 2024; Qin et al. 2025), and expert- or context-parallel inference [17, 1] (Zhu et al. 2025; Zuo et al. 2025; Yang et al. 2025). These approaches improve accelerator utilization and throughput, but do not remove the latency sensitivity of low-batch or high-interactivity¹ decode.

^1Salience Labs Ltd, 100 Park Drive, Oxfordshire, OX14 4RY, United Kingdom ^2NVIDIA Corporation, Santa Clara, CA, USA ^3Hardware and Artificial Intelligence Lab, Heidelberg University, Germany. Correspondence to: Maximilian Thürmer <[email protected]>.

OCS has been explored for datacenter networks [32] (Poutievski et al. 2022), scale-out fabrics (Wang et al., 2023), and distributed training [27, 28, 31] (Shou et al. 2025), typically for provisioning, topology optimization, or workload-phase-dependent reconfiguration. Decode poses a different challenge: communication occurs at fine temporal granularity, while OCS reconfiguration is generally too slow to follow the token-generation loop. This makes per-token optical reconfiguration unattractive for latency-critical inference.

The co-design of decode parallelization, collective communication, and optical scale-up topology remains under-explored. For inference decode, the central question is not whether OCS can follow token-level communication changes dynamically, but whether the communication pattern can be regularized and scheduled such that an optical topology can be configured once and reused throughout serving.

This work studies workload-static OCS as a scale-up network design point for large-scale LLM inference decode. The OCS fabric is configured at deployment time and kept fixed during token generation, acting as a workload-specific static optical topology that realizes the logical communication structure of distributed decode. For a fixed model, context regime, accelerator allocation, service-level objective (interactivity target) and parallelization strategy, communication patterns are known at deployment time. Even in mixture-of-experts inference, token routing only modulates volume, not connectivity: production all-to-all kernels communicate with every peer of a fixed expert-parallel group during each call, with statically allocated per-peer buffers, and with batches padded to pre-captured accelerator execution graph sizes (NVIDIA, 2026c; vLLM Project, 2026a).

Evaluating distributed AI workloads on compute systems requires exploring model architectures, parallelization strategies, accelerator properties, collective algorithms, and network topologies that are difficult to benchmark directly at full scale. Physical access to hundreds of future-generation accelerators is limited, and production traces are tied to specific hardware and software stacks. A flexible modeling framework is therefore needed to study the design space while grounding assumptions on kernel and communication behavior measured on real hardware.

This work makes the following contributions.

1. We introduce workload-static OCS for LLM inference decode: optical circuits are configured once per deployment and co-designed with model parallelization and topology-aware collective schedules to realize recurring communication demand with low-latency logical overlays.

2. We introduce GXNET, a graph-based distributed LLM workload emulation framework that transforms model-level computation graphs into distributed execution traces with explicit compute, memory, and communication operators.

3. We combine analytical and event-based network modeling for all-EPS, hybrid EPS/OCS, and all-OCS scale-up fabrics. The analytical model supports broad design-space exploration, while a fit-level simulator estimates collective execution time, routing behavior, credit effects, and link utilization.

4. We evaluate 1T- and 5T-parameter MoE decode across 128–512 serving accelerators in a 576-accelerator scale-up domain. All-OCS improves interactivity by up to 65 % and system-level energy efficiency by up to 119 % over all-EPS, while sensitivity analysis over bandwidth, radix, and accelerator latency identifies the regimes in which workload-static OCS is most beneficial.

More broadly, we argue that inference specialization should extend beyond accelerators to the scale-up network: communication structure known before execution need not be rediscovered dynamically by a general-purpose packet fabric.

Background and Related Work

There is a large corpus of prior literature which addresses many separate pieces of the work presented here. Some work is devoted to related use cases - e.g. training rather than inference. Other contributions focus on a subset of the design space addressed here - for instance ignoring the co-optimization of collective schedules and underlying network topology.

Large scale distributed LLM training has received wide attention in the past. The parallelization strategies were studied [36] and cast into analytical, roofline-based models [25]. Other approaches which seek to model computation and communication separately are partially based on real hardware traces or synthetic ones that are then feeding a network simulator (Won et al., 2023). These approaches, however, have been developed for training. Calculon [25] models the scale-up and scale-out networks with single-level latency and bandwidth parameters and assumes bandwidth-optimal collective implementations, which captures step time for large, bandwidth-dominated collectives but not the small, latency-bound collectives of decode. Network simulators such as ASTRA-sim (Won et al., 2023) resolve communication in more detail, but do not treat parallelization strategy, collective schedule, and optical circuit topology as a joint design space. Inference-specific simulators such as Vidur [3] and LLMServingSim [14] address serving-level questions such as scheduling, batching, and deployment configuration; their communication model is either profiled on existing hardware [3] or inherited from ASTRA-sim [14], so the scale-up fabric is not a design variable.

Inference serving and disaggregation methods have been widely described [29, 37] (Yu et al. 2022; Zhong et al. 2024; Patel et al. 2024; Qin et al. 2025). In particular, advances in prefill/decode disaggregation (Zhong et al., 2024; Patel et al., 2024) have identified the difference in computation, memory access and network access footprints between prefill and decode phases as a root cause of inefficient inference serving. Consequently, recent work explores dedicated hardware for these two phases of inference and extends this approach by further decomposing the decode phase into attention and feed-forward-network portions, resulting in heterogeneous inference hardware [37] (Zhu et al. 2025), while others argue that shifting phase ratios favor a single balanced accelerator (Ho et al., 2026). In these cases, the network architectures remain packet-switched [35] (Ho et al. 2026) or use fixed direct-connect topologies [8, 21] whose physical connectivity is not tailored to the workload, with many accelerator links reaching into the scale-up network.

Extensive work has examined the role of OCS in AI compute clusters ever since the inception of Google’s TPUv4 [27]. There has been work focusing on optimizing the network topology through OCS [31] (Wang et al. 2023) and investigations into the required reconfiguration speeds of OCS control planes and optical transceiver relock times to exploit the very predictable and long-running nature of large collective operations resulting from AI training workloads such that an OCS operates more like an optical burst switch (OBS) (Wu et al., 2025). Harvest (Rahman et al., 2026) and its precursors [22, 2] go further and synthesize optimal

circuit schedules within individual collectives of scale-up domains, which requires fast OCS reconfiguration and tight coupling of the OCS and accelerator control planes, advancing the OCS towards something close to an optical packet switch. Other work has tried to leverage the dual network approach, scale-up inside the rack/box, scale-out between them - to reshape scale-out traffic within the scale-up domain such that the scale-out fabric reduces to rails (Wang et al., 2024) that can in turn be realized with OCS [19]. Apart from Harvest, all this work addresses training use cases and focuses on collective schedules for bandwidth optimality as a natural consequence.

In latency-sensitive decode, bandwidth-optimal collective schedules are not necessarily latency-optimal, and resulting collective schedules on scale-up networks are already showing an increasing specialization by expected traffic volume as a consequence [37]. In decode, tensor, KV-cache, and expert parallelism can induce frequent collectives with small payloads, exposing accelerator-to-accelerator round-trip time rather than only aggregate bandwidth. Optimized communication kernels based on remote-store operations reduce software overhead, but not the underlying sensitivity to fabric latency. In parallel, collective acceleration engines [21, 34] are moving collective execution into dedicated accelerator hardware. Serving stacks already select collective implementations, but through fixed, offline-tuned heuristics [37]. Exposing communication explicitly to an automated mapping and scheduling search is a step towards co-optimizing serving and collective implementation (Ho et al., 2026), but the repercussions on the ideal network fabric serving the chosen collectives, or even network topology co-optimization for the chosen collective, remain open. The work presented here exploits this specialization more systematically by extending it from the collective schedule to the scale-up topology. It extends the idea of short-circuiting ring collectives [22] but, in contrast to per-collective reconfiguration (Rahman et al., 2026), derives one topology for the composite decode schedule, i.e., for all collectives of inference decode at once, which the OCS then holds throughout serving, removing reconfiguration from the decode critical path.

Workload-Static OCS Topology Design

Workload-static OCS maps the recurring collective communication demands of distributed decode onto optical circuits configured once at deployment time. Different model regions, notably attention and MoE/FFN, induce distinct logical connectivity requirements over the same multi-port accelerators. We jointly plan the corresponding logical overlays, collective schedules, and their embedding into the physical fabric, deriving fixed OCS configurations and precomputed accelerator routing tables. Communication may involve data relay through intermediate accelerators, allowing the fixed circuits to support multiple logical topologies. We consider rail-optimized two-level scale-up fabrics Wang et al., 2024, whose physical construction is detailed in Appendix § 13.§ 7.

Static Communication Demand in Decode

We consider a prefill/decode-disaggregated deployment in which the scale-up fabric serves autoregressive decode. At deployment time, an inference framework selects the model, accelerator allocation, parallelization scheme (such as Helix or DP/TP/EP) and degrees (e.g. TP4 EP8), and operating targets such as maximum context length and interactivity. For a fixed configuration, the communication demand of each decode step is known structurally: participating rank groups, mesh axes, collective primitives, and invocation counts are determined by the model and parallelization. Payloads may depend on batch size and expert routing, but the communication support, the set of possible communicating rank pairs, required by the topology remains fixed.

Inference frameworks already select a collective implementation from a set of viable options: TensorRT-LLM picks between one-shot, two-shot and NCCL all-reduce from offline-tuned tables keyed on tensor-parallel group size, hidden size and token count. As scale-up domains grow, additional factors, such as placement-dependent latency, will have to be factored in. Workload-static switching as described here is a more systematic approach to optimize network communication under the same constraints.

We use Helix-style distributed decode Bhatia et al., [1], which induces different communication demands across model regions. Attention uses KV-parallel (KVP) and tensor-parallel (TPA) axes, whereas the MoE/FFN region uses expert-parallel (EP) and tensor-parallel (TPE) axes over the same accelerators. Table 1 illustrates the resulting demand: its structural columns are deployment-static, while batch size changes the payload per collective invocation. The KVP all-to-all exchanges partial attention outputs and log-sum-exp statistics across KV-cache shards, as required by Helix-style attention parallelism. Some axes induce no communication in a given region. In this configuration, context length affects KV-cache computation and memory traffic, but not the collective structure or payload.

Axis	Coll.	$G \times R$	Calls	$B=1$	$B=4$	$B=16$
Attention
world	AR	$1 \times512$	128	4 KiB	16 KiB	64 KiB
KVP	A2A	$128 \times4$	128	66 B	264 B	1056 B
TPA	none	$4 \times128$	0	–	–	–
FFN
EP	AG	$8 \times64$	128	256 KiB	512 KiB	768 KiB
TPE	AR	$64 \times8$	128	4 KiB	8 KiB	12 KiB
LM head
world	AG	$1 \times512$	1	25 KiB	100 KiB	400 KiB

Token routing may change traffic volume across peers, but need not change the communication support. Our evaluated expert axis uses fixed-group collectives with buffers padded to the busiest expert-parallel rank, so FFN payloads need not scale linearly with batch size (Appendix § 12.§ 2). Other implementations can regularize routing-dependent traffic through padding or bucketing, at the cost of additional traffic (Appendix § 9.§ 4). Thus, the fabric can be planned against recurring collective demands rather than topology-agnostic worst-case connectivity.

From Communication Demand to Logical Topology

We convert the static communication demand of an inference decode forward pass into logical communication graphs. Table 1 gives a worked example of such a demand pattern. Each graph corresponds to one mesh axis in one decode region; its vertices are the participating accelerator ranks, and its edges are the direct communication links available to the collective schedule. The topology planner embeds these graphs into the physical accelerator links.

Since different decode regions induce different logical meshes over the same accelerators, these graphs must be embedded jointly into a shared physical host graph. Its circuits remain fixed during serving, while different regions may reuse them through different logical views.

For power-of-two accelerator counts, we describe logical overlays through Cayley-style graphs. Ranks are labeled by bit strings, and generators connect ranks whose labels differ by a fixed mask. Unit generators yield cube-like structures; additional generators introduce folds or chords that reduce diameter. Folds extend the idea of short-circuiting ring collectives Hammer et al., [22]: instead of reconfiguring the OCS per collective step, the shortcut circuits are provisioned once at deployment, so their cost is accelerator port degree rather than reconfiguration delay on the critical path. Different mesh axes can thus be viewed as different partitions or colorings of one shared host graph, allowing several phase-specific logical topologies to be served by one workload-static set of optical circuits. Figure 2(a) illustrates this for a $Q_4$-style host graph: the same physical connec­tion tivity induces different logical mesh views by changing the partition of rank-label bits. Appendix § 9.§ 1 gives the formal construction.

Port Allocation as Latency/Bandwidth Trade-off

The previous subsections turn a decode workload into a set of axis requirements over a shared accelerator set. We now expose the main degree of freedom of OCS-based scale-up fabrics with multi-port accelerators: each accelerator has a finite port budget $L$, and every port can be spent either to reduce the number of serialized communication phases or to increase bandwidth along an already selected logical edge.

For one logical axis $i$, assume a regular overlay of degree $d_i$ and uniform edge multiplicity $p_i$, i.e. each logical edge on that axis is realized by $p_i$ parallel circuits. We estimate a collective schedule with a Hockney-style model

where $h_i$ is the number of serialized schedule phases, $\lambda_i$ is the per-phase accelerator-to-accelerator latency along the selected physical path, $v_i$ is the bottleneck volume, and $R$ is the bandwidth of one physical link. For an EPS path, $\lambda_i$ includes the latency of the electronic switch stages traversed; for an OCS path, it includes endpoint, link, and optical pass-through latency. Folds reduce $h_i$ by shortening the logical graph; parallel circuits increase $p_i$ and reduce serialization time, as shown in Figure 2(b) and Figure 2(c).

More generally, let $H = (V,E)$ denote the physical host graph and $p_e$ the multiplicity of edge $e$. Since each circuit consumes one port at its endpoint, the per-accelerator port budget is

where $\delta(u)$ is the set of edges incident to accelerator $u$.

For disjoint axis-specific edge families of degree $d_i$ and uniform multiplicity $p_i$, this simplifies to $\sum_i d_i p_i \le L$. In the shared-host construction, physical edges reused across different logical views are counted only once. Thus, the OCS fabric exposes an explicit exchange rate between latency and bandwidth: ports spent on additional generators make the graph shallower ($h_i \downarrow$), while ports spent on multiplicity make selected edges wider ($p_i \uparrow$).

A fold that reduces the schedule by $\Delta h$ phases saves approximately $\Delta h\lambda_i$, provided the schedule exploits it. Increasing multiplicity from $p_i$ to $p_i+1$ instead saves $v_i/[Rp_i(p_i+1)]$.

With a locally linear volume model $v_i \approx c_i m$, where $m$ is the number of tokens per forward pass and $c_i$ the bottleneck volume per token, the crossover occurs at

Below this threshold, reducing schedule depth is preferable; above it, additional bandwidth becomes more valuable. High-interactivity decode therefore favors latency-shaped topologies for many, but not necessarily all, axes.

Collectives as Topology-Specific Schedules

A logical topology improves collective performance only if the communication schedule exploits its connectivity. We describe each collective implementation by a logical overlay, a validity predicate specifying which overlays support the collective, and an ordered schedule of communication phases. Each phase specifies communicating rank pairs, the logical edges used, payload slices, and dependencies on preceding phases. The schedule determines the serialized phase count, bottleneck volume, and link utilization for the analytical model, as well as the communication events initiated by a network simulator (such as Simlette, see § 5).

We have implemented a planner, conceptually part of the inference deployment framework, which jointly searches over axis partitions, folding generators, edge multiplicities, and compatible collective schedules under the shared accelerator port budget. It selects the topology–schedule combination minimizing predicted communication time for the deployment’s target message-size regime. Folds reduce latency only when schedules exploit their additional connectivity, while parallel circuits improve bandwidth only when schedules distribute traffic across them. Appendix § 9.§ 2 details the topology search, and Appendix § 9.§ 3 describes the topology-aware collective algorithms.

GXNET: WORKLOAD MODELING FOR DISTRIBUTED LLM DECODE

GXNET converts a model-level computation graph and a deployment, i.e., accelerator specification, system size, precision, and parallelization strategy, into a shape-exact distributed decode execution with explicit compute regions, memory traffic, collectives, device groups, and timing dependencies. It provides the deployment-time communication demand for topology planning and the timed execution for network simulation. Appendix § 12 provides additional details.

Model Graph and Sharding Transformations

One decode step is a computation graph whose tensors carry shape, element type, and logical device placement. A decode step is specialized to generating one token (or verifying multiple tokens) per active request against an existing KV cache. Communication is derived structurally from this placement: wherever adjacent operators require different tensor placements, GXNET inserts a collective, whose group follows from the mesh axes, payload from tensor shape and element width, and occurrence count from the layer structure. For a fixed model and parallelization, this yields the deployment-time demand of Table 1 before execution.

The same sharded graph yields the per-rank weight and KV-cache footprint, which bounds the batch size feasible in accelerator memory. For MoE regions, we model dropless routing that is uniform in expectation: routing affects expert compute and FFN collective payloads, but not collective group membership; routing skew is not modeled.

Per-rank Execution Trace and Cost Model

GXNET lowers the communication-explicit graph into a per-rank execution trace, akin to traces recorded on real hardware with tools such as Chakra (Sridharan et al., 2026): the compute regions, memory accesses, collectives, and dependencies a single accelerator carries out or is exposed to during a decode step. Where a collective schedule (§ 3.4) describes how one collective uses a logical overlay, the trace determines when collectives occur, which ranks participate, what payload they carry, and whether their latency is exposed or hidden behind computation.

The dependency representation can express communication–computation overlap. We use Helix’s attention/FFN sharding without HOP-B, i.e., without splitting the attention into microbatches whose all-to-all overlaps with the attention of the next, so communication latency is fully exposed in the decode-step timeline. A sensitivity analysis (Appendix § 12.§ 5) shows that HOP-B does not improve decode-step time in the evaluated configurations: for GPT-OSS-5.3-like, micro-batching adds more all-to-all communication than HOP-B can hide, and for Megatron 5T, it yields no net benefit.

Each compute region’s execution time is estimated with a roofline model taking the maximum of compute time and HBM-transfer time, using tensor-derived FLOPs and bytes together with size-dependent compute and memory efficiencies. Operators between communication points are grouped into fused execution windows and charged one launch overhead per window. Exact efficiency functions and accelerator parameters are given in Appendix § 12.§ 3, and Appendix § 15.§ 3 relates the model to kernel measurements on current hardware.

GXNET is analytical rather than cycle-accurate; projected accelerators use specification-level throughput and bandwidth parameters, while measured kernel durations can replace analytical estimates for existing hardware.

Coupling GXNET to the Network Model

GXNET interfaces with the network model at two points. Before execution, it emits the complete communication demand of a decode step, against which the OCS fabric of § 3 is mapped and routed once at setup. In practice, an inference framework such as vLLM [29] or SGLang (Zheng et al., 2024), which already holds this information, would perform this step. During execution, GXNET registers each collective with its group, primitive, payload, axis, and earliest start time. It does so until all further events in its scheduler are blocked by dependencies to pending/registered collective operations. It then calls an advance function to delegate control to the network simulator. The network simulator returns the collective’s actual start and completion time on the specified network topology and for the given collective schedule. Once the completion time is known, dependent operators in GXNET’s internal scheduler are released. Whether a collective is exposed on the decode critical path is thus determined by the coupled execution, not by summing independently timed messages.

All collective instances whose dependencies are satisfied may be in flight simultaneously; Simlette advances them jointly and therefore captures contention and credit stalls between concurrent groups, whereas the analytical backend estimates collectives independently. A run produces a timed decode-step timeline, from which we derive token-to-token latency, per-user interactivity, and system throughput, and, with the power model of Appendix § 15.§ 4, energy-normalized throughput.

Communication Model and Grounding

Evaluating workload-static OCS fabrics requires estimating the completion time of collective communication kernels on hardware and topologies that are not yet available at scale. We use two models for this: a fast analytical model to evaluate candidate topologies and schedules, and a flit-level simulator to capture effects that the analytical model misses, such as credit stalls and link contention. Both use latency parameters based on measurements from current hardware.

Simlette: a Flit-Level Event-Based Simulator

We evaluate collective schedules and routing on EPS and OCS fabrics using Simlette, a discrete-event simulator built on SimPy (Team SimPy). Inspired by the UALink specification (Ultra Accelerator Link Consortium, 2025), it models memory-semantic transactions at flit granularity, capturing serialization, link contention, and credit stalls during execution. Simlette supports single-level and two-level spine-leaf fabrics, dynamic EPS packet switching and spraying, and workload-static OCS routing without runtime reconfiguration.

The simulator separates workload mapping into four independent layers: physical topology, worker allocation, collective schedule, and routing (Appendix § 13.§ 1). This modular design allows us to vary each component independently while keeping the others fixed, enabling controlled comparisons of fabrics, placements, schedules, and routing strategies.

UALink Transactions and Credit-Based Flow Control

Simlette implements the transaction and credit-based flow control system of the UALink specification, the same mechanisms used in practice for direct accelerator-to-accelerator communication. Each transaction pairs a request (e.g., a read or a write) issued by an Originator with the response returned by a Completer. One or more 64-byte Transaction Layer (TL) flits are grouped into a 640-byte Data Link (DL) flit before being put on the link, and switching and flow control operate at DL-flit granularity. Each port maintains independent credit channels for request and response traffic, so that traffic in one direction can never block the other. A port must hold a credit on the corresponding channel before sending a DL flit; credits are returned once the receiver has processed the data, so a buffer is never sent more data than it can store.

Latency Decomposition

The fixed per-hop latency is divided into four components: endpoint latency inside the accelerator (processor to cache to scale-up port), protocol latency of the scale-up port including credit transmission, link propagation delay, and switch forwarding delay.

For the latency comparison, we apply common endpoint, protocol, and propagation parameters, while distinguishing EPS forwarding (approx. 300 ns (UALink Spec, Sec. 10.9.2) unloaded pin-to-pin latency) from OCS optical pass-through (approx. 10 ns).

Figure 3 shows the latency of a 512-accelerator all-reduce on an example single-level fabric. The buffer size is the amount of data, in bytes, that each accelerator holds at the end of the collective. For the low-batch payloads of Table 1 (4 to 64 KiB), switch latency accounts for about a third of each EPS bar, while OCS removes almost all of it (2% of the bar).

For buffers of 1 to 64 KiB, which cover the low-batch payloads of Table 1, the completion time is 3.18–3.26 $\mu\mathrm{s}$ for OCS, compared with 5.21–5.26 $\mu\mathrm{s}$ for EPS. As the payload grows beyond 64 KiB, serialization takes over on OCS, reaching 41% of the time at 1 MiB (2.07 $\mu\mathrm{s}$), where OCS remains faster with 5.10 $\mu\mathrm{s}$ compared with 6.41 $\mu\mathrm{s}$ for EPS.

Analytical Model and Consistency Checks

The analytical model times each candidate port allocation of § 3.3 by Eq. (1), with the bottleneck volume $v$ obtained as the sum over all phases of the data carried by the most loaded edge (Appendix § 13.§ 2). Simlette then builds the circuits of the allocation with the lowest predicted time and simulates the collectives on them flit by flit, including credit waits and the load on each link.

The analytical model assumes that phases do not overlap, that no transfer shares its link with another, and that a sender never waits for buffer space at the receiver. Without contention and with unlimited buffers, Simlette reproduces the analytical completion time for all message sizes and radices on a single-level OCS fabric (Appendix § 13.§ 4). When the buffer per port is too small to keep a link busy for one hop, senders wait for credits and Simlette results are up to $1.9 \times$ slower; when several flows reach the same switch port, as for the two-shot all-reduce on EPS, the analytical model underestimates completion time by up to $2 \times$.

The analytical model is accurate for contention-free schedules with sufficient buffering. Simlette captures additional delays from credit stalls and concurrent traffic, which the analytical model omits. We therefore use the analytical model for design-space exploration and Simlette to evaluate the selected configurations with a more detailed representation of contention and credit effects.

Grounding

We parameterize Simlette using hardware measurements and published interconnect specifications (Appendix § 13.§ 6). Custom accelerator-to-accelerator ping-pong kernels estimate end-to-end communication latency, while pointer-chasing microbenchmarks characterize the local memory hierarchy ([33]). On NVIDIA Blackwell B300 for example, memory-hierarchy measurements yield access latencies of approximately 14 ns (shared memory), 17 ns (L1), and 140 ns (L2), compared with a measured accelerator-to-accelerator remote-access round trip of 1.61 $\mu$s. These measurements constrain the accelerator-side latency budget, while the attribution of the remaining latency to protocol, interconnect, and switching components requires additional modeling assumptions (Appendix § 13.§ 6). We characterize accelerator-to-accelerator latency using several ping-pong kernel implementations, and we select the minimum measured duration as a minimum upper bound. Appendix § 13.§ 6 details further measurements and implementation-dependent variation.

End-to-end measurements do not uniquely determine the latency contribution of every internal component. We therefore state the assumed component latencies explicitly and examine their influence through the sensitivity analysis of Appendix § 17. Simlette produces deterministic completion times for the specified hardware and workload parameters; run-to-run jitter and tail-latency distributions are outside the scope of this evaluation.

Evaluation

We first present results for two model architectures, Megatron 5T MoE and GPT-OSS-5.3-like, for very large context lengths of 5 million tokens. The choice of large models with large context is motivated by current hardware limitations and the resulting need to scale to fabrics larger than what a single datacenter rack can provide, even with accelerators such as the upcoming Vera Rubin platforms ((NVIDIA, c)). Model architectures and accelerator properties are given in Appendices § 15.§ 1 and § 15.§ 3.

Optimized Interactivity-Throughput Envelope

Figure 4 plots achieved interactivity, the per-user throughput metric, against total system throughput for various batch sizes. A batch of 16 concurrent users is the largest that the 512 serving accelerators can hold for Megatron before KV cache pages have to be swapped out of an accelerator’s HBM. The three network scenarios are two-level networks using only EPS (all-EPS), OCS on the second level (hybrid EPS/OCS), or only OCS (all-OCS); the first two are $4:1$ oversubscribed towards the second level, while all-OCS is non-blocking (Appendix § 13.§ 7). The EPS-based fabrics use $4:1$ leaf-to-spine oversubscription to maintain a practical system-power envelope. A sensitivity experiment with GPT-OSS-5.3-like at batch size 1 shows that removing oversubscription requires four times as many spine switches (288 vs. 72), yet reduces collective completion time by only 3.2% (13.61 to 13.18 $\mu$s). Consistently, the bandwidth sensitivity in Figure 28 from Appendix § 17 shows that even quadrupling per-port bandwidth barely changes the all-OCS interactivity advantage (64.8% to 62.8%), supporting the interpretation that the observed gains are primarily latency-driven rather than an artifact of EPS oversubscription.

The Pareto front for each scenario is found by searching the Helix parallelization space (Appendix § 12.§ 6), so the optimal parallelization for a given batch size may differ between fabrics: reduced collective completion time can make a higher degree of parallelism along a mesh dimension viable. For instance, batch size 1 on Megatron calls for a Helix scheme (KVP4, TPA128, EP16, TPE32) on all-EPS, reaching 458 tokens/s/user, while all-OCS reaches 698 tokens/s/user with (KVP4, TPA128, EP8, TPE64). The larger FFN tensor-parallel group is enabled by our collective schedule optimization for large groups, whereas a group of 64 on this physical topology forces EPS traffic through both levels, and thus three switching devices.

Bottlenecks Change with Batch Size

In high-interactivity/low-batch regimes of Figure 4, collective communication volume is small and network latency contributes substantially to decode-step time, so the difference between all-EPS and all-OCS fabrics is significant: for GPT-OSS-5.3-like, 775 versus 1277 tokens/s/user, 1.65× the interactivity. In practice, achievable values may be lower due to unmodeled system properties such as accelerator-to-accelerator jitter or GEMM inefficiencies for certain operand aspect ratios; on the other hand, this analysis does not factor in EPS-specific disadvantages such as tail latency due to buffer congestion and arbitration delays. The lower power consumption of OCS fabrics further amplifies their advantage, so the energy per generated token decreases through both lower system power and faster token generation.

As batch size increases, compute kernel time and network transaction sizes grow, which reduces the share of exposed latency in the token generation time (time per output token,

TPOT). Figure 5 shows that this effect is strong for Megatron but weak for GPT-OSS-5.3-like. Attention time scales with the KV bytes read per decode step, i.e., with the product of batch size and context length. Megatron keeps 1 MiB of KV cache per token of context, GPT-OSS-5.3-like only 48 KiB, since its interleaved sliding-window attention reads the full context in only half of its layers, with 16 instead of 128 KV heads (Table 10, Appendix § 15.§ 1); the hidden size, 8192 in both models, does not explain the difference. At 5M context, Megatron’s attention time therefore grows from 0.53 ms at batch 1 to 8.3 ms at batch 16, while communication stays nearly constant, so the OCS saving of 0.71 to 0.74 ms per step falls from 34 % to 7 % of the all-EPS step. For GPT-OSS-5.3-like, attention reaches only 0.41 ms, and communication remains the largest term throughout. The remainder of each step is the fixed invocation cost of its fused execution windows (§ 4), identical across fabrics and batch sizes.

Model	Total	Active	Layers	Hidden	Type	Q / KV heads × dim	Total / top-$k$	Width	Vocab	KV cache per token
Megatron 235B	0.23 T	17 B	96	1536	GQA, full	64/16 × 96	128 / 8	4096	51,200	144 KiB
Megatron 1T	1.10 T	38 B	128	2048	GQA, full	128/64 × 32	256 / 8	5440	51,200	256 KiB
Megatron 5T	5.00 T	207 B	128	8192	GQA, full	256/128 × 64	256 / 8	6144	51,200	1,024 KiB
GPT-OSS-120B	0.12 T	5.7 B	36	2880	GQA, SW / full	64/8 × 64	128 / 4	2880	201,088	9 KiB
GPT-OSS-1T-like	1.05 T	21 B	64	4608	GQA, SW / full	72/8 × 64	256 / 4	4608	201,088	16 KiB
GPT-OSS-5.3-like	4.97 T	95 B	96	8192	GQA, SW / full	128/16 × 64	256 / 4	8192	201,088	48 KiB

Scaling and Energy Efficiency

Figure 6 shows the scaling with system size for each of the three physical topologies; each point is again the result of a parallelization search. The maximum context is fixed at 64K tokens to allow scaling across a range of world sizes; after all, growing scale-up domains is not always motivated by higher interactivity but may also be required to provide more low-latency, high-bandwidth memory. At batch sizes 1 and 4, interactivity first rises with world size and then falls once collectives span the second level: all-EPS peaks at 64 accelerators, all-OCS only at 128, 26 % and 41 % higher than the all-EPS peak. At batch 16, all-OCS fabrics show the most favorable interactivity, and thus throughput, scaling, reaching $1.5 \times$ the interactivity of all-EPS at 512 accelerators.

Finally, Figure 7 takes a broader perspective across three scenarios of model size and maximum context length. Small models with small context are usually the free-tier variants that AI providers offer, while large models with long context are a paid service because they aim to provide high-value work such as agent supervision and long-term reasoning. Instead of total system throughput, the y-axis shows tokens per second per megawatt, the perspective of datacenter operators whose token volume, and thus profitability, is usually limited by available power. The proposed scheme is of general utility across the three very different regimes evaluated here: owing to the lower per-port power of OCS (Appendix § 15.§ 4), all-OCS fabrics outperform their EPS counterparts in throughput per unit energy by 35 to 37 % even for the 8- and 16-accelerator deployments of Megatron 235B and 1T, where interactivity improves by only 1 to 3 %, and by up to 102 % in the high-value, large-model regime where the interactivity advantage rises to 52 %; across the GPT-OSS family, the energy efficiency gain ranges from 32 to 119 % (Appendix § 15.§ 2). GPT-OSS-5.3-like at batch 1 is also the baseline of the sensitivity analysis over per-port bandwidth, accelerator radix, and switch and accelerator latency in Appendix § 17.

Discussion and Conclusion

Distributed LLM decode exhibits recurring collective communication patterns that can be served by optical circuits configured once per deployment. We show that jointly optimizing circuit connectivity, accelerator port allocation, and topology-aware collective schedules allows OCS-based fabrics to support these patterns without runtime reconfiguration. Allocating ports to reduce communication phases or increase effective bandwidth improves modeled interactivity and energy efficiency over EPS-based fabrics, particularly in latency-sensitive decode regimes. For Megatron 5T MoE, all-OCS fabrics improve modeled interactivity by up to 52 % and energy efficiency by up to 102 % over the oversubscribed all-EPS baseline; for GPT-OSS-5.3-like, the corresponding improvements reach 65 % and 119 %.

Given a physical fabric and accelerator port budget, our planner constructs candidate logical overlays from Cayley-graph generators. It jointly explores mesh-axis partitions, shortcut connections that reduce collective schedule depth, and parallel circuits that increase bandwidth. Compatible collective schedules are evaluated using an analytical latency–bandwidth model, and the configuration minimizing workload-weighted communication time is mapped onto physical ports as a deployment-static circuit topology.

These benefits of our approach are conditional on the workload and hardware assumptions. Their magnitude depends on the communication exposed on the decode critical path, accelerator endpoint latency, model architecture, and the power and oversubscription assumptions of the compared fabrics. With co-packaged optics (CPO), accelerators may see a rise in both per-port data rates, e.g., through WDM-capable optical PHYs such as OCI ((OCI MSA, 2026)), and port count (accelerator radix). Both can be shown to improve the benefits of workload-static switching in terms of system performance and energy efficiency (Appendix § 17).

Our main evaluation excludes compute–communication overlap, although HOP-B microbatching is examined separately (Appendix § 12.§ 5). We also assume balanced expert routing and do not model routing skew or stochastic execution-time variations.

More broadly, our results motivate treating scale-up topology, model parallelization, and collective schedules as a joint optimization problem, including for optically switched fabrics. Future work should validate the predicted performance and energy benefits on physical hardware, extend the optimization to overlapping computation and communication, and assess robustness to dynamic traffic and routing imbalance. Workload-static OCS thus offers a promising alternative to general-purpose packet switching when communication requirements are predictable at deployment time.

AI USAGE DISCLOSURE

Assistance of AI was extensively used in the context of this work. It was used to assist in writing code for the Simlette network simulator. It was used to help in formalizing the mathematical concepts underlying the algorithmic ideas of mesh fabric creation and collective schedule implementation. It was used to check public availability of performance and power data used in this work. It was used to draft text snippet suggestions for certain paragraphs that were then manually edited and revised. It was used for consistency checks between data and text. It was used for a consistent formatting style of figures.

Workload-Static Topology and Collective Design

This appendix details the workload-static topology design and collective schedules. It covers graph construction, joint topology–schedule optimization, and variable MoE traffic, followed by worked examples, fabric comparisons, alternative port allocations, and limitations.

Throughout this appendix we use the notation of § 3 and drop the axis index $i$ where only one axis is considered: $h$ is the number of serialized schedule phases, $\lambda$ the per-phase latency, $v$ the bottleneck volume, $p$ the edge multiplicity, $R$ the bandwidth of one physical link, and $m$ the number of tokens per forward pass. We denote the payload of one collective call by $\sigma$, with $\sigma= cm$ for a per-token volume $c$; we also call this the buffer size of the collective, i.e., the larger of its input and output buffers (§ 3.1). It differs from the messages a schedule sends: each phase transfers per-port messages whose sizes are determined by the schedule and, e.g., for coset-factored schedules, are not uniform across phases.

Shared-Host Graph Construction

Under Helix parallelism [1], every layer of a decode step passes through a sequence of regions, each of which arranges the same $N$ accelerators into its own logical device mesh (Figure 8). The attention region uses a mesh of shape (KVP, TPA) and ends in an all-to-all over its KV-parallel axis; the output projection is followed by an all-reduce over a one-dimensional mesh spanning all accelerators; and the FFN/MoE region uses a mesh of shape (TPE, EP), with a tensor-parallel all-reduce and an expert-parallel all-gather. Each dimension of a mesh is a mesh axis, and its extents form the mesh shape. The accelerators that differ only in their coordinate along one axis form a collective group of that axis, so in a two-axis mesh every accelerator belongs to exactly two collective groups, one per axis.

Helix is one of several parallelization schemes for decode, and its region structure is more demanding than that of the common TP/EP scheme. There, attention and FFN typically share one mesh and one tensor-parallel axis; under Helix, the attention and FFN meshes split the same accelerators along different axes, since TPE generally differs from TPA. The shared-host allocation must therefore serve several conflicting mesh shapes with one host graph, in which every mesh axis uses a subgraph and the subgraphs of different meshes share edges. The subgraphs have to be chosen in conjunction with the collective schedules, since a densely connected logical topology only pays off where a schedule can exploit its additional edges. We build every subgraph from a hypercube spanning the collective groups of its axis, the regular topology in which each of $2^a$ accelerators has one link per dimension to $a$ neighbors. Hypercubes suit this purpose because the hypercubes of all mesh axes are sub-cubes of one hypercube spanning all accelerators, so a single wiring serves every mesh shape. To them we add either folds, i.e., shortcuts across the cube that lower its diameter, or parallel circuits that raise its bandwidth, both at the cost of additional ports. Choosing them for all meshes at once is a joint optimization problem, in which the alignment of folds across meshes (Appendix § 9.§ 1) and the joint choice of partitions and folds (Appendix § 9.§ 2) become necessary.

Bit labels, cosets, and the hypercube invariant We label each of the $N = 2^n$ accelerators with its index in binary. A generator is an $n$-bit mask $g$ that induces the edge $x \leftrightarrow x \oplus g$ at every node. A set $S$ of generators thus defines the Cayley graph $\operatorname{Cay}((\mathbb{Z}_2)^n, S)$ of the group of $n$-bit labels under bitwise XOR: its vertices are the labels, and every label $x$ is connected to $x \oplus s$ for each $s \in S$ [20]. Since every label is its own inverse under XOR, the edges are undirected. Three standard properties of such graphs are used throughout: for distinct non-zero generators, the graph is $|S|$-regular, i.e., every node has one neighbor per generator; it is vertex-transitive, i.e., it looks identical from every node, so per-node port counts and schedules carry over to all nodes; and it is connected exactly when $S$ spans $(\mathbb{Z}_2)^n$. We write labels and masks as bit strings, most significant bit first, so that a label reads $x = b_{n-1}\cdots b_0$; the generator $g = 111$, for instance, sets bits 0, 1 and 2 and connects every label $x$ to $x \oplus111$, i.e., to the label with these three bits flipped. The unit masks $e_j = 2^j$ generate the hypercube $Q_n$, whose degree and diameter are both $n$ and whose graph distance is the Hamming distance (Table 2).

notation	meaning
Hamming weight $|g|$	number of 1-bits of a mask; the Hamming distance of two labels $x$, $y$ is $|x \oplus y|$, the number of bit positions in which they differ
$Q_n$	hypercube on $n$ bits; its generators are the $n$ unit masks, i.e., the masks of Hamming weight 1
$Q_{n+g}$	$Q_n$ with the additional generator $g$, a fold
$FQ_n$	folded cube $Q_{n+1}\ldots1$, in which every node is joined to its antipode
$K_s$	complete graph on $s$ nodes
$G \mathbin{\Box} H$, $K_s^\ell$	Cartesian product: two nodes are adjacent if they agree in one factor and are adjacent in the other; $K_s^\ell= K_s \mathbin{\Box}\cdots\mathbin{\Box} K_s$ with $\ell$ factors
block, sub-block	the bit positions an axis owns; a subset of them
block subgroup	all masks supported on the bits of a block, i.e., all ways of flipping only these bits
coset (of a block)	the labels $x \oplus s$ for a fixed $x$ and all $s$ in the block subgroup, i.e., all labels that agree with $x$ outside the block; each coset is one collective group of the axis that owns the block
complete layer on $k$ bits	all $2^k-1$ non-zero masks supported on a sub-block of $k$ bits; it turns every coset of that sub-block into $K_{2^k}$
$Q_a[k_1,\ldots,k_\ell]$	block fold: the $a$ bits of an axis split into sub-blocks of $k_j$ bits, each carrying a complete layer; equals $K_{2^{k_1}}\mathbin{\Box}\cdots\mathbin{\Box}K_{2^{k_\ell}}$, with degree $\sum_j(2^{k_j}-1)$ and diameter $\ell$
uniform block fold	all sub-blocks of equal size, $k_j=a/\ell$, e.g., $Q_6[3,3]=K_8\mathbin{\Box}K_8$
mixed-radix block fold	sub-blocks of different sizes, e.g., $Q_7[4,3]=K_{16}\mathbin{\Box}K_8$; reading a label as a number whose digit $j$ is the value of sub-block $j$, every digit has its own base $2^{k_j}$

A 2-D mesh of shape $(2^{a_0}, 2^{a_1})$ partitions the bit positions into two blocks. The coordinate of a node along an axis is the value of that axis’s bits, and a collective group is the set of nodes that agree on all bits outside the block. Each group is therefore a coset of the block subgroup: there are $N/s_i$ of them, each of size $s_i = 2^{a_i}$. This is why we restrict axis extents to powers of two: grid coordinate and label slice become the same object. Other sizes could be handled by planning the enclosing power of two and marking the surplus labels as absent, which we do not implement.

Unit masks over disjoint blocks together form the unit-mask set of the larger cube. For $N = 16$, the meshes $(2, 8) = Q_1 \square Q_3$ and $(4, 4) = Q_2 \square Q_2$ are therefore the same 32-edge cube $Q_4$, colored into axes in two different ways (Figure 2(a)). One fixed wiring thus serves every power-of-two mesh shape at diameter $n$. Every other configuration the shared-host planner considers keeps this cube as its base and only adds to it, either additional generators that fold it (Appendix § 9.§ 1) or parallel circuits that widen its edges.

The planner treats the bit partition of each mesh as its placement decision and chooses it jointly with the folds (Appendix § 9.§ 2).

Table 3 summarizes the topologies a single axis could take. Reducing the degree below $d=a$ differs qualitatively from increasing it: the axis can no longer remain in $(\mathbb{Z}_2)^a$ and has to change group. The shared-host planner therefore stays in the cube family, $d\geq a$; rings and circulants appear only in the stacked-link allocation of Appendix § 9.§ 6 and for a ring schedule on a single mesh (Appendix § 9.§ 5.1).

degree $d$	topology	group	diameter
$d=a$	hypercube $Q_a$	$(\mathbb{Z}_2)^a$	$a$
$d>a$	folded cube	$(\mathbb{Z}_2)^a$	$<a$
$2<d<a$	circulant	$\mathbb{Z}_{2a}$	intermediate
$d=2$	ring	$\mathbb{Z}_{2a}$	$2^{a-1}$

Folding generators, the support rule, and alignment

The hypercube uses $n$ ports per accelerator, one per unit mask. Any further port, i.e., a host degree $D>n$, can be spent on an additional generator of Hamming weight at least two. Whereas a unit mask connects labels that differ in a single bit, a generator of weight $\omega$ connects labels that differ in $\omega$ bits at once, a pair the cube alone separates by $\omega$ hops. Such generators act as chords that fold the cube. Adding $g=111$ to an axis of size 8 makes antipodal nodes adjacent, raising the degree from 3 to 4 and lowering the diameter from 3 to 2 (Figure 9). Whether a fold helps a given axis is governed by its support: a folding generator only helps a mesh axis if all of its set bits lie inside that axis’s bit block. If its support spans two blocks, the circuits are still built and occupy ports, but the collectives of that mesh leave them idle.

Since a generator adds one port on every node, folds are a global resource shared by all meshes, and they are the main source of coupling between them. For $N = 16$ with the concurrent mesh shapes $(2, 8)$ and $(4, 4)$ and their partitions $b_3 \mid b_2b_1b_0$ and $b_3b_2 \mid b_1b_0$ of Figure 2(a), the mask 0011 lies inside axis B of both meshes and reduces the diameter of both from 4 to 3. The mask 1100 helps only $(4, 4)$, whose axis A it turns into $K_4$; it leaves $(2, 8)$ at diameter 4, since that partition places bits 3 and 2 on different axes. Together, the two masks bring $(4, 4)$ to diameter 2 and $(2, 8)$ to 3 (Figure 10). Choosing partitions such that one chord serves the latency-bound axes of several meshes is therefore a genuine optimization problem, and the reason why partitions are part of the search.

Two cases simplify the problem. With a single mesh there is nothing to align to, so an axis could leave the cube family, e.g., become a degree-2 ring that frees ports for edge multiplicity; our planner offers this only to the ring schedule, which it can evaluate, since an analytical cost model exists, but which the simulator does not execute (Table 7). This is because in the latency-bound inference decode regime, the ring algorithm typically is the worst of all schedule choices. A logical-only axis, i.e., a mesh dimension that carries no collective, such as a replica or pipeline-stage axis, receives neither circuits nor collective groups. Its bits still label nodes so that the other axes can index their cosets by them, but the ports it would otherwise occupy remain available.

schedule	case	topology	deg	$p$	axis	$T$
ring (planning only)	multi-mesh $[(2, 32), (8, 8)]$	$Q_5$ sub-cube	5	7		$3660\ \mu\mathrm{s}$
ring (planning only)	single mesh $(2, 32)$	ring	2	17		$1526\ \mu\mathrm{s}$
parallel RS+AG	multi-mesh $[(2, 32), (8, 8)]$	$Q_5$ sub-cube	5	7		$731\ \mu\mathrm{s}$
parallel RS+AG	single mesh $(2, 32)$	$Q_5$	5	7		$731\ \mu\mathrm{s}$

Realizing the host graph: edge coloring Mapping the host graph onto the OCS planes is an edge coloring problem in which colors correspond to ports. For a Cayley graph of $(\mathbb{Z}_2)^n$, this coloring is immediate: the edges $\{x, x \oplus g\}$ of one generator $g$ form a perfect matching, so coloring every edge by its generator is proper and uses exactly $D$ colors, and each of the $p_e$ circuits of a multiplied edge

receives its own copy of that color. Every aligned host graph therefore fits the port budget of Eq. (2) without a coloring search, whatever the weights of its generators. Only when the combined demand graph is not a Cayley graph of $(\mathbb{Z}_2)^n$, for instance for meshes that are not aligned to one host, can it require one color more than its maximum degree (Vizing’s theorem); a port then has to be reserved.

Joint Topology–Schedule Optimization

The impact of folds and multiplicity The planner spends the ports of Eq. (2) on two resources: fold generators, which cost one port per node and reduce the phase count $h$ of the axes whose schedules can use them, and parallel circuits, which raise the multiplicity $p$ of an axis. Each axis is evaluated using Eq. (1), where $h$ counts the phases the selected schedule actually traverses, not the graph diameter, and an axis completes at the pace of its least-multiplied generator, its effective multiplicity $p_i$. An edge shared by several axes takes the largest multiplicity any of them requires; together with the global nature of folds, this is the only coupling between meshes. The planner minimizes $\sum_i w_i T_i$ subject to Eq. (2), where $w_i$ counts how axis $i$’s collective occurs per decode step.

The diameter frontier and the schedulable frontier The number of phases a given degree can buy is a staircase (Figure 11). The diameter frontier bounds every schedule; the schedulable frontier of the coset-factored schedules (Appendix § 9.§ 3) lags behind it, because these schedules need generator sets that factor into non-overlapping layers: on an axis of 64 accelerators, diameter 2 is reached at degree 13, but two coset layers only at degree 14. Both frontiers are precomputed per axis size.

Search structure and the planner The planner proceeds in the stages of Figure 12: it selects candidate schedules per axis, derives each axis’s ladder of candidate folds from its schedule, searches bit partitions, fold sets and multiplicities jointly, and colors the resulting circuits onto ports. For large instances, the search is heuristic. The planner is deterministic, so a plan is a pure function of its request and can be cached and reused.

Binding the cost model to execution The cost model is only trustworthy if $h$ and $v$ are properties of the actual schedule rather than of an idealization of it. We therefore specify every collective implementation as a triple of an executable schedule, a validity predicate, and a closed-form cost (§ 3.4), and check that the three agree. The closed-form cost takes the topology and the payload and returns $(h,v,u)$: serialized phases, bottleneck-link bytes, and ports driven per node. It does not receive $\lambda$, $R$ or $p$; the planner alone owns these deployment parameters and forms $T$ by Eq. (1). We run the resulting check for every collective schedule the planner can select: under the assumptions of the closed form, i.e., a fixed per-phase latency $\lambda$, link bandwidth $R$, multiplicity $p=1$ and no competing traffic, the simulated completion time of the schedule has to match $h\lambda+ v/(pR)$.

The check prevents two failure modes that would otherwise be undetectable. First, crediting recursive doubling with a folded diameter it cannot use would make the selector prefer it on every folded cube. Second, charging RS+AG a full message per phase instead of its genuine two-shot total would make the selector never choose it. The validity predicate keeps invalid combinations of schedule and topology out of the search entirely, rather than having the cost model extrapolate a value for a schedule that would not run.

The scope of this check is internal consistency between the closed form and a simulated execution of the same schedule under the same idealizing assumptions; it is not a hardware validation. It currently covers $p=1$ only: with parallel circuits, the closed form is known to over-predict the simulated time, so plans that use $p>1$ are, if anything, estimated pessimistically. Grounding of the constants themselves ($\lambda$, $R$, endpoint and protocol terms) is treated separately in § 5.5.

Topology-Aware Collective Schedules

All-reduce operating points Two lower bounds hold for any all-reduce schedule: the diameter of the topology, since every node’s result depends on all inputs, and $\lceil\log_{d+1} N \rceil$ phases at degree $d$. The achievable minimum is the larger of the two.

Recursive doubling cannot use folding edges, since merging partial results with overlapping source sets would count inputs twice. Our coset-factored schedules respect this constraint: they group the generators of a fold into layers $S_1,\ldots,S_\ell$ that tile the group without overlap, $\prod_j (1+|S_j|)=N$, and exchange over one layer per phase, so a fold of $\ell$ layers needs $\ell$ phases. On a block fold, these layers are exactly its complete layers (Table 2). Where no such tiling is as short as the diameter, as on $Q_4+1111$, the diameter can still be reached by reusing links across phases at a higher volume, as the one-shot schedule does (Table 4). The parallel variant of the coset-factored schedule keeps the $\ell$ phases and reduces the bottleneck volume from $\ell\sigma$ to $\sigma$. RS+AG on the same layers trades phases for volume: a reduce-scatter and an all-gather of $\ell$ phases each, with a volume that shrinks geometrically in the layer radices $r_j = 1 + |S_j|$, the number of nodes a layer connects,

topology (deg, diam)	min. $h$	rec. doubling	one-shot	coset-fact.	coset-fact. par.
$Q_3$ (3, 3)	3	3 / 3$\sigma$	3 / 12$\sigma$	3 / 3$\sigma$	3 / $\sigma$
$FQ_3$ (4, 2)	2	3 / 3$\sigma$	2 / 6$\sigma$	2 / 2$\sigma$	2 / $\sigma$
$Q_4$ (4, 4)	4	4 / 4$\sigma$	4 / 32$\sigma$	4 / 4$\sigma$	4 / $\sigma$
$Q_4+1100$ (5, 3)	3	4 / 4$\sigma$	3 / 20$\sigma$	3 / 3$\sigma$	3 / $\sigma$
$Q_4+1111$ (5, 2)	2	4 / 4$\sigma$	2 / 7$\sigma$	3 / 3$\sigma$	3 / $\sigma$

which on the plain cube is the familiar $2^{N-1}\sigma$; its parallel variant divides this volume by $\ell$ on a uniform fold.

The planner selects by $\operatorname{arg\,min} T$ among the schedules whose validity predicate holds and that are not dominated (Table 5). In the latency-bound regime the coset-factored parallel schedule usually wins on the deepest fold the budget affords; in the bandwidth-bound regime parallel RS+AG does, whose volume is lower than $\sigma$ but which needs twice as many phases. The worked plan of Appendix A.5.1 illustrates the former. Figure 13 compares the resulting all-reduce completion times with those of the all-EPS fabric over a range of buffer sizes (Appendix A.5.2).

schedule	phases	bottleneck vol.	regime
recursive doubling	$\log_2 N$	$\log_2 N \cdot\sigma$	dominated
coset-factored	$\ell$	$\ell\sigma$	latency
coset-factored parallel	$\ell$	$\sigma$	latency
RS+AG	$2\ell$	Eq. (4)	bandwidth
RS+AG parallel	$2\ell$	Eq. (4)/$\ell$	bandwidth

All-to-all

The all-to-all generalizes the radix-$r$ Bruck algorithm (Bruck et al., [12]) to multi-port accelerators on the same coset layers and takes $\ell$ phases on a fold of $\ell$ layers. For axes whose number of bits is prime, such as 128 accelerators, uniform folds are scarce, so we additionally admit balanced mixed-radix block folds such as $Q_7[4,3] = K_{16} \mathbin{\Box} K_8$ (Table 2); their parallel schedules are priced by the narrowest layer. Figure 13 shows the corresponding all-to-all completion times (Appendix A.5.2).

Variable MoE Traffic and Regularization

MoE token routing and deterministic all-to-all The workload-static argument requires the communication of decode to be fixed at deployment time. Token routing is the only mechanism in the model that varies from token to token, so we state here what has to be regularized, what does not, and what regularization costs.

With $N_{\exp}$ experts distributed over N_{\ep} expert-parallel groups and top-$k$ routing, each decode step produces a dispatch traffic matrix W \in\mathbb{N}^{N_{\ep}\times N_{\ep}}, whose entry $W_{uv}$ counts the rows sent from group $u$ to group $v$; a combine phase with the transposed pattern follows. The router determines the entries of $W$, but not its support: no routing decision is excluded a priori, and over any nontrivial number of steps every ordered pair occurs. The logical graph the circuits must serve is therefore the complete graph on the expert-parallel axis, i.e., an all-to-all, independent of the router. The topology can thus be static without any regularization. Regularization is only needed for the schedule, since a fixed phase structure and a fixed planner input both require the per-pair payloads to be fixed.

Capacity padding. We assign each expert a fixed capacity

with capacity factor $\gamma\ge1$, and pad every buffer between a source group and a destination expert to $C$ rows. Every ordered pair then exchanges exactly C\cdot(N_{\exp}/N_{\ep})\cdot d_{\mathrm{model}} elements per step, independent of the routing outcome, where $d_{\mathrm{model}}$ is the hidden size. The dispatch becomes a uniform all-to-all over the expert-parallel axis that maps directly onto the Bruck/coset schedule of Appendix § 9.§ 3, with $h$ and $v$ fixed and known. Tokens exceeding the capacity are handled on an overflow path outside the static schedule, either dropped, as is common in training, or deferred to the next step. Production inference kernels take the opposite end of this trade-off: they size per-peer receive buffers for the worst case, so that no token is dropped (NVIDIA, 2026d; vLLM Project, 2026a). In our terms, this corresponds to the largest capacity factor, for which padding is only incurred in buffer memory and not necessarily on the wire.

Bucketing. Padding is the special case of a more general scheme in which per-pair counts are quantized to a small set of levels $C_1 < \cdots< C_q$ and the schedule is planned for the largest. Bucketing recovers volume when the routing histogram is reliably skewed, at the cost of $q$ cached plans instead of one. Since plans are pure functions of their request and are cached (Appendix § 9.§ 2), the additional planning cost is paid once per bucket at deployment. Our simulator implements neither routing, capacity nor overflow: it runs the dispatch as a uniform all-to-all of the declared per-pair payload, which is exactly the padded dispatch with $q=1$.

Cost of padding in decode. Padding increases the bottleneck volume from $v$ to $\gamma v$, so by Eq. (1) the time of the axis becomes $T = h\lambda+ \gamma v/(pR)$. With $v = cm$, the relative penalty is

where $\hat{m}$ is the token count at which the latency and bandwidth terms of Eq. (1) are equal. For $m \ll\hat{m}$ the penalty vanishes: an axis operating an order of magnitude below $\hat{m}$ pays less than $10\%$ of $\gamma-1$. Regularization is therefore affordable in exactly the regime that requires it. At small $m$, the routing histogram is least balanced and $\gamma$ has to be largest, but this is also where $m$ is furthest below $\hat{m}$. Conversely, at large batch sizes, where $\gamma$ approaches 1 because the histogram concentrates, the axis has moved toward the bandwidth-bound regime in which padding would be expensive. Both effects act in the same direction.

Regularized token routing. Where an implementation may constrain the router, for instance through balanced assignment or expert-choice routing, the histogram becomes deterministic by construction and $\gamma=1$ is attainable without dropping tokens. Since this changes model quality, it is out of scope for this paper; we note it because it removes the only remaining non-determinism in the communication pattern.

What remains variable. Regularization fixes the dispatch and combine patterns, but not the compute time of the expert MLPs, which still depends on the realized token counts unless the implementation also computes on padding. Similar variability arises from speculative decoding, which varies the number of tokens verified per step and thus compute times and payloads, but not which accelerators communicate. How such variability, and the regularization strategies that bound it, affect the decode step is a question for the workload model (§ 4) rather than for the network model: the circuits serve the collective schedule, whose structure stays fixed, so this variability changes the execution trace but not the network fabric.

Worked Examples and Fabric Comparisons

Worked Multi-Mesh Plans

A latency-bound multi-mesh plan. We consider $N=256$ ($n=8$) accelerators with $L=18$ ports, $R=5\times10^{10}$ B/s ($400$ Gb/s) and $\lambda=0.52\,\mu\mathrm{s}$ on a single-level OCS fabric. The fabric serves the mesh $(4,64)$ with all-reduce payloads of $4$ kB and $32$ kB and the mesh $(16,16)$ with $4$ kB on both axes, and the planner selects every schedule automatically. Every axis is latency-bound, so the budget is spent on degree. The resulting plan has host degree $D=17$, consisting of the 8 cube generators and 9 folding generators, uses 17 of the 18 ports, and achieves a total of $T=4.54\,\mu\mathrm{s}$ (Table 6; Figure 12 traces it through the planner’s stages).

mesh	axis	size	topology	deg	$h$	$p$	axis $T$
(4, 64)	0	4	$K_4$	3	1	1	$0.60\,\mu\mathrm{s}$
(4, 64)	1	64	$Q_6[3,3] = K_8 \square K_8$	14	2	1	$1.70\,\mu\mathrm{s}$
(16, 16)	0	16	$Q_4[2,2] = K_4 \square K_4$	6	2	1	$1.12\,\mu\mathrm{s}$
(16, 16)	1	16	$Q_4[3,1] = K_8 \square K_2$	8	2	1	$1.12\,\mu\mathrm{s}$

The size-64 axis carries the coset-factored parallel schedule on two complete layers of seven generators each. In terms of Figure 11, it operates at degree 14, the first point of the schedulable frontier with two phases, rather than at degree 13, where the diameter frontier already reaches 2 but no two-layer factorization exists. The size-4 axis is folded into a complete graph and carries the volume-optimized one-shot schedule, which ties with coset-factored parallel there. Both axes of (16, 16) reach two phases without additional ports: the folds they use are a subset of those already present for (4, 64), which is why the second axis even receives a degree-8 fold where degree 6 would suffice. The eighteenth port remains unused, since no further fold reduces a phase count and a multiplicity increase on any axis costs at least three ports.

The cost of alignment. Table 7 contrasts a bandwidth-bound axis with and without a second mesh to align to. With two or more meshes, the fabric is one aligned $Q_6$, so every axis must be a sub-cube and a degree-2 ring is not available. A ring schedule would have to run over a Gray-code Hamiltonian cycle of the $Q_5$, driving only 2 of the 5 links per node; each multiplicity unit then costs 5 ports instead of 2, and the multi-mesh plan reaches $p = 7$ where a single-mesh ring reaches $p = 17$. Alignment costs nothing, however, for the schedule the planner actually selects here. Parallel RS+AG drives all $d$ links of the cube in every phase, so it gains nothing from a ring and both regimes arrive at the same plan, at half the time of the best ring. The ring schedule serves only as a reference: the planner can evaluate it, but the simulator does not execute it.

Collective Performance Across Fabrics

Figure 13 compares single collectives spanning all 512 accelerators on the two fabrics of § 6: the all-EPS fabric, two levels and 4:1 oversubscribed towards the second level, with dynamic routing and the schedules of current inference frameworks, and the all-OCS fabric, non-blocking, with the topology and schedule planned for every buffer size.

Every all-reduce on both fabrics is charged at least $2\frac{N-1}{N}\sigma$, the volume of a valid sum all-reduce.

For small buffers, both collectives are latency-bound, and all-OCS completes them about twice as fast: a 1 KiB all-reduce takes 1.78 $\mu$s instead of 3.77 $\mu$s, a 1 KiB all-to-all 1.70 $\mu$s instead of 3.02 $\mu$s. This advantage does not stem from bandwidth and would therefore persist against a non-blocking EPS fabric. A packet-switched fabric lets any node reach any other, which determines which edges are usable and therefore affects the bandwidth term; it cannot remove serialized phases from a collective, and the switch stages it traverses add to the latency term. Folding instead converts ports into fewer phases. For the all-reduce, the coset-factored parallel schedule is best up to 50 KiB and parallel RS+AG beyond, while the all-EPS fabric moves from one-shot to two-shot and, from 5 MiB on, halving-doubling.

For large buffers, the all-reduce advantage grows to 3.6× at 50 MiB (68 instead of 242 $\mu$s), which mostly reflects the 4:1 oversubscription of the all-EPS fabric that all-OCS avoids (Appendix § 13.§ 7); a non-blocking EPS fabric would close most of this part of the gap. For the all-to-all, the advantage peaks at 3.4× at 512 KiB (3.05 instead of 10.4 $\mu$s) and is 1.84× at 250 MiB (0.66 instead of 1.21 ms).

The following Subsection A.5.3 separates the contributions of schedule, subscription, and switch technology to this gap.

Contributions of Schedule, Subscription, and Switch Technology

The EPS fabric of our comparison is oversubscribed 4:1 between its leaf (level-1) and spine (level-2) switches, since a fully subscribed EPS spine with pluggable conversion exceeds a reasonable power envelope; the OCS spine needs no conversion and is fully subscribed. The all-to-all is the collective that suffers most from oversubscription: with $C$ racks, a fraction $(C - 1)/C$ of every accelerator’s data is addressed to other racks and has to cross the spine, whatever the schedule. To separate the effect of the schedule from that of the fabric, we also run our parallel Bruck schedule on the EPS fabric, with the logical edges of the fold carried as packets over all physical links of each accelerator.

Figure 14 compares four configurations to examine the contributions of collective scheduling, spine subscription, and switch technology. The transition to the fully subscribed EPS reference also changes transport from packet routing to static circuits, so this step does not isolate subscription alone. The EPS fabric at 4:1 with its native all-to-all schedules is the deployment that still fits a reasonable power envelope. Running our parallel Bruck schedule on the same fabric changes only the schedule and accounts for about two thirds of the gap at large buffers: at 250 MiB, it reduces the completion time from 1.21 to 0.85 ms, against 0.66 ms on the OCS fabric. Raising the EPS spine to full subscription changes only the subscription; this configuration can only be built with static circuits, since circuits routed through an EPS spine are colorable only at full bisection, and we include it as a reference that exceeds a reasonable power envelope. At 250 MiB, this step accounts for the remaining third of the gap (0.85 to 0.66 ms). The remaining step replaces the EPS switches of that reference by OCS and changes only the switch technology; at large buffers the two perform alike (0.661 and 0.657 ms at 250 MiB). At small buffers they do not: every hop through an EPS switch still pays its switch latency, so the static reference takes 5.3 instead of 1.8 $\mu\mathrm{s}$ at 1 KiB, which is the latency advantage discussed in Appendix § 9.§ 5.2. At large buffers, the OCS fabric therefore does not gain on bandwidth at equal subscription. It gains because full subscription is affordable for it, whereas on the EPS fabric the step from 4:1 to 1:1 is exactly the part of the gap that the oversubscribed spine imposes.

Newer inference hardware architectures devote physical subsets of their scale-up links to a given mesh axis, e.g., TP or EP in TP/EP parallelism (Ho et al., 2026). Stacked-link allocation is the workload-aware variant of this concept, which partitions the ports at deployment time rather than at system design time.

Shared-Host vs. Stacked-Link Allocation

An alternative to the shared-host construction of § 3.2 is to treat each communication axis independently and allocate it a dedicated subset of accelerator ports. We call this stacked-link allocation.

Each axis $i$ receives $L_i$ ports, with $\sum_i L_i \le L$: one port each, and the rest in proportion to the full-connectivity degree $N_i - 1$, rounded by largest remainder. If $\sum_i(N_i - 1) \le L$, every axis is fully connected and the allocated ports become parallel circuits. For each axis over its $N_i$ ranks, we choose a circulant Cayley graph on $\mathbb{Z}_{N_i}$ at degree $L_i$ that minimizes the diameter, using diameter as a latency-oriented proxy for collective completion time. The search is exhaustive for small axes and sampled for large ones, so for large axes the stacked-link circulant is a heuristic, not necessarily the minimum-diameter one. Since a circulant is not a cube, only the one-shot schedules run on it.

Figure 15 compares the two allocation approaches on a single-level OCS fabric of 512 accelerators under an 8 KiB all-reduce on a 2-D mesh (8 × 64) with 72 collective groups: 64 groups of 8 accelerators along the first axis, and 8 groups of 64 accelerators along the second. The two approaches use the $L$ links of each accelerator differently. Stacked-link fixes the split beforehand: one set of ports is assigned to the 8-rank axis and another to the 64-rank axis. Each axis can then use its ports only to reduce the diameter of its own graph; until both axes are fully connected, ports assigned to one axis can neither be used by the other nor increase bandwidth. Shared-host instead uses $\log_2 N = 9$ ports to build the base hypercube, whose generators it splits between the two axes, three for the 8-rank axis and six for the 64-rank one. The remaining ports form a common pool, and the planner chooses folds and parallel circuits for both axes jointly under that budget: fold generators to remove collective phases when latency is the main bottleneck, parallel circuits to increase bandwidth when it is. The automatically selected collective schedule then uses whichever the plan provides. This gives shared-host a clear advantage at intermediate radix, from $L = 16$ to $L = 64$ in Figure 15. At higher radix, the two curves converge again: once $L \ge(8 - 1) + (64 - 1) = 70$, stacked-link has enough ports to fully connect both axes independently, and both approaches reach a network diameter of one with the same completion time.

Shared-host allocation is therefore the more efficient in most cases, since one budget serves both axes and surplus ports can go to whichever of latency and bandwidth gains more. Stacked-link allocation is more flexible in the topology it can realize. Our current shared-host construction requires $L \ge\log_2 N$, since its base host graph is defined over binary rank labels, and therefore assumes power-of-two axis extents; stacked-link only needs enough ports $L_i$ per axis to connect its own $N_i$ ranks. Both allocations determine which logical adjacencies and bandwidth are available to a collective, but neither alone determines the resulting execution time. Their benefits materialize only if the collective schedule can exploit the provided connectivity (Appendix § 9.§ 3).

Scope and Limitations

The multi-mesh search is heuristic and does not establish global optimality. Only one- and two-dimensional meshes are supported, all meshes of a deployment must span the same power-of-two world size, and non-power-of-two sizes are not supported at all. The conformance check covers $p = 1$ only (Appendix § 9.§ 2). The coset-factored schedules require that the partial sums combined in successive phases never overlap, which holds exactly when the generators of a fold tile the group as layers. On irregular folds, such as the diameter-optimal generator set of degree 13 on an axis of size 64, no such tiling exists, so these schedules need more phases than the diameter.

Details on GXNET

This appendix details how GXNET translates a model graph into a distributed decode-step execution. It describes the graph transformations, MoE load assumptions, compute and memory cost model, and coupling to the network simulator. It then evaluates sensitivity to HOP-B microbatching and explains the search over parallelization configurations used in the evaluation.

From Model Graph to Distributed Decode Step

A model graph consists of the token embedding, the decoder layers, and the final norm and LM head. The decoder layers are one repeated block with a repetition count; Megatron 5T repeats one layer, GPT-OSS-5.3-like a pair of a sliding-window and a full-attention layer. Tensor shapes are SymPy expressions in the batch size and in the mesh coordinates of the rank. A sequence of graph-to-graph passes turns the model into a timed distributed decode step:

1. Decode. The graph is specialized to generating one token per user against a KV cache that holds the context.

2. Precision. Activations, weights, and the KV cache are stored in FP4, and collectives communicate activations in FP4; Appendix § 17 repeats the evaluation with BF16 communication.

3. Helix sharding. Attention is placed on a KVP × TPA mesh, which shards the KV cache by sequence over KVP and by KV head over TPA, and the FFN on an EP × TPE mesh over the same accelerators. Collectives are inserted as described in § 4 (Figure 26).

4. Kernels. Operators are grouped into fused kernels, e.g., the QKV projection, flash-decode attention, and the grouped expert GEMM.

5. Resources and time. Each kernel is annotated with its FLOPs and HBM traffic, and then with a duration (Appendix § 12.§ 3).

6. Schedule. Each layer is split into an attention and an FFN region (four regions for GPT-OSS-5.3-like), which execute in order without HOP-B microbatching (Bhatia et al., [1]) (Appendix § 12.§ 5).

The same sharded graph also yields a per-accelerator memory estimate (weights and KV cache), which bounds the batch size that fits in HBM.

MoE Load Model

Since the all-reduce after the attention output projection replicates the expert input on every accelerator, token routing changes no collective (§ 3.1). It only sets how many tokens each expert processes, and with it the expert compute time and the payload of the TPE all-reduce and EP all-gather. We model dropless routing that is uniform in expectation. With $B$ users and top-$k$ routing over $N_{\mathrm{exp}}$ experts, the number of distinct active experts is approximately

rounded to the nearest integer. The active experts are spread over the EP ranks, so each rank holds $\left\lfloor N_{\mathrm{act}}/\mathrm{EP}\right\rfloor$ or $\left\lceil N_{\mathrm{act}}/\mathrm{EP}\right\rceil$ of them, and its grouped expert GEMM processes $\left\lceil Bk/N_{\mathrm{act}}\right\rceil$ tokens per active expert. The collectives after the experts carry the rows of the busiest rank, its share of the routed tokens,

rounded up once for the rank rather than once per expert. MoE payloads are therefore sublinear in $B$ (Table 1). We do not model routing skew, which would raise expert compute and payload on the most loaded EP rank.

Compute and Memory Cost Model

The duration of a kernel is a roofline estimate,

where $F$ is the kernel’s FLOPs, $D$ the bytes it moves to and from HBM, $P_{\mathrm{FP4}}$ and $W_{\mathrm{HBM}}$ the peak FP4 throughput and HBM bandwidth of Table 11, and $\eta_F$ and $\eta_D$ size-dependent efficiencies. For RX00, $\eta_D$ is 0.3 for transfers up to 1 MB, 0.6 up to 10 MB, 0.75 up to 100 MB, and 0.9 above; $\eta_F$ is 0.15 for kernels up to 7 GFLOP, 0.36 up to 16 GFLOP, 0.53 up to 127 GFLOP, and 0.73 above. A grouped expert kernel reads the weights of every active expert on its rank. The efficiency functions are a step approximation of curves measured on H100 (Appendix § 15.§ 3).

Accelerator	FP4 peak	HBM capacity	HBM bandwidth	Scale-up ports	Per-port bandwidth (unidirectional)	Kernel launch	In-graph launch
NVIDIA B200	20 PFLOP/s	186 GB	8 TB/s	18	400 Gb/s	2.5 $\mu$s	0.8 $\mu$s
NVIDIA RX00 (Rubin-like)	50 PFLOP/s	288 GB	22 TB/s	36	400 Gb/s	2.5 $\mu$s	0.8 $\mu$s

Kernel launches are charged per rank: the first kernel after each collective pays one launch overhead, and the kernels that follow until the next collective run back to back, as in an accelerator execution graph or persistent-kernel implementation. We use 0.8 $\mu$s, the per-kernel gap of a replayed CUDA graph on B200 (Cheng et al., [13]).

Execution and Network Coupling

The schedule is compiled into a plan of compute entries with fixed durations and communication entries whose durations are left to the network model. Each accelerator issues its compute entries in order, as from a single stream. Ranks with identical durations and dependencies share an entry, so the plan size depends on the number of distinct rank behaviors, which arise only where the active experts do not divide evenly over the EP ranks.

An entry starts when its last predecessor ends. A communication entry registers its collective with the network model at its start time, and its end stays open until the network model reports it. When no further collective can be registered, the network model advances to the next completion, and the reported end times are substituted into the entries that depend on them. The network model therefore holds every collective that is in flight at a given time, e.g., the concurrent groups of one mesh axis, which share second-level links on the two-level fabrics. Simlette simulates these collectives together, so link contention and credit stalls between them lengthen their completion times, and the compute that waits for them starts later. Pricing each collective in isolation would miss both effects; the analytical model does so. Before the walk, the network model receives the traffic demand of Table 1, against which the OCS planner builds circuits (§ 3); a registered collective outside this demand is an error.

Without HOP-B, no communication overlaps with computation, so every collective lies on the critical path and a decode step is the sum of kernel durations, launch overheads, and collective completion times. The OCS saving per step is the reduction in collective completion time. This also matches the public implementations of Helix: TensorRT-LLM provides Helix parallelism, but its documented implementation does not yet include HOP-B’s batch-wise pipelining of attention computation and all-to-all communication, which NVIDIA lists as future work [37], and the Helix all-to-all backends recently added to vLLM (vLLM Project, 2026b) and SGLang (SGLang Project, 2026) do not describe such overlap. The following subsection Appendix § 12.§ 5 shows that HOP-B would not shorten the evaluated decode steps.

Sensitivity to HOP-B

HOP-B [1] splits the attention of a decode step into microbatches and overlaps the KV-parallel all-to-all of one microbatch with the attention computation of the next. The QKV projection, the FFN, and their collectives, i.e., the all-reduce after the output projection, the FFN all-reduce and expert all-gather, and the logit all-gather, still run once on the whole batch and are not overlapped. Figure 16 evaluates it for both models at batch 16 and 5M context, the largest batch and context of § 6, which give HOP-B the most computation to overlap with.

Microbatching does not reduce the bytes of the all-to-all, but it issues the all-to-all once per microbatch in every layer. Its payload is divided among the calls, but each additional call adds the fixed latency of a collective, which dominates for the small payloads of decode (Table 1), so the communication time before overlap grows with the number of microbatches. HOP-B can hide this communication only behind attention, the all-to-all of the last microbatch stays exposed, and the decode step becomes shorter only if the hidden communication exceeds the added communication.

For GPT-OSS-5.3-like, it does not. A decode step computes for only 0.77 ms, half of its layers attend to a 128-token window, and HOP-B hides at most 0.34 ms of communication, while eight microbatches add 1.6 ms of communication on all-EPS. Against the step without HOP-B, the decode step grows by 6, 22, and 70% on all-EPS with 2, 4, and 8 microbatches, and by 4, 11, and 29% on all-OCS, whose collectives have the lowest latency per call. For Megatron 5T, attention over the 5M-token context dominates the 8.8 ms of computation per step and hides all added all-to-alls, but the exposed communication does not shrink: the all-to-all of the last microbatch takes as long as the single all-to-all without HOP-B, and the other collectives are not overlapped. The decode-step time is therefore unchanged on every fabric.

For the evaluated model-parallel configurations, microbatching with HOP-B does not improve decode-step time relative to a single microbatch. For GPT-OSS-5.3-like, the additional collective startup costs exceed the communication hidden through overlap. For Megatron 5T, the additional communication is almost entirely hidden, but the remaining exposed communication is unchanged. The single-microbatch configuration is therefore optimal or equivalent among the evaluated schedules, supporting its use in § 6.

Searching the Parallelization Space

On $N$ accelerators, a Helix split (KVP, TPA, EP, TPE) satisfies $\mathrm{KVP}\cdot\mathrm{TPA}=\mathrm{EP}\cdot\mathrm{TPE}=N$, with power-of-two degrees, TPA dividing the number of KV heads, EP dividing the number of experts, and TPE dividing the expert width. At $N = 512$, this gives 72 splits for Megatron 5T and 45 for GPT-OSS-5.3-like.

For Figure 4, we choose a split per cell, i.e., per model, context (32K and 5M tokens), batch size (1, 2, 4, 8, and 16), and fabric. GXNET with the analytical network model of § 5.4 times every split in every cell: 2160 points for Megatron 5T and 1350 for GPT-OSS-5.3-like, each simulated in under a second. The analytical model ignores contention and credit stalls (§ 5.4), so we re-simulate with Simlette every split whose analytical interactivity is within 7% of the best split of its fabric in that cell, and report the best Simlette result per cell. This amounts to 758 points for Megatron 5T and 338 for GPT-OSS-5.3-like, with a median simulation time of about 10 minutes per point.

Collective Algorithms and Time Consumption in the Network Simulator

This appendix details how Simlette models collective communication on the evaluated network fabrics. It describes the simulator’s mapping stages, derives the analytical timing model, and explains the collective schedules and their consistency checks. It then presents simulation optimizations, hardware measurements used to parameterize latency, and the physical fabric configurations used in the evaluation.

Simlette Layers

Simlette separates the OCS-specific problem of mapping an ML workload to the physical network fabric into four layers, applied in the following order. The first is the topology: the physical host graph $H = (V, E)$ of devices, switches, and the ports that connect them, under the per-accelerator port budget $L$ introduced in § 3.3. The second is

worker allocation: the mapping of logical ranks onto this physical graph, i.e., the embedding of a logical overlay (§ 3.2) into the accelerators and links that will actually carry its traffic. The third is the collective, which supplies the schedule defined in § 3.4: the ordered sequence of phases, the rank pairs and edges used at each phase, and the payload carried on each. This layer jointly searches over the collective and its logical topology, keeps the lowest-cost pair, and thereby determines the number of phases $h_i$. The fourth is routing, which takes the chosen logical topology together with the physical topology and the allocation, and maps it onto concrete ports, paths, and circuit assignments. These four layers stay independent in the implementation.

Derivation of the Analytical Model

We compute the terms of Eq. (1) from the schedule of § 3.4. Let $b_{k,e}$ be the amount of data, in bytes, sent on logical edge $e$ during phase $k$. All transfers of a phase start at the same time, and the phase ends when the transfer on the most loaded edge arrives. This transfer takes the latency $\lambda$ of one hop, plus the time to put its data on the edge. The edge is made of $p$ parallel circuits of bandwidth $R$ each, expressed in bytes per second (400 Gb/s corresponds to $R = 50\,\mathrm{GB/s}$), so sending $b_{k,e}$ takes $b_{k,e}/(pR)$ and phase $k$ lasts $\lambda+ \max_e b_{k,e}/(pR)$. Adding the $h$ phases gives Eq. (1), with the bottleneck volume

i.e., the sum, over all phases, of the data carried by the most loaded edge of each phase. This assumes that phases do not overlap, and that no other collective uses the same links at the same time.

Collective Algorithms for Workload-static Switching

Each collective algorithm in Simlette computes $h$, the schedule phase count, and $v$, the bottleneck data volume, from its own phases, for a buffer size $\sigma$ on a logical graph of $N$ ranks.

All-reduce schedules

Simlette implements the all-reduce schedules of Appendix § 9.§ 3: the one-shot schedule, which reaches every rank within the diameter of the graph, and its data-volume-optimized variant (one-shot DVO), as well as the coset-factored and RS+AG schedules and their parallel variants, which exchange over one coset layer per phase (Table 4 and Table 5).

Example: Analytical completion time of the all-reduce algorithms Figure 17 shows the completion time of the all-reduce algorithms on $N = 512$ accelerators, from 1 KiB to 16 MiB. For small buffers the latency term $h\lambda$ is the largest, and the time depends mostly on the number of phases. For example, the two RS+AG algorithms take 18 phases, and their latency term is $18 \times0.52 = 9.4\,\mu\mathrm{s}$. The other algorithms take only 9 phases, and their latency term is $9 \times0.52 = 4.7\,\mu\mathrm{s}$.

For large buffers the bandwidth term $v/(pR)$ is the largest, and the algorithm that sends the least data is the fastest. Coset-factored sends the full buffer in each of its 9 phases. RS+AG sends only a part of the buffer in each phase, and about twice the buffer in total. RS+AG takes twice as many phases, but sends less data. The two curves cross at the buffer size where the extra phases of RS+AG cost as much time as the extra data of coset-factored. Below this size coset-factored is faster, and above it RS+AG is faster. Their parallel variants keep the same number of phases, but send 9 times less data on the most loaded link.

One-shot is the slowest at every size. At each phase, a rank sends to each neighbor every contribution it has received so far. This number grows from one phase to the next, and the same contributions cross the same links many times.

Topology-schedule interaction Figure 18 shows the topology-schedule interaction at network level. The collective is a single all-reduce over $N = 64$ accelerators, with buffer sizes from 1 KiB to 16 MiB in powers of two. The schedule is coset-factored parallel, and the physical fabric is a single-level all-OCS with a radix of 13 ports per accelerator, of which $L = 12$ are used for circuits, and a bandwidth of $R = 400$ Gb/s per link.

All-to-all schedules

r-Bruck (radix-$r$ Bruck all-to-all on a block fold, r-Bruck takes $h = \ell$ phases on a fold of $\ell$ layers. The all-to-all is uniform: every rank starts and ends with the same payload $\sigma$, so $v$ is the same for all ranks. r-Bruck implements the Helix all-to-all of Table 1 in our evaluation.

Analytical Model against Simlette

The analytical model describes what we expect when nothing gets in the way, so without congestion Simlette should

give the same results. To compare the two, we run single all-reduces on 32 accelerators connected by a single-level fabric with 400 Gb/s links, with buffer sizes from 1 KiB to 1 MiB and 8, 12, 16 or 24 ports per accelerator. All runs use persistent kernels, so a collective is complete when its last data arrives, as in the analytical model.

OCS. Figure 19 uses the one-shot all-reduce, whose schedule never sends two flows on the same link at the same time, so there is no link contention. With an unlimited buffer, Simlette and the analytical model give the same results for all buffer sizes and radices, which shows that the simulator’s behavior is consistent with the model. If the buffer on each port is too small to keep a link busy for one hop, senders have to wait for credits, and Simlette is up to $1.9\times$ slower than the analytical model.

EPS: credit stalls. Figure 20 uses the same one-shot all-reduce on EPS, again without link contention. With an unlimited buffer, Simlette and the analytical model agree within 5% for small buffers and for buffers of 256 KiB and larger. For 16–64 KiB, however, Simlette is up to 29% slower: a buffer this small yields per-port messages of only a few flits, which cannot be divided evenly across the links, so the busiest link carries more data than the analytical model assumes. With the buffer of every port reduced below what one hop needs to keep its link busy, senders wait for credits, which makes Simlette 1.6–1.7$\times$ slower than the analytical model. For example, with a 1 MiB buffer and 8 ports, the analytical model gives 82.1 $\mu$s and Simlette 135.0 $\mu$s; Simlette is 52.9 $\mu$s slower, and its most delayed port spends 53.4 $\mu$s waiting for credits.

EPS: link contention. Figure 21 uses the two-shot all-reduce on EPS with an unlimited buffer, so there are no credit stalls. Two-shot consists of a reduce-scatter followed

ends_mid=1 by an all-gather. In each phase, all accelerators send to the others in the same order, so they all target the same destination at the same time, and the switch port leading to that destination becomes congested. From 16 KiB onwards, Simlette is 1.5-$2 \times$ slower than the analytical model, which counts the right amount of data on each link but cannot capture the time flows spend waiting for each other. The optimized mode of Simlette (Appendix § 13.§ 5) gives almost the same results as the flit-by-flit mode, with less than 2 % difference.

Simlette Simulation Procedure Optimizations

Most of the latency along a hop is not paid at the point where it is decided in the simulation. When a switch or a scale-up port adds its share of delay, that delay is added to a timestamp carried on the message, while the operation handing the message to the next stage happens immediately, at the same simulated instant. Simulated time only moves forward at a few points: right before a port puts a message on the link, during serialization, and when a flit reaches its destination.

This deferred accounting matters because several flits are usually in flight at once: by the time a port is ready to send one, part of the delay it owes may already have elapsed while other flits were being processed elsewhere, so only the remaining gap, if any, still needs to be paid as a real wait. Simlette computes latency this way so that it always knows exactly how much of that gap is left to catch up, and never pays for the latency twice. The simulation stays exact even though every delay is resolved only when needed.

Simulating a message flit by flit means creating one event for every flit a device sends, rather than a single event for the whole message. At scale, this overhead itself costs real time to execute. A single all-reduce on a simple single-level topology with 256 accelerators and a 40 KiB buffer already takes about 14 minutes to simulate flit by flit. Larger buffers and more devices in the topology push this further, up to several hours. This is why Simlette also supports an optimized mode: the entire message is handed to the link in a single event, and the delay it would have accumulated flit by flit is charged at once on arrival. This way, only one event is created per message per transfer, instead of one event per flit per transfer. A flit is 640 bytes and a message can be tens of kilobytes, so flit-by-flit mode schedules dozens of events where the optimized mode schedules one; with the 40 KiB buffer of this experiment, a message spans up to 64 flits. As shown in Figure 21, both modes give almost the same completion times. Figure 22 compares the runtime of both modes as a function of accelerator count, on a fixed-radix topology (8 ports per accelerator) with a 40 KiB buffer, and shows the resulting speedup directly. From 32 to 256 accelerators, the optimized mode is consistently 7-$9 \times$ faster.

For the 256-accelerator topology, this cuts runtime from 827 seconds down to 90 seconds.

Determinism The simulation is currently fully deterministic: given the same topology, routing, and workload, every run produces the exact same completion time, with no randomness in link timing, arrivals, or failures. This makes results easy to reproduce and attribute to a single cause, but it does not capture the variability real hardware shows, so the completion times we report should be read as a best case rather than a distribution. This is particularly true for EPS based fabrics, where network-internal dynamics such as arbitration are well known to cause collective completion tail latencies.

Scale-up Latency Profiles

End-to-end measurements. We characterize accelerator-to-accelerator communication latency using several ping-pong kernel implementations. Figure 23 illustrates the substantial implementation dependence of measured round-trip times. For model parameterization, we use the minimum observed latencies as an optimistic estimate of the hardware communication path.

Latency decomposition. Figure 24 shows the latency component decomposition of network transactions on scale-up networks as modeled by the analytical network backend of GXNET and as used for the parametrization of our flit-based network simulator interfacing with GXNET. Note that this is not official NVIDIA data. The bare metal measurements carried out in § 5.5 can only provide a limited number of known values to a set of equations that will then be underspecified. Specifically, allocating latency to the sub-components “GPU MMU”, “TX”, “RX”, “SWITCH” (core switch latency) and “LINK TLB” is a matter of educated guesses. We believe that budgeting more latency into RX rather than TX sections is well justified by the FEC decode overhead alone.

Results for collective completion times in inference decode, especially for low-batch/high-interactivity cases, will be very susceptible to rebalancing spent latency between TX/RX stages and switch core. A lower switch pin-to-pin latency than the unloaded 300 ns assumed in this paper (UALink Spec., Sec. 10.9.2) would translate into a smaller performance advantage for all-OCS based network fabrics - and this for two separate reasons: first, the latency difference between EPS and OCS switches as such would reduce. More importantly however, OCS fabrics rely on relaying information through the accelerator itself. A lower EPS pin-to-pin latency directly translates into higher accelerator ingress/egress latency because the absolute PE-to-PE latency must stay the same.

However, the opposite is of course also true: accelerators with very low ingress/egress latencies will emphasize the advantage of all-OCS fabrics. In light of recent industry trends to offload processing elements from collective communication tasks and introduce hardware based “collective acceleration engines” into the architectures (e.g., TPU 8i (Google, [21])), the importance to lower switch pin-to-pin latencies in the network will grow in future systems.

Memory-hierarchy latency. The accelerator-specific terms of the scale-up latency decomposition cover the path from the PE through the cache hierarchy to the scale-up port. We anchor them by replicating the pointer-chasing results of Table 3 in (Luo et al., [33]) with the open-source microbenchmarks accompanying that paper, run on H100 and B300 (Table 8). On H100 we measure ≈41 cycles (L1), ≈29 cycles (shared memory) and ≈276 cycles (L2), matching the published H800 values for shared memory and L2 and slightly exceeding the published L1 latency. On B300 the same binaries yield ≈34, ≈28.6 and ≈277 cycles. Reading the GPU global timer from PTX instead of the cycle counter, and averaging over 30,000 samples, gives ≈14.0 ns, ≈16.7 ns and ≈139.7 ns for shared memory, L1 and L2 respectively, without assuming an SM clock frequency. Two observations matter for the model: on-SM and L2-hit latency is effectively unchanged across two GPU generations, so it can be carried into future-accelerator scenarios with little risk; and L2, the deepest on-chip stage a remote access traverses, sits below 150 ns and is therefore a small fraction of the accelerator-to-accelerator path (Figure 24).

	RTX4090	A100	H800	H100	B300
	[33]	[33]	[33]	(ours)	(ours)
L1 cache	32.0	33.0	32.0	41.1	34.1
Shared memory	30.1	29.0	29.0	29.0	28.6
L2 cache	273.0	202.8–408	264.5–502	276.2	276.8

Scale-up latency parametrization variants The latency profile break-down presented above has to work with a set of assumptions for the per-component latencies on the data path. In order to study sensitivity of our results to latency attribution variations, we choose three different attributions that either emphasize or de-emphasize the contribution of the EPS switch core (Table 9).

Case	Switch total	Switch core	MMU + TLB	GPU MMU	Link TLB	Total
Low	200	100	350	217	133	810
Nominal	300	200	250	155	95	810
High	500	400	50	31	19	810

Physical Fabrics

We evaluate three scale-up fabric families as reference points for where optical circuits enter the accelerator-to-accelerator path. In all cases, each accelerator exposes $L$ links of bandwidth $R$. We describe both packet- and circuit-switched fabrics by the paths available to collective schedules: hop count, per-hop latency, available bandwidth, and port consumption.

The all-EPS baseline represents a conventional electronic-packet-switched scale-up fabric. It provides flexible any-to-any connectivity, but traffic pays the latency of the electronic switching stages it traverses. The hybrid EPS/OCS fabric keeps local connectivity electronic while replacing the level-2 switches with deployment-time optical circuits, making it an incremental path from current EPS fabrics toward optical scale-up. The all-OCS fabric connects accelerator links directly to an optical circuit-switched fabric whose circuits are planned before serving starts and kept fixed throughout deployment.

The fabrics therefore differ in runtime routing flexibility and in how much electronic switching latency remains on accelerator-to-accelerator paths. In the all-OCS case, the full port budget $L$ can be assigned to the logical overlays induced by decode, either to reduce serialized communication phases, widen selected logical edges, or support multiple phase-specific overlays over the same accelerator set.

Concretely, each accelerator exposes $L = 36$ links of $R = 400$ Gb/s and all switching devices have a radix of 72, following NVL72-class systems. We instantiate a two-level electronic packet switch (EPS) based network (all-EPS), the same network with the devices on level 2 replaced by OCS (hybrid EPS/OCS), and a third topology in which all switching devices are OCS (all-OCS). In the all-EPS and hybrid fabrics, each level-1 switch uses 32 ports towards the accelerators and 8 towards level 2, i.e., a 4:1 oversubscription towards the second level, since a fully subscribed EPS spine with pluggable conversion exceeds a reasonable power envelope. The all-OCS fabric instead uses 32 ports in each direction and is thus non-blocking: its OCS devices require no E/O conversion and consume so little power per port (Appendix § 15.§ 4) that the full uplink bandwidth still fits into a lower system power than either EPS-based fabric. Consequently, part of the all-OCS advantage for collective groups that span level 2 stems from this higher level-2 bandwidth rather than from the switching technology alone; the energy-efficiency results of § 6 account for the full port count of each fabric.

Note that while the EPS based fabric uses all traits well-known to EPS based scale-up networks such as multi-path routing and packet spraying in accelerators and switches, the all-OCS fabric runs on the collective schedules presented in § 3.4, which in turn make use of the topologies described in § 3.2. The hybrid EPS/OCS fabric configures the OCS on the topmost level once at deployment, using the same planner to place workers such that bandwidth demand is balanced across the level-2 OCS devices, and otherwise relies on the same collective operations as implemented by current inference frameworks such as TensorRT-LLM [37], vLLM [29] or SGLang (Zheng et al., 2024).

Models and Accelerators

This appendix documents the model architectures, accelerator parameters, and power assumptions underlying the evaluation in § 6, with model and accelerator specifications summarized in Table 10 and Table 11. It also presents energy-efficiency results across the GPT-OSS family and examines the compute cost model against measured kernel durations, clarifying the assumptions and limitations of the projected accelerator results.

Models

Figures 26a and 26b show one decoder layer of each model. GPT-OSS-5.3-like is our extrapolation of the open-weight GPT-OSS-120B to the 5T scale. It keeps the GPT-OSS architecture recipe, namely interleaved sliding-window and full attention with a 128-token window and attention sinks, eight query heads per KV head, experts as wide as the hidden size with four active per token, and the 201,088-token vocabulary. It scales the hidden size from 2880 to 8192, the depth from 36 to 96 layers, the attention from 64 query and 8 KV heads to 128 and 16, and the expert count from 128 to 256.

The two models share their overall shape. Both are prenorm decoders of about 5T parameters with a hidden size of 8192, in which grouped-query attention is followed by a mixture-of-experts FFN of 256 SwiGLU experts. Under Helix parallelism both issue the same five collectives at the same points of the layer, so the deployment-static structure of Table 1 is identical between them apart from the layer count. Because the hidden size is the same, the all-reduces and the expert all-gather also carry the same 4 KiB per token in both models.

The models differ above all in their attention, which decides how a decode step responds to context length.

• Interleaved sliding-window attention. GPT-OSS-5.3-like alternates between layers that attend only to the most recent 128 tokens and layers with full causal attention. The KV cache and attention time of a sliding-window layer do not depend on context length, so only 48 of its 96 layers read the whole context. Every one of Megatron’s 128 layers does. Removing the sliding window from GPT-OSS-5.3-like in GXNET doubles the part of its attention time that grows with context, from 0.39 to 0.77 ms per decode step between 32K and 5M context at batch 16.

• Narrow grouped-query attention. The full-attention layers of GPT-OSS-5.3-like keep 16 KV heads, eight query heads each. Megatron keeps 128 KV heads under 256 query heads, which also makes its attention 16,384 wide, twice the hidden size. Giving GPT-OSS-5.3-like Megatron’s 128 KV heads multiplies its context-dependent attention time by eight, to 3.1 ms in the same setting.

• KV cache. Together, the two points above give Megatron 1 MiB of KV cache per token of context against 48 KiB for GPT-OSS-5.3-like, a factor of 21. At 5M tokens one Megatron user occupies 5.2 TB of HBM and one GPT-OSS-5.3-like user 246 GB. This is what bounds the evaluated batch to 16 for Megatron on 512 accelerators, and it is why decode attention time grows much faster with batch and context for Megatron than for GPT-OSS-5.3-like (§ 6).

• Experts. Megatron’s experts are fine-grained (6144 wide, 0.75 of the hidden size) with eight active per token; those of GPT-OSS-5.3-like are as wide as the hidden size with four active. Megatron therefore activates 207 B parameters per token against 95 B.

• Vocabulary and details. GPT-OSS-5.3-like has a four times larger vocabulary, which only enlarges the LM head and the final logit all-gather. Its attention sinks and projection biases do not change the communication structure.

Energy Efficiency across the GPT-OSS Family

Figure 27 repeats the comparison of Figure 7 for three models of the GPT-OSS family: the open-weight GPT-OSS-120B on 8 accelerators at 32K context and batch 3413, GPT-OSS-1T-like on 64 accelerators at 128K context and batch 1024, and GPT-OSS-5.3-like on 512 accelerators at 5M context and batch 1. GPT-OSS-1T-like is an intermediate extrapolation of GPT-OSS-120B built like GPT-OSS-5.3-like, with a hidden size of 4608, 64 layers, 72 query and 8 KV heads, and 256 experts (Table 10). All-OCS fabrics improve throughput per unit energy by 32%, 67% and 119%, respectively. For GPT-OSS-120B, whose deployment spans only eight accelerators, interactivity is practically unchanged (−1%), so the gain stems from system power alone; for GPT-OSS-1T-like, interactivity improves by 26%, and for GPT-OSS-5.3-like, whose decode step is dominated by communication (Figure 5), by 65%.

Accelerators

RX00 is a projected accelerator modeled on announced Rubin figures (as announced at GTC in March 2026 ([37])), so its kernel durations cannot be measured. Its parameters in Table 11 are peak throughput and bandwidth; the efficiency functions of Appendix § 12.§ 3 are a step approximation of curves measured on H100, which we use unchanged for B200 and RX00.

The cost model is a roofline with size-dependent efficiencies and no fixed cost per kernel. Operators between two collectives form one execution window that pays a single in-graph launch overhead, which assumes they run back to back as in a persistent kernel (Cheng et al., [13]). The model therefore gives a lower bound on compute time rather than a prediction of a particular kernel implementation.

We choose a roofline deliberately. Predicting kernel durations more precisely requires either profiling operators on the target accelerator, as in Vidur (Agrawal et al., [3]), or models fitted to measurements of existing kernels, such as PipeWeave (Zhang et al., 2026). The first is impossible for an unreleased accelerator. The second generalizes to held-out GPUs of architectures seen in training, e.g., from H20 and H800 to H100 and H200, but has not been shown to carry over to a new architecture generation, and reports no separate error for the small batch sizes of decode, where durations depend on kernel setup, tiling, and occupancy rather than on FLOPs and bytes. Such models can be more accurate on the hardware they were fitted to, but their error on a new architecture has neither a known size nor a known sign. The error of a roofline has a known sign: it underestimates.

We compared the model with Nsight Systems traces of GPT-OSS-120B decode on four B200 with tensor parallelism and CUDA-graph replay, at batch sizes 1 to 16. The decode kernels there read a few to a few tens of MB per accelerator and reach 12 to 53 % of the peak HBM bandwidth, below the efficiencies of Appendix § 12.§ 3, so the H100 curves remain optimistic one generation later. Even under CUDA-graph replay, no kernel finishes in less than about 3 $\mu$s. These decode kernels therefore take 1.4 to 5 times as long as their roofline estimate, and more for the smallest ones. A persistent-kernel implementation avoids part of this per-kernel cost, which is the execution our fused windows assume.

Since the model underestimates compute time, the share of communication in a decode step, and with it the relative OCS gain of § 6, is an upper estimate, while the absolute reduction of collective completion time per step does not depend on the compute model. For existing accelerators, GXNET can replace the roofline estimate of any operator with a measured kernel duration without changing the graph or the sharding.

Power Model

Our power models are based on publicly available data. In conjunction with the topologies described in Appendix § 13.§ 7, we always assume that moving between scale-up network levels 1 and 2 requires optical cabling (due to rack-to-rack distance) and that all-OCS based topologies make use of CPO at the accelerator. The power figures we use for the system components are given in Table 12, together with their origin wherever applicable.

Component	Variant	Value	Unit	Derivation / source
XPU (incl. host share)	B200 (GB200 NVL72)	1454	W	$(P_{\mathrm{rack}} - 9P_{\mathrm{sw}})/72$, $P_{\mathrm{rack}} = 120\,\mathrm{kW}$ (NVIDIA, a;b), $P_{\mathrm{sw}} = 1.7\,\mathrm{kW}$
	B300 (GB300 NVL72)	1663	W	as above, $P_{\mathrm{rack}} = 135\,\mathrm{kW}$ (NVIDIA, b; Lenovo Press)
	Rubin (VR200 NVL72)	2631	W	as above, $P_{\mathrm{rack}} = 220\,\mathrm{kW}$ (Max-P) (SemiAnalysis, c; Supermicro), $P_{\mathrm{sw}} = 3.4\,\mathrm{kW}$ (4 ASICs/tray (NVIDIA, c))
EPS (scale-up), per port	400 G	11.81	W	switch tray: $P_{\mathrm{sw}}/(2 \times72)$; cross-check (Broadcom, 2025; TechInsights; IP Infusion)
	800 G	15.35	W	extrapolated: 2× rate, −35 % energy/bit (Arista Networks, a;b)
	1600 G	19.95	W	extrapolated: 2× rate, −35 % energy/bit
OCS, per port	200 G	1.36	W
	400 G	1.50	W	internal projections; unidirectional, two fibers per port
	800 G	1.65	W
	1600 G	1.82	W
Pluggable module	400 G DR4	9	W	vendor datasheet, max (Cisco)
	800 G DR4	17	W	(SemiAnalysis, a)
	1.6 T	30	W	(SemiAnalysis, b)
CPO, per port	400 G (6.9 pJ/bit)	2.75	W	Broadcom Bailly, 5.5W per 800 G (Broadcom)
	800 G (5.6 pJ/bit)	4.50	W	NVIDIA Quantum-X Photonics, 4–5 W per 800 G (SemiAnalysis, a)
	1600 G (4.4 pJ/bit)	7.00	W	Broadcom TH6 Davisson, 3.5 W per 800 G (The Next Platform, 2025)

Table 12 lists the power figures used by the model. All values are thermal-design (maximum) figures, so the model gives an upper bound on power independent of traffic. Per-XPU power is not the accelerator TDP. We take the designed rack power of an NVL72-class system ((NVIDIA, b)), subtract the scale-up switch trays, and divide by the 72 accelerators. The resulting figure therefore includes each XPU’s share of host CPU, NICs, DPU, fans and voltage-regulation losses.

No public figure exists for the GB200/GB300 switch tray, so we assume 1.7 kW. We cross-check this value by scaling the published power of a 51.2 Tb/s scale-up Ethernet switch ASIC (800 W (Broadcom, 2025)) to the two 28.8 Tb/s switch ASICs per tray (SemiAnalysis, b). We then apply the system-to-ASIC overhead observed for Tomahawk 5 (913 W system without optics (IP Infusion) versus about 500 W ASIC (TechInsights)). This yields about 1.64 kW, within 4 % of our assumption.

No rack power has been published for Vera Rubin NVL72. We use the 220 kW Max-P estimate of (SemiAnalysis, c), which agrees with the 227 kW per-rack cooling sizing of an OEM reference design (Supermicro). The rack’s power shelves (4 × 110 kW, N+1) define a provisioning ceiling, not a TDP. The Rubin tray holds four switch ASICs instead of two (NVIDIA, c). Each ASIC keeps the per-ASIC bandwidth of 28.8 Tb/s (SemiAnalysis, c), so we double the tray power. Per 400 G-equivalent port, the scale-up switching power is therefore the same as for GB200.

EPS ports at 800 G and 1.6 T are extrapolated from the 400 G anchor assuming a 35 % energy-per-bit reduction per doubling of port rate. This is the reduction observed between the 25.6 and 51.2 Tb/s generations of switch systems measured on the same basis (typical power, excluding optics) (Arista Networks, a;b).

OCS figures are internal projections for a switch whose ports each use two unidirectional fibers. The power per port is 1.5 W at 400 G.

Every switching device, EPS or OCS, is evaluated with all of its physical ports, whether a port is used or not. On the level-1 switches of the EPS-based fabrics, which use 40 of their 72 ports, this is conservative, but the per-port figure amortizes device-level power, such as that of the switch ASIC core, which does not scale down with unused ports.

Optical links attach either through one pluggable transceiver per port or through co-packaged optics (CPO). For CPO we use the energy per bit published for switch-side optical engines, including the external laser source, and multiply by the unidirectional rate. We assume the same CPO energy per bit when an accelerator attaches directly to the level-1 OCS switches.

Sensitivity Analysis

This appendix examines the sensitivity of the evaluation results to network bandwidth, accelerator radix, switch and endpoint latency, and communication precision. These analyses assess how the relative performance and energy efficiency of the evaluated fabrics depend on the baseline configuration and modeling assumptions.

Figure 28 varies one fabric or accelerator parameter at a time for GPT-OSS-5.3-like at batch 1 and 5M context, the latency-bound regime of § 6. The parallelization is re-optimized for every point, and all four sweeps share the baseline of § 6, with a 64.8 % interactivity and a 118.7 % energy-efficiency advantage of all-OCS over all-EPS. Since energy efficiency is throughput per system power, the energy advantage is the interactivity advantage multiplied by the ratio of all-EPS system power to that of the respective fabric, which is 1.33 for all-OCS at the baseline.

Per-port bandwidth. At batch 1, the interactivity advantage of all-OCS is nearly insensitive to bandwidth, staying between 62.8 % and 65.2 % between 400 Gb/s and 1.6 Tb/s. Since the all-EPS baseline is oversubscribed towards its second level, it carries a bandwidth-bound component that shrinks with per-port bandwidth, whereas the non-blocking all-OCS fabric has little to gain. The energy advantage of all-OCS nevertheless grows from 119 % to 161 %, as the power ratio rises from 1.33 to 1.60: per rate doubling, EPS port power grows by 30 % and pluggable power almost doubles, while OCS port power grows by only 10 % and the energy per bit of CPO decreases (Appendix § 15.§ 4). The hybrid fabric does not benefit, since it retains electronic level-1 switches and pluggables on every link between the two levels.

Accelerator radix. Co-packaged optics allow the accelerator radix to grow. For inference decode, additional ports pay off substantially at first: the interactivity advantage of all-OCS rises from 65 % at 36 ports to 88 % at 48 ports and 100 % at 64 ports, but then saturates at 104 % and 103 % for 96 and 128 ports. Along the way, the planner changes the parallelization of all-OCS: it keeps the baseline split up to 48 ports (32/16/2/256 in KVP/TPA/EP/TPE), spreads the FFN over a single tensor-parallel group of all 512 accelerators (32/16/1/512) at 64 and 128 ports, and uses 32/16/8/64 at 96 ports, so additional ports buy freedom of parallelization as much as speed. The saturation follows from the schedulable frontier of Appendix § 9.§ 2: phase counts drop in discrete steps, each further step costs considerably more ports than the previous one, e.g., two phases on an axis of 64 accelerators cost 14 ports and one phase 63, and ports between two steps buy no phase reduction. At a fixed system size, more accelerator radix therefore affords larger scale-up domains, or links for network-attached memory, rather than higher interactivity directly. In energy efficiency, however, the advantage of all-OCS keeps growing to 293 %, as the power ratio rises from 1.33 to 1.94 with every port added to the packet-switched fabric.

Switch latency attribution. Our measurements (§ 5.5) cannot separate accelerator from switch latency, and the unloaded 300 ns pin-to-pin latency of the UALink specification (Ultra Accelerator Link Consortium, 2025) may be exceeded under load. We therefore keep the 810 ns per switch hop of Table 9 constant and shift latency between accelerator and switch core. Since the all-OCS path by-passes the switch core, every nanosecond moved into the switch shortens it: the interactivity advantage grows from 30 % to 65 % and 160 % for switch core latencies of 100, 200, and 400 ns, at a constant power ratio of 1.33. The results are thus highly sensitive to where latency is spent, which explains the effort of recent architectures to keep latency-critical traffic within a single switch level: Jalapeño bifurcates its links into separate single-level domains for tensor and expert parallelism (Ho et al., 2026), a design-time counterpart of the stacked-link allocation of Appendix § 9.§ 6.

Accelerator latency. Reducing accelerator egress and ingress latency benefits both fabrics, but all-OCS considerably more, since its per-hop latency consists almost entirely of accelerator latency: its interactivity advantage changes by about 9 to 12 percentage points per 100 ns, from 88 % at 300 ns to 55 % at 600 ns. The hybrid fabric gains almost nothing, as it runs the same packet-switched collective schedules over the same electronic level-1 paths as all-EPS. Collective acceleration engines (Google, [21]) that shorten the accelerator-side path therefore directly increase the advantage of workload-static OCS, and so does lighter forward error correction: an EPS switch terminates FEC at every hop, whereas a passive optical path is terminated once at its endpoints. Scale-Up Ethernet names RS-272 as a candidate to reduce receive latency (Broadcom Inc., Sec. 3.5.4), with link-level retransmission as the hardware-based safety net for residual errors.

Communication precision. Our baseline models activation communication in FP4, consistent with the assumed activation storage precision (Appendix § 12.§ 1). This is a forward-looking assumption: many serving collectives still communicate in BF16, including the default MoE combine paths in DeepEP (DeepSeek-AI, [16]) and TensorRT-LLM (NVIDIA, [37]). However, lower-precision transport is already supported for MoE dispatch (FP8/NVFP4) and optionally for combine (NVFP4) (DeepSeek-AI, [16]; NVIDIA, [37] (2025)).

To assess sensitivity to communication precision, we repeat the evaluation of Figure 4 using BF16 activation transfers, quadrupling the bytes per activation element relative to FP4, while Helix softmax statistics are communicated in FP32. For each precision, we independently repeat the full parallelization search of Appendix § 12.§ 6, selecting the best-performing configuration for each workload and fabric. We account for the resulting communication volumes but do not model precision-conversion overheads or numerical accuracy effects.

All-OCS continues to outperform all-EPS in every configuration, with an interactivity advantage of 9.6–75.7 %, compared with 7.3–81.6 % under FP4. Up to batch size 8, the advantage generally decreases by up to 16 percentage points. At batch size 16, it instead increases by 1–3 points, consistent with additional bandwidth pressure on the oversubscribed EPS fabric. The hybrid fabric changes by at most 3 points, except for Megatron 5T at 32K context and batch size 16, where its advantage decreases from 12.5 to 5.7 %.

Thus, the relative benefits depend on communication precision, but the conclusions of § 6 remain unchanged. The HOP-B findings (Appendix § 12.§ 5) are also robust: repeated all-to-all startup costs continue to outweigh overlap savings for GPT-OSS-5.3-like, whose decode-step time under BF16 increases by 58 % on all-EPS and 24 % on all-OCS with eight microbatches.