betaFeedback

WorksComputer Science

Multi-GPU MoE Inference with Adaptive Expert Offloading

Abstract

Modern Mixture-of-Experts (MoE) inference is constrained by GPU memory. This constraint is mitigated by expert offloading, which stores part of the model's weight in host memory and executes on demand. Though MoE architecture has computation sparsity, more GPUs are needed for serving increasingly larger frontier models. However, existing expert offloading systems overlook this need and fail to scale performance with additional GPUs. We present AdaOffload, a general and efficient multi-GPU hybrid expert execution framework that effectively handles heterogeneous hardware and workloads. We co-design GPU-CPU parallelism and GPU-CPU co-execution for decode and prefill with online workload observation that adapts to different hardware, workload, and batch size heterogeneity. We evaluated AdaOffload on five different platforms with varying GPU capability (L4, Pro6000, B200), number of GPUs, CPU capability, and NUMA topology on three different frontier open-weight models (284B to 753B). AdaOffload achieves 1.9×1.9\times to 15.5×15.5\times throughput speedups across different models over state-of-the-art systems.

Introduction

Increasingly capable open-weight models make self-hosted inference attractive to organizations seeking greater control over their data and infrastructure. Frontier open-weight Mixture-of-Experts (MoE) models offer strong capabilities across a range of tasks (Moonshot AI, [19]; Z.ai, [33]; DeepSeek-AI, [6]) and can be deployed on local hardware or rented cloud GPUs. For small organizations handling sensitive or regulated data, self-hosting offers greater control over data processing while avoiding the service quotas and recurring costs of external providers (Anthropic, [2]; thsottiaux, 2026).

Limited GPU memory, however, remains a key barrier to practical self-hosting. Large models commonly require multiple GPUs for their combined compute and memory capacity, and growing models and rising memory prices make fully GPU-resident deployment increasingly costly. For example, DeepSeek-V4-Flash’s weights occupy approximately 145 GiB, while its successor, DeepSeek-V4.1-Flash-1017, requires 475 GiB—more than three times as much. KV caches and runtime buffers add to this footprint, causing memory requirements to grow further as context lengths and concurrency increase. These demands lead many users to access open-weight models only through commercial inference APIs.

The sparsity of MoE models makes expert offloading a promising way to reduce their GPU memory requirements. Since each token activates only a small subset of experts, systems can retain expert weights in host memory and cache a subset on GPUs, prefetching or fetching missing experts on demand. These transfers, however, can incur substantial overhead over bandwidth-constrained PCIe links. To reduce this overhead, recent systems employ hybrid expert execution (Kamahori et al., [13]), executing selected experts on the CPU and transferring smaller activations and outputs instead of their weights.

Unfortunately, existing MoE offloading systems provide limited support for interactive serving at modest concurrency across multiple GPUs. Specifically, prior work has primarily focused on hybrid expert execution in single-GPU settings and for single-request inference (Kamahori et al., [13]; Chen et al., [5]; Yang et al., [30]; Yu et al., 2026b) or in exclusively throughput-oriented settings suited for large-batch offline processing (Xu et al., 2026; Cao et al., [4]).

Our goal is to build a general and efficient hybrid expert execution engine for MoE inference across diverse hardware configurations, models, and batch sizes. We target interactive services within small organizations that must meet service-level objectives (SLOs) for time to first token (TTFT) and time per output token (TPOT), typically at modest concurrency. The system should adapt to the available CPU and GPU resources and the requests being served to

Preprint. Copyright 2026 by the authors. Licensed under CC BY 4.0. make effective use of each deployment’s hardware.

Building a general MoE offloading engine that remains efficient across hardware configurations, models, and batch sizes requires addressing three forms of heterogeneity. The first is in how CPU and GPU resources are organized. Distributing expert computation across multiple GPUs requires balancing both the computation and the weight transfers so that each GPU and its PCIe link can contribute effectively. On the CPU, efficient execution depends on NUMA topology, memory locality, and how expert tensors are partitioned among workers. A partitioning strategy that works well for GPUs may therefore be poorly suited to CPUs, and simply assigning equal amounts of work does not ensure efficient execution. The engine must accommodate the partitioning requirements of each resource while coordinating their work to avoid transfer bottlenecks and stragglers.

The second form of heterogeneity is in the compute and transfer capabilities of different hardware platforms. The relative benefit of executing an expert on the CPU or fetching its weights for GPU execution depends on the CPU and GPU capabilities, the available PCIe bandwidth, and the amount of work assigned to that expert. These costs also interact: CPU computation and expert transfers share host memory bandwidth, while multiple GPUs may contend for the same PCIe links. Consequently, execution times measured in isolation can misrepresent the costs of concurrent CPU–GPU execution. Choosing an effective division of work requires a model that accounts for expert loads, the number of distinct experts, and contention among computation and transfers.

The third form of heterogeneity is in workload demands, which vary across models, batches, and inference phases. During decode, routing patterns and batch size determine how many experts are activated and how much work each receives, while cache residency determines which experts require fetching or CPU execution. A fixed placement or execution policy cannot accommodate these changes: executing all cache-missed experts on the CPU can leave GPU capacity idle, while fetching too many can make transfers the bottleneck. Prefill introduces another tradeoff. Longer prefills can benefit from overlapping next-layer expert prefetching with GPU computation, whereas shorter prefills can benefit from hybrid expert execution. A general engine must therefore observe the workload at runtime and adapt its execution policy as these demands change.

We present AdaOffload, an expert-offloading engine for multi-GPU MoE inference that adapts to differences in resource organization, hardware capability, and workload demands. To accommodate CPU and GPU resource organization, AdaOffload uses heterogeneous tensor parallelism: it shards experts across GPUs to balance computation and weight transfers, while choosing CPU partitions independently based on NUMA topology and kernel efficiency. To account for hardware capabilities, it profiles expert computation and fetching under concurrent CPU–GPU execution, capturing contention for shared resources. To adapt to workload changes, it combines these profiles with lightweight runtime observations of expert loads and cache residency to select execution policies for both decode and prefill. Put together, these mechanisms enable AdaOffload to make effective use of available CPU and GPU resources across diverse deployments and changing workloads.

We evaluate AdaOffload across three MoE models and five hardware platforms. The models range from MoB parameters, and the platforms span L4, PRO 6000, and B200 GPUs with varying GPU counts, CPU capabilities, and interconnect topologies. On decode-heavy reasoning workloads, AdaOffload achieves an average of 1.9×–15.5× decode throughput and 54.6%–85.1% lower TTFT than the strongest evaluated baseline across DeepSeek-V4-Flash, GLM-5.3-Flash, and GLM-5.2. On agentic coding tasks, it improves decode throughput by an average of 4.82× at modest concurrency. For DeepSeek-V4-Flash, its adaptive decode policy achieves 94.8% and 99.2% of the throughput of the best measured fixed policy at concurrency 1 and 8, respectively, approaching the performance of policies selected through an exhaustive sweep without requiring that search at runtime.

In summary, this paper makes the following contributions:

  1. A cost model for multi-GPU hybrid expert execution that accounts for expert loads, distinct experts, and contention between computation and transfers.

  2. An inference engine, AdaOffload, combining heterogeneous tensor parallelism with workload-adaptive CPU–GPU execution.

  3. Extensive evaluation across five platforms and three MoE models, demonstrating average of 1.9×–15.5× decode throughput and a 54.6%–85.1% reduction in TTFT on reasoning workloads.

Background and Motivation

Self-hosted Inference

Privacy concerns and stronger open-weight models motivate self-hosted inference. Accessing proprietary LLMs through external APIs may expose sensitive data and interaction traces to service providers, creating privacy and compliance risks. Changes in API quotas, pricing, or availability also introduce operational uncertainty. Meanwhile, open-weight models such as Kimi-K3, GLM-5.3-Flash, and DeepSeek-V4.1-Flash are narrowing the capability gap with frontier proprietary models [19, 33, 6]. Self-hosting offers access to these capabilities while retaining control over data and infrastructure.

Limited GPU memory remains a key barrier to self-hosted inference. Capable open-weight models often have substantial memory requirements. For example, the weights of GLM-5.3-Flash and DeepSeek-V4.1-Flash require 176 and 475 GiB of memory under FP4 quantization, respectively, excluding runtime buffers and KV caches. For agentic and long-context workloads, KV caches can require substantial additional memory [27, 15]. An organization-wide inference service must handle concurrent requests while maintaining acceptable response latency and throughput. Thus, practical self-hosted inference requires fitting models within limited GPU memory while serving concurrent requests efficiently.

MoE Inference and Expert Offloading

MoE sparsity makes expert offloading practical. Mixture-of-Experts (MoE) models activate only a few experts per token, a design widely adopted by recent open-weight models. This sparsity makes MoE models particularly well suited to expert offloading: expert weights can reside in host memory, with only a subset cached on GPUs. Fetch-based approaches transfer the weights of missing experts to GPUs for execution, incurring PCIe overhead when transfers cannot be overlapped with computation. Prior work uses expert prediction and prefetching to reduce cache misses [29, 23]. Alternatively, hybrid expert execution systems execute missing experts on CPUs by transferring activations instead of expert weights [13]. Recent work combines expert caching with bandwidth-aware scheduling to partition missing-expert execution between CPUs and GPUs [30]. As Figure 1 shows, a carefully designed expert offloading system can retain a large fraction of full-residency performance as GPU resources increase.

Mean TTFT and decode throughput when serving 16 concurrent requests with L4 GPUs with expert offloading

Figure 1. Mean TTFT and decode throughput when serving 16 concurrent requests with L4 GPUs with expert offloading (ADAOFFLOAD). 8-fit indicates fitting all experts in VRAM.

Expert offloading beyond model fitting. As models continue to scale, experts increasingly dominate the memory footprint of MoE models, accounting for over 90% of the parameters in many recent open-source models. Expert offloading is therefore essential when the model exceeds aggregate GPU memory. Yet its value extends beyond making the model fit. Even when all experts can reside on GPUs,

keeping them resident may not be the best use of scarce VRAM. In the example with eight L4 GPUs in Figure 1, full expert residency leaves just 5.5 GiB for KV caches, enough for 124K tokens, whereas expert offloading leaves 49 GiB, enough for 1.1M tokens. As Figure 2 shows, this tradeoff becomes increasingly favorable for longer contexts: VRAM is better spent on repeatedly accessed KV entries than on sparsely accessed experts, as avoiding KV offloading can outweigh the overhead of fetching experts. The same trade-off also arises under higher concurrency, as more concurrent requests further increase KV demand.

Figure 2. Decode throughput of DeepSeek-V4-Flash on eight L4 GPUs with 16 concurrent requests. Cold-KV offloading keeps all experts on GPU, with indexer keys on GPU or CPU as indicated. Expert offloading instead uses a half-VRAM expert cache while keeping KV and indexer keys on GPU.

Limitations of Existing Systems

Given the demonstrated advantages of hybrid expert execution over pure expert offloading, we focus hereafter on hybrid expert execution systems. Existing systems, such as KTransformers [5] and Ollama [20], statically partition experts between CPUs and GPUs during decode. This partitioning makes CPU execution the bottleneck while leaving H2D PCIe bandwidth underutilized, missing opportunities to fetch CPU-resident experts for GPU execution. These limitations worsen as the number of GPUs increases: static placement underutilizes aggregate PCIe bandwidth and exacerbates CPU–GPU imbalance, degrading performance.

FreeToken (Yang et al., [30]) and Fiddler (Kamahori et al., [13]) orchestrate expert execution across CPUs and GPUs to balance execution times in single-GPU settings. Extending this approach to multiple GPUs, however, is nontrivial: execution time depends on the parallelization strategy and the resulting contention for shared resources, including PCIe switch and NUMA-node bandwidth. These interactions complicate both execution-time modeling and the search for an optimal hybrid expert execution policy.

Finally, prior systems primarily target single-request inference and overlook emerging agentic workloads. Larger batches and changing workload characteristics substantially alter expert load patterns during decode, as shown in Figure 3. Agentic workloads also exhibit variable prefill lengths that prior systems fail to accommodate. For example, some systems always favor prefetching, missing opportunities to reduce TTFT for short prefills with hybrid expert execution. Table 1 compares ADAOFFLOAD with prior work.

Decode expert footprint across batch sizes for DSV4-Flash on AIME, Chemistry, and ShareGPT workloads

Figure 3. Decode expert footprint across batch sizes for DSV4-Flash on AIME, Chemistry [9], and ShareGPT (anon8231489123, 2023) workloads. Left: number of unique experts; right: median token load per expert.

SystemHardware awareBatch-size awareMulti GPUSupported parallelism
KTransformers––✓TP
Ollama––✓PP
llama.cpp––✓PP
FreeToken✓–––
ADAOFFLOAD (ours)✓✓✓HTP

Table 1. Comparison of ADAOFFLOAD with prior works.

Therefore, to build an efficient multi-GPU hybrid expert execution system, we must jointly consider parallelism design (§ 3.2), hardware heterogeneity (§ 3.3), and dynamic workload patterns (§ 4) to select the most efficient hybrid expert execution plan.

ADAOFFLOAD: DESIGN

Overview

We now introduce ADAOFFLOAD, an inference framework for hybrid expert execution that adaptively dispatches expert computation across CPUs and multiple GPUs. ADAOFFLOAD adopts heterogeneous tensor parallelism (HTP) to efficiently coordinate CPU and GPU resources (§ 3.2). HTP balances GPU workloads through tensor parallelism. For CPU execution, it selects the degree of parallelism based on NUMA topology and kernel efficiency to reduce host-memory contention and mitigate stragglers. To account for differences in hardware performance, ADAOFFLOAD then profiles the hardware offline to accurately estimate CPU and GPU worker execution times under different expert workloads and contention conditions (§ 3.3). Variations in workload and batch size, however, must be observed during inference. Thus, ADAOFFLOAD combines these profiles with lightweight runtime observations of expert loads and cache residency to adapt the CPU–GPU execution policy to varying batch sizes in both the prefill and decode phases (§ 4). Figure 4 summarizes the system design.

Overview of ADAOFFLOAD

Figure 4. Overview of ADAOFFLOAD. HTP employs distinct TP domains for CPUs and GPUs. The CPU is split by NUMA node; hardware profiling selects intra-NUMA sharding, resulting in TP4. Offline profiling provides hardware- and model-specific cost tables. Online expert-residency and token-load samples guide asynchronous policy updates. In this example, q=1/3q = 1/3, Expert 3 is fetched to the GPU for minimum completion time.

Heterogeneous Tensor Parallelism

Extending hybrid expert execution to multiple GPUs is not straightforward. It requires carefully balancing CPU and GPU parallelism while reconciling their differing partitioning requirements.

Observation 1: TP better matches hybrid expert execution than PP and EP. We consider pipeline parallelism (PP), expert parallelism (EP), and tensor parallelism (TP). PP splits layers across workers, but requires sufficient micro-batches to keep pipeline stages busy (Huang et al., [10]), making it inefficient for interactive workloads with modest concurrency. Moreover, increasing heterogeneity across model layers makes it harder to balance workloads across PP stages. EP assigns complete experts to workers and uses two all-to-all operations to dispatch and combine tokens. It is prone to load imbalance, especially with few tokens (Lepikhin et al., [16]), and hybrid execution further translates routing skew into imbalanced weight transfers and CPU memory accesses: some PCIe links and NUMA nodes become overloaded while others remain underutilized. Rebalancing experts requires additional weight transfers, further stressing the bandwidth-constrained path. In contrast, TP shards each expert across all workers in a group, distributing computation and weight transfers evenly across PCIe links and NUMA nodes. In addition, the allreduce introduced by TP has a much more regular communication pattern and is more amenable to optimization over PCIe than EP’s alltoall (Kim et al., [14]).

Observation 2: CPU and GPU tensor-parallelism domains should be decoupled. A naive design does not partition CPU experts and therefore suffers substantial performance degradation from NUMA memory access (Chen et al., [5]). Applying the GPU TP degree to CPUs, or merely partitioning CPU experts by NUMA node, is also suboptimal. As shown in Figure 5, CPU kernel efficiency within a NUMA node depends significantly on both the hardware and the intra-NUMA sharding degree. Based on these observations, ADAOFFLOAD adopts heterogeneous tensor parallelism, as illustrated in Figure 4. HTP partitions CPU experts independently of GPU experts: it first shards host-expert tensors across NUMA nodes, then profiles CPU kernels using the model’s tensor shapes to determine the optimal intra-NUMA sharding degree. The CPU TP degree, TPcTP_c, is the product of the NUMA-node count and the intra-NUMA degree. In HTP, a GPU associated with each NUMA node accumulates the partial results produced by that NUMA node’s CPUs, followed by a single GPU-wide allreduce.

Figure 5. CPU execution efficiency and task latency. (a) CPU kernel efficiency on two different single-NUMA-node configurations at different expert intermediate widths. Intermediate widths 2048/1024/512/256 correspond to NUMA-local sharding factors of 1/2/4/8. (b) CPU task wall-time distributions with spin waiting off/on.

Mitigating CPU worker stragglers. HTP distributes computation evenly across CPU workers, but does not guarantee equal completion times. Since the HTP reduction depends on every rank’s contribution, CPU scheduling delays can expose a straggler even when ranks process equal-sized shards. We observe a long tail in CPU task completion time when CPU workers are preempted by the OS scheduler. We therefore enable spin polling for worker task queues and completion waits to reduce blocking and wake-up overhead, at the cost of additional CPU usage. Figure 5(b) shows the reduction in CPU task tail latency.

Profiling Compute and Transfer Costs

Hardware platforms exhibit substantially different characteristics, as shown in Table 2. While HTP balances computation across GPUs and CPUs and reduces NUMA contention,

ProviderGPUsEff. compute (TFLOP/s)CPU (cores)Eff. compute (TFLOP/s)DRAM (GiB)PCIe/link (GB/s)Aggregate H2D (GB/s)
MiburaL4 ×43.390Xeon 4410Y (24)0.428256Gen4 (32)98.0
GCPL4 ×86.317Xeon (48)0.510402Gen3 (16)33.3
NVIDIAPRO 6000 ×26.829TR PRO 7955WX (16)0.314480Gen5 (64)103.2
AWSPRO 6000 ×28.751Xeon 8559C (24)0.463500Gen5 (64)50.8

Table 2. Hardware specifications across different multi-GPU nodes. Effective compute is measured under one representative workload.

ADAOFFLOAD must also account for the capabilities of individual components and contention among them. To characterize these capabilities and interactions, ADAOFFLOAD performs comprehensive offline system profiling.

Observation 3: Expert loads and the number of unique experts significantly affect execution time. For an expert ee, matrix multiplication requires Fe=6neHHEF_e = 6n_eH H_E FLOPs, and its weights occupy Se=3HHEseS_e = 3H H_E s_e bytes, where nen_e is the number of tokens routed to the expert, ses_e is the number of bytes per expert parameter, and HH and HEH_E are the activation and expert hidden dimensions, respectively. We use EdE_d, d∈{c,g,f}d \in\{c,g,f\}, to denote the sets of experts executed on CPUs, executed on GPUs, and fetched from host memory, respectively. Let Ld=∑e∈EdneL_d = \sum_{e\in E_d} n_e denote the expert compute load (expert-token assignments) and Ud=∣Ed∣U_d = |E_d| denote the number of unique experts. Then Texec,dT_{exec,d}, the time required to execute the experts assigned to device d∈{c,g}d \in\{c,g\}, is approximated by

Texec,d≈max⁡(6HHELdCompd,3HHEUdseMemBWd)(1)T_{exec,d} \approx\max\left(\frac{6H H_E L_d}{Comp_d}, \frac{3H H_E U_d s_e}{MemBW_d}\right) \tag*{(1)}

Similarly, the MoE execution time is approximated by

TMoE,hybrid≈max⁡(Texec,c,Texec,g+3HHEUfseHtoDBW)(2)T_{MoE,hybrid} \approx\max\left(T_{exec,c}, T_{exec,g} + \frac{3H H_E U_f s_e}{HtoD BW}\right) \tag*{(2)}

On GPUs, CompdComp_d and MemBWdMemBW_d remain stable, leaving LdL_d and UdU_d as the workload-dependent factors. On CPUs, MemBWdMemBW_d may also vary due to memory-bandwidth contention between expert computation and concurrent expert transfers to GPUs. Under a fixed HTP configuration, however, this contention is determined by the workload characterized by LdL_d and UdU_d. We therefore profile expert execution as a function of these two dimensions.

Expert execution. For each supported decode batch size BB, we enumerate feasible (L,U)(L,U) pairs, where LL denotes the expert-token load and UU the number of distinct experts. We replay a single MoE layer using the actual model shape and inference kernels, with all CPU and GPU workers running concurrently under the configured HTP setup. The resulting measurements form lookup tables T^CPU(L,U)\hat{T}_{\mathrm{CPU}}(L,U) and T^GPU(L,U)\hat{T}_{\mathrm{GPU}}(L,U) for estimating expert execution time.

Fetching and resource interference. Expert fetching and CPU execution share host-memory resources. Consequently, their costs cannot be estimated independently from isolated bandwidth measurements alone. We profile each CPU workload both without expert fetching and under sustained concurrent fetching from all GPUs. We measure fetching latency as T^fetch(F∣Lc,Uc)\hat{T}_{\mathrm{fetch}}(F \mid L_c,U_c), where FF is the number of fetched experts and (Lc,Uc)(L_c,U_c) describes the concurrent CPU workload. Measurements are collected with all participating GPUs fetching concurrently to capture the actual contention under HTP. These measurements provide CPU execution costs with and without concurrent fetching, as well as fetching costs conditioned on the CPU workload.

Heuristics to reduce profiling space. A batch of bb tokens with top-kk expert routing contains bkbk expert-token routes. The expert-token load LL and unique expert count UU satisfy 1≤L≤bk1 \le L \le bk and ⌈Lb⌉≤U≤min⁡(∣E∣,L)\left\lceil\frac{L}{b} \right\rceil\le U \le\min(|E|,L), where ∣E∣|E| is the number of routed experts per layer. For DSV4-Flash with b=32b=32, k=6k=6, and ∣E∣=256|E|=256, this yields 18,048 feasible (L,U)(L,U) pairs. This space is prohibitively expensive to profile exhaustively. When the number of unique experts is fixed, computation time scales linearly with the expert-token load LL. For batch sizes greater than 8, we use linear regression for extrapolation. As shown in Figure 6, four-point linear regression captures the relationship between execution latency and expert-token load LL. Across the configurations that we exhaustively profiled, the average WAPE of the four-point linear regression is 1.99%.

Figure 6. Four-point latency prediction on L×8L \times8 for DSV4-Flash at U=8U=8. Held-out WAPE is 2.6% and 4.0%, respectively.

Online Adaptation to Workload Changes

Figure 3 shows how expert footprints vary across decode batch sizes and workloads. For instance, at a batch size of 20, the ShareGPT workload activates 3.6 times as many unique experts as AIME, while its median token load per expert is significantly lower. However, workloads and batch sizes change during inference and cannot be fully characterized offline. Therefore, AdaOFFLOAD uses low-overhead online instrumentation to capture the expert footprint at runtime. Instrumented forward passes write statistics into preallocated device buffers, which are copied asynchronously to host buffers for policy optimization. For each sampled forward pass, we record expert and cache statistics, including per-

layer unique experts, expert load, cache-missed experts, and the fetching order determined by the fetching policy (e.g., most recently used first or highest load first). Experiments on DSV4 show that instrumentation slows instrumented forward passes by 1.3–6.6%. These statistics provide the expert load LL and unique expert count UU for the current workload. The sampling interval and the number of historical samples retained for policy selection are configurable. A shorter sampling interval allows the policy to adapt more quickly to workload changes, at the cost of greater instrumentation overhead. Retaining fewer historical samples makes policy selection more responsive to recent changes, but potentially less stable and more sensitive to noise.

Asynchronous Policy Selection and Execution. The hybrid execution policy is determined by the fraction of cache-missed experts fetched to GPUs, with the objective of minimizing TMoE,hybridT_{\mathrm{MoE,hybrid}}. Denote this fraction by q∈[0,1]q \in[0,1]. Let QQ denote the candidate set of qq values considered by the policy selection algorithm. In our implementation, AdaOFFLOAD uses Q={0,1/64,2/64,…,62/64,63/64,1}Q = \{0, 1/64, 2/64, \ldots, 62/64, 63/64, 1\}. Policy selection runs asynchronously without blocking the main inference path. The main inference path adopts updated policies at forward-pass boundaries.

Decode The policy selector uses the recorded forward-pass statistics to evaluate candidate qq values. Formally,

a∗=arg⁡min⁡a∈A∑ℓ=1NlayersT^ℓ(S∣a),A={hybrid(q):q∈Q}(3)a^* = \arg\min_{a \in A} \sum_{\ell=1}^{N_{\mathrm{layers}}} \hat{T}_{\ell}(S \mid a), \quad A = \{\mathrm{hybrid}(q): q \in Q\} \tag*{(3)}

Here T^ℓ(S∣a)\hat{T}_{\ell}(S \mid a) represents the estimated execution time of layer ℓ\ell given the sampled workload SS and the execution policy aa. It is obtained by reconstructing the expert assignment induced by aa from the sampled statistics. For a candidate qq, we determine the number of experts to fetch and select them according to the fetching priority order. This yields the expert sets EgE_g and EcE_c assigned to each device, and the instrumentation provides their expert loads and unique expert counts. These quantities index the offline profiles T^CPU(Lc,Uc)\hat{T}_{\mathrm{CPU}}(L_c,U_c), T^GPU(Lg,Ug)\hat{T}_{\mathrm{GPU}}(L_g,U_g), and T^fetch(F∣Lc,Uc)\hat{T}_{\mathrm{fetch}}(F \mid L_c,U_c). The CPU estimate accounts for contention during concurrent fetching. We substitute these component estimates into Eq. (2) to obtain T^ℓ(S∣a)\hat{T}_{\ell}(S \mid a). When multiple historical samples are available, we choose the policy aa that minimizes the average estimated execution time across the samples.

Prefill. An alternative to on-demand hybrid execution is to prefetch all expert weights for the next layer, so weight transfer overlaps with computation for the current layer. This suits prefill, where the many routed tokens activate nearly all experts. Prior work instead prefetches only the experts predicted to be activated (Zhong et al., [36]). However, the performance of these approaches varies substantially with prediction accuracy, and prediction complicates integration with CUDA Graphs. Thus, we consider prefetching all experts in the next layer. The execution time of an MoE layer is then

TMoE,prefetch=max⁡(Texec,g,∑e∈ESeHtoDBW)(4)T_{MoE,prefetch} = \max\left(T_{exec,g}, \frac{\sum_{e \in E} S_e}{HtoDBW}\right) \tag*{(4)}

The prefill phase uses the same policy-selection formulation, with A={prefetch}∪{hybrid(q):q∈Q}\mathcal{A} = \{\mathrm{prefetch}\} \cup\{\mathrm{hybrid}(q):q \in Q\}.

Evaluation

Experimental Setup

Testbed. We conduct experiments on four different machines as listed in Table 2. The GCP and AWS machines are rented from the corresponding cloud service providers. Other machines are workstations that are accessed through research partnerships. We conduct experiments on them without modifying any of their hardware configurations. These machines exhibit substantial heterogeneity in PCIe bandwidth and CPU compute capacity. For example, NVIDIA Pro6000 has GPUs in different PCIe switch resulting in higher PCIe aggregate bandwidth while AWS Pro6000 has both GPUs sharing the same PCIe switch. This allows us to evaluate how effectively ADAOFFLOAD accommodates diverse hardware characteristics.

Baselines. We compare ADAOFFLOAD against llama.cpp (llama.cpp Contributors, 2026) (b76b133), KTransformers (Chen et al., [5]) (5632f71), and Ollama (Ollama Contributors, [20]) (d919272), three SOTA hybrid expert execution systems that support multi-GPUs. KTransformers integrates the SGLang (Zheng et al., [35]) backend to support TP, while llama.cpp and Ollama both use PP. We use their default multi-GPU supports and keep the model checkpoints and quantization identical. For a fair comparison, all systems use the same VRAM expert cache size.

Model. All systems serve DeepSeek-V4-Flash with MXFP4 quantization, with half of the available GPU memory reserved for the expert cache. We primarily focus on DeepSeek-V4-Flash because it’s the largest model that can fit into all machines’ DRAM while all baseline systems have native support on. We also evaluate GLM-5.3-Flash

with NVFP4 quantization and leave the result figures in Appendix to show the generalizability of our results across different model architectures.

Workloads. We evaluate the end-to-end performance of all systems on two representative LLM workloads: (1) mathematical reasoning and (2) agentic coding. For reasoning, we use AIME24 (HuggingFaceH4, n.d.) and AIME25 (Zhang & Math-AI Team, [34]), two challenging mathematical reasoning benchmarks. For agentic coding, we use SWE-bench Verified (Jimenez et al., [12]) with Claude Code (Anthropic, [2]) as the agent harness. AIME is a single-turn, decode-heavy workload with 206 (495) median (p99) input sequence length (ISL) and 4032 (23902) median (p99) output sequence length (OSL). In contrast, agentic coding is a multi-turn workload with long initial input to describe the task and multiple rounds of tool call and response. For example, for our evaluation of ADAOFFLOAD at concurrency of 8, the initial task description has median of 9339 tokens. The subsequent tool call results have median of 332 tokens and median OSL per turn is 155. We randomly sample a set of problems from each benchmark and issue them in the same order to all systems to ensure a consistent workload. We use 1, 4, and 8 as concurrency levels.

Metrics. We primarily focus on decode throughput (per-request average token per second) and time-to-first-token (TTFT) (per-request average time to receive the first token).

End-to-end Result on Realistic Workloads

Mathematical reasoning. As shown in Figure 7, ADAOFFLOAD achieves an average improvement across four different platforms in decode throughput (TTFT reduction) of 1.30×1.30\times (41%), 1.88×1.88\times (57%), and 2.47×2.47\times (67%) over the strongest baseline at three different concurrency levels. ADAOFFLOAD outperforms the baselines at small batch size due to hybrid execution. At larger batch sizes, ADAOFFLOAD outperforms the baselines more because it adaptively selects the hybrid execution policy.

Figure 7. End-to-end performance of ADAOFFLOAD and baselines on AIME reasoning tasks. Columns show concurrency 1, 4, and 8; rows show streaming decode throughput and mean TTFT.

Agentic coding. Figure 8 shows the end-to-end performance of ADAOFFLOAD and baselines on coding tasks. As concurrency increases, all baseline systems suffer from prefill cache misses and poor prefill efficiency, leading to low session decode throughput overall. For almost all cases, other frameworks reach the 30-minute session timeout limit. In contrast, ADAOFFLOAD consistently performs efficient prefill via adapting between hybrid execution and full-layer prefetching. This leads to the substantial gap in overall decode throughput for the baseline systems.

Figure 8. End-to-end performance of ADAOFFLOAD and baselines on coding tasks. Columns show concurrency 1, 4, and 8; rows show streaming decode throughput and mean visible TTFT. Ollama timeouts all session except for concurrency 1.

Effectiveness of Adaptive Hybrid Execution

We next examine whether ADAOFFLOAD adjusts hybrid expert execution policy efficiently under varying workloads and hardware configurations. We study its policy decisions separately for decode and prefill, respectively.

Decode. We first evaluate the accuracy of ADAOFFLOAD’s adaptive policy in selecting the fraction qq of experts to fetch to GPUs. Figure 9 compares ADAOFFLOAD’s selection with the best and worst fixed-qq policies identified through brute-force search. Averaged across the four platforms, ADAOFFLOAD attains 94.8% and 99.2% of the throughput achieved by the best fixed-qq policy at C=1C = 1 and C=8C = 8, respectively, demonstrating that it accurately identifies near-optimal execution policies without exhaustive search. Meanwhile, ADAOFFLOAD outperforms the worst fixed-qq policy by 49.1% and 33.9%.

Figure 9. Decode throughput of ADAOFFLOAD versus the best and worst measured fixed-qq policies at concurrency C=1C = 1 (left) and C=8C = 8 (right).

The optimal qq varies substantially across hardware configurations and concurrency levels, making a fixed policy ineffective. In most cases, q=0.1q = 0.1 performs the worst because it fetches too few experts, underutilizing PCIe bandwidth and GPU compute. By contrast, the Mibura testbed (L4×4) favors a smaller qq under high concurrency. Its smaller expert cache incurs 2–3× more expert cache misses per forward pass, making expert fetching more expensive than CPU execution. These contrasting cases demonstrate that ADAOFFLOAD accurately adapts its policy to different hardware configurations and runtime conditions.

Prefill. For prefill, ADAOFFLOAD dynamically switches between CPU co-execution and next-layer prefetch based on the prefill length. Figure 10 compares its prefill forward latency with KTransformers on the coding workload from our end-to-end evaluation. We omit Ollama and llama.cpp, as frequent prefix cache misses force them to repeatedly prefill long contexts, resulting in poor performance. On the two platforms, ADAOFFLOAD switches to next-layer prefetch at 300 and 400 tokens, respectively, based on the runtime observation and estimated computation and PCIe transfer costs. On Mibura, the hybrid execution policy chooses qq-value of around 0.6 to leverage the CPU compute capacity when prefill length is between 128 and 300. On NVIDIA, given its powerful PCIe bandwidth, the hybrid execution policy chooses qq-value of 1 to fully utilize the GPU resources. This adaptive switching keeps prefill latency low across varying prefill lengths.

Figure 10. Prefill forward latency of ADAOFFLOAD and KTransformers for coding tasks at concurrency C=8C = 8.

Scalability of Hybrid Expert Execution

We next evaluate how ADAOFFLOAD scales hybrid expert execution across multiple GPUs with HTP (§ 3.2). We vary the number of GPUs and compare ADAOFFLOAD against KTransformers and an offload-only variant of ADAOFFLOAD that disables hybrid expert execution. We run eight concurrent reasoning tasks from the AIME dataset and measure decode throughput and mean TTFT. For all configurations, both KTransformers and ADAOFFLOAD are allowed to utilize all available CPU resources.

Decode. As shown in Figure 11(top), KTransformers fails to translate additional GPUs into higher decode throughput, which even decreases as the GPU count increases. This is because its fixed CPU–GPU expert placement during decoding cannot adapt to additional GPU capacity, leaving the added GPU resources underutilized. In contrast, ADAOFFLOAD’s offload-only variant executes all experts on GPUs by fetching miss expert on demand and scales from 4 tok/s with one GPU to 86.2 tok/s with eight GPUs, but leaves CPU compute resources underutilized. With HTP, ADAOFFLOAD jointly exploits CPUs and GPUs while optimizing their parallelism independently, scaling from 28.5 to 90.8 tok/s. In summary, ADAOFFLOAD achieves 1.84×–9.61× higher decode throughput than KTransformers across 1–8 GPUs.

Figure 11. Decode throughput and mean TTFT for different numbers of GCP L4 GPUs. Framework can use all CPUs.

Prefill. As shown in Figure 11(bottom), KTransformers scales with additional GPUs, reducing TTFT from 24.1 s to 6.8 s during prefill. However, its fixed hybrid CPU–GPU execution is suboptimal for most cases, where GPU prefetch is more efficient. ADAOFFLOAD instead adapts to the prefill length, using GPU prefetch in most cases and hybrid expert execution only when beneficial. For the latter, HTP and the adaptive execution policy efficiently distribute expert execution across CPUs and GPUs. With 1, 2, 4, and 8 GPUs, ADAOFFLOAD achieves 1.14×1.14\times, 2.24×2.24\times, 1.61×1.61\times, and 3.67×3.67\times TTFT speedup over KTransformers, respectively. The adaptive configurations use 32 CPU expert workers: one cross-NUMA pool with interleaved expert weights at one GPU, and two NUMA-local 16-worker pools at 2–8 GPUs. KTransformers uses 48 CPU expert workers throughout.

System Overhead

When performing instrumentation, ADAOFFLOAD shows an average of 4.04% extra latency on sampled forwards. ADAOFFLOAD captures CUDA graph for supported decode batch size. On average, the CUDA graph consumes 0.47 GiB of GPU memory when support 4 different batch sizes (1, 2, 4, 8) in our evaluation. The asynchronous decode policy solver takes an average of 4.1 ms to decide the policy for a batch size.

Performance on Other MoE Models and GPUs

We further evaluate ADAOFFLOAD on GLM-5.3-Flash across the same four platforms and GLM-5.2 on B200 to test if ADAOFFLOAD could generalize to other models and newer platforms. (See appendix for detailed results.) GLM-5.3-Flash and GLM-5.2 have 320B and 753B parameters respectively and each token is routed to top-8 experts while DeepSeek-V4-Flash routes to top-6 experts.

For GLM-5.3-Flash, Averaged across the four platforms, ADAOFFLOAD achieves 3.86×3.86\times, 7.64×7.64\times, and 9.78×9.78\times decode throughput over the strongest baseline at concurrency 1, 4, and 8, respectively, while reducing mean TTFT by 79.6%, 70.0%, and 83.9%. Comparing with the best fixed-qq policy, ADAOFFLOAD attains 94.7% and 99% of the best measured throughput. GLM-5.2 is tested with two NVIDIA B200 GPUs with half of the HBM reserved for expert cache (Result figure in Appendix). On this two-GPU configuration, ADAOFFLOAD achieves 6.42×6.42\times, 17.98×17.98\times, and 22.17×22.17\times decode throughput over the strongest baseline at concurrency 1, 4, and 8, respectively, while reducing mean TTFT by 84.8%, 84.4%, and 86.0%.

Related Work

Expert caching and prefetching. Expert offloading uses GPU memory as a cache for a larger expert pool. Mixtral-offloading combines expert caching with speculative prefetching [7], while MoE-Infinity uses request-level activation traces to guide cache replacement and prefetching [29]. ProMoE predicts subsequent expert use from intermediate results [23]; FineMoE additionally exploits fine-grained expert-selection patterns and prompt semantics (Yu et al., 2026b). LayerScope coordinates prediction and cross-layer scheduling to account for prefetch cost and PCIe bandwidth competition (Yu et al., 2026a). These techniques reduce demand misses or move transfers off the critical path, but prefetched weights still consume device memory and link bandwidth. A separate direction reduces the work or bytes required for a miss: HOBBIT loads less critical experts at lower precision [25], and SPICE combines speculative prefetching with approximate expert surrogates and exact CPU recovery [18]. These approximation choices are distinct from deciding where to execute a given expert computation.

CPU–GPU collaborative execution. Host-resident experts need not always be transferred to a GPU. Fiddler executes selected expert work on the CPU to avoid expensive weight movement [13]. KTransformers develops optimized CPU expert kernels and heterogeneous CPU/GPU execution [5], while HybridMoE combines dynamic intra-layer scheduling with inter-layer prefetching and cache management [36]. CoX-MoE coalesces expert computation across a batch and coordinates AMX-enabled CPU execution, GPU-resident experts, and attention offloading [22]. We build on FreeToken, which combines an elastic expert cache with a measured-bandwidth model that divides residual expert work between CPU execution and GPU loading, within a graph-compatible serving runtime [30]. Its evaluation focuses on single-GPU deployments. We extend this execution setting to a node in which multiple GPUs share host resources and also communicate with one another. The CPU/GPU split must therefore be considered together with expert placement and the selected parallelism.

Multi-GPU offloading and pipelining. Batching can amortize weight movement across more tokens. FlexGen optimizes tensor placement and execution for latency-insensitive batched inference [21]. MoE-Lightning pipelines CPU, GPU, and I/O work using paged weights and a hierarchical roofline model; it also serves large MoE models on multiple low-cost T4 GPUs [4]. Klotski schedules multiple batches around expert activation and loading requirements to reduce pipeline idle time [8]. These systems focus on large batch offline processing, while we primarily focus on interactive serving with medium batch sizes. In our case, batch size will not be large enough to cover the overhead of pipelining.

Conclusion

We present ADAOFFLOAD, an expert-offloading engine for multi-GPU MoE inference that adapts to differences in resource organization, hardware capability, and workload demands. We co-design GPU–CPU parallelism and GPU–CPU co-execution for decode and prefill with online workload observation that adapts to different hardware, workload, and batch size heterogeneity. Across a range of heterogeneous GPU platforms ranging from low-power GPU to high-end GPU, ADAOFFLOAD achieves 1.9×–15.5× throughput speedup over state-of-the-art systems across three different frontier MoE models, DeepSeek-V4-Flash, GLM-5.3-Flash, and GLM-5.2.

AI USAGE DISCLOSURE

We used OpenAI Codex as the generative-AI assistant across several stages of this work. It assisted with the implementation and debugging guided by human design and review. For manuscript, it assisted generating experiment result figures, proofreading, and formatting.

ModelExpert formatTotal params (B)Active params (B)Expert weights (GiB)Non-expert weights (GiB)Total weights (GiB)
DSV4-FlashMXFP428413137.068.24145.30
GLM-5.3-FlashNVFP432018159.4716.62176.09
GLM-5.2NVFP475340379.6934.65414.34

Table 3. Model specification.

Model specifications

Detailed model specification for DeepSeek-V4-Flash, GLM-5.3-Flash, and GLM-5.2 we evaluated. These two families of models are representative frontier open-weight models with different architecture. GLM routes each token to 8 experts while DeepSeek model routes each token to 6 experts. For all models, routed experts occupy more than 90% of the total model weights.

GLM-5.3-Flash AIME evaluation

Figure 12 extends the AIME evaluation to GLM-5.3-Flash with NVFP4 routed experts, comparing ADAOFFLOAD with llama.cpp and Ollama on the four platforms. KTransformers does not have official support for GLM-5.3-Flash FP4 so we omit it in the comparison.

Figure 12. End-to-end performance of ADAOFFLOAD and baseline on AIME reasoning tasks with GLM5.3-Flash NVFP4.

Decode policy sensitivity. Figure 13 compares ADAOFFLOAD’s adaptive decode policy with the best and worst measured fixed-qq policies for GLM-5.3-Flash. We sweep q=0.1,0.2,…,0.9q = 0.1, 0.2, \ldots, 0.9 and then run the adaptive policy, using one persistent server per platform with matched priming and cache resets.

Figure 13. Decode throughput of ADAOFFLOAD versus the best and worst measured fixed-qq policies at concurrency C=1C=1 and C=8C=8 for GLM5.3-Flash.

For GLM-5.3-Flash, Averaged across the four platforms, ADAOFFLOAD achieves 3.86×3.86\times, 7.64×7.64\times, and 9.78×9.78\times decode throughput over the strongest baseline at concurrency 1, 4, and 8, respectively, while reducing mean TTFT by 79.6%, 70.0%, and 83.9%. Comparing with the best fixed-qq policy, ADAOFFLOAD attains 94.7% and 99% of the best measured throughput.

GLM-5.2 AIME EVALUATION ON B200

Figure 14 compares ADAOFFLOAD with llama.cpp and Ollama when serving GLM-5.2 with NVFP4 routed experts on two NVIDIA B200 GPUs. KTransformers does not natively support GLM-5.2 and is therefore omitted from comparison. The configuration reserves half of the available HBM for the expert cache. We use 2, 8, and 16 AIME questions at concurrency 1, 4, and 8, respectively, with scheduled arrivals every 3 seconds.

Figure 14. AIME performance of ADAOFFLOAD, llama.cpp, and Ollama with GLM-5.2 NVFP4 on two B200 GPUs as concurrency varies. (a) Aggregate streaming decode throughput; (b) mean TTFT (log scale). The output cap is 512 tokens. TTFT includes the first request.

With two B200, the aggregate GPU HBM capacity is 360 GiB, smaller than the routed expert size for GLM-5.2. This example illustrates an important role ADAOFFLOAD plays in inference ecosystem. Renting B200 is very expensive. For people who want to serve frontier model like GLM-5.2, ADAOFFLOAD gives the option of avoiding renting 4 B200 to fully fit the model’s weight. This provides a budget option for small-concurrency inference of strong model on a small number of powerful GPUs.

References

  1. [2]Anthropic. Claude Code overview. Claude Code documentation, 2026a. URL https://code.claude.com/docs/en/overview. Accessed October 30, 2026.
  2. [4]Cao, S., Liu, S., Griggs, T., Schafhalter, P., Liu, X., Sheng, Y., Gonzalez, J. E., Zaharia, M., and Stoica, I. MoE-Lightning: High-throughput MoE inference on memory-constrained GPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (ASPLOS ’25), pp. 715–730. Association for Computing Machinery, 2025. doi: 10.1145/3669940.3707267. URL https://doi.org/10.1145/3669940.3707267.
  3. [5]Chen, H., Xie, W., Zhang, B., Tang, J., Wang, J., Dong, J., Chen, S., Yuan, Z., Lin, C., Qiu, C., Zhu, Y., Ou, Q., Liao, J., Chen, X., Ai, Z., Wu, Y., and Zhang, M. KTransformers: Unleashing the full potential of CPU/GPU hybrid inference for MoE models. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP ’25), pp. 1014–1029. Association for Computing Machinery, 2025. doi: 10.1145/3731569.3764843. URL https://doi.org/10.1145/3731569.3764843.
  4. [6]DeepSeek-AI. DeepSeek-V4.1-Flash model card, 2026. URL https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash. Accessed September 29, 2026.
  5. [7]Eliseev, A. and Mazur, D. Fast inference of mixture-of-experts language models with offloading. arXiv preprint arXiv:2312.17238, 2023. URL https://arxiv.org/abs/2312.17238.
  6. [8]Fang, Z., Huang, Y., Hong, Z., Lyu, Y., Chen, W., Yu, Y., Yu, F., and Zheng, Z. Klotski: Efficient mixture-of-expert inference via expert-aware multi-batch pipeline. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’25), pp. 574–588. Association for Computing Machinery, 2025. doi: 10.1145/3676641.3716261. URL https://doi.org/10.1145/3676641.3716261.
  7. [9]Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2009.03300.
  8. [10]Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, Z. GPipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/093f65e080a295f8076b1c5722a46aa2-Abstract.html.arxiv.org/abs/1811.06965
  9. [12]Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html.arxiv.org/abs/2310.06770
  10. [13]Kamahori, K., Tang, T., Gu, Y., Zhu, K., and Kasikci, B. Fiddler: CPU-GPU orchestration for fast inference of mixture-of-experts models. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2402.07033.
  11. [14]Kim, H., Ryu, J., and Lee, J. TCCL: Discovering better communication paths for PCIe GPU clusters. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS ’24, pp. 999–1015. Association for Computing Machinery, 2024. doi: 10.1145/3620666.3651362. URL https://doi.org/10.1145/3620666.3651362.
  12. [15]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). Association for Computing Machinery, 2023. URL https://arxiv.org/abs/2309.06180.
  13. [16]Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb.
  14. [18]Lyu, Y., Li, N., and Jia, B. SPICE: Speculative prefetching with low-rank expert surrogates and heterogeneous orchestration for MoE inference acceleration. arXiv preprint arXiv:2608.21240, 2026. URL https://arxiv.org/abs/2608.21240. Accepted to ASP-DAC 2027.
  15. [19]Moonshot AI. Kimi-K3 model card, 2026. URL https://huggingface.co/moonshotai/Kimi-K3. Accessed September 30, 2026.
  16. [20]Ollama Contributors. Ollama frequently asked questions, 2026. URL https://docs.ollama.com/faq. Accessed October 19, 2026.
  17. [21]Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31094–31116. PMLR, 2023. URL https://proceedings.mlr.press/v202/sheng23a.html.
  18. [22]Son, M., Chen, Y., Yoo, S., Choi, S., and Kim, J.-Y. CoX-MoE: Coalesced expert execution for high-throughput MoE inference with AMX-enabled CPU-GPU co-execution. In Proceedings of the 63rd ACM/IEEE Design Automation Conference (DAC), 2026. URL https://arxiv.org/abs/2605.17889.
  19. [23]Song, X., Zhong, Z., Chen, R., and Chen, H. ProMoE: Fast MoE-based LLM serving using proactive caching. arXiv preprint arXiv:2410.22134, 2024. URL https://arxiv.org/abs/2410.22134.
  20. [25]Tang, P., Liu, J., Hou, X., Pu, Y., Wang, J., Heng, P.-A., Li, C., and Guo, M. HOBBIT: A mixed precision expert offloading system for fast MoE inference. arXiv preprint arXiv:2411.01433, 2024. URL https://arxiv.org/abs/2411.01433.
  21. [27]Wang, Z., Tang, C., Gong, L., Wang, C., Lou, W., Wang, T., and Zhou, X. ActKV: Efficient LLM agents through action-guided KV cache management. arXiv preprint arXiv:2609.31395, 2026. URL https://arxiv.org/abs/2609.31395.
  22. [29]Xue, L., Fu, Y., Lu, Z., Mai, L., and Marina, M. MoE-Infinity: Efficient MoE inference on personal machines with sparsity-aware expert cache. arXiv preprint arXiv:2401.14361, 2024. URL https://arxiv.org/abs/2401.14361.
  23. [30]Yang, S., Fan, X., Pan, M., Xi, H., Wang, Z., Sun, S., Keutzer, K., Han, S., Zaharia, M., Xu, C., and Stoica, I. FireToken: Efficient edge-native MoE serving with bandwidth-adaptive execution. arXiv preprint arXiv:2608.16157, 2026. URL https://arxiv.org/abs/2608.16157.
  24. [33]Z.ai. GLM-5.3-Flash model card, 2026. URL https://huggingface.co/zai-org/GLM-5.3-Flash. Accessed September 28, 2026.
  25. [34]Zhang, Y. and Math-AI Team. American invitational mathematics examination (AIME) 2025. Hugging Face dataset repository, 2025. URL https://huggingface.co/datasets/math-ai/aime25. Accessed October 30, 2026.
  26. [35]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, volume 37, pp. 62557–62583, 2024. URL https://arxiv.org/abs/2312.07104.
  27. [36]Zhong, S., Sun, Y., Liang, L., Wang, R., Huang, R., and Li, M. HybridMoE: Hybrid CPU-GPU scheduling and cache management for efficient MoE inference. In Proceedings of the 62nd ACM/IEEE Design Automation Conference (DAC), 2025. URL https://arxiv.org/abs/2504.05897.

Paper details

vlnc:W_ttnqk6hu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex

View statement
Looking for related papers…
Contents