betaFeedback

WorksComputer Science

CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs

Abstract

Despite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident. To address this, we identify three key observations for on-device MoE inference: (a) cache hit rates are fundamentally bounded even under Belady's optimal policy; (b) these bounds are insufficient for low-latency on-device inference; and (c) prior cache-aware routing based on binary cache presence can induce routing instability. Based on these insights, we propose CECAR (Cache & Expert Co-Aware Routing), a MoE inference system that jointly considers expert routing and cache management. Unlike prior cache-aware routing methods, CECAR uses an ML-based cache policy to predict the expected reuse distance of each expert and prioritizes routing and execution toward experts with smaller predicted reuse distances. On the Qwen3-30B-A3B model, CECAR achieves 12.5 tokens/s on a consumer-grade GPU with only 16 GB of VRAM, where non-resident experts are fetched from SSD, delivering a 4.5x speedup over conventional MoE inference and exceeding the decode speed of baselines that keep the entire model fully resident in DRAM. Code is available at https://github.com/hoonki2002/CECAR.

Introduction

Large language models (LLMs) have achieved significant advances, but their deployment on user-centric devices remains challenging due to strict memory and latency constraints. While on-device inference is increasingly preferred for privacy, latency, and cost reasons [Li et al., 2024], limited hardware resources restrict executable model size to a few billion parameters (e.g., Mistral 7B [10]) [Xu et al., 2024]. This trend is already reflected in commercial systems such as Microsoft Copilot and Samsung Galaxy devices, which increasingly provide on-device AI functionalities for tasks including summarization, translation, and voice interaction.

Mixture-of-Experts (MoE) architectures [Shazeer et al., 2017] mitigate computation by activating only a subset of experts per token, enabling large model capacity with reduced per-token cost. However, the total parameter footprint of all experts remains large, introducing a critical bottleneck in on-device environments. As illustrated in Figure 1, the limited DRAM capacity of typical devices (8–16 GB) prevents most experts from residing in memory, forcing them to be stored in lower memory tiers such as SSDs [1]. Consequently, dynamically loading top-kk experts incurs frequent memory transfers, making memory access latency the dominant factor limiting throughput (e.g.,

Figure 1. Memory bottlenecks for MoE inference on consumer devices.

40th Conference on Neural Information Processing Systems (NeurIPS 2026). ∼10\sim10 tokens/s). In particular, even a small number of uncached experts can dominate decoding time, highlighting the importance of efficient cache utilization. As a result, MoE architectures alone are insufficient to enable practical on-device inference.

Prior work addresses this challenge through expert pruning and cache-aware routing. Pruning methods such as DES [Lu et al., 2024] and ODP [9] reduce activated experts, improving efficiency at the cost of reduced model capacity and degraded accuracy on benchmarks such as MATH [8] and HumanEval [Chen et al., 2021]. Cache-aware routing methods, such as MoCCE [Skliar et al., 2025], target on-device settings by improving cache hit rates through preferential selection of cached experts.

However, these approaches fail to address key challenges of on-device inference. Even with pruning, many experts must still be repeatedly loaded from SSDs, and existing methods do not effectively exploit temporal locality by reusing cached experts. Moreover, pruning requires aggressive reduction under small cache budgets, further degrading accuracy, while cache-aware routing relies on binary cache signals and cannot distinguish eviction priorities among cached experts. As a result, existing methods remain insufficient to simultaneously achieve high efficiency and stable accuracy under constrained cache capacity.

To address these challenges, we propose CECAR (Cache & Expert Co-Aware Routing), which jointly considers expert cache residency and predicted eviction likelihood under constrained cache capacity. CECAR integrates cache residency and predicted eviction likelihood into the routing process, maximizing inference speed while minimizing accuracy degradation in on-device environments. It is composed of three tightly coupled components:

Cache-Aware Routing. Cache residency is explicitly incorporated into the routing process, biasing expert selection toward those already present in VRAM. Unlike conventional MoE inference that strictly follows router logits, this mechanism reduces expensive expert loading from lower memory tiers (e.g., SSD), which dominates token-level latency in constrained environments. Furthermore, it applies cache-conditioned pruning during expert selection, skipping uncached low-priority experts. Importantly, cache awareness is applied during expert selection without altering the final routing weights, preserving the model’s original behavior.

Expert-Aware Routing. Semantic preference of the original router is preserved by maintaining alignment with its ranking over experts. Rather than replacing routing decisions based solely on cache status, it ensures that selected experts remain consistent with the model’s learned routing distribution. To achieve this, CECAR applies margin-based adjustments as controlled logit perturbations within the top-ranked candidates, allowing flexible reordering without deviating significantly from the original routing. This design maintains semantic consistency while preventing routing instability, avoiding the cascading errors observed in binary cache-aware approaches during autoregressive decoding.

ML-Based Cache Policy. A lightweight learning-based cache policy is used to predict the reuse likelihood of each expert. The policy leverages normalized recency and frequency signals to produce fine-grained eviction scores, enabling differentiation between frequently reused (“hot”) and rarely reused (“cold”) experts. These scores are directly integrated into the routing stage as eviction-aware bonus terms, allowing CECAR to jointly optimize cache replacement and expert selection under varying cache sizes and workloads.

We demonstrate that, in an on-device environment with 16 GB VRAM and 1 GB system DRAM, our approach enables inference of a 30B-parameter LLM at 1.5 tokens/s using SSD-based storage (Figure 2), achieving a 4.5× speedup over conventional MoE inference and outperforming DES (1.7×) and MoCCE (3.1×). Moreover, compared to a baseline that executes the same MoE model on the same GPU (16 GB VRAM) with all expert parameters (approximately 56.7 GB) fully resident in system DRAM, achieving 7 tokens/s, our approach delivers a 1.8× higher inference throughput while reducing the total memory footprint (VRAM + system DRAM) by approximately 78.8%. These results demonstrate that 30B-scale LLMs, which typically require more than 60 GB of system DRAM, can be executed at practical speeds even in memory-constrained on-device environments, and further suggest the potential for scaling to even larger models.

Figure 2. Decoding Speed (Tokens/s) Across MoE Models under Different Routing Strategies.

Related Works

Expert Caching

To overcome memory capacity limitations in MoE inference systems, prior work has proposed caching recently used experts. Mixtral-Offloading[7] was the first to apply LRU caching to expert layers in MoE models, achieving a 2.2×2.2\times speedup compared to naïve offloading approaches designed for dense models. FlashMoE[Kim et al., 2026] further improved accuracy by leveraging a calibration set based on Belady’s optimal policy[3], achieving up to a 2.6×2.6\times speedup over heuristic cache policies such as LRU and LFU. These studies demonstrate that expert caching is feasible in MoE inference systems and can yield substantial speed improvements when properly managed. However, they do not fully exploit a key property of MoE models—that routing to alternative experts, even when not exact, does not necessarily incur significant accuracy degradation.

Expert Pruning

Expert pruning techniques have also been proposed, in which inference does not strictly follow the experts learned during training, but instead selectively omits or replaces certain experts. In MC-MoE[9], Dynamic Expert Skipping (DES) dynamically skips experts with low contribution among the selected experts, while Online Dynamic Pruning (ODP) further mitigates accuracy degradation by incorporating attention-based key token protection. These approaches require prior calibration or threshold tuning to minimize accuracy loss after pruning, which can introduce additional overhead in terms of latency and memory usage in practical on-device environments. More importantly, these studies do not consider cache locality during the routing stage, and therefore cannot be regarded as optimized for on-device settings.

Efficient MoE Inference

Beyond caching and pruning, a variety of techniques have been proposed to accelerate MoE inference by optimizing hardware resource utilization or improving model representations. Fiddler[Kamahori et al., 2024] dynamically determines where computation should be performed for each token and expert, alleviating CPU–GPU communication bottlenecks and achieving up to a 1.26×1.26\times inference speedup. HOBBIT[Tang et al., 2024] reduces both memory footprint and computational cost of expert layers by applying mixed-precision techniques based on expert importance. Beyond these, recent expert-offloading systems intervene at three distinct points of the pipeline. (i) Exact-routing caching and prefetching: MoE-Infinity[Xue et al., 2025] targets single-user, batch-size-one inference on personal machines with a sparsity-aware expert cache and activation-trace-based prefetching, and FineMoE[Yu et al., 2025] guides asynchronous prefetching and eviction per request using semantic and trajectory signals. These systems leave the router’s decision untouched and hide I/O by predicting which experts will be needed. (ii) Scheduling and deferral: KTransformers[4] keeps selected experts in host memory, runs them with optimized CPU kernels, and defers the outputs of low-score experts to a later layer to overlap CPU and GPU work; it assumes enough host DRAM for the experts rather than trying to reduce it. (iii) Cache-aware expert selection: CECAR adjusts the routing decision itself so that fewer non-resident experts are requested in the first place. A system that keeps exact routing cannot exceed the hit rate of Belady’s policy on the original routing; CECAR is not bound by it, and the two approaches can be combined (Appendix § 16).

Motivation

When performing MoE LLM inference in on-device environments, limited memory capacity prevents all experts from being kept resident in memory. For example, under a setting where only 24 out of 128 experts can be stored in memory, conventional cache replacement policies such as Least Recently Used (LRU) and Least Frequently Used (LFU) exhibit substantially limited average layer-wise cache hit rates, as illustrated in Figure 3a.

Latency and memory implications of cache hit rate and expert cache size

Figure 3. Latency and memory implications of cache hit rate and expert cache size.

Using the Qwen3-30B-A3B[Yang et al., 2025] model, we measured cache hit rates across multiple datasets during inference and observed that the average hit rates of LRU and LFU are only 67% and 45%, respectively. This is approximately 10–30 percentage points lower than the hit rate of 80% achieved by the ideal Belady’s optimal policy under the same conditions. Similar patterns were observed for the OLMoE-1B-7B-0125-Instruct[Muennighoff et al., 2025] and DeepSeek-V2-Lite-Chat[6] models, and repeated experiments across diverse datasets consistently showed that LRU/LFU-based approaches yield significantly lower hit rates than Belady’s optimal policy.

A more critical issue is that even when assuming Belady’s optimal policy, cache hit rates remain insufficient in practical on-device settings with limited cache capacity. For the Qwen3-30B-A3B model, when only 24 out of 128 experts can be kept in memory, the average layer-wise hit rate reaches only 84.2% even under the ideal Belady policy (Figure 3a). As shown in Figure 3b, this hit rate still results in an I/O-dominated execution, where expert loading accounts for over 75% of the per-token latency and limits the achievable decoding speed to around 5 tokens/s. Therefore, further improvements to cache replacement alone are fundamentally insufficient, motivating a routing-stage approach that explicitly accounts for cache locality.

However, cache-aware routing that relies solely on the binary status of cache presence (hit or miss) can introduce significant challenges in memory-constrained environments. To better understand the limitations of such approaches under tight memory constraints, we analyze the behavioral impact of routing-stage approximation. While prior work[Skliar et al., 2025] reports minimal degradation in perplexity, we find that cache-aware routing based on binary cache presence can induce a stickiness effect, where certain experts persist in the cache despite limited reuse. Under MoCCE-style routing, 6.1% of cache residency periods exhibit prolonged idle behavior (remaining in cache for over 20 tokens with fewer than five uses), with idle residency extending up to 53 tokens, indicating a long-tail risk of cache lock-in.

This cache-induced bias also affects routing stability during generation. In free-running decoding after cache warm-up, small routing perturbations introduced by binary cache-aware routing are amplified through autoregressive generation, causing the output to diverge from the exact-routing baseline after 23 tokens on average, with only 6.8% token-level agreement. Although such divergence does not necessarily lead to immediate benchmark degradation, it reveals an instability where cache bias can cascade over time. These observations suggest that effective cache-aware routing must go beyond binary cache presence and selectively account for expert reuse characteristics to improve cache locality while preserving semantic stability.

Accordingly, the goal of this work is to leverage the memory constraints and expert flexibility observed above to substantially improve the inference efficiency of MoE-based LLMs in highly resource-constrained on-device environments. To this end, we propose CECAR, a novel routing policy that jointly accounts for cache state and expert reuse status, overcoming the structural limitations of existing exact-routing-based MoE inference methods.

Method

Preliminary

The CECAR inference system targets a fine-grained Mixture-of-Experts (MoE) architecture. Each MoE layer consists of 64 to up to 128 experts, and 6∼86 \sim8 experts are selected per token. This architecture poses a practical challenge in that, due to memory constraints, it is difficult to keep all experts resident in memory. Conventional heuristic-based exact caching policies (e.g., LRU and LFU) are simple and deterministic, but fail to capture the nonlinear expert selection patterns across tokens, resulting in limited cache hit rates. Accordingly, we propose a pre-trained, learning-based cache policy that can preserve model response accuracy while maintaining high hit rates, even without guaranteeing exact routing.

CECAR

CECAR is motivated by the observation that MoE models can retain a reasonable level of accuracy even without exact expert routing. This method leverages the output of a learning-based scoring model, and we experimentally confirm that it can achieve higher cache hit rates than LRU. Based on this score, CECAR excludes uncached experts from the router-selected top-kk set and reassigns weights to the remaining experts. For example, if the 7th and 8th experts in a top-8 selection are not resident in the cache, normalized weights are reassigned to the remaining top-1 to top-6 experts. In this way, CECAR preserves routing sparsity while maximizing cache hit rates.

Figure 4 illustrates an example of CECAR under a cache size of 24. In the illustrated top-8 routing decision, three experts are initially not resident in the cache, resulting in an expected cache hit rate of 5/8. During expert selection, the bonus mechanism excludes Expert 63 from the routing candidates due to its low expected reuse. In addition, top-kk expert skipping (with k=4k = 4 in the example) further removes Expert 4 from consideration. As a result, the effective routing set contains only cached experts, increasing the expected hit-rate from 5/8 to 6/7.

Figure 4. Illustration of the CECAR Routing Algorithm. Darker colors denote experts that CECAR expects to be routed in the near future.

Algorithm 1: CECAR: Cache & Expert Co-Aware Routing

Input: Layer index ℓ\ell, expert logits {ze}e=1N\{z_e\}_{e=1}^{N}, cache state CℓC_\ell, recency scores rnormer^{e}_{\mathrm{norm}}, frequency scores fnormef^{e}_{\mathrm{norm}}, top-KK, margin index mm, prune threshold τ\tau

(1) Compute routing margin Let z[1]≥z[2]≥⋯≥z[N]z_{[1]} \ge z_{[2]} \ge\cdots\ge z_{[N]} be the sorted values of {ze}\{z_e\}, Δ←z[1]−z[m]\Delta\leftarrow z_{[1]} - z_{[m]}

(2) Eviction score computation foreach e∈Cℓe \in C_\ell do sℓ(e)←ML_Scorer(rnorme,fnorme)s_\ell(e) \leftarrow\mathrm{ML\_Scorer}(r^{e}_{\mathrm{norm}}, f^{e}_{\mathrm{norm}})

(3) Eviction-aware cache bonus foreach e∈Cℓe \in C_\ell do δe←1−sℓ(e)max⁡e′∈Cℓsℓ(e′), ze′←ze+Δ⋅δe\delta_e \leftarrow1 - \frac{s_\ell(e)}{\max_{e' \in C_\ell} s_\ell(e')},\ z'_e \leftarrow z_e + \Delta\cdot\delta_e

(4) Cache-aware top-kk selection Ecand←TopK⁡(z′,K)\mathcal{E}_{\mathrm{cand}} \leftarrow\operatorname{TopK}(z', K)

(5) Cache-conditioned pruning E←∅\mathcal{E} \leftarrow\varnothing foreach (j,e)∈Ecand(j,e) \in\mathcal{E}_{\mathrm{cand}} do if e∈Cℓe \in C_\ell or j≤τj \le\tau then E←E∪{e}\mathcal{E} \leftarrow\mathcal{E} \cup\{e\}

(6) Cache replacement foreach e∈Ee \in\mathcal{E} do if e∉Cℓe \notin C_\ell then e∗←arg⁡max⁡e′∈Cℓsℓ(e′), Cℓ←(Cℓ∖{e∗})∪{e}e^* \leftarrow\arg\max_{e' \in C_\ell} s_\ell(e'),\ C_\ell\leftarrow(C_\ell\setminus\{e^*\}) \cup\{e\}

return E\mathcal{E}

As shown in Algorithm 1, CECAR decomposes the routing stage into separate expert selection and weighting steps. A bonus is computed by measuring the margin between the top-1 and top-kk router logits, and logit adjustments are applied only to experts that are resident in the cache. The magnitude of this bonus is modulated according to an ML-based eviction score, and does not affect the routing probabilities themselves. The adjusted logits are used exclusively for expert selection (top-kk), while the final softmax weights are computed from the original logits. This design steers expert selection toward cache-friendly choices while preserving the original weighting behavior of the model.

During the decoding stage, top-kk experts are selected using the bonus-adjusted logits; if any of the selected experts are not resident in the cache, they are removed from the selection. Weights are then assigned to the remaining experts by normalizing them with respect to the sum over the original top-8 experts. The top-kk threshold used in this process is empirically fixed per model, and is set to 4 for Qwen3-30B-A3B and 3 for DeepSeek-V2-Lite-Chat in our study (see Appendix § 11). Although this approach introduces a degree of approximation compared to exact top-kk routing, it is compensated by token-level soft weighting. As a result, the method achieves high flexibility without incurring performance degradation due to cache misses.

The scoring model used in CECAR leverages the scores produced by the ML-based cache policy predictor described earlier. This score represents the eviction priority of each expert and is directly incorporated into the computation of the logit adjustment coefficient, denoted as δe\delta_e in Algorithm 1. Through this design, CECAR enables efficient, scoring-based bonus assignment within the existing MoE architecture without introducing additional parameters.

Experimental results (Table 3) show that CECAR consistently outperforms MoCCE in terms of both cache hit rate and decode speed. In particular, CECAR consistently maintains an average cache hit rate exceeding 95% with negligible accuracy degradation in Qwen3-30B-A3B. Furthermore, CECAR is highly practical in that it can be directly applied to existing MoE architectures without requiring any structural modifications.

Table 3: Benchmark accuracy comparison & performance of cache-aware MoE inference methods. Accuracy is mean ± SD over 3 decoding seeds; bold and underline mark the best and second-best cache-aware or pruning method (baseline excluded).

ModelMethodHumanEvalMBPPGPQA-DMATH-500Avg. top-kk*Hit-Rate (%)Tokens/s
Qwen3-A3BBaseline67.7 ±1.264.4 ±0.936.7 ±1.556.7 ±2.28.0068.32.8 ±0.31
DES65.4 ±2.152.1 ±0.734.0 ±0.838.0 ±1.24.6771.34.7 ±0.97
ODP66.3 ±0.955.4 ±1.231.0 ±0.844.1 ±0.84.7173.15.0 ±1.43
MoCCE67.5 ±1.961.8 ±0.538.0 ±2.855.3 ±2.88.0092.38.6 ±1.80
CECAR67.9 ±3.061.1 ±1.436.4 ±1.856.4 ±1.77.1396.612.5 ±1.19
DeepSeek-A2.4BBaseline47.4 ±0.945.7 ±0.416.8 ±1.824.1 ±0.86.0058.24.4 ±4.87
DES45.3 ±0.944.6 ±0.610.8 ±0.822.9 ±1.45.0762.25.0 ±3.48
ODP44.5 ±2.744.0 ±0.911.8 ±1.523.1 ±1.35.1160.44.6 ±2.19
MoCCE44.9 ±2.544.2 ±0.616.7 ±1.323.9 ±2.36.0083.97.6 ±0.55
CECAR45.9 ±1.344.8 ±1.317.5 ±2.323.5 ±1.85.5187.110.3 ±3.59
OLMoE-A1BBaseline25.8 ±1.326.5 ±0.625.9 ±0.819.0 ±0.78.0074.58.5 ±1.97
DES21.7 ±2.722.7 ±0.320.4 ±0.817.7 ±0.35.7977.616.2 ±2.93
ODP24.6 ±0.923.7 ±0.321.2 ±2.218.3 ±0.35.7978.618.4 ±5.69
MoCCE25.6 ±2.426.6 ±0.926.1 ±1.116.7 ±1.08.0091.427.7 ±6.96
CECAR25.2 ±3.125.5 ±1.028.8 ±1.017.3 ±0.57.6693.529.9 ±4.99
  • Average number of routed experts per layer (excluding shared experts). Baseline and MoCCE use fixed top-kk routing (kk=8 for Qwen3-A3B and OLMoE-A1B, kk=6 for DeepSeek-A2.4B), while DES, ODP, and CECAR dynamically adjust the effective top-kk. Accuracy is reported as mean ± SD over 3 decoding seeds on the full benchmarks; tokens/s are reported as mean ± SD across repeated on-device runs (Appendix F). The single-seed HumanEval sweeps in Appendices B–D use one of these seeds and therefore differ slightly from the 3-seed means here.

Table 3.

ML-Based Cache Policy

Existing cache policies, such as LRU and LFU, fail to capture the complex expert reuse patterns that arise during MoE inference. FlashMoE addresses this limitation by learning an ML-based eviction policy trained with Belady’s optimal targets, achieving improved hit rates on in-distribution workloads.

However, we observe that FlashMoE’s ML cache policy exhibits two critical limitations when applied in conjunction with expert-aware routing. First, the learned policy fails to generalize to out-of-distribution (OOD) inputs, often underperforming even LRU. Second, its performance is highly sensitive to cache size, degrading significantly under small or varying cache configurations, which limits its applicability in memory-constrained settings such as on-device inference.

To address these issues, we design an ML-based cache policy tailored for CECAR. Our approach decouples eviction prediction from expert indices, preventing overfitting to dataset-specific expert access patterns observed during training and thereby improving robustness to out-of-distribution inputs. It produces fine-grained, expert-specific eviction likelihood scores that can be directly incorporated into routing decisions. Moreover, we account for the fact that expert reuse characteristics vary across cache regimes. Rather than relying on a single reuse signal, CECAR adjusts the relative influence of multiple reuse predictors according to the cache configuration. We refer to these predictors as the hot expert detector (FFNHot\mathrm{FFN}_{\mathrm{Hot}}) and the cold expert detector (FFNCold\mathrm{FFN}_{\mathrm{Cold}}), which together enable eviction decisions to adapt to different memory constraints.

Figure 5 illustrates the principle of hot and cold expert detection used in the proposed ML-based cache policy. The figure assumes a model with 16 total experts and top-2 routing. The hot expert detector is trained to identify experts that are likely to be reused in the near term (e.g., Expert Index 3, 5, and 11), whereas the cold expert detector is trained to identify experts that are unlikely to be reused (e.g., Expert Index 1, 4, and 13). When the cache size is small, caching decisions must carefully distinguish experts that are likely to be reused in the near term from those that were only recently accessed, and the hot detector is therefore weighted more heavily. As the cache size increases, identifying experts with low reuse likelihood becomes more important, shifting emphasis toward the cold detector. This cache-size-aware weighting enables robust performance across workloads and cache sizes.

Figure 5. Merging FFNHot\mathrm{FFN}_{\mathrm{Hot}} and FFNCold\mathrm{FFN}_{\mathrm{Cold}} for Eviction Score.

The precise weighting strategy, training targets, and an analysis of the limitations of FlashMoE and how they are addressed by our approach are described in Appendix § 9.

Experiments

Experiment Setup

All experiments are conducted under a controlled inference setting using stochastic decoding with temperature 0.7, top-pp sampling threshold of 0.95, and a maximum generation length of 2048 tokens. We evaluate Qwen3-30B-A3B, DeepSeek-V2-Lite-Chat, and OLMoE-1B-7B-0125-Instruct on HumanEval, MBPP[2], GPQA-Diamond, and MATH-500. Accuracy is measured on the full benchmark datasets using a virtual-cache executor of Algorithm 1 with all expert weights resident in VRAM, since routing decisions and cache state depend only on expert identities and not on whether weights are materialized; hit-rate and decode speed are measured on the actual SSD-offloading setup by sampling 100 instances per benchmark (400 samples in total), with host

DRAM pinned to ∼1\sim1 GB (see Appendix § 14). A fixed cache budget of 24 experts per layer is used across all methods. While the cache size is constant, the specific experts stored are dynamically updated during decoding. Decoding throughput (tokens/s) is measured during the decode phase only. For the hardware configuration of the experiments, please refer to Appendix § 14.

Component-wise Ablation of CECAR

Table 1 presents a component-wise ablation study quantifying the effects of the ML-based cache policy, cache-aware bonus routing, and expert pruning on HumanEval accuracy, cache hit rate, and decoding speed. The baseline configuration achieves 47.56 accuracy, a cache hit rate of 54.83%, and a decoding speed of 3.40 tokens/s.

MLBonusPruningHumanEvalHit-rate (%)Decoding Speed
×\times×\times×\times47.5654.833.40
✓\checkmark×\times×\times47.5657.953.54
×\times✓\checkmark×\times44.5175.915.45
×\times×\times✓\checkmark47.5663.477.35
✓\checkmark✓\checkmark×\times51.8376.575.60
✓\checkmark×\times✓\checkmark49.3966.877.40
×\times✓\checkmark✓\checkmark49.3982.827.07
✓\checkmark✓\checkmark✓\checkmark49.3986.878.98

Table 1. Ablation study on ML-based cache policy, CECAR bonus routing, and expert pruning on DeepSeek-V2-Lite-Chat. Accuracy, hit-rate, and decoding speed are all measured on HumanEval only, whereas Table 3 averages hit-rate and speed over four benchmarks.

Introducing the ML-based cache policy alone increases the cache hit rate from 54.83% to 57.95% while leaving accuracy unchanged (47.56), resulting in a modest throughput improvement to 3.54 tokens/s. Applying cache-aware bonus routing significantly boosts the cache hit rate to 75.91% and decoding speed to 5.45 tokens/s; however, this comes at the cost of reduced accuracy (44.51), indicating semantic distortion when routing decisions are biased without eviction awareness.

Combining the ML-based cache policy with bonus routing restores and even improves accuracy to 51.83, while sustaining a high cache hit rate of 76.57% and decoding speed of 5.60 tokens/s. Similarly, combining the ML-based cache policy with expert pruning improves both accuracy (49.39) and throughput (7.40 tokens/s) compared to pruning alone. When all components are enabled, CECAR achieves the best overall performance, reaching a cache hit rate of 86.87% and a decoding speed of 8.98 tokens/s, while maintaining strong accuracy (49.39).

These results demonstrate that while bonus routing and expert pruning are effective at improving cache efficiency and throughput, their benefits can be fully realized only when guided by the ML-based cache policy. Learned eviction scores play a critical role in preventing semantic degradation and enabling stable integration of cache-aware routing and pruning under constrained memory budgets.

ML-Based Cache Policy for CECAR

As shown in Table 2, on the TriviaQA[11] in-distribution (ID) dataset, ML(FlashMoE) achieves higher cache hit rates than LRU and LFU policies. However, on out-of-distribution (OOD) datasets such as HumanEval and MBPP, ML(FlashMoE) exhibits lower hit rates than LRU, indicating limited robustness to workload shifts. In contrast, the proposed ML(CECAR) consistently achieves the highest cache hit rates across all evaluated datasets, demonstrating strong robustness under OOD workloads.

PolicyCache Size (TriviaQA)Dataset Generalization (Cache Size = 24)
8162432TriviaQAHumanEvalMBPPGPQA-DiamondMATH-500
Belady’s Optimal52.3072.4782.5688.6882.5679.5179.6282.3779.49
ML (CECAR){sup}**44.1862.4273.6382.0473.6368.3769.3173.7369.63
ML (FlashMoE)34.4158.6373.2282.2373.2244.8446.1753.0746.17
LRU33.3958.4569.4978.2869.4965.5065.2670.8666.04
LFU11.3330.6547.5360.9647.5339.5047.2545.6643.49
LIFO9.6027.8743.7156.8943.7139.2045.3039.7539.13

Table 2. Comparison of cache hit rates (%) across replacement policies.

Table 2 also highlights that the performance of ML (FlashMoE) is highly sensitive to cache size. In particular, when the cache size is small, ML(FlashMoE) underperforms even LRU, including on the TriviaQA in-distribution dataset. By contrast, ML(CECAR) maintains stable and superior hit rates across a wide range of cache sizes, indicating improved robustness to cache capacity variations.

On its own, the ML-based cache policy improves the hit rate over LRU by only 3–4 points (Table 2; 54.83% to 57.95% in Table 1). Its value lies in robustness: unlike ML(FlashMoE), it does not fall below LRU on out-of-distribution workloads or at small cache sizes, so its scores can be used in the routing stage without the failure mode seen in Table 2. This robustness does not come from the scorer’s size: a sweep over 15 architectures spanning a 1000× range of parameter counts changes the OOD hit rate by less than 0.5% (Appendix K).

Overall Performance of CECAR

As shown in Table 3, CECAR consistently achieves the highest cache hit rates and decoding throughput across all evaluated models. For the Qwen3-30B-A3B model, CECAR reaches a cache hit rate of 96.6% with a decoding speed of 12.5 tokens/s, outperforming both LRU-based routing and prior cache-aware methods. In comparison, MoCCE attains a lower hit rate of 92.3% and 8.6 tokens/s, while DES and ODP achieve only around 70–73% hit rates with substantially reduced throughput (4–5 tokens/s).

The speedup is not a result of activating fewer experts. On Qwen3-30B-A3B, DES and ODP activate fewer experts on average (4.67 and 4.71) than CECAR (7.13) but run at 4.7–5.0 tokens/s against CECAR’s 12.5. The difference is the hit rate (70–73% vs. 96.6%). DES and ODP prune by routing weight alone, so many of the experts they keep still have to be read from SSD; CECAR’s cache-conditioned pruning drops the low-priority experts that are not resident. Appendix I breaks the per-token latency down and shows that reduced expert loading accounts for nearly all of the gain.

We observe a strong positive correlation between cache hit rate and token decoding speed. Methods with hit rates above 90% consistently operate at significantly higher throughput, whereas strategies with hit rates around 70% experience frequent stalls due to expert cache misses. Notably, the throughput improvement achieved by CECAR is disproportionate to its absolute hit-rate gain over LRU-based methods, suggesting that eliminating tail cache misses—rare but expensive expert loads—has a dominant impact on decoding latency.

Combined with the accuracy results, CECAR achieves higher cache hit rates and up to 4.5×4.5 \times speedup while keeping accuracy close to the exact-routing baseline: over the 12 model–benchmark pairs the mean change is −0.5 points, 8 pairs are within 1 point of the baseline, and CECAR is the best or second-best cache-aware or pruning method on 11 of the 12 Table 3.

Conclusion

We identify expert residency as a key bottleneck in MoE-based LLM inference in on-device environments, and experimentally analyze the necessity of cache-aware routing and cache management to mitigate this issue. Our results show that existing ML-based cache policies fall short of Belady’s optimal, and that this limitation becomes even more pronounced under constrained cache capacity. We further demonstrate that approximate expert selection that accounts for cache locality at the routing stage can simultaneously improve cache hit rates and inference performance while locally constraining semantic distortion. CECAR operates orthogonally to commonly used model compression techniques, such as expert quantization, and can be directly combined with them to further reduce memory requirements. Based on these observations, this paper highlights the need for a new routing paradigm that jointly considers cache and expert reuse state, and argues that the proposed CECAR provides a practical and generalizable solution for on-device MoE inference.

Impact Statements

This work improves the efficiency of Mixture-of-Experts (MoE) inference under constrained memory budgets by reducing expert cache misses. The proposed cache-aware routing enables more practical deployment of large MoE models on memory-limited hardware, reducing inference latency. The approach operates purely at the inference and system level and does not modify model architectures. Consequently, it does not introduce new model capabilities or increase risks beyond those inherent to the evaluated models.

Acknowledgments

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korean government (MSIT) under Grant Nos. RS-2025-02218704 (Development of Chiplet-based DNN-SNN Hybrid Computing Processor for Large-Language-Models, 50%) and RS-2025-02218733 (Full-Stack Development of a 1-POPS Digital Neuromorphic Event Processor, 50%).

Detailed Analysis of ML-Based Cache Policies

This appendix provides a detailed analysis of the limitations of existing ML-based cache policies and describes the design of the ML-based cache policy proposed for CECAR.

Limitations of Existing ML-Based Cache Policies

Limited Out-of-Distribution Robustness

FlashMoE trains a feed-forward network (FFN) to predict eviction scores using normalized LRU and LFU statistics as inputs. The FFN jointly processes statistics from all experts within a layer and outputs eviction scores for all experts simultaneously. This joint prediction structure introduces a strong coupling between eviction decisions and expert indices. As a result, the model learns dataset-specific expert activation biases during training. When the routing distribution changes under OOD workloads, these learned biases no longer hold, leading to degraded cache performance. Consequently, although FlashMoE improves cache hit rates on the in-distribution TriviaQA dataset, it often underperforms even LRU on OOD datasets such as HumanEval and MBPP.

Sensitivity to Cache Size

FlashMoE constructs training targets by approximating Belady’s optimal cache replacement policy. At each timestep, the NEN_E experts in the layer are ranked according to their reuse distance, and binary labels are assigned by setting the top 75 percent of experts to 1 and the remaining experts to 0. This binarization collapses fine-grained reuse distance information into a coarse eviction signal. Under small cache sizes, where eviction decisions depend on subtle differences in reuse distance, this coarse labeling prevents the model from learning accurate eviction priorities. As a result, FlashMoE exhibits significant performance degradation as cache size decreases, even on in-distribution workloads.

ML-Based Cache Policy Design for CECAR

Expert State Representation (Input Features)

The eviction scorer in CECAR operates on a compact, expert-level state representation derived from routing history. Based on this observation, we represent the state of each expert using two scalar statistics: a recency score rter_t^e and a frequency score ftef_t^e.

The recency score represents the number of steps since the expert was last accessed, while the frequency score measures how often the expert has been routed so far. These statistics are updated as follows:

rte={1,if expert e is accessed at time t,rt−1e+1,otherwise,fte={ft−1e+1,if expert e is accessed at time t,ft−1e,otherwise.(1)r_t^e = \begin{cases} 1, & \text{if expert } e \text{ is accessed at time } t,\\ r_{t-1}^e + 1, & \text{otherwise,} \end{cases} \qquad f_t^e = \begin{cases} f_{t-1}^e + 1, & \text{if expert } e \text{ is accessed at time } t,\\ f_{t-1}^e, & \text{otherwise.} \end{cases} \tag*{(1)}

If an expert has never been accessed, its recency score is initialized to infinity. To ensure stable training and scale-invariant input, both access-recency and access-frequency statistics are normalized before being fed into the FFNs. Experts with smaller recency scores or higher frequency scores are more likely to be routed again in the next steps. The final input to the eviction scorer is constructed for each expert independently, ensuring a fixed input dimension regardless of the number of experts:

rnorme=1rte,fnorme=ftemax⁡efte,xte=[rnorme∥fnorme].(2)r_{\mathrm{norm}}^{e} = \frac{1}{r_t^{e}}, \qquad f_{\mathrm{norm}}^{e} = \frac{f_t^{e}}{\max_e f_t^{e}}, \qquad\mathbf{x}_t^{e} = [r_{\mathrm{norm}}^{e} \Vert f_{\mathrm{norm}}^{e}]. \tag*{(2)}

More complex representations (e.g., routing-pattern statistics or hidden-state-based features) do not consistently improve eviction decisions and often introduce instability across datasets and cache regimes. In contrast, recency and frequency provide a minimal yet robust signal that generalizes well across models.

However, these signals alone are insufficient to deterministically specify optimal eviction decisions: experts with similar (rte,fte)(r_t^{e}, f_t^{e}) may exhibit different future reuse depending on layer-specific and contextual factors. This motivates a learned eviction predictor that captures non-linear, context-dependent interactions beyond rule-based policies such as LRU or LFU.

Expert-Independent FFN Architecture

To eliminate expert-specific bias, we adopt an expert-independent FFN architecture. Instead of jointly predicting eviction scores for all experts, the model predicts the eviction score of each expert independently using a fixed set of expert-agnostic features. Two FFNs are employed: FFN-Hot for short-term Hot targets and FFN-Cold for long-term Cold targets, capturing fine-grained temporal patterns at different scales. Each network is shared across all experts in a layer, ensuring expert-independent predictions. This design prevents the model from encoding dataset-specific expert indices and enables robust generalization across workloads.

In contrast, prior work such as FlashMoE processes all experts jointly and relies on routing statistics collected from a single dataset (e.g., TriviaQA). As shown in Table 4, the input to the FFNs consists of normalized recency and frequency scores aggregated across all experts. Because the model observes fixed expert indices during training, it can inadvertently learn dataset-specific routing patterns. When evaluated on out-of-distribution workloads, this bias manifests as degraded cache performance, even under otherwise simple policies such as LRU or LFU. By using expert-independent inputs and the FFN-Hot / FFN-Cold design shared across experts, CECAR mitigates this issue and achieves robust generalization across diverse datasets.

HyperparameterML Cache (FlashMoE)ML Cache (CECAR)Notes
# FFNs per MoE Layer12Hot and Cold networks for CECAR
# Layers of FFN33Base depth per FFN; sweep over 2–4 in Appendix K
Input Dimension2NE2N_E22 features per expert (recency, frequency)
Hidden Size12832Base size; sweep over 8–128 in Appendix K
Output DimensionNEN_E1Eviction score per expert
ActivationReLUSiLUNon-linear activation
LossMSELossMSELossRegression target
OptimizerAdamWAdamW-
Learning Rate1e−31\mathrm{e}{-3}2e−32\mathrm{e}{-3}Constant; no warm-up or decay
Weight Decay-0.01AdamW default
Epochs-≤200\leq200Early stopping on training loss, patience 5
Dcold/CcapD_{\mathrm{cold}} / C_{\mathrm{cap}}-18 / 36See Appendix K
Training TargetsBinary Belady (top 75%)Hot & Cold targetsHot = short, Cold = long-term reuse

Table 4. Training Hyperparameters for ML-Based Cache Policies. Full training details for CECAR are given in Appendix K.

Training Data & Target Construction

Routing histories were collected from 512 input sequences sampled from the TriviaQA dataset, each containing 512 tokens; the last 20 sequences are held out and used only for the evaluation in Appendix § 19. For every expert and time step, recency and frequency scores were computed as described above, and these statistics were used to calculate the reuse distance of each expert.

From the computed reuse distances, two complementary training targets were derived: a short-term Hot Target for fine-grained prioritization among frequently reused experts, and a long-term Cold Target for coarse-grained discrimination among rarely reused experts. FFN-Hot and FFN-Cold were trained on their respective targets, capturing caching dynamics across multiple temporal scales while remaining shared across all experts.

Fine-Grained Training Targets

To capture expert reuse patterns at multiple temporal scales, we introduce two complementary training targets: a Hot Target for short-term reuse and a Cold Target for long-term reuse. Two separate networks, FFN-Hot and FFN-Cold, are trained on these targets and shared across all experts in a layer. The Hot and Cold Targets together provide a complete, multi-scale view of eviction likelihood.

Hot Target. The Hot Target focuses on experts likely to be reused in the near future. Let DeD_e denote the reuse distance of expert ee, and let DhotD_{\mathrm{hot}} be the reuse distance of the expert ranked kk-th in ascending order among all experts. This value is used as the cutoff for constructing the Hot Target. The Hot

Target ThotT_{\mathrm{hot}} is defined as:

Thot(De)={De−1Dhot−1,if De<Dhot,1,if De≥Dhot.(3)T_{\mathrm{hot}}(D_e) = \begin{cases} \frac{D_e-1}{D_{\mathrm{hot}}-1}, & \text{if } D_e < D_{\mathrm{hot}},\\ 1, & \text{if } D_e \ge D_{\mathrm{hot}}. \end{cases} \tag*{(3)}

This formulation preserves the relative ordering among frequently reused experts while saturating the target for those unlikely to be reused within the cache window.

Cold Target. The Cold Target captures long-term eviction likelihood that the Hot Target alone cannot distinguish. Let DcoldD_{\mathrm{cold}} denote a predefined long-term reuse threshold, chosen such that Dcold>DhotD_{\mathrm{cold}} > D_{\mathrm{hot}}. The Cold Target TcoldT_{\mathrm{cold}} is defined as:

Tcold(De)={0,if De<Dcold,0.5,if Dcold≤De<2Dcold,1,if De≥2Dcold.(4)T_{\mathrm{cold}}(D_e) = \begin{cases} 0, & \text{if } D_e < D_{\mathrm{cold}},\\ 0.5, & \text{if } D_{\mathrm{cold}} \le D_e < 2D_{\mathrm{cold}},\\ 1, & \text{if } D_e \ge2D_{\mathrm{cold}}. \end{cases} \tag*{(4)}
DeD_eFlashMoECECAR
Binary TargetHot Target (Dhot=6D_{\mathrm{hot}} = 6)Cold Target (Dcold=15D_{\mathrm{cold}} = 15)
100.00.0
200.20.0
300.40.0
400.60.0
500.80.0
611.00.0
1011.00.0
1511.00.5
2011.01.0

Table 5. Example Training Targets for FlashMoE and CECAR.

Complementary Nature. The Hot Target emphasizes short reuse distances (De<DhotD_e < D_{\mathrm{hot}}) while the Cold Target emphasizes long reuse distances (De≥DcoldD_e \ge D_{\mathrm{cold}}). By combining these targets, the model captures both short-term and long-term eviction likelihood, allowing fine-grained prioritization for small caches and coarse-grained discrimination for large caches.

Eviction Score Composition. The outputs of FFN-Hot and FFN-Cold are combined to produce the final eviction score:

sℓ(e)=ML_Scorer(rnorme,fnorme)=(1−ω)⋅FFNhot(rnorme,fnorme)+ω⋅FFNcold(rnorme,fnorme),(5)s_{\ell}(e) = \mathrm{ML\_Scorer}(r_{\mathrm{norm}}^e, f_{\mathrm{norm}}^e) = (1-\omega)\cdot\mathrm{FFN}_{\mathrm{hot}}(r_{\mathrm{norm}}^e, f_{\mathrm{norm}}^e) + \omega\cdot\mathrm{FFN}_{\mathrm{cold}}(r_{\mathrm{norm}}^e, f_{\mathrm{norm}}^e), \tag*{(5)}

where the weight ω∈[0,0.5]\omega\in[0, 0.5] is the contribution of the long-term (Cold) signal, and 1−ω1-\omega that of the short-term (Hot) signal. Smaller caches prioritize short-term reuse, while larger caches gradually increase the influence of long-term patterns. Formally, ω\omega is adjusted based on the cache size CC as

ω=min⁡(0.5,CCcap⋅0.5),(6)\omega= \min\left(0.5, \frac{C}{C_{\mathrm{cap}}}\cdot0.5\right), \tag*{(6)}

where CcapC_{\mathrm{cap}} is the cache size at which the Hot and Cold contributions reach a 1:1 ratio. For C→0C \to0 we have ω→0\omega\to0, so the Hot detector dominates; as CC grows toward CcapC_{\mathrm{cap}}, long-term eviction signals are incorporated progressively. The upper bound of 0.5 enforces the maximum 1:1 weighting between Hot and Cold targets, independent of actual RAM size. (In the submitted version, the roles of ω\omega and 1−ω1-\omega in Eq. (5) were printed in the opposite order; the released code implements the form given here.)

Implementation Details of Reproduced Baseline Methods

To comprehensively evaluate the efficacy of the proposed CECAR framework, we reproduced and analyzed four distinct configurations. These consist of two groups of prior baselines—MoCCE (Baseline 1) and pruning-based methods, DES and ODP (Baseline 2)—alongside CECAR-LRU (Baseline 3), an ablation variant of our method. While the first two baseline groups represent established methodologies in the field, CECAR-LRU is introduced as an ablative benchmark. It enables us to quantify the performance gains specifically attributable to our predictive eviction model by substituting it with a standard Least Recently Used (LRU) policy. All evaluations were conducted within a unified simulation environment to ensure a fair comparison across different cache management and expert pruning strategies.

MoCCE

In reproducing MoCCE, we strictly followed the architectural design proposed in the original work, while performing hyperparameter tuning tailored to our target models: Qwen3-30B-A3B and DeepSeek-V2-Lite-Chat. MoCCE introduces a scaling factor λ\lambda to modulate the degree of cache-awareness during expert routing. We performed a parameter sweep over λ∈{0.2,0.4,0.6,0.8}\lambda\in\{0.2, 0.4, 0.6, 0.8\} using the HumanEval benchmark.

Based on the results, we selected λ=0.2\lambda= 0.2, which minimized accuracy degradation. This aligns with the findings in MoCCE [Skliar et al., 2025], where excessive λ\lambda values lead to accuracy loss due to over-utilization of cached experts.

For fine-grained expert architectures such as Qwen and DeepSeek, we adopted J=2J = 2 as the default setting, following the recommendations of the original MoCCE study.

λ\lambdaQwen3-30B-A3BDeepSeek-V2-Lite-Chat
Baseline67.6847.56
0.269.5147.56
0.464.0246.95
0.662.1946.34
0.860.3646.34

Table 6. MoCCE parameter sweep on HumanEval.

DES and ODP

Unlike CECAR, which improves cache locality via routing bonuses, DES and ODP are pruning-based optimizations that reduce computation by skipping experts with negligible contributions. We reproduced these methods using the algorithmic framework of MC-MoE [9], while adhering to the experimental protocols of Ban&Pick [5], which are specifically optimized for fine-grained MoE models such as Qwen and DeepSeek.

DES. DES dynamically prunes experts based on routing weight concentration. To establish a common baseline, we first conducted an ablation study on vanilla models by sweeping fixed Top-kk expert counts from 1 to 8 without any pruning or cache-aware routing.

Top-kkQwen3-30B-A3BTop-kkDeepSeek-V2-Lite-Chat
867.68648.78
768.29548.78
665.85445.12
564.02339.63
457.93229.88
336.5916.71
222.5600.00

Table 7. Top-kk ablation on HumanEval.

From this study, we identified k=5k = 5 as the optimal trade-off point for both architectures. Accordingly, DES was reproduced with dynamic pruning range of Top-4 to Top-6 experts, ensuring an average activation level consistent with Ban&Pick.

Layer-wise pruning thresholds μ1(l)\mu_1^{(l)} and μ2(l)\mu_2^{(l)} were computed using medians of routing weight ratios:

r=wjwj+1+ϵ,ϵ=10−6.r = \frac{w_j}{w_{j+1} + \epsilon}, \qquad\epsilon= 10^{-6}.

Calibration was performed using domain-specific datasets: MMLU (Code), MathQA (Math), and ARC-Challenge (General Knowledge), enabling pruning behavior to adapt to domain-dependent expert utilization patterns.

ODP. ODP extends DES by incorporating attention-based token significance to prevent pruning of semantically critical tokens. We adopted the Token Significance Index IjI_j from MC-MoE:

Ij=∥tj∥1⋅1H∑h=1H(1L−j∑i=jL−1ai,j(h))(7)I_j = \lVert t_j \rVert_1 \cdot\frac{1}{H}\sum_{h=1}^{H}\left(\frac{1}{L-j}\sum_{i=j}^{L-1}a_{i,j}^{(h)}\right) \tag*{(7)}

By calculating this index for each layer during the prefill stage, we identify the top 2% of tokens as core protection targets. These selected tokens are mandated to activate the maximum expert capacity (Top-6), regardless of whether they satisfy the weight-based pruning condition (r≥μr \ge\mu). This mechanism effectively preserves essential semantic information and mitigates accuracy degradation caused by the loss of critical experts during the pruning process.

CECAR Top-kk Threshold Selection

CECAR dynamically modulates the number of experts involved in routing by establishing a top-kk threshold. While experts within this threshold are always activated to ensure a baseline accuracy floor, any experts exceeding the threshold are conditionally included based on their cache residency. To achieve the most aggressive pruning configuration that does not induce significant accuracy degradation, we configured the thresholds at 4 for Qwen and 3 for DeepSeek. These specific values were identified through a comprehensive parameter sweep as presented in Table 8.

Top-kkQwen3-30B-A3BTop-kkDeepSeek-V2-Lite-Chat
Baseline67.68Baseline47.56
668.90449.39
567.68350.61
470.73242.07
355.49132.32

Table 8. CECAR Top-kk threshold sweep on HumanEval.

CECAR-LRU vs. CECAR-ML

The impact of ML within the CECAR framework extends far beyond a simple increase in cache hit rates. To isolate the specific contribution of our ML model, we benchmarked it against CECAR LRU, a variant that employs a standard Least Recently Used (LRU) policy to assign cache bonuses to recently activated experts. Across four comprehensive benchmarks, CECAR ML outperformed CECAR LRU. Our comparative analysis demonstrates that the ML approach facilitates more sophisticated cache replacement and routing decisions than conventional heuristic-based methods, thereby driving substantial gains in model performance and accuracy.

Top-kkPolicyHumanEvalMBPPGPQA-DMATH-500
4LRU43.9043.4020.7122.80
ML50.6144.0017.6826.60
3LRU48.7842.4020.7126.00
ML49.3946.2019.7028.40

Table 9. Effect of cache policy on DeepSeek-V2-Lite-Chat.

Routing Behavior Analysis of CECAR

We analyze how CECAR alters routing behavior relative to the baseline router. Specifically, we examine routing stability, perturbation magnitude, and the preservation of expert specialization.

Routing Stability

We measure routing stability using the average first divergence token (FDT), defined as the generation step at which routing decisions first deviate from the baseline router.

On Qwen3-30B-A3B (averaged over HumanEval, MBPP, and MATH-500), CECAR achieves a higher FDT (25.4) than MoCCE (20.7) and ODP (9.0), indicating that it preserves the original routing behavior for a longer prefix of generation.

Routing Perturbation Magnitude

We quantify the extent of routing modifications using two metrics: (i) the global influence ratio, i.e., the fraction of routing decisions altered by cache-aware adjustments, and (ii) the normalized weight gap, which measures relative changes in routing weights.

MethodGlobal Influence RatioNormalized Weight Gap
CECAR24.30%3.12%
MoCCE29.67%4.73%

Table 10. Routing perturbation analysis.

CECAR produces fewer routing modifications and smaller weight perturbations than MoCCE, indicating that its adjustments remain localized around the original top-kk candidates.

Expert Specialization Preservation

We assess whether CECAR preserves expert specialization by measuring the relative usage of the top experts identified by the baseline router. Across all datasets, the average usage ratio remains above 1 (1.16–1.42), indicating that dominant experts are consistently retained.

Moreover, the identity of top experts varies significantly across datasets, reflecting domain-specific specialization. CECAR closely follows these patterns, suggesting that it preserves both expert diversity and domain-adaptive routing behavior.

CECAR improves performance while maintaining routing stability and preserving expert specialization. This indicates that its gains arise from controlled, localized adjustments rather than large deviations from the original routing decisions.

Hardware Configuration of CECAR

Table 11 summarizes the hardware and software configuration used for evaluating CECAR. All experiments were conducted on a consumer-grade desktop system with 64 GB of system DRAM and a 16 GB GPU. This configuration was chosen to allow efficient experimentation while retaining practical relevance to on-device deployment scenarios with limited GPU and system memory.

ComponentSpecification
CPUAMD Ryzen 5 9600X (Zen 5, 6 cores, PCIe 5.0 support)
GPUNVIDIA RTX 5070 Ti (16 GB GDDR7, PCIe 5.0)
MotherboardGigabyte X870 AORUS EAGLE WiFi7 (PCIe 5.0)
SSDSK hynix Platinum P51 NVMe SSD (PCIe 5.0, 7.4 GB/s sequential read)
RAM16 GB DDR5-6000MHz × 4 (Dual Channel, 64 GB total installed)
OSUbuntu 24.04 LTS
EnvironmentPython 3.11, PyTorch 2.7, CUDA 12.9
DRAM Usage During Inference∼ 1 GB reserved for OS and bookkeeping;
Expert StorageNon-cached experts fetched from SSD to emulate edge-device constraints

Table 11. Specifications of Desktop System used for CECAR. The system includes ample DRAM to facilitate experimentation, but during inference only a small portion is used to emulate memory-constrained edge environments.

Emulating Memory-Constrained On-Device Environments

Although the host machine provides ample DRAM (64 GB), real-world edge devices such as laptops, tablets, or mobile phones typically have far less memory available for model inference. To realistically simulate such constraints, we deliberately restrict the usage of host DRAM for expert parameters during inference. Specifically, we cap system DRAM at 1GB for operating system stability and runtime bookkeeping. The remaining DRAM is not used for storing expert weights.

Under this configuration, experts that are not currently resident in the cache are fetched directly from SSD storage. This setup emulates the slower storage tier typical in edge devices, where it is not feasible to keep all expert parameters in RAM. By explicitly controlling DRAM usage in this way, we ensure that the observed performance—such as cache hit rates, throughput, and latency—reflects on-device memory limitations rather than the full capacity of the desktop host system.

Concretely, for Qwen3-30B-A3B, each expert contains approximately 4.7M parameters, corresponding to 9.44 MB under BF16 precision. Storing all experts therefore requires approximately 56.7 GB of memory, making it infeasible to keep expert parameters in host DRAM under realistic on-device constraints. Accordingly, in our setting with a cache size of 24, expert parameters occupy approximately 10.6 GB of the 16 GB GPU VRAM, while the remaining VRAM is used to store non-expert model parameters and the KV cache. As a result, CECAR operates with only approximately 1 GB of host DRAM reserved for operating system stability and runtime bookkeeping, whereas a baseline configuration would require 64 GB of host DRAM to keep all experts resident.

In short, although the experiments run on a desktop with sufficient memory, the behavior of the system is constrained to mirror realistic edge deployment conditions. This design allows us to evaluate CECAR’s performance under practical memory limitations without requiring access to specialized hardware with limited RAM.

Measurement Methodology and Reproducibility

Two measurement paths. The numbers in Table 3 come from two programs that run the same rule (Algorithm 1). Accuracy, hit rate, and average top-kk come from the evaluation path in the released code (./evaluation --bonus_strategy cecar). It keeps every expert in VRAM on a separate large-memory GPU, tracks the cache as a set of expert identities, applies the same selection, top-KK, cache-conditioned pruning (rank ≤τ\le\tau or resident, no refill), and admit/evict steps as the offloading engine, and runs the full MoE forward pass. Whether an expert is a hit or a miss depends only on identities and cache state, not on where the weights are stored, so this path reports the same hit rate and average top-kk as the offloading engine while running fast enough to cover the full benchmarks. Decode speed is the one quantity that needs wall-clock timing of the offloading engine. It is measured on the testbed of Table 11 with host DRAM limited to about 1 GB with mlock(), on 100 prompts per benchmark. Without the lock, the OS uses free host memory as a page cache, the experts end up resident in DRAM, and no SSD read takes place. Table 1 and Table 8 use the same two paths, restricted to HumanEval.

Run-to-run variance. Expert selection in Algorithm 1 is deterministic for a given prompt and cache state, but wall-clock speed varies between runs because of token sampling and I/O timing. Table 12 gives the decode speed of each method as mean ±\pm SD over repeated runs of the 100-prompt measurement; Table 3 reports the same values.

StrategyQwen3-A3BDeepSeek-A2.4BOLMoE-A1B
Baseline2.8 ± 0.314.4 ± 4.878.5 ± 1.97
DES4.7 ± 0.975.0 ± 3.4816.2 ± 2.93
ODP5.0 ± 1.434.6 ± 2.1918.4 ± 5.69
MoCCE8.6 ± 1.807.6 ± 0.5527.7 ± 6.96
CECAR12.5 ± 1.1910.3 ± 3.5929.9 ± 4.99

Table 12. Decode speed (tokens/s), mean ± SD across repeated on-device runs (cache size 24 experts/layer, 100 prompts per benchmark).

Released code. The repository (https://github.com/hoonki2002/CECAR) contains the evaluation path, the SSD-offloading engine, the throughput harness used for the 100-prompt measurements, and the mlock() script that limits host DRAM. Its default decoding settings are those of Section 5.1 (temperature 0.7, top-pp 0.95). An earlier version of the engine refilled the pruned expert set to the full budget after the keep test; the released engine follows the no-refill rule of Algorithm 1, as the evaluation path does.

Robustness Across Hardware Interfaces

We evaluate CECAR under different interconnect configurations (PCIe 5.0 vs. PCIe 4.0) to assess hardware sensitivity. Table 13 reports expert loading latency and effective throughput.

ModelSizePCIe 5.0 Latency (ms)PCIe 4.0 Latency (ms)PCIe 5.0 TP (GB/s)PCIe 4.0 TP (GB/s)
OLMoE-1B-7B-0125-Instruct12.6M5.1376.2562.452.01
DeepSeek-V2-Lite-Chat17.3M7.1118.1872.432.11
Qwen3-30B-A3B9.4M4.4644.4242.122.13

Table 13. Expert loading latency and throughput across PCIe generations.

Across all models, the latency difference between PCIe 5.0 and PCIe 4.0 is limited (0.99×–1.22×), indicating weak sensitivity to interconnect bandwidth. This is because individual expert parameters are relatively small (e.g., ∼9.4 MB for Qwen3-30B-A3B), which prevents saturation of peak PCIe bandwidth.

Consequently, expert loading latency is dominated by per-transfer overhead rather than bandwidth, and the relative performance of CECAR remains stable across PCIe generations. This suggests that CECAR’s gains are primarily driven by routing and cache decisions, rather than hardware-specific bandwidth advantages, and thus generalize across commodity systems. This further indicates that improving expert locality is more critical than increasing interconnect bandwidth in on-device MoE inference.

Strategy Predictions on Sample Problems

For illustration purposes, we include example problems to demonstrate how different caching strategies generate outputs. Each example shows, for each strategy (Baseline, DES, ODP, MoCCE, and CECAR), how the generated tokens differ in both content and quantity. These results were obtained by running two representative Math problems through DeepSeek-V2-Lite-Chat, highlighting the differences in the generated tokens produced by each strategy.

StrategyExample 1Example 2Avg. top-kk*Hit-Rate (%)Tokens/s
BaselineFalseTrue6.0058.783.24
DESFalseFalse4.9857.083.74
ODPFalseFalse5.0856.953.79
MoCCETrueFalse6.0087.548.78
CECARTrueTrue5.4888.4710.12

Table 14. Comparison of strategy outputs on Examples.

  • Average number of routed experts per layer, excluding shared experts.

Example 1

Question: If f(x)=(3x−2)/(x−2)f(x) = (3x - 2)/(x - 2), what is the value of f(−2)+f(−1)+f(0)f(-2) + f(-1) + f(0)? Please reason step by step and put your final answer within [answer].

Baseline (False): To find the value of f(−2)+f(−1)+f(0)f(-2) + f(-1) + f(0), we need to substitute x=−2x = -2, x=−1x = -1, and x=0x = 0 into the function f(x)f(x) and then add the results. First, let’s find f(−2)f(-2): f(−2)=(3(−2)−2)/(−2−2)=(−6−2)/(−4)=−8/−4=2f(-2) = (3(-2) - 2)/(-2 - 2) = (-6 - 2)/(-4) = -8/-4 = 2. Next, let’s find f(−1)f(-1): f(−1)=(3(−1)−2)/(−1−2)=(−3−2)/(−3)=−5/−3=5/3f(-1) = (3(-1) - 2)/(-1 - 2) = (-3 - 2)/(-3) = -5/-3 = 5/3. Finally, let’s find f(0)f(0): f(0)=(3(0)−2)/(0−2)=−2/−2=1f(0) = (3(0) - 2)/(0 - 2) = -2/-2 = 1. Now, we add these results together: f(−2)+f(−1)+f(0)=2+5/3+1=6/3+5/3+1/3=12/3=[4]f(-2) + f(-1) + f(0) = 2 + 5/3 + 1 = 6/3 + 5/3 + 1/3 = 12/3 = [4]. DES (False): It seems like you’re asking for the value of a function f(x)f(x) at two specific points, and then adding those values together. However, the function f(x)f(x) is defined as a fraction, and we can’t simply substitute the values into the fraction to get the result. The function f(x)f(x) is defined as a fraction, and we can’t simply substitute the values into the fraction to get the result.

ODP (False): To solve this problem, we need to find the value of f(−2)f(-2) and f(0)f(0) and then substitute them into the equation. Step 1: Find f(−2)f(-2). We have f(x)=(3x−2)/(x−2)f(x) = (3x - 2)/(x - 2). Substituting x=−2x = -2 into the equation, we get f(−2)=(3(−2)−2)/(−2−2)=−8/−4=2f(-2) = (3(-2) - 2)/(-2 - 2) = -8/-4 = 2. Step 2: Find f(0)f(0). We have f(x)=(3x−2)/(x−2)f(x) = (3x - 2)/(x - 2). Substituting x=0x = 0 into the equation, we get f(0)=(3(0)−2)/(0−2)=−2/−2=1f(0) = (3(0) - 2)/(0 - 2) = -2/-2 = 1. Step 3: Substitute f(−2)f(-2) and f(0)f(0) into the equation. We have f(−2)+f(0)=2+1=3f(-2) + f(0) = 2 + 1 = 3. Therefore, the answer is [3].

MoCCE (True): To find the value of f(−2)+f(−1)+f(0)f(-2)+f(-1)+f(0), we need to substitute x=−2x = -2, x=−1x = -1, and x=0x = 0 into the function f(x)f(x). First, let’s find f(−2)f(-2): f(−2)=(3(−2)−2)/(−2−2)=(−6−2)/(−4)=−8/−4=2f(-2) = (3(-2) - 2)/(-2 - 2) = (-6 - 2)/(-4) = -8/-4 = 2. Next, let’s find f(−1)f(-1): f(−1)=(3(−1)−2)/(−1−2)=(−3−2)/(−3)=−5/−3=5/3f(-1) = (3(-1) - 2)/(-1 - 2) = (-3-2)/(-3) = -5/-3 = 5/3. Finally, let’s find f(0)f(0): f(0)=(3(0)−2)/(0−2)=−2/−2=1f(0) = (3(0)-2)/(0-2) = -2/-2 = 1. Now, we add these values together: f(−2)+f(−1)+f(0)=2+5/3+1=2+5/3+3/3=2+8/3=6/3+8/3=14/3f(-2) + f(-1) + f(0) = 2 + 5/3 + 1 = 2 + 5/3 + 3/3 = 2 + 8/3 = 6/3 + 8/3 = 14/3. So, the value of f(−2)+f(−1)+f(0)f(-2) + f(-1) + f(0) is [14/3].

CECAR (True): To find the value of f(−2)+f(−1)+f(0)f(-2)+f(-1)+f(0), we need to substitute x=−2x = -2, x=−1x = -1, and x=0x = 0 into the function f(x)f(x). First, let’s find f(−2)f(-2): f(−2)=(3(−2)−2)/(−2−2)=(−6−2)/(−4)=−8/−4=2f(-2) = (3(-2) - 2)/(-2 - 2) = (-6 - 2)/(-4) = -8/-4 = 2. Next, let’s find f(−1)f(-1): f(−1)=(3(−1)−2)/(−1−2)=(−3−2)/(−3)=−5/−3=5/3f(-1) = (3(-1) - 2)/(-1 - 2) = (-3-2)/(-3) = -5/-3 = 5/3. Finally, let’s find f(0)f(0): f(0)=(3(0)−2)/(0−2)=−2/−2=1f(0) = (3(0)-2)/(0-2) = -2/-2 = 1. Now, we can add these values together: f(−2)+f(−1)+f(0)=2+5/3+1=2+5/3+3/3=2+8/3=6/3+8/3=14/3f(-2) + f(-1) + f(0) = 2 + 5/3 + 1 = 2 + 5/3 + 3/3 = 2 + 8/3 = 6/3 + 8/3 = 14/3. So, the value of f(−2)+f(−1)+f(0)f(-2) + f(-1) + f(0) is [14/3].

Example 2

Question: When the positive integers with exactly three positive divisors are listed in ascending order, what is the fifth number listed? Please reason step by step and put your final answer within [answer].

Baseline (True): The positive integers with exactly three positive divisors are the squares of prime numbers. This is because the divisors of such a number are the number 1, the prime number itself, and the square of the prime number itself. The first four squares of prime numbers are 4, 9, 25, and 49. Therefore, the fifth number with exactly three positive divisors is [121].

DES (False): A positive integer with exactly three positive divisors must be prime. This is because a prime number has exactly two positive divisors: 1 and the number itself. So, we need to find the fifth prime number. The first four prime numbers are 2, 3, 5, and 7. Therefore, the fifth prime number is 11. So, the fifth number listed is [11].

ODP (False): A positive integer with exactly three positive divisors must be in the form p2p^2 or p3p^3, where pp is prime. We can list the numbers in ascending order as follows: 4=224 = 2^2, 9=329 = 3^2, 8=238 = 2^3, 27=3327 = 3^3, 16=2416 = 2^4, 81=3481 = 3^4, 32=2532 = 2^5, 243=35243 = 3^5, 64=2664 = 2^6, 729=36729 = 3^6, 128=27128 = 2^7, 2187=372187 = 3^7, 256=28256 = 2^8, 4993=384993 = 3^8, 512=29512 = 2^9, 10000=21010000 = 2^{10}, 29169=31029169 = 3^{10}, 1024=2111024 = 2^{11}, 30000=21230000 = 2^{12}, 30576=31030576 = 3^{10}, 104976=213104976 = 2^{13}, 205376=214205376 = 2^{14}, 311731=311311731 = 3^{11}, 1073744=2151073744 = 2^{15}, 2111088=2162111088 = 2^{16}, 3229225=3123229225 = 3^{12}, 10995112=21710995112 = 2^{17}, 21990240=21821990240 = 2^{18}, 33554433=31333554433 = 3^{13}, 112589994=219112589994 = 2^{19}, ...

MoCCE (False): The positive integers with exactly three positive divisors are the square numbers. This is because the divisors of a square number are 1, the number itself, and the square root of the number. Since the square root must be a positive integer, there can be at most three divisors. The first few square numbers are 12=11^2 = 1, 22=42^2 = 4, 32=93^2 = 9, 42=164^2 = 16, 52=255^2 = 25, 62=366^2 = 36, 72=497^2 = 49, 82=648^2 = 64, 92=819^2 = 81, 102=10010^2 = 100, and so on. We can see that the fifth square number is 52=255^2 = 25. Therefore, the fifth number listed is [25].

CECAR (True): The positive integers with exactly three positive divisors are the squares of prime numbers. This is because the divisors of a number are determined by its prime factorization. If a number has exactly three divisors, it must be the square of a prime number, because the square of a prime number has exactly three divisors: 1, the prime number itself, and the square of the prime number. The first four squares of prime numbers are 22=42^2 = 4, 32=93^2 = 9, 52=255^2 = 25, and 72=497^2 = 49. The fifth square of a prime number is 112=12111^2 = 121. Therefore, the fifth number listed when the positive integers with exactly three positive divisors are listed in ascending order is [121].

Comparison with MoE-Infinity

MoE-Infinity[Xue et al., 2025] is open-source and targets the same single-request, personal-machine setting as CECAR. We replayed its cache logic (ExpertTracer, find_most_similar) on routing traces under the same conditions as our experiments (Qwen3-30B-A3B, 24 experts per layer, top-kk 8) and compared the resulting hit rate with LRU and with Belady’s optimal policy computed on the original routing. Belady on the original routing is an upper bound for any system that keeps exact routing and only decides which experts to cache or prefetch.

DatasetMoE-InfinityLRUBelady (original routing)CECARnn (trace / CECAR)
MBPP0.516 ± 0.0280.642 ± 0.0210.797 ± 0.0130.950 ± 0.01950 / 100
HumanEval0.670 ± 0.0260.627 ± 0.0270.731 ± 0.0160.949 ± 0.0216† / 100
MATH-5000.598 ± 0.0420.629 ± 0.0170.718 ± 0.0130.968 ± 0.0146† / 100
GPQA-Diamond0.604 ± 0.0300.646 ± 0.0330.737 ± 0.0160.982 ± 0.0116† / 100

Table 15. Hit rate on the four evaluation benchmarks (Qwen3-30B-A3B, 24 experts/layer), mean ± std over sequences.

† The MBPP trace has 50 sequences. The HumanEval, MATH-500, and GPQA-Diamond traces have 6 sequences with 32 decode tokens each and should be read as indicative only. CECAR’s numbers come from its own 100-sample run, not from the same trace.

CECAR’s hit rate (0.95–0.98) is above the Belady bound on the original routing (0.72–0.80). A caching or prefetching system cannot pass this bound without changing the routing; CECAR passes it because it changes the routing (about 28% of selections differ from the baseline router). The two kinds of system therefore act at different points of the pipeline and can be combined: CECAR reduces the number of non-resident experts that are requested, and a prefetcher can hide the latency of the ones that remain.

We could not compare end-to-end throughput. MoE-Infinity allocates every expert in host DRAM (AllocateMemory in model_topology.cpp) and has no budget or eviction mechanism, so it needs host memory on the order of the full model, about 57 GB for Qwen3-30B-A3B. It ran out of memory when loading this model with 60 GB of RAM, in two attempts. CECAR runs with about 1 GB of host DRAM and reads from SSD on every token, so there is no host-memory budget under which both systems run.

Runtime Overhead Analysis

Table 16 breaks the per-token decode latency of Qwen3-30B-A3B (24 experts per layer) into expert loading, fixed compute, and the cost CECAR adds. The inputs are per-operation costs measured on the testbed of Table 11: 2.62 ms per expert miss, 384 expert activations per token (48 layers × 8 experts), and 42.73 ms of compute per token that does not depend on the cache policy (expert compute 18.61 ms, attention 12.13 ms, norm and gate 11.99 ms).

Operating pointMissesExpert load (ms)Fixed compute (ms)CECAR scorer (ms)Total (ms)Tokens/s
On-demand (0% hit)3841006.142.73–1048.80.95
LRU (64.4% hit, MBPP)137358.242.73–400.92.49
CECAR (96.6% hit)1334.242.735.2782.212.16

Table 16. Per-token latency at three operating points (Qwen3-30B-A3B, 24 experts/layer).

The model gives 12.16 tokens/s at CECAR’s operating point, close to the 12.5 tokens/s measured in Table 3. Expert loading is the only term that depends on the policy. Going from on-demand loading to CECAR removes 972 ms of loading per token. What CECAR adds is small: the cache-aware selection and logit adjustment cost a few microseconds per layer, the same order as the gate itself, and the eviction scorer costs 5.27 ms per token in the per-layer implementation used for the results in this paper (48 kernel launches per token, dominated by launch overhead) or 0.088 ms when the 48 layers are scored in one batched launch. In either case the added compute is under 0.6% of the loading it saves. The 2.62 ms miss cost is calibrated against the SSD; if experts were served from the OS page cache instead, the true SSD cost would be higher and the share of the speedup due to loading would be larger still.

Robustness to Cache Budget

Static Capacity Sweep

Table 2 varies the cache size but reports only hit rate on TriviaQA. On DeepSeek-V2-Lite-Chat, HumanEval accuracy under CECAR is 43.90 at C=8C=8, 45.73 at C=16C=16, 49.39 at C=24C=24, and 50.00 at C=32C=32. Accuracy falls gradually as the budget shrinks and is flat above C=24C=24.

Budgets That Change During Decoding

The memory available on a device is not constant, and ω(C)\omega(C) in Eq. (6) is defined for a changing cache size. We ran three schedules in which the per-layer budget CtC_t changes during generation, on DeepSeek-V2-Lite-Chat over all 164 HumanEval problems. When the budget shrinks, the excess experts are evicted at once in order of eviction score; ω\omega is recomputed from CtC_t at every decode step.

  • S1: 24→824 \to8 during decode steps 192–384, then back to 24.

  • S2: 32→1632 \to16 at step 256.

  • S3: alternating between 24 and 12 every 64 steps.

† Most generations end before step 256, so the realized average budget of S2 is 31.7; we compare against the nearest fixed budget.

Under all three schedules, accuracy and hit rate are within one problem and about 1 pp of a fixed budget with the same time-averaged size (Table 17). Right after a shrink, the hit rate is already at or above the steady-state hit rate of the smaller budget (Table 18), because the experts that survive the shrink are the ones with the best eviction scores. Freezing ω\omega at its C=24C=24 value throughout S1 changes the hit rate by at most 0.3 pp and the accuracy by no problems, so the adaptation comes from the scorer rather than from the ω\omega schedule.

Schedulepass@1Hit rateFixed-budget reference
S1 (24→8→2424 \to8 \to24)72/16485.4%C=23C=23 (realized average 22.8): 73/164, 86.5%
S2 (32→1632 \to16)71/16490.4%C=32C=32 (realized average 31.7†31.7^{\dagger}): 71/164, 90.7%
S3 (24↔1224 \leftrightarrow12)72/16482.1%C=18C=18 (realized average 20.6): 70/164, 81.4%

Table 17. Accuracy and hit rate under changing budgets, with a fixed budget of the same time-averaged size for reference (DeepSeek-V2-Lite-Chat, HumanEval, 164 problems).

TransitionAfter transitionSteady state at new CC
S1: 24→824 \to859.2%56.9%
S2: 32→1632 \to1681.4%78.3%
S1: 8→248 \to2480.2%87.2% (reached as the cache refills)

Table 18. Hit rate immediately after each budget change, compared with the steady-state hit rate of the new budget.

Eviction-Scorer Training and Architecture Sweep

FFNHot\mathrm{FFN}_{\mathrm{Hot}} and FFNCold\mathrm{FFN}_{\mathrm{Cold}} are per-layer MLPs trained jointly on the summed MSE loss with AdamW, one 512-token sequence per step. Traces come from 512 TriviaQA sequences of 512 tokens, the last 20 held out. Targets use Dcold=18D_{\mathrm{cold}}=18, Dcap=36D_{\mathrm{cap}}=36, and a DhotD_{\mathrm{hot}} under which the 98 farthest-reused experts of 128 receive a Hot target of 1. To test whether the scorer’s OOD robustness depends on capacity, we sweep h∈{8,16,32,64,128}h \in\{8,16,32,64,128\} and d∈{2,3,4}d \in\{2,3,4\}, each with 3 seeds, and measure hit rates on Qwen3-30B-A3B at C=24C=24 under exact routing, averaged over 13 MoE layers (0–7, 15, 23, 31, 39, 47), on the 20 held-out TriviaQA sequences (ID) and 10 prompts of 512 tokens each from KoBEST/KLUE, MBPP, and GSM8K (OOD). Because trace, layers, and prompts differ from Table 2, Table 19 reports hit rates relative to a retrained h=32,d=3h=32,d=3 scorer.

Policy# ParamsTriviaQA (ID)KoreanMBPPGSM8K
Retrained (h=32h=32, d=3d=3), 3 seeds2,3701.0001.0001.0001.000
Deployed scorer (h=32h=32, d=3d=3)2,3700.9950.9950.9940.996
Sweep range (15 configs ×\times 3 seeds)66 – 67,0740.998 – 1.0040.999 – 1.0010.999 – 1.0030.999 – 1.002
Belady (upper bound)–1.1351.0741.2021.186

Table 19. Hit rate relative to the retrained h=32h=32, d=3d=3 scorer.

Limitations

CECAR is evaluated on three representative MoE LLMs and four standard benchmarks under a controlled on-device inference setup. While the results show consistent improvements across these settings, performance may vary under substantially different model architectures, workloads, or hardware platforms. Our testbed is a memory-constrained desktop GPU with SSD offloading and host DRAM pinned to ∼1\sim1 GB rather than a phone- or tablet-class SoC; we chose it because even the smallest evaluated model (∼14\sim14 GB in BF16) exceeds phone-class memory, and because a mobile runtime would introduce confounds from the storage stack and power/thermal limits. Since CECAR’s benefit derives from the latency gap between cached and non-resident experts, we expect it to be maintained or amplified on lower-bandwidth mobile storage such as UFS, but this is not verified here. Accuracy and throughput are measured on two different paths (Appendix § 14); accuracy is reported as mean ±\pm SD over three decoding seeds and throughput as mean ±\pm SD over repeated runs, whereas multi-seed variance for the hit rates in Table 3, sensitivity to DhotD_{\mathrm{hot}}, DcoldD_{\mathrm{cold}} and CcapC_{\mathrm{cap}}, and a training-data-scale ablation remain to be reported. The architecture sweep in Appendix § 19 shows that the scorer’s out-of-distribution hit rate is unchanged across a 1000×1000 \times range of parameter counts, so its robustness is not an artifact of limited capacity; whether it is entirely due to the expert-independent input design has not been tested by ablating that design in isolation. Expert activation patterns are input-dependent, and we have not evaluated workloads with abrupt semantic shifts within a single request. Extending CECAR to batched serving, long-context generation, and quantized expert storage, and composing it with exact-routing prefetchers, remains future work.

References

  1. [1]Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Efficient large language model inference with limited memory. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12562–12584, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.678. URL https://aclanthology.org/2024.acl-long.678/.DOI
  2. [2]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732.
  3. [3]L. A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Systems Journal, 5(2):78–101, 1966. doi: 10.1147/sj.52.0078.DOI
  4. [4]Hongtao Chen, Weiyu Xie, Boxin Zhang, Jingqi Tang, Jiahao Wang, Jianwei Dong, Shaoyuan Chen, Ziwei Yuan, Chen Lin, Chengyu Qiu, Yuening Zhu, Qingliang Ou, Jiaqi Liao, Xianglin Chen, Zhiyuan Ai, Yongwei Wu, and Mingxing Zhang. Transformers: Unleashing the full potential of cpu/gpu hybrid inference for moe models. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, 2025a. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374.
  5. [5]Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, and Jian Cheng. Ban&pick: Enhancing performance and efficiency of moe-llms via smarter routing, 2025b. URL https://arxiv.org/abs/2509.06346.
  6. [6]DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecheng Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhaung Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyu Jin, Xianzhu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhijpeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zhihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. URL https://arxiv.org/abs/2405.04434.
  7. [7]Artyom Eliseev and Denis Mazur. Fast inference of mixture-of-experts language models with offloading, 2023. URL https://arxiv.org/abs/2312.17238.
  8. [8]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874.
  9. [9]Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAOJUAN QU. Mixture compressor for mixture-of-experts LLMs gains more. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=hheFYj0sWO.arxiv.org/abs/2410.06270
  10. [10]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825.
  11. [11]Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https://arxiv.org/abs/1705.03551.

Paper details

vlnc:W_89rfuzt7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents