betaFeedback

WorksComputer Science

Ripple: Deadline-Aware CPU-GPU Sharing for Embodied Inference

Abstract

Unified-memory edge devices increasingly run LLM/VLM inference alongside time-critical CPU tasks for perception and control. GPU memory traffic can overwhelm these tasks, yet platforms such as DGX Spark expose neither bandwidth counters nor direct bandwidth regulation. We present Ripple, a CPU-GPU co-scheduler that combines model-derived GPU memory volumes with CPU completion feedback to regulate interference. Ripple divides inference into pausable, layer-based units and translates each CPU job's deadline slack into a GPU-byte budget, admitting work within the budgets of all overlapping jobs. Online updates track changing interference, while deadline-aware pacing and judicious SM footprint narrowing preserve inference progress. Implemented with SGLang engine hooks and a central scheduler, we evaluate Ripple on DGX Spark with four robotic CPU task mixes and LLM/VLM inference. It is the only evaluated approach that consistently meets the 2% CPU deadline-miss target and VLM deadlines across the tested deadline settings while retaining best-effort LLM throughput.

Introduction

AI models increasingly run on edge devices ([8, 49]), with privacy protection, low/consistent latency, low cost, and device portability. Many of these devices adopt a unified memory architecture (UMA), where the CPU and GPU share one physical memory on one package, as with Apple’s M-series processors and NVIDIA’s DGX Spark. UMA is cheaper, smaller, and easier to build and maintain, eliminating host-device memory copies and providing a decent capacity (128GB on DGX Spark) to hold reasonably large models (e.g., 32B at FP16).

With the rapid growth of agentic and embodied AI, edge AI devices are increasingly versatile, often pairing model inference with CPU computation tasks for perception and control, which typically come with timing requirements. Consider a home robot working on clearing a dinner table while interacting with human users on travel planning. Every few 10s of milliseconds its CPU must work out how far away things are and where it can safely move, or the robot may walk into a person who has just stepped into its path; every few seconds a vision-language model (VLM) examines the table again; meanwhile a large language model (LLM) plans the next robot movements, retrieves family travel history/preference and calendar events using retrieval-augmented generation (RAG) (Lewis et al., [34]), and generates responses to the user. A missed obstacle-avoidance cycle can cause an accident, a late scene update stalls the next action, and a slower answer merely arrives later — all three run on one chip and share its memory bus ([37, 22, 54, 20]).

Our study finds that memory bandwidth is the contention point, in a lopsided fight, dominated by the less time-critical inference tasks (user response generation in the above scenario). Figure 1 shows this with a workload mix on the GB10 processor of DGX Spark, with 4 routine robotic/agentic CPU tasks — costmap updates, nearest-neighbor queries, stereo matching, and vector retrieval over the robot’s memory. The left chart reports the mutual slowdown when this CPU mix co-runs with a GPU-side LLM decode task, with growing CPU thread allocations (CPU-side performance saturating at 12). The more deadline-sensitive CPU tasks suffer much higher loss. The right chart provides a reference by running the same test on an H200 server, with separate HBM. The CPU tasks sees no impact of concurrent LLM inference there, as opposed to 1.7–2.5× on the Spark.

CPU–GPU interference under LLM decode

Figure 1. CPU–GPU interference under LLM decode.

Managing bandwidth on these devices is harder than on a conventional CPU, where we have homogeneous cores and a rich set of hardware counters. On edge AI devices with UMA, bandwidth management faces four major challenges: (1) the platform lacks bandwidth counters and exposes no interface that reports per-requester memory bandwidth, neither that regulates bandwidth consumption; (2) modern transformer-based AI models are bandwidth-intensive even at token granularity: on GB10 one LLM token has a 69-74ms forward pass and one VLM image a 285ms one, longer than the 45 to 70ms deadlines for recurrent CPU robotic tasks; (3) the CPU side is not designed to take most of the unified memory bandwidth, consuming at most 60% of it even with all cores; (4) the power-aware design of edge devices perform aggressive and black-box, temperature-triggered clock-throttling, further complicating performance monitoring and resource regulation.

These constraints rule out most existing (and relatively mature) multi-tenant bandwidth management solutions ([64, 2, 57, 13, 14, 53, 46]), because the interfaces they rely on are missing. The one scheduler to our knowledge that work without, published recently, is SERENO ([62]), which protects foreground mobile apps from background NPU inference. It targets a phone SoC, where the bandwidth relinquished by the NPU can be almost entirely absorbed by the CPU. Edge UMA devices are not zero-sum in this way. Also, SERENO pairs one foreground app with one background model, where our setting has many concurrent workloads on both sides, each with its own timing requirement.

This work is based on the observation that “what the CPU and GPU cannot do symmetrically, they can do complementarily.” GPU inference is opaque in time and transparent in volume: when a forward pass will finish depends on contention and thermal throttling, but its memory access volume is well known. A real-time CPU task is the exact opposite — transparent in time and opaque in volume: its release time and deadlines are set in stone, whether each deadline is met can be easily known, but access volume remains not observable without detailed source code instrumentation. Together, the two pieces of information give sufficient guidance for our scheduling, compensating for the lack of bandwidth counters (or even surpassing their capability).

To this end, we propose Ripple, a CPU-GPU co-scheduling framework that performs fine-grained GPU inference task admission, to protect CPU-side deadline-sensitive tasks while retaining overall bandwidth utilization and inference speed as allowed by the deadlines. Ripple overcomes the lack of counters by leveraging the semantic knowledge on today’s mainstream transformer-based LLM/VLM workflow to decompose the inference forward pass into smaller, layer-based work units with known memory access volumes, for GPU-side bandwidth regulation. It performs continuous modeling by observing the GPU-side memory traffic on CPU-side deadline-sensitive tasks, leading to a pricing model that releases metered volume budgets to admit GPU work units, dynamically adjusted to current clock throttling levels and CPU-side contentions.

We implement Ripple as a standalone scheduler with engine hooks in SGLang and evaluate it on DGX Spark with four mixes of robotic CPU tasks plus LLM/VLM inference. It is shown to consistently meet CPU deadline-miss targets and VLM deadlines, while retaining substantial best-effort LLM throughput across workload mixes and deadline settings.

Novelty. To the best of our knowledge, we are the first to target the bandwidth asymmetry on DGX Spark. Also, Ripple is the first system that makes independent CPU tasks’ declared deadlines and miss tolerances explicit constraints for LLM/VLM co-serving on UMA devices. It introduces the first counter-free scheduler that translates CPU deadline slacks into GPU memory-byte budgets, with the first theoretical throughput bound for best-effort inference under these CPU timing constraints.

Scope. This proof-of-concept study targets declared deadline miss tolerances and batch-one, pausable inference.

Background and Motivation

Unified Memory at the Edge

Unified Memory Architecture (UMA) ([26]) began as a power- and cost-driven solution in integrated graphics ([30, 17]), enabling a small GPU to “borrow” system memory instead of carving its own. It has since become the common design for AI-capable edge devices: Apple’s M-series processors, AMD’s Ryzen AI Max, NVIDIA’s Jetson modules as well as DGX Spark all come with a single memory, of up to 128GB, directly accessed by both the GPU and CPU sides. Table 1 summarizes these recent models and their major parameters ([38, 8, 9, 4, 44 e;i]).

DeviceMemorySize (GB)BW (GB/s)Power /GBShared w/ CPU
Edge, UMA
M4 MaxLPDDR51285464.3–
Ryzen AI Max+LPDDR5X1282562.00.9
Jetson OrinLPDDR5642053.20.9
Jetson ThorLPDDR5X1282732.11.0
DGX SparkLPDDR5X1282732.11.1
Discrete
RTX 5090GDDR73217925618
Datacenter
H100 NVLHBM3943900414.3
H200 NVLHBM3e1414800344.3
MI300AHBM31285300414.3
GB300ᵃHBM3e2888000284.9

Table 1. Memory and power of representative GPUs. “–” denotes missing vendor specifications [9, 48].

All the above devices with UMA adopt LPDDR (Low-Power Double-Data-Rate memory) ([31]). Unlike the DDR modules of desktops and servers, LPDDR is designed for battery-powered devices: it runs at lower voltages, enters deep power-saving states when idle, and is soldered next to the processor rather than socketed. It offers large capacities at low power and price, with modest bandwidth, as reflected by the bandwidth/GB column and power/GB columns. Edge GPUs therefore raise their bandwidth by widening the LPDDR bus, to 256 bits on DGX Spark, or by raising its pin rate, as Jetson Thor does over Orin ([31]) ᵃ Per GPU of GB300 NVL72. Its Grace CPU has its own LPDDR5X [11, 41].

2023; 2025; 2026. Such limited bandwidth is further shared between GPU and CPU.

This stands in stark contrast to datacenter GPUs, also shown in the table, where the more familiar HBM (High Bandwidth Memory) delivers 13 to 20× the bandwidth/GB of DGX Spark, with rated chip/module power of 4.3–4.9 W per GB of memory. Among the datacenter models listed, only the AMD MI300A shares its memory between GPU and CPU, and its bandwidth/GB is around 20× that of DGX Spark. Discrete accelerators, such as the NVIDIA RTX 5090, adopt GDDR (Graphics DDR) memory, which drives each pin several times faster at higher power and delivers a bandwidth/GB even higher than the datacenter GPUs (Table 1; NVIDIA, 2025a; Wahlgren et al., 2025; Aubrey & Stam, 2025; NVIDIA, 2026b;c; AMD, 2026b;c).

Memory Bandwidth Contention on DGX Spark

For the rest of the paper, we focus our discussion on the NVIDIA DGX Spark architecture (“GB10” for short). Released in October 2025, it has rapidly grown into a product line sold by seven major PC makers and become a mainstream edge AI device [40]. Neither the bandwidth contention problem on UMA devices (Table 1) nor our proposed approach, meanwhile, is specific to GB10.

DGX architecture GB10, the DGX Spark processor, places a CPU and a GPU die on a single package (Figure 2). The CPU has 20 cores in two clusters, each containing 5 performance (P) and 5 efficiency (E) cores. The 48-SM GPU die reaches the 128GB shared LPDDR5X DRAM through NVLink-C2C and the memory controllers on the CPU die. Our benchmarking achieves 190-257GB/s combined

Figure 2. GB10 package and memory topology

DRAM throughput out of the 273GB/s from NVIDIA specification, depending on the access pattern (Appendix § 10).

Sharing introduces contention, especially with the low overall bandwidth availability relative to capacity. GB10 provides no direct bandwidth allocation mechanism between its CPU and GPU sides: how bandwidth divides appears to depend on how many memory requests each keeps in flight. This gives the GPU side an easy advantage, where thousands of memory accesses can be issued with its massive parallelism, burying the small number of time-critical and latency-sensitive CPU requests. The above asymmetry is too strong to mitigate through high-level concurrency parameters, in other words, CPU core or GPU SM counts, as shown earlier in Figure 3.

Figure 3. (a) Slowdown brought by GPU inference (with different SM counts) on CPU time-sensitive tasks; (b) bandwidth with concurrent CPU+GPU memory streaming

Bandwidth Management Challenges In addition to such significant CPU-GPU imbalance, UMA devices like GB10 pose unique challenges for bandwidth management:

No observation or regulation Unlike server CPUs, with mature per-workload memory bandwidth monitoring/allocation mechanisms such as Intel RDT or ARM MPAM ([57, 10]), the GB10 firmware exposes no memory-controller counters and does not enable MPAM. Our tests found that NVIDIA DCGM’s DRAM-activity metric returns no reading.

Limited GPU control An MPS SM limit is set when a process starts and remains throughout its execution. Lowering the GPU clock reduces GPU application throughput while freeing little bandwidth. The recent CUDA green contexts ([47]) can confine a stream’s kernels to a subset of the SMs. Our measurement (Figure 3(a)), however, demonstrates that reducing LLM’s GPU SM count from the full 48 to 8 (minimum allowed) has very limited effect on mitigating CPU slowdown. Meanwhile, with the GB10 CPU clusters unable to saturate the memory bus, when both the GPU and CPU sides run memory streaming benchmarks, the bandwidth yielded by reducing the GPU SM count cannot be absorbed by the CPU, resulting in low overall bandwidth utilization, as shown in Figure 3(b).

Run-to-run performance variability Being power- and heat-aware, ddge devices constantly adjust CPU/GPU clock rate to temperature, further complicating bandwidth control.

Major Edge Workloads

CPU-side Representative CPU tasks on edge AI devices, in the current pursuit of AGI, are diverse and often tightly timed. Embodied systems run their periodic perception, localization, and control loops on CPU: constructing LiDAR-based occupancy grids, stereo disparity and image features, nearest-neighbour queries for scan matching, and model-predictive control. These loops run with fixed frequency, typically at 10-50Hz with deadlines of tens of milliseconds. Agentic systems retrieve from their memory before responding: in a spoken conversation that retrieval must return within the few hundred milliseconds of a turn.

Table 2 lists representative CPU tasks in embodied AI workloads. Unlike GPU tasks, they tend to be complex yet resource-light. Each of the above tasks has its algorithm and access pattern, but moves at most 1.6GB of data per iteration, about 1/10 of what an LLM moves in generating one token (e.g., 15.2GB for Qwen3-8B). A task typically tolerates a small fraction of misses, as in the weakly-hard model ([16]) of real-time systems. The tasks run with independent intervals and phases, with overlapping deadline windows.

TaskDescription
CostmapMaps nearby obstacles from LiDAR scans.
KDNNFinds nearby points to help align scans and track position.
SGMEstimates object distances from two camera images.
ControlComputes quadrotor control commands using model-predictive control (OSQP).
ORBExtracts visual features from camera images.

Table 2. Common edge robotic CPU tasks ([15, 28, 60, 55, 24, 37, 59, 51, 50]).

GPU-side GPU workloads on edge AI devices, on the other hand, are computationally intensive, with rather elastic timing requirements. Examples include LLMs for planning and answering and VLMs (Vision-Language Models) for scene understanding, reasoning, and task planning in embodied robots. With all mainstream models, a forward pass goes through a regular sequence of layers, with per-layer model weights streamed from DRAM. Edge LLM/VLM inference processing is particularly bandwidth-hungry during decode, where a small batch (mostly 1) reads GBs per token. The small batch sizes, limited model sizes, and limited context lengths in edge AI scenarios also make KV cache access much lighter than in datacenter services: our 8192-token context used with Qwen3-32B-FP8 yields KV cache memory access at around 6.5% of model weight read volume.

Between LLM and VLM, the latter requires timely updates to its understanding of the environment to support reasoning and task planning, though with an interval often much longer than CPU-side perception/control tasks. Also, they spend host time differently: a VLM decode step issues about a thousand kernels lasting 28 μ\mus on average, making CUDA graph replay indispensable to avoid a host bottleneck; the LLM kernels are fewer and last much longer, receiving negligible performance gain from CUDA graphs. Finally, running the LLM and VLM engines on GB10 side by side has little GPU SM-level interference: DRAM bandwidth is the clear contention point (Appendix § 11).

Approach Overview

Rationale In this work, we leverage several opportunities to overcome the lack of UMA bandwidth monitoring or regulating mechanisms. First, the time-critical CPU tasks do not require full dedication of CPU resources or DRAM bandwidth, so uniform GPU throttling would be an overkill. Second, the many layers of LLM/neural-network models provide a natural granularity of GPU-side task admission, with known data volume, to allow bandwidth regulation without wasting resources on aborted work. Finally, one does not need a detailed hardware counter for our purpose: the coarse observation based on the completion times of past jobs, plus the GPU-side DRAM access volume during each deadline window, provide sufficient information for task-level coordination. Note that a task’s completion time already reflects how much the job itself read and how fast the clocks ran, giving a complete CPU-side task profile.

Based on these observations, we present Ripple, a CPU-GPU task scheduler for UMA devices that protects time-critical CPU tasks while retaining the progress of bandwidth-hungry GPU inference. Most importantly, it does so without requiring a bandwidth counter/arbitrator. Instead, it focuses on informed GPU task admission, predicting the delay the task adds to every active CPU job.

Note that GPU work could also be deadline-bound, such as VLM inference for the scene updates that guide the robot’s next action. However, due to the biased memory arbitration discussed in § 2.2, only the CPU tasks need bandwidth protection: deadline-bound GPU tasks are safe as long as Ripple gives them scheduling priority.

System model and objective Following the above reasoning, we term the time-critical CPU workloads protected tasks (PTs). Each PTiPT_i releases a sequence of jobs and is associated with an inter-arrival interval TiT_i: known and fixed for periodic tasks such as costmap updates and stereo matching, and a declared minimum for sporadic tasks such as memory retrieval for a spoken reply. PTiPT_i also comes with a relative deadline Di≤TiD_i \le T_i and a tolerated deadline miss rate (DMR) εi\varepsilon_i. Every job reports its completion; Ripple neither modifies nor throttles the task.

Each GPU inference workload is either deadline-bound (with a per-request deadline), or best-effort. Ripple observes only the release/completion time of a CPU job and the DRAM access volume and duration of each GPU job. It learns about the CPU tasks online from their completions and requires little offline profiling. The more predictable GPU inference tasks are profiled once per device and model, recording each unit’s completion time, each engine’s overhead per grant, and the peak byte rate ρ\rho.

In priority order, its objectives are (1) that in every window of nin_i consecutive jobs of a protected task, at most a fraction εi\varepsilon_i miss DiD_i, except with probability at most δ\delta (a chance constraint [21]), where PTi,jPT_{i,j} misses when its completion time CTi,jCT_{i,j}, measured from its release Ri,jR_{i,j}, exceeds DiD_i; (2) that every deadline-bound GPU workload meets its deadline; and (3) that best-effort GPU work makes as much progress as possible. Ripple acts only on the GPU work it schedules: when each piece is dispatched and on how many SMs. Non-time-critical CPU tasks or the division of service among several best-effort workloads (CPU or GPU) is beyond the scope of this work.

Main idea Ripple leverages the common software architecture for GPU model inference today for scheduling GPU work at an appropriate granularity. It cuts each engine’s forward pass into units, the units at which execution can pause with state information in memory, and authorizes a contiguous run of units, called a grant, at a time.

Each CPU job receives an allowance Ai,jA_{i,j}, the total GPU memory access volume (“volume” for short) assigned from its predicted deadline slack. Its budget Bi,j(t)B_{i,j}(t) is the volume available for additional GPU work at time tt: the allowance released by then, less the volume already charged. When an engine finishes a grant, it reports its execution time. The scheduler then takes the global budget B(t)B(t), the minimum across active uncertified CPU jobs, selects the longest grant that fits it, and charges its volume to every active CPU job. When a CPU job completes, the scheduler updates that task’s model of its perceived interference from GPU volumes. When the same PTPT’s next job is released, Ripple adjusts that job’s allowance from the updated model.

Software architecture Ripple comprises three major modules, as illustrated in Figure 4. It adds engine hooks in SGLang (Zheng et al., [65]) that divide forward passes at unit boundaries, pause engines between grants while preserving state, apply SM allocations through CUDA green contexts, and report each grant’s CUDA time. Its scheduler runs as a separate process occupying one performance core; it maintains per-job byte accounts, orders GPU work, and admits grants within the CPU jobs’ budgets. It controls the SGLang engines (also running on CPU cores), one per inference service, via the Ripple hooks. The Ripple estimator runs within the scheduler, updating each CPU task’s model on job completion (dashed feedback edges) and computing allowances for subsequent jobs.

Figure 4. Ripple’s modules and their interactions, with solid lines indicating control flow while dashed ones feedback

Ripple Design

Software Architecture and Scheduling Workflow

Ripple manages UMA bandwidth through GPU-side task admission, with no interference with the scheduling of CPU-side tasks. It deploys a central scheduler, along with its scheduling-relevant components through SGLang modification. Such a Ripple-customized SGLang scheduler performs bandwidth-aware LLM/VLM task decomposition/regrouping at layer boundaries, as well as the GPU task monitoring/reporting duties required by Ripple. The Ripple scheduler, which makes all admission decisions, occupies one GB10 performance core by default, due to its heavy processing and time-sensitive task nature; the SGLang processes of the inference engines share a separate set of cores, disjoint from those of the PTs.

The chief goal of the Ripple scheduler is to maximize overall DRAM bandwidth utilization (and consequently the GPU inference throughput), while keeping each time-sensitive task’s deadline misses within its tolerance: PTs on CPU and deadline-bound GPU tasks. It makes admission decisions at scheduling points triggered by the following events: (1) a CPU PT job starts or completes; (2) a GPU grant completes; (3) an engine announces new work; or (4) the budget accrues enough to schedule the next unit.

On the CPU side, as mentioned earlier, Ripple does not interfere with CPU tasks’ scheduling or execution, but observes the slowdown they experience from GPU memory bandwidth usage (major) and other CPU tasks or environment variances such as thermal throttling (minor). Ripple builds a per-PT interference model through linear regression and updates it constantly by new data points collected upon PT job completion. Based on the model, it assigns each PT job a total volume allowance Ai,jA_{i,j} for its lifetime. At time tt, PTi,jPT_{i,j}’s budget Bi,j(t)B_{i,j}(t) is the released part of Ai,jA_{i,j} minus the GPU volume already charged to the job. § 5 details the slowdown modeling and allowance/budget calculation.

On the GPU side, Ripple breaks an LLM/VLM forward pass into units at the natural boundaries of modern transformer-based models: transformer layers, the output head, and the attention/MLP blocks of a vision encoder. At each scheduling point, if possible, Ripple admits the next grant, the longest stretch of one engine’s units, that fits the global budget B(t)B(t), the minimum over active uncertified PT jobs. Deadline-bound and best-effort units are ordered by the Ripple scheduler by their own deadlines: a deadline-bound request’s work gets due times spread evenly over a fraction γ\gamma of the interval to its deadline, and a best-effort grant runs before the next due time only if it ends by then and leaves enough budget for the request’s next grant.

As mentioned in § 2.3, of the two engines, only the VLM gains from CUDA-graph replay, so Ripple keeps SGLang’s breakable graphs and pauses between their segments. The LLM, in contrast, runs eagerly and can leave and re-enter its forward pass at any unit boundary. A grant that an engine has started runs to its end, since work submitted to the GPU cannot be recalled.

Pricing GPU Work Against CPU Deadlines

Ripple models how GPU volume delays PTiPT_i’s jjth job:

CTi,j=ai+κiCi,j+ei,j,(1)CT_{i,j} = a_i + \kappa_i C_{i,j} + e_{i,j}, \tag*{(1)}

where aia_i estimates the job’s interference-free duration, κi\kappa_i its delay by GPU volume, and ei,je_{i,j} the residual execution time not explained by the model. Upon the job’s completion, Ripple refits aia_i and κi\kappa_i by least squares with exponential forgetting [36], following changes in measured completion times. Using a conservative CPU duration estimate aˉi,j\bar{a}_{i,j} and slope estimate κˉi,j\bar{\kappa}_{i,j}, PTi,jPT_{i,j} has the nominal allowance

Ai,j=min⁡(max⁡(0,Di−aˉi,j)κˉi,j,Aimax⁡),(2)A_{i,j} = \min\left(\frac{\max(0, D_i-\bar{a}_{i,j})}{\bar{\kappa}_{i,j}}, A_i^{\max}\right), \tag*{(2)}

converting the remaining deadline slack into allowed GPU volume. Here aˉi,j\bar{a}_{i,j} reserves time for the job’s own execution, CPU, wider inter-task interference (very minor), thermal throttling, and run-to-run variances.1 Aimax⁡=ρDiA_i^{\max}=\rho D_i caps tasks with κ\kappa near zero (insensitive to bandwidth contention), where ρ\rho is the peak GPU memory access rate.

Ai,jA_{i,j} is the total GPU volume allowed by job PTi,jPT_{i,j}, released linearly over its predicted interference-free duration a^i,j\hat{a}_{i,j}. At time tt within this interval following PTi,jPT_{i,j}’s release time Ri,jR_{i,j}, the job has a remaining budget

Bi,j(t)=Ai,j(t−Ri,j)a^i,j−Ci,j(t),(3)B_{i,j}(t) = \frac{A_{i,j}(t-R_{i,j})}{\hat{a}_{i,j}} - C_{i,j}(t), \tag*{(3)}

where Ci,j(t)C_{i,j}(t) is the volume already charged to the job.

At a scheduling point (time tt), Ripple identifies the next grant, the maximum sequence of units whose combined volume fits under the global budget

B(t)=min⁡Bi,j(t), for all active PTi,j(4)B(t) = \min B_{i,j}(t),\ \text{for all active } PT_{i,j} \tag*{(4)}

so that the volume is allowed by the most sensitive PT. Once issued, the GPU grant is charged in full to every PT it overlaps with, as it simultaneously delay all of them.

Figure 5(a) illustrates the allowance/budget calculation with 3 PTs and no active GPU workloads. The purple solid line marks the global budget B(t)B(t), which follows the lowest active per-PT budget line (dashed lines). Figure 5(b) simplifies with only one PT, but adding two LLM/VLM inference tasks, with grants admitted subject to B(t)B(t) (the illustrated LLM and VLM grants require 0.45 GB and 1 GB, respectively). When PT1PT_1 completes, Ripple is able to release more LLM units in a single grant. Different height of the grant boxes indicate SM narrowing applied only to LLM decode.

Ripple’s control loop illustrated with sample budget calculation and grant issuing

Figure 5. Ripple’s control loop illustrated with sample budget calculation and grant issuing

Dynamic SM Regulation and Overhead Mitigation

Dynamic SM allocation Ripple is built with the recognition that common high-level concurrency tuning knobs, such as through the CPU core and GPU SM allocations, do not effectively protect CPU PTs, as explained in § 2.2. However, within the framework of Ripple’s model-based pricing and fine-grained GPU job admission, we re-examined the potential of dynamic SM allocation, through the relatively recent NVIDIA green context mechanism.

We observe that the effect of reducing the SM footprint (referred to as SM narrowing in our discussion) depends on the GPU workload. Intuitively, it slows compute-heavy operators substantially. A bandwidth-bound decode job, on the other hand, can retain much of its speed when narrowed to fewer SMs. Interestingly, it can maintain most of its throughput while reducing CPU delay per GB of its GPU traffic, in other words, a flatter interference slope κi\kappa_i. This is because that fewer SMs would reduce the intensity of concurrent GPU memory accesses, allowing more breathing room for CPU tasks. A lower interference slope κi\kappa_i can increase the allowance and let the budget Bi,j(t)B_{i,j}(t) accrue faster, allowing earlier GPU admission.

Ripple therefore runs prefill at full width and retains full-width CUDA graphs for the VLM, including its compute-heavy vision-encoder units. For continuous LLM decode jobs overlapping active CPU jobs, Ripple uses a fixed SM width (typically 8 or 16), based on its quick profiling during warm-up. CPU-free decode uses all SMs. Figure 8(a) in the next Section gives related evaluation results.

Breakdown and adaptivity on the Robotic1, LLMS, VLML mix

Figure 8. Breakdown and adaptivity on the (Robotic1Robotic_1, LLMSLLM_S, VLMLVLM_L) mix

This allows Ripple to expand its solution to a 2-D space of admission/scheduling and SM footprint management. While a more comprehensive investigation is planned for our future work, these initial results reveal that under careful task decomposition and admission, mechanisms such as green context could deliver a smoothing effect on GPU bandwidth consumption under edge CPU–GPU co-running, not previously observed at inference engines’ native scheduling granularity.

Grant pipelining To hide the scheduler round trip, Ripple authorizes and delivers one successor grant while the current grant runs. At the next grant boundary, the engine checks the cached grant’s work identity, lease and CPU-release limit, and discards it if stale. A successor delivered ahead for a CPU release waits until that release before it can start.

Evaluation

We implemented Ripple’s three modules (engine hooks, scheduler, and estimator) in Python, with approximately 10.5k, 3.7k, and 2.2k LoC, respectively.

Our evaluation examines four properties: (1) Ripple’s capability to meet the CPU and VLM targets across representative CPU and GPU workload combinations (§ 6.2); (2) its capability to retain GPU-side inference throughput while meeting given DMR limits (§ 6.3); (3) its overhead and adaptation to changes (§ 6.3); and (4) the impact of individual design/optimization (§ 6.3).

Experimental Setup

Testbed We evaluate on an ASUS Ascent GX10, an NVIDIA DGX Spark system. Its GB10 SoC has a 20-core Arm CPU (10 Cortex-X925 and 10 Cortex-A725 cores) and a Blackwell GPU with 48 SMs, which share 128 GB of LPDDR5X at 273 GB/s. The machine runs Ubuntu 24.04.3 LTS with Linux 6.14, NVIDIA driver 580.95.05 and CUDA 13.0, and SGLang 0.5.12.

Workloads and configurations Our evaluation runs three robotic perception/control tasks on the CPU (from those listed in Table 2), with an LLM and a VLM inference engine on the GPU. We create two CPU workload mixes, Robotic1_1 (Costmap, KDNN, and SGM), and Robotic2_2 (Control, ORB, and SGM). For LLM and VLM, we test two model configurations each: LLMLLLM_L with Qwen3-32B-FP8, LLMSLLM_S with Qwen3-8B, VLMLVLM_L with Qwen3-VL-30B-A3B-FP8, and VLMSVLM_S with Qwen3-VL-8B ([63, 12]). We evaluate four workload mixes to examine how workload composition affects protection and throughput.

CPU PTs use 100 ms periods, following KITTI’s 10 Hz sensor rate ([24]), except Control, which runs every 20 ms, a 50 Hz cadence used in embedded quadrotor MPC ([56]). We set the deadlines of Costmap, KDNN, and SGM to kk times their co-run p99, measured with the other CPU tasks running and both GPU engines resident but idle, and sweep kk from 1.1 to 2.0 to cover different amounts of timing slack. For detailed experiments, we select k=1.5k = 1.5, giving these tasks 50% slack over their co-run p99; results at other settings are labeled separately. Control and ORB use deadlines equal to their periods, requiring completion within the control cycle or before the next frame; the Robotic2_2 sweep therefore varies only SGM’s deadline. In our experiments, the VLM period and deadline are both 5 s for VLMLVLM_L and 10 s for VLMSVLM_S; the LLM runs continuously without a deadline.

We use a DMR limit of 2% for each CPU task in the scored window and require every VLM frame to meet its deadline. We report mean throughput with one-standard-deviation error bars (often invisible).

Baselines As the first framework designed for real-time CPU tasks that share a unified memory with GPU inference, we have a hard time identifying baseline systems. Though there are no direct competitors, we do a best-effort comparison with several existing systems or alternative strategies: (1) Sereno ([62]), designed for mobile SoCs that run background model inference beside foreground apps, pauses the LLM between decode steps to yield time shares to UI tasks (we run its “Balanced” and “UI-First” policies); (2) SiGAMMA ([18]) uses a high-priority blocking kernel to suppress GPU memory traffic while a PT job is active; (3) Pause pauses both engines while any PT job is active, using SGLang’s own pause command, which takes effect after the forward pass in flight; (4) Cap-xx uses the same command and gives each PT job a budget fraction xx of its solo median time to tolerate the engines’ co-run (x=1/8,1/4,1/2x = 1/8, 1/4, 1/2), pausing the latter whenever a active PT job has spent its budget;

Sereno and SiGAMMA run using their published configuration, untuned, on SGLang 0.5.12 with the same engine flags used by Ripple. Only Ripple react to the deadline. For reference, we also have the unmanaged strategy, where all GPU-side engine jobs are admitted at once while the PT workloads run unprotected.

Effectiveness in CPU PT Protection

Figure 6 (a) gives Ripple’s overall performance, with 4 CPU-GPU workload mixes, at different deadline scaling factors (kk along the xx axis). Markers are filled only when all CPU tasks meet the 2% DMR limit and every VLM frame meets its deadline, otherwise hollow.

Figure 6. End-to-end results. (a) Worst-task DMR of CPU PTs and LLM tokens/s; hollow markers indicate a CPU DMR limit or VLM deadline was missed. (b) Response times with deadlines at k=1.5k = 1.5 on the (Robotic1,LLMS,VLML)(\mathrm{Robotic}_1, \mathrm{LLM}_S, \mathrm{VLM}_L) mix; each inset magnifies where the curves cross the 98th percentile.

The first row plots the worst deadline miss rate (DMR) across all three CPU PTs (yy axis in log scale). Ripple is the only solution meeting the DMR requirement consistently. A distant second is SiGAMMA, getting below the 2% target with the most relaxed 2-3 deadline settings, due to its partial suppression of inference through time-slicing with its blocking kernel. All the other baselines not only miss the target in all or nearly all tests, but produce close-to-100% DMRs, with the worst CPU task missing most of its deadlines even at k=1.5k = 1.5. This indicates existing approaches’ failure in regulating the GPU-side aggressive memory bandwidth usage. In particular, the Sereno strategies fail to respond to the remaining slack of individual CPU jobs, since their controller observes the LLM’s slowdown rather than CPU deadlines. Even the pause strategy, which is in theory the harshest on the GPU workloads, is unable to contain the inference engines’ memory traffic due to the pause request taking effect only after the in-flight forward pass completes.

The second row plots the GPU-side throughput, by LLM token generation speed. Here we add a theoretical throughput bound (diamonds), computed by replacing each GPU work unit with a fixed-rate memory-access stream with the same volume. Therefore it also removes Ripple overheads associated with discrete grants, VLM deadlines, and the scheduling protocol. As expected, Ripple’s result trails behind this theoretical throughput bound, due to the conservative nature of its budget-based admission, its safety margins, as well as Ripple’s overhead. Meanwhile, it clearly outperforms any solutions that meet the DMR requirement (SiGAMMA, which delivers lower LLM throughput due to its 48-SM “blocking kernel” wasting GPU resources even with relatively safe CPU deadlines). With more relaxed deadlines (k=1.8k=1.8 or higher), it delivers higher LLM throughput than a few approaches that yields much higher DMR, as it admits more GPU work as deadline slack increases, whereas these baselines retain deadline-independent throttling.

Among the four CPU-GPU workload combinations, the LLML+VLMLLLM_L + VLM_L GPU combination is most aggressive in bandwidth use, rendering most baselines miss the DMR target significantly, even with the most forgiving 2×2\times deadline setting. Robotic2Robotic_2, meanwhile, is slightly less vulnerable than Robotic1Robotic_1, due to its lower bandwidth demand (Control and ORB has a per-job data volume of 10s of MBs, while Costmap and KDNN are at 100MB-GB level.) Overall, the performance behavior of Ripple and the baselines are quite consistent across the board.

Figure 6(b) shows PT response-time CDFs at k=1.5k=1.5 for Robotic1Robotic_1, LLMSLLM_S, and VLMLVLM_L; the vertical dashed lines mark deadlines. Under Ripple, the 98th-percentile response times of Costmap, KDNN, and SGM fall 8.6, 3.4, and 4.9 ms below their respective deadlines, significantly outperforming the baselines. The shapes of the CDF curves here also clearly reflect the vulnerability of each CPU PT, with SGM most sensitive to GPU bandwidth occupation.

Figure 7 examines Ripple’s capability to work under varied DMR targets, with ε\varepsilon varied between 0.5 and 5%, under 1.5×1.5\times and 2×2\times deadlines, respectively. For reference, we provide the LLM throughput of three baselines, SiGAMMA, Pause (SGLang), and Unmanaged, who deliver flat lines as they are oblivious to deadlines. Ripple, on the other hand, is able to relax its GPU work admission and produces higher inference performance while staying within the DMR tolerance. Consistent with Figure 6(a), Ripple allows better LLM throughput in general, compared to baselines that meet the DMR target. The dip of Ripple throughput from 4% to 5% DMR target with 1.5×1.5\times deadline shows its sensitivity to misses, which cost bandwidth utilization with the admission decision adjusted more conservative.

Figure 7. Sensitivity to ε\varepsilon on the (Robotic1Robotic_1, LLMSLLM_S, VLMLVLM_L) mix. Labels show worst-task CPU DMR.

Breakdown Analysis

Impact of individual techniques Figure 8(a) gives the DMR and LLM throughput results by incrementally adding our proposed techniques: “pricing” (base-variation with pricing only), “grant” (issuing contiguous units), and “dynamic SM” (SM narrowing with green context) on LLM decode jobs). For comparison, we have two versions of the “Pause” strategy, one native from SGLang (pausing entire forward passes), the other at our refined unit level.

The SGLang native Pause significantly exceeds the 2% DMR limit (84.8%), although it stops the GPU whenever a CPU job runs. When pausing at unit level, it becomes extra-conservative. With each of its optimizations, the green Ripple bars safely stay under the 2% DMR limit to protect the CPU PTs, while allowing more efficient bandwidth utilization by the GPU tasks, finally reaching 90.4% of the SGLang Pause’s throughput.

Adaptivity With its pricing model observing the CPU PTs’ performance, Ripple is able to adapt to variability due to CPU-side contention or thermal throttling. The latter factor is beyond our control while the former can be tested. Figure 8(b) evaluates Ripple by injecting two sporadic CPU tasks (memory readers) during a 200s run. The top plot shows how the KDNN’s allowance automatically lowers during the two perturbation intervals and recovers thereafter, thanks to the aˉi,j\bar{a}_{i,j} term in our pricing model. The bottom plot shows the DMR readings (binned at 3s), comparing Ripple with a variation that has its adaptation component disabled. (dashed line). The difference is dramatic: KDNN’s DMR averages at 0.6% and 1.0% in the two intervals, respectively, while the non-adaptive version reaches 60.4% and 67.0%.

Overhead We then examine the management and scheduling overhead of Ripple, by measuring the percentage of time during the GPU workloads’ active execution that Ripple has no grant running. Our detailed profiling reveals the breakdown of such “gap time”, plotted in Figure 9. With a more stringent deadline (1.5×1.5 \times), the gap is dominated by “budget wait”, the time Ripple waits for sufficient budget to release the next unit, a price it has to pay to regulate GPU bandwidth usage and protect CPU PTs. This item becomes much smaller with a 2×2 \times deadline, as Ripple schedules units more aggressively. The other categories, such as Ripple decision wait and delivery times, are its overhead. They also shrink when the deadline is relaxed, as there is a higher chance more units are issued within each grant, reducing this cost. Together, such gap times contribute to the inference throughput loss against the “unmanaged” reference in Figure 6. At 2×2 \times deadline, only 5% of GPU time is not running inference.

Figure 9. Breakdown of inter-grant gap time as percentages of scored time on the (Robotic1Robotic_1, LLMSLLMS, VLMLV LML) mix.

Other results After a 3 s GPU copy, allowances recovered within 20 s for KDNN and SGM and 24 s for Costmap in both runs. Without a CPU-task profile, Ripple misses no more deadlines in its first 10 s than when started with one. The same configuration meets the CPU DMR limits and VLM deadlines across all four workload mixes without per-workload tuning. In two one-hour runs at k=1.5k = 1.5, every 100 s window met CPU DMR limits, with no VLM deadline misses.

Related Work

Real-time memory regulation Memory regulators use budgets or coordinated phases. MemGuard [64] budgets per-core memory accesses, and RT-Gang [3] throttles best-effort cores while a real-time gang runs. BWLOCK++ [2] throttles CPU traffic to protect GPU kernels; SiGAMMA [18] blocks GPU accesses during PREM [52] CPU memory phases. These require counters and throttling or phase separation. Ripple learns from CPU completions, admitting overlapping inference within per-job allowances without bandwidth counters or CPU memory-phase annotations.

Inference preemption and co-serving ConServe [53] preempts offline inference at layer boundaries, while Valve [35] bounds preemption latency and rate to protect GPU requests. Sereno [62] yields background NPU inference according to draft-subgraph latency to protect foreground applications. Ripple admits work at fine boundaries against each independent CPU job’s declared deadline; inference latency alone does not reveal that job’s slack.

GPU spatial partitioning libsmctrl [13] and nvtaskset [14] provide compute partitioning; CUDA green contexts [47] restrict a stream’s kernels to selected SMs. Duet-Serve (Gao et al., 2025) adaptively divides SMs between prefill and decode to meet GPU latency targets. Ripple combines per-grant SM control through green contexts with temporal admission to protect CPU deadlines. Narrowing can lower CPU interference per GPU byte and increase the allowance; throughput improves only when enough additional admitted work offsets slower execution.

Prediction and feedback Clockwork [27] predicts inference duration, SOLA [29] updates latency estimates using measured-to-predicted ratios, and FC-EDF [58] adjusts service levels and admission from deadline-miss feedback. Ripple combines duration prediction with a model of CPU delay per GPU byte and adaptive conformal inference [25] for its CPU prediction margin (Appendix § 12); together they turn deadline slack into a GPU-byte allowance.

Conclusion and Future Work

Ripple demonstrates that with UMA edge AI devices, the CPU can accommodate diverse time-critical robotic/agentic workloads while contemporary LLM/VLM inference runs on the GPU. We protect the less intensive, but highly sensitive CPU-side memory accesses by leveraging semantic knowledge as well as available resource management mechanisms on the GPU workloads to overcome the lack of bandwidth counters on these edge devices with unified memory.

Two questions merit further study. First, narrowing the SM footprint can reduce CPU interference per GPU byte, but the throughput benefit depends on the workload and deadline slack. Jointly choosing the SM count and admission timing could exploit this trade-off more fully. Second, long prefill operations call for finer execution boundaries; the challenge is to shorten each uninterrupted interval without excessive weight rereads or scheduling overhead.

AI Usage Disclosure

We used OpenAI Codex and Anthropic Claude iteratively under author direction to explore design alternatives, discover and summarize literature, propose experimental contrasts, implement and debug runtime and experiment-harness code, develop data-reduction and figure-generation scripts, and draft, revise, and format manuscript text. This assistance included executing author-authorized commands and helping interpret retained experiment outputs. Reported hardware measurements are drawn from experiment records rather than AI-generated numerical observations. The authors retain responsibility for the design decisions, experimental methodology, references, analysis, and all content of the paper.

Hardware and Interference Measurements

Shared-memory contention. The selected co-run measurements give about 257 GB/s for sequential CPU reads beside Qwen3-8B decode and 190 GB/s for read–write copying, against the nominal 273 GB/s. Table 3 summarizes how the rates are computed. GPU decode traffic is calculated from its output rate and per-token byte count; CPU traffic is calculated by the benchmark from its processed data and elapsed time. In separate measurements, Nsight Compute L2 read misses agree with checkpoint-based decode-byte counts within 0.7 % for Qwen3-8B and Qwen3-32B in BF16 and AWQ. Beside Qwen3-8B decode, a CPU thread that reads 30 GB/s alone obtains 6.6 GB/s, while the combined rate remains about 50 GB/s below the peak. Thus the peak is not a fixed pool either processor can fully reclaim when the other withdraws. In a separate experiment, an eight-thread CPU reader shares memory with a GPU reader reduced by duty cycling, SM count, or arithmetic intensity. The comparison uses three matched GPU rates. The CPU recovers 0.401, 0.572, and 0.570 GB/s per GB/s withdrawn, respectively.

QuantityDefinition
16.4 GB/tokenQwen3-8B weight bytes per batch-1 decode token used in the co-run calculation
GPU rateDecode tokens/s × 16.4 GB/token
CPU rateBenchmark bytes processed divided by elapsed time
Combined rateSum of GPU and CPU rates
273 GB/sNominal GB10 memory bandwidth

Table 3. Inputs and calculations for the shared-memory bandwidth measurements.

DRAM rather than cache interference. A CPU reader sustains 150–151 GB/s beside GPU data fitting in the GPU’s 24 MB L2, but only 85–86 GB/s when the GPU data spill to DRAM. Thrashing the 16 MB system-level cache does not shift this threshold. A dependent CPU load chain’s mean access latency rises from 126.1 to 283.7 ns under a GPU stream. Its last-level read miss ratio changes by only +0.5%, and memory accesses per instruction remain unchanged. A dependent integer chain without memory traffic retains its rate. The dependent-load experiment thus identifies longer memory service time rather than additional cache misses or a slower CPU.

Available interfaces. Table 4 describes the tested firmware and software. The administrator-level inventory finds CPU-core, statistical-profiling, and SMMU counters, but no uncore or memory-controller PMU: firmware provides neither an APMT table for those units nor an MPAM table, although the kernel has both drivers.

MeansEffect on CPUShortcoming
Uncore / memory-controller PMU–Not exposed by the firmware
DCGM and Nsight DRAM metrics–None for this GPU
CPU core PMU, GPU L2 counters–One side’s cache traffic only
MPAM, MIG, OS regulation–Not enabled, or blind to the GPU
GPU clock 2.4→1.1 GHzp50 −7%Too small; locking halves decode
CPU clock–Limit accepted, not applied
SMs 48→24p50 −9%, tokens −14%Static, deadline-blind
CPU cores, engine prioritiesCPU side / GPU queue onlyCannot move the GPU’s share

Table 4. Memory and GPU control interfaces on the evaluated GB10 stack.

CPU traffic and narrowing. With the GPU idle, core bus counters give about 1.1 GB per Costmap job, 370 MB for SGM, 250 MB for KDNN, 11 MB for ORB, and 2 MB for Control. The two smallest include cache hits and are upper bounds. With Costmap, KDNN, and SGM together, average CPU traffic is 18.7 GB/s and the 1 ms p99 is 64–65 GB/s; their p99 completion times rise 2–7% over running alone. In the synthetic GPU-stream experiment, narrowing FP8 work to 16 of 48 SMs reduces its byte rate by 8% and CPU delay per GB by 9–35%. BF16 still saturates memory at 16 SMs, with harm per byte unchanged within run-to-run spread.

GPU Volume Accounting and Execution

Operation granularity. Table 5 compares the duration of complete forwards with individual operations: Qwen3-8B decode takes 69–74 ms per forward, while an operation spans 1.68–8.40 ms. Table 6 identifies the operation boundaries used by the two engines: Ripple can admit one or more operations in a grant, with eager execution for the LLM and breakable CUDA graphs for VLM decode.

Model / phaseWhole forward (ms)Single operation (ms)
Qwen3-8B Prefill76.611.83–7.37
Qwen3-8B Decode69.00–74.041.68–8.40
Qwen3-VL Image extend285.230.05–4.72
Qwen3-VL Decode19.48–21.100.02–4.69

Table 5. Diagnostic CUDA-event spans on GB10, including host enqueue gaps. Ranges show observed minima and maxima.

Engine and phaseOperationsBoundary groupsExecution mode
Qwen3-8B Prefill38Input preparation; 36 decoder layers; output headEager
Qwen3-8B Decode3736 decoder layers (preparation included with the first); output headEager
Qwen3-VL Image/prompt108Preparation; vision input, attention/MLP regions, and merger; language input, 48 layers, norm, and outputEager
Qwen3-VL Decode52Preparation; language input, 48 layers, norm, and outputBreakable CUDA graphs

Table 6. Selected single-request SGLang paths for Qwen3-8B and Qwen3-VL-30B-A3B-Instruct-FP8. Counts are registered operations per forward pass; one grant can admit several.

Byte charges. A decode step’s weight and KV traffic grows with model size and context length; Figure 10 converts the counted bytes at a 16,384-token context to step time using a 220 GB/s streaming rate. A decoder layer at sequence length TT is charged Wℓ+Aℓ(T+1)KW_{\ell} + A_{\ell}(T + 1)K: weights, activation traffic, and KV reads for TT positions plus one write. Counts come from checkpoint tensors and the engine’s sequence length; the output head, final norm, and embedding row are charged separately. Qwen3-8B has Wℓ=0.386W_{\ell} = 0.386 GB and K=4096K = 4096 bytes, so each thousand context tokens adds about 1% to step bytes. A short-context step reads 15.19 GB in 69–74 ms, or 205–220 GB/s. MoE decode charges the router and selected experts; image prefill charges all experts.

Figure 10. Time one batch-1 decode step takes at a 16,384-token context when its bytes stream at 220 GB/s. Appendix § 11 gives the method.

One-metric-per-run Nsight Compute checks match computed decode bytes to L2 read misses within 0.7% for Qwen3-8B/32B in BF16 and AWQ; a known-size stream agrees within 0.02%. An initial VLM check corrected its per-step charge by 17%. Subsequent Qwen3-32B-FP8 and Qwen3-VL-30B-A3B/8B checks agree within 0.5% for decode, 0.4% for routed/all-expert MoE charges, and 1% for a frame. These checks are offline; admission reads no counter. Prefill can reread weights across GEMM tiles: Qwen3-8B reads 1.6 times its weights at 1024 input tokens and 18.9 times at 16,384, versus 1.009 for the evaluation’s 37-token prompt.

Execution boundaries. LLM execution is eager; VLM decode uses SGLang’s breakable CUDA graphs, with eager image processing and prefill. Figure 11 compares graph replay with eager execution in each engine alone. Replay speeds VLM decode by 30–38% and a frame by 16–21% in engine-only tests, versus 0–3.4% for the tested dense LLMs. Segmenting a graph without admission callbacks increases a fixed frame sequence’s duration by 0.4%. VLM decode issues roughly 1000 kernels averaging 28 μ\mus; tested LLM kernels average 166 μ\mus. A CUDA event marks grant completion, so reports measure execution rather than submission. Stream dependencies preserve serial order when switching SM widths. Separate engine processes time-slice on the tested GB10 stack; restricting one does not reserve SMs for the other. In a single-process BF16 matrix-kernel microbenchmark, concurrent decode-like streams took 0.9–1.6% longer than serial execution; a decode-like stream paired with a prefill-like stream took 5.4–5.7% longer.

Figure 11. CUDA-graph ablation, engine alone on GB10. (a) Throughput change of graph replay over eager execution, one contrast per bar; the hatched bar is 48 graph segments per decode step against one. (b) Host-dispatch idle per VLM decode token.

Pricing and Estimator Details

Each task’s model uses recursive least squares with exponential forgetting: effective windows of W=50W = 50 jobs for the intercept and Wκ=200W_{\kappa} = 200 for the slope. The held slope changes only when the new estimate is positive and at least two standard errors above zero. The adaptive margin uses the last Wq=200W_q = 200 residuals, starts after 50 samples, and uses one adaptive level across VLM phases. Before then, solo profiles seed the fit and margin; the warm-up margin uses the larger of the phase and pooled quantiles over up to 50 residuals. The phase split separates releases in the first 10% of a VLM period from the rest.

The conservative CPU duration is a^i,j=a^i,j+qi,j\hat{a}_{i,j} = \hat{a}_{i,j} + q_{i,j}: the fitted interference-free duration plus a residual margin.

Margin. The margin is a rank among the ni,j≤Wqn_{i,j} \le W_q retained residuals whose level αi,j\alpha_{i,j} adapts after every job,

qi,j=e^(m),m=⌈(1−αi,j)(ni,j+1)⌉,αi,j+1=αi,j+η(ϵi′−err⁡i,j),(5)q_{i,j} = \hat{e}_{(m)}, \qquad m = \left\lceil(1-\alpha_{i,j})(n_{i,j}+1) \right\rceil, \\ \alpha_{i,j+1} = \alpha_{i,j} + \eta\left(\epsilon'_i-\operatorname{err}_{i,j}\right), \tag*{(5)}

with η=0.01\eta= 0.01, where err⁡i,j=1\operatorname{err}_{i,j} = 1 if the job exceeds its saved release-time pricing line a^i,j+qi,j+si,jCi,j\hat{a}_{i,j} + q_{i,j} + s_{i,j}C_{i,j}; an unfinished job exceeds any finite line. Each residual e^i,j=CTi,j−a^i,j−κ^i,jCi,j\hat{e}_{i,j} = CT_{i,j} - \hat{a}_{i,j} - \hat{\kappa}_{i,j}C_{i,j} is computed before its job enters the fit. This adaptive conformal inference update [25, 7] widens the margin by lowering the level when a job runs longer than its saved prediction plus margin, and otherwise raises it gradually. The feedback error and the deadline miss CTi,j>DiCT_{i,j} > D_i are scored separately; rank endpoints are specified in Appendix § 12.

Window target. To leave room for variation within the contract’s window (§ 3), Ripple sets ϵi′=max⁡(10−6,ϵi−zϵϵi(1−ϵi)/ni)\epsilon'_i = \max(10^{-6}, \epsilon_i - z_{\epsilon}\sqrt{\epsilon_i(1-\epsilon_i)/n_i}) with zϵ=Φ−1(1−ϵi)z_{\epsilon} = \Phi^{-1}(1-\epsilon_i) and ni=max⁡(1,⌊L/Ti⌋)n_i = \max(1,\lfloor L/T_i\rfloor) for L=100L = 100 s. The subtraction uses a normal approximation to an independent binomial count [19] as the reference buffer; actual window misses are evaluated from job outcomes. With ϵi=δ=2%\epsilon_i = \delta= 2\%, ϵi′\epsilon'_i is 1.09% for the 100 ms tasks (ni=1,000n_i = 1{,}000) and 1.59% for the 20 ms Control task in Robotic2_2 (ni=5,000n_i = 5{,}000).

Pessimistic slope. Pricing reserves slack for residual variation and slope uncertainty. The margin covers the scatter of jobs around the fit of Eq. (1) at the GPU volumes they received, but not the error of the slope itself, which grows with the volume. The uncertainty estimate is κi,ju=max⁡(0,κ^i,j)+zεSE⁡i,j\kappa^{u}_{i,j} = \max(0,\hat{\kappa}_{i,j}) + z_{\varepsilon}\operatorname{SE}_{i,j} with zε=Φ−1(1−εi)z_{\varepsilon} = \Phi^{-1}(1-\varepsilon_i), using the current least-squares slope. Pricing uses si,j=max⁡(κ^i,j,κi,ju)s_{i,j} = \max(\hat{\kappa}_{i,j},\kappa^{u}_{i,j}), where κ^i,j\hat{\kappa}_{i,j} is the held slope, or si,j=κ^i,j/0.85s_{i,j} = \hat{\kappa}_{i,j}/0.85 until a positive uncertainty estimate is available. Equivalently, βi,j=κ^i,j/si,j\beta_{i,j} = \hat{\kappa}_{i,j}/s_{i,j} is the share of residual slack used for GPU volume. The cap Aimax⁡=ρDiA_i^{\max} = \rho D_i uses the profiled peak rate ρ\rho: the larger of the highest per-unit byte rate and the rate measured with both engines decoding concurrently.

Issuing the allowance. The nominal allowance is rate-limited to a base that changes by at most ±25%\pm25\% per release, using one unit as the growth reference below that size, and remains capped at Aimax⁡A_i^{\max}. Downward rate limiting can retain bytes above the new nominal allowance. While slope uncertainty exceeds 10%10\% or is not yet estimable, one job in eight of the task with the smallest allowance receives 40%40\% fewer bytes. The one-unit probes below also affect the issued Ai,jA_{i,j}. A best-effort grant before a deadline-bound request’s due time must finish 1 ms before it.

Pacing. Releasing a job’s allowance linearly over a^i,j\hat{a}_{i,j} (§ 5) is what budget-pacing regulators do (Aghilinasab et al. [1]). With floor ff, the budget is

Bi,j(t)=min⁡(Ai,j,max⁡(f,Ai,j(t−Ri,j)a^i,j))−Ci,j(t),(6)B_{i,j}(t) = \min\left(A_{i,j}, \max\left(f, \frac{A_{i,j}(t-R_{i,j})}{\hat{a}_{i,j}}\right)\right) - C_{i,j}(t), \tag*{(6)}

with a^i,j\hat{a}_{i,j} floored at 1 ns. An allowance below one unit receives a one-unit probe on alternate releases, subject to the guard below. The probe size is the same for every task: the largest, over the bound engines, of each engine’s smallest nonzero decode unit, so that every engine can issue one unit of its steady-state work under it (385,978,880 bytes, one Qwen3-8B layer, on (Robotic1,LLMS,VLML)(\mathit{Robotic}_1,\mathit{LLM}_S,\mathit{VLM}_L)); a job issued exactly Aimax⁡A_i^{\max}, because its raw allowance exceeds it, has its whole allowance released at entry. Inherited grants remain charged separately, as described below.

Charging. Bytes are charged at admission and never refunded once a grant has executed, so splitting work into smaller grants does not relax a job’s limit; a grant returned unexecuted is refunded. With no CPU job active, no budget limits GPU work.

The crossing bound. The crossing check of § 5 models the response time of a job PTi,jPT_{i,j} released at Ri,jR_{i,j} under a grant with predicted end tet^{e} and cc bytes as (te−Ri,j)+max⁡(a^i,j+qi,pi)+κ^i,jmax⁡(0,Aˉi−c)(t^{e}-R_{i,j})+\max(\hat{a}_{i,j}+q_i,p_i)+\hat{\kappa}_{i,j}\max(0,\bar{A}_i-c); after the grant the job can receive at most Aˉi−c\bar{A}_i-c further bytes, and max⁡(⋅,pi)\max(\cdot,p_i), with pip_i the task’s solo p99, keeps a low intercept from shrinking its busy time. Aˉi\bar{A}_i bounds the allowance the job can be issued whenever completes before Ri,jR_{i,j}; it includes the 25%25\% growth rule and the one-unit probe, both capped at Aimax⁡A_i^{\max}. A job’s charge can still exceed Aimax⁡A_i^{\max} by the grants in flight that it inherits at its release. Capping the crossing at TiT_i keeps a grant from reaching the task’s next release.

Rank endpoints and the tight-deadline guard. If the rank exceeds the retained sample count, the margin update uses qi,j=Diq_{i,j}=D_i to represent an unbounded margin; a saved qi,j≥Diq_{i,j}\ge D_i records erri,j=0err_{i,j}=0. At αi,j≥1\alpha_{i,j}\ge1 it uses the smallest residual. The updates use integer parts per million. Jobs assigned this margin follow the same rate limit and probe rules, and their deadline outcomes are scored normally. At the tightest deadlines Ripple adds one rule: if the predicted delay from GPU volume ff exceeds a job’s remaining slack, that is, ⌈κ^i,jf⌉>Di−max⁡(a^i,j+qi,j,pi)\lceil\hat{\kappa}_{i,j}f\rceil>D_i-\max(\hat{a}_{i,j}+q_{i,j},p_i), evaluated with the estimates that price it, then that job gets neither the probe nor the floor ff nor a crossing into its release.

Task exemptions. The model-based certification check requires κi,juρ<1\kappa^{u}_{i,j}\rho<1 and

hi,j+2κi,juρH1−κi,juρ≤min⁡(Di,a^i,j+qi,j+βi,j[Di−a^i,j−qi,j]+)\frac{h_{i,j}+2\kappa^{u}_{i,j}\rho H}{1-\kappa^{u}_{i,j}\rho}\le\min\left(D_i,\hat{a}_{i,j}+q_{i,j}+\beta_{i,j}\left[D_i-\hat{a}_{i,j}-q_{i,j}\right]_+\right)

where hi,j=a^i,j+qi,j+[κ^i,j−κi,ju]+Cˉih_{i,j}=\hat{a}_{i,j}+q_{i,j}+\left[\hat{\kappa}_{i,j}-\kappa^{u}_{i,j}\right]_+\bar{C}_i reanchors the intercept at its fit’s weighted mean charge Cˉi\bar{C}_i. After 50 consecutive passes the task is exempt. A failed check or an observed response above the saved line hi,j+κi,juCh_{i,j}+\kappa^{u}_{i,j}C ends exemption; a deadline miss while exempt, or a new lower slope estimate κ^i,jLS−2SEi,j\hat{\kappa}^{\mathrm{LS}}_{i,j}-2\mathrm{SE}_{i,j} above the previous κi,ju\kappa^{u}_{i,j}, disables exemption for the rest of the run.

Supplementary Evaluation

Protocol and baselines. The Ripple grid uses 140 s preparation; the external baselines use 30 s. Both use a 100 s score. A point is feasible only if every run meets every CPU task’s 2% DMR limit and every VLM deadline; all Ripple grid runs meet these criteria. Each workload mix covers seven deadline settings. The controller uses the same parameter settings across mixes, with the SM width chosen automatically during warm-up. Table 7 reports LLM throughput and the highest pooled CPU-task DMR for every deadline and workload. For example, at k=1.5k=1.5 on (Robotic1,LLMS,VLML)(\mathit{Robotic1},\mathit{LLM}_{S},\mathit{VLM}_{L}), Ripple produces 7.00 tokens/s at 0.78% DMR, while SiGAMMA produces 6.16 tokens/s at 21.50% DMR.

Table 7. End-to-end results by workload and deadline. Each cell gives LLM throughput (tokens/s) / CPU DMR (%). DMR is the highest task miss rate after pooling jobs across runs. Cap-xx selects the highest-throughput level meeting the CPU and VLM targets, or the lowest-DMR level if none meets them.

MixScheduler1.1×\times1.2×\times1.3×\times1.4×\times1.5×\times1.8×\times2.0×\times
Robotic1_1 LLMS_S VLML_LRipple4.06/0.824.59/0.855.45/0.556.10/0.457.00/0.788.08/0.108.44/0.08
Robotic1_1 LLMS_S VLML_LSereno UI-First7.25/85.507.25/80.477.25/76.157.25/71.677.25/66.957.25/26.607.25/4.57
Robotic1_1 LLMS_S VLML_LSereno Balanced9.53/100.009.53/100.009.53/100.009.53/100.009.53/99.859.53/63.959.53/24.78
Robotic1_1 LLMS_S VLML_LSiGAMMA6.16/99.986.16/99.806.16/98.836.16/62.926.16/21.506.16/0.186.16/0.02
Robotic1_1 LLMS_S VLML_LPause8.03/98.928.03/98.658.03/91.258.03/86.338.03/84.208.03/28.938.03/1.08
Robotic1_1 LLMS_S VLML_LCap-xx8.21/99.988.37/99.888.21/99.558.21/93.428.21/84.058.21/19.838.21/1.28
Robotic1_1 LLMS_S VLML_LUnmanaged9.55/100.009.55/100.009.55/100.009.55/100.009.55/99.729.55/48.779.55/17.27
Robotic1_1 LLML_L VLML_LRipple1.68/1.021.91/0.782.27/0.582.52/0.402.76/0.353.34/0.203.56/0.25
Robotic1_1 LLML_L VLML_LSereno UI-First3.27/83.503.27/80.753.27/78.283.27/75.753.27/72.503.27/36.023.27/13.15
Robotic1_1 LLML_L VLML_LSereno Balanced4.23/100.004.23/99.884.23/99.674.23/99.484.23/98.084.23/46.654.23/15.30
Robotic1_1 LLML_L VLML_LSiGAMMA2.71/100.002.71/99.502.71/98.052.71/53.682.71/12.352.71/0.122.71/0.00
Robotic1_1 LLML_L VLML_LPause3.83/98.853.83/98.103.83/95.733.83/90.923.83/84.703.83/47.753.83/12.60
Robotic1_1 LLML_L VLML_LCap-xx3.98/99.783.98/99.623.98/99.304.01/96.303.98/89.104.01/40.304.01/9.50
Robotic1_1 LLML_L VLML_LUnmanaged4.21/100.004.21/99.954.21/99.704.21/99.524.21/98.104.21/62.004.21/32.34
Robotic1_1 LLMS_S VLMS_SRipple3.33/0.883.79/0.854.64/0.525.22/0.585.81/0.506.92/0.157.38/0.02
Robotic1_1 LLMS_S VLMS_SSereno UI-First7.08/89.857.08/85.457.08/80.907.08/77.387.08/73.127.08/33.277.08/6.90
Robotic1_1 LLMS_S VLMS_SSereno Balanced8.72/100.008.72/100.008.72/100.008.72/100.008.72/99.788.72/55.458.72/14.92
Robotic1_1 LLMS_S VLMS_SSiGAMMA5.50/99.985.50/99.755.50/99.125.50/70.555.50/35.925.50/0.085.50/0.02
Robotic1_1 LLMS_S VLMS_SPause7.21/99.907.21/99.907.21/99.807.21/99.707.21/98.657.21/28.307.21/3.93
Robotic1_1 LLMS_S VLMS_SCap-xx7.31/99.907.31/99.787.31/98.787.31/93.587.31/88.387.37/23.657.41/2.22
Robotic1_1 LLMS_S VLMS_SUnmanaged8.76/100.008.76/100.008.76/100.008.76/100.008.76/99.728.76/50.408.76/18.55
Robotic2_2 LLMS_S VLML_LRipple5.11/0.856.33/1.056.86/0.507.80/0.858.08/0.458.83/0.429.44/0.20
Robotic2_2 LLMS_S VLML_LSereno UI-First8.14/86.658.14/79.728.14/71.658.14/62.878.14/50.808.14/12.208.14/0.65
Robotic2_2 LLMS_S VLML_LSereno Balanced10.55/99.4210.55/98.9210.55/96.7810.55/88.6010.55/69.3210.55/21.1510.55/1.15
Robotic2_2 LLMS_S VLML_LSiGAMMA6.55/96.436.55/83.706.55/45.936.55/12.356.55/0.826.55/0.006.55/0.00
Robotic2_2 LLMS_S VLML_LPause9.06/98.589.06/96.239.06/94.059.06/89.889.06/79.889.06/10.189.06/0.00
Robotic2_2 LLMS_S VLML_LCap-xx9.21/99.808.74/98.808.74/94.458.74/93.938.74/66.888.74/4.609.21/0.00
Robotic2_2 LLMS_S VLML_LUnmanaged10.50/100.0010.50/99.8810.50/98.6510.50/91.2510.50/78.1010.50/27.1510.50/2.40

Table 7.

Each 10 Hz task contributes 1000 jobs per score, and Control contributes 5000. Costmap, KDNN, and SGM use cores 15–18, 11–14, and 7–9; Control and ORB use 15 and 16. Both engines use cores 0–6. The LLM repeatedly serves a 37-token prompt with 51 output tokens. VLM periods/deadlines are 5 s for VLML\mathit{VLM}_{L} and 10 s for VLMS\mathit{VLM}_{S}. CPU deadlines use the co-run p99 profile associated with each repetition, with both engines resident but idle. Published baselines use the same engine flags as Ripple, including eager LLM execution. Pause and Cap act at stock forward-pass boundaries; Ripple uses finer hooks and reacts to the declared CPU deadline. The VLM due-time fraction is γ=0.95\gamma=0.95.

Baseline configuration. For Sereno, we tune PI gains separately for each workload and host using its Ziegler–Nichols procedure: proportional-only gain runs are (1,2,4,…,64)×106(1,2,4,\ldots,64)\times10^{6} in parts-per-million units; the ultimate gain and period come from sustained oscillation or the relay fallback, and Kp=0.45KuK_p=0.45K_u, Ki=0.54Ku/TuK_i=0.54K_u/T_u. UI-First uses a contention target of 0.07; Balanced uses 0.15 and a 12-token/s guardrail. The contention score compares streamed LLM decode-step intervals with their solo median; the controller pauses only the LLM between steps, without speculative decoding. For SiGAMMA, a separate process launches a highest-priority spin kernel with 512 threads per block and enough blocks to fill the thread slots on all 48 SMs. Each CPU job waits for the blocker to become resident before executing; the blocker remains active while any CPU job is active. For Cap-xx, each job’s GPU exposure budget is xx times its task’s solo median duration, with x∈{1/8,1/4,1/2}x\in\{1/8,1/4,1/2\}. Exposure accumulates while the job is active and the engines are commanded to run; when any active job exhausts its budget, the controller pauses both engines at their next forward-pass boundary.

For the CPU-load comparison, both arms use 8 decode SMs, 48 prefill SMs, 140 s preparation, and a 200 s score. Both restore the same saved initial state and update for 30 s; the control then freezes CPU adaptation state. Two 128 MiB readers on cores 4 and 10 run during 30–80 s and 130–180 s, with 500 KDNN offers per interval. Each session learns separately before the control freezes its adaptation state.

The following experiments use (Robotic1,LLMS,VLML)(\mathit{Robotic1},\mathit{LLM}_{S},\mathit{VLM}_{L}) at k=1.5k=1.5 and 140 s preparation unless specified otherwise.

Sensitivity. Table 8 retains the three settings discussed in the body. Each changes one parameter from H=18H=18 ms, a 20 ms lease, and W=50W=50. SM selection runs during preparation, so results include its selection. Every run meets CPU and VLM targets. Mean throughput changes by at most 1.7%; the changes are relative to the current Ripple control.

Table 8. Sensitivity: mean tokens/s per setting.

SettingMean tokens/sChange
Default7.130—
H=12H=12 ms7.1300.0%
Lease 26 ms7.010−1.7%
W=100W=1007.070−0.8%

Table 8.

Transient recovery. A 3 s GPU copy reaches 187.8–194.4 GB/s. Costmap’s allowance first returns within 10% of its pre-copy value after 21.5–23.9 s, KDNN after 8.5–19.3 s, and SGM after 14.9–15.2 s. Sustained settling is observed for Costmap in a disturbed run within the post-copy interval. Pooled CPU misses over the 100 s scores are 64/73/73 out of 2000 per task (Costmap/KDNN/SGM), and VLM misses are 4/40. The disturbed runs fail the CPU and VLM requirements.

Cold start. Table 9 compares the first-entry times and initial misses for starts with and without a CPU profile. Profiled and no-CPU-prior starts disable certification and its dependent drift handling in both arms, retaining other calibration. All starts have zero CPU misses in the first 10 s (100 At k=1.5k=1.5 for Robotic1_1 / LLMS_S / VLML_L, Ripple delivers 7.00 tokens/s at 0.78% DMR; SiGAMMA delivers 6.16 tokens/s at 21.50% DMR. The same table shows how throughput and miss rates change across workloads and deadlines.

Table 9. Cold start at k=1.5k=1.5: first entry within 10% of the final allowance and CPU deadline misses in the first 10 s, for Costmap / KDNN / SGM.

StartFirst entry (s)Misses, 10 s
Profiled57.4 / 97.3 / 45.60
No profile8.4 / 60.5 / 63.60

Table 9.

jobs per task per run); all 100 s scores meet CPU and VLM targets. In a no-prior start, the full pre-score interval has 41/1470 KDNN and 36/1470 SGM misses, including three dropped SGM jobs, both above 2%. Profiled/no-prior mean first-entry times are 57.4/8.4 s for Costmap, 97.3/60.5 s for KDNN, and 45.6/63.6 s for SGM.

One-hour serving. The experiment uses a 3600 s score after 140 s preparation, selecting 8 decode SMs. Table 10 reports the whole-hour outcomes and the worst per-task DMR over consecutive 100 s windows. All 216 task-windows meet 2%, and all 1440 VLM frames meet their deadlines.

RunWhole-hour DMR (%)Worst window (%)
10.281 / 0.625 / 0.7081.1
20.253 / 0.719 / 0.5721.1

Table 10. One-hour serving at k=1.5k = 1.5: CPU deadline miss rates for Costmap / KDNN / SGM and worst task DMR across consecutive 100 s windows.

Both starts have zero misses in the first 10 s. In a no-profile start, the full pre-score interval has 41/1470 KDNN misses and 36/1470 SGM misses.

No throttle flag is active in the 7195 GPU samples.

References

  1. [1]Aghilinasab, H., Ali, W., Yun, H., and Pellizzoni, R. Dynamic memory bandwidth allocation for real-time GPU-based SoC platforms. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(11):3348–3360, 2020. doi: 10.1109/TCAD.2020.3012210.DOI
  2. [2]Ali, W. and Yun, H. Protecting real-time GPU kernels on integrated CPU-GPU SoC platforms. In 30th Euromicro Conference on Real-Time Systems (ECRTS 2018), volume 106 of LIPIcs, pp. 19:1–19:22, 2018. doi: 10.4230/LIPIcs.ECRTS.2018.19.DOI
  3. [3]Ali, W. and Yun, H. RT-Gang: Real-time gang scheduling framework for safety-critical systems. In IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 143–155, 2019. doi: 10.1109/RTAS.2019.00020.DOI
  4. [4]AMD. AMD Ryzen AI Max+ 395. Official product specifications, 2026a. URL https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html.
  5. [7]Angelopoulos, A. N. and Bates, S. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023. doi: 10.1561/2200000101.DOI
  6. [8]Apple. Apple unleashes M1. Apple Newsroom, November 2020. URL https://www.apple.com/newsroom/2020/11/apple-unleashes-m1/. Accessed September 23, 2026.
  7. [9]Apple. Apple introduces M4 Pro and M4 Max. Apple Newsroom, October 2024. URL https://www.apple.com/newsroom/2024/10/apple-introduces-m4-pro-and-m4-max/.
  8. [10]Arm Limited. Arm Architecture Reference Manual Supplement: Memory System Resource Partitioning and Monitoring (MPAM), for A-profile Architecture. Arm Limited, 2022.
  9. [11]Aubrey, K. and Stam, N. Inside NVIDIA Blackwell Ultra: The chip powering the AI factory era. NVIDIA Technical Blog, August 2025. URL https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/. Published August 22, 2025. Accessed October 23, 2026.
  10. [12]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Meng, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, J., Tu, J., Wan, J., Wang, P., Wang, P., Wang, Q., Wang, Q., Xie, T., Xu, Y., Xu, H., Xu, J., Yang, Z., Yang, M., Yang, J., Yang, A., Yu, B., Zhang, F., Zhang, H., Zhang, X., Zheng, B., Zhong, H., Zhou, J., Zhou, F., Zhou, J., Zhu, Y., and Zhu, K. Owen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. URL https://arxiv.org/abs/2511.21631.
  11. [13]Bakita, J. and Anderson, J. H. Hardware compute partitioning on NVIDIA GPUs. In Proceedings of the 29th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 54–66. IEEE, 2023. URL https://www.cs.unc.edu/~jbakita/rtas23.pdf.DOI
  12. [14]Bakita, J. and Anderson, J. H. Hardware compute partitioning on NVIDIA GPUs for composable systems. In Proceedings of the 37th Euromicro Conference on Real-Time Systems (ECRTS), volume 335 of LIPIcs, pp. 21:1–21:25. Schloss Dagstuhl — Leibniz-Zentrum für Informatik, 2025. doi: 10.4230/LIPIcs.ECRTS.2025.21. URL https://www.cs.unc.edu/~jbakita/ecrts25.pdf.DOI
  13. [15]Bentley, J. L. Multidimensional binary search trees used for associative searching. Communications of the ACM, 18 (9):509–517, 1975. doi: 10.1145/361002.361007.DOI
  14. [16]Bernat, G., Burns, A., and Lamosí, A. Weakly hard real-time systems. IEEE Transactions on Computers, 50 (4):308–321, 2001. doi: 10.1109/12.919277.DOI
  15. [17]Branover, A., Foley, D., and Steinman, M. AMD Fusion APU: Llano. IEEE Micro, 32(2):28–37, 2012. doi: 10.1109/MM.2012.2.DOI
  16. [18]Capodici, N., Cavicchioli, R., Valente, P., and Bertogna, M. SiGAMMA: Server based integrated GPU arbitration mechanism for memory accesses. In Proceedings of the 25th International Conference on Real-Time Networks and Systems, pp. 48–57. Association for Computing Machinery, 2017. doi: 10.1145/3139258.3139270. URL https://hipert.unimore.it/people/marko/papers/RNST17.pdf.
  17. [19]Casella, G. and Berger, R. L. Statistical Inference. Duxbury, 2nd edition, 2002.
  18. [20]Chang, M., Chhablani, G., Clegg, A., Cote, M. D., Desai, R., Hlavac, M., Karashchuk, V., Krantz, J., Mottaghi, R., Parashar, P., Patki, S., Prasad, I., Puig, X., Rai, A., Ramrakhya, R., Tran, D., Truong, J., Turner, J. M., Undersander, E., and Yang, T.-Y. PARTNR: A benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2411.00081
  19. [21]Charnes, A. and Cooper, W. W. Chance-constrained programming. Management Science, 6(1):73–79, 1959.DOI
  20. [22]Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Thompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 8469–8488. PMLR, 2023. URL https://proceedings.mlr.press/v202/driess23a.html.arxiv.org/abs/2303.03378
  21. [24]Geiger, A., Lenz, P., and Urtasun, R. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  22. [25]Gibbs, I. and Candès, E. Adaptive conformal inference under distribution shift. In Advances in Neural Information Processing Systems 34 (NeurIPS), 2021.arxiv.org/abs/2106.00170
  23. [26]Glaskowsky, P. N. Intel innovates with integrated graphics. Microprocessor Report, 13(6), May 1999. URL https://www.cecs.uci.edu/~papers/mpr/MPR/19990510/130604.pdf.
  24. [27]Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, J. Serving DNNs like Clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 443–462. USENIX Association, 2020. URL https://www.usenix.org/conference/osdi20/presentation/gujarati.arxiv.org/abs/2006.02464
  25. [28]Hirschmüller, H. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2): 328–341, 2008. doi: 10.1109/TPAMI.2007.1166.DOI
  26. [29]Hong, K., Li, X., Chen, L., Mao, Q., Dai, G., Ning, X., Yan, S., Liang, Y., and Wang, Y. SOLA: Optimizing SLO attainment for large language model serving with state-aware scheduling. In Proceedings of Machine Learning and Systems (MLSys), volume 7, 2025.
  27. [30]Intel. Intel 810 and 815 chipset family dynamic video memory technology. Technical Specification Revision 3.0, Intel Corporation, March 2002. URL https://www.intel.com/content/www/us/en/content-details/840352/intel-810-and-815-chipset-family-dynamic-video-memory-technology-product-specifications.html.
  28. [31]JEDEC. Low power double data rate (LPDDR) 5/5x. Standard JESD209-5C, JEDEC Solid State Technology Association, July 2023. URL https://www.jedec.org/standards-documents/docs/jesd209-5c.
  29. [34]Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pp. 9459–9474, 2020.arxiv.org/abs/2005.11401
  30. [35]Liu, F., Liu, H., Lyu, X., Ai, S., Liang, H., Chen, L., Hu, Z., Zha, C., Jin, X., Luo, H., and Chen, P. Valve: Production online-offline inference colocation with jointly-bounded preemption latency and rate, 2026. URL https://arxiv.org/abs/2604.07874.
  31. [36]Ljung, L. and Söderström, T. Theory and Practice of Recursive Identification. MIT Press, Cambridge, MA, 1983.
  32. [37]Macenski, S., Martín, F., White, R., and Ginés Clavero, J. The marathon 2: A navigation system. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020. doi: 10.1109/IROS45743.2020.9341207. URL https://arxiv.org/abs/2003.00368.
  33. [38]Mittal, S. and Vetter, J. S. A survey of CPU-GPU heterogeneous computing techniques. ACM Computing Surveys, 47(4):1–35, 2015. doi: 10.1145/2788396. URL https://doi.org/10.1145/2788396.
  34. [40]NVIDIA. NVIDIA DGX Spark arrives for world’s AI developers. NVIDIA Newsroom, October 2025b. URL https://nvidianews.nvidia.com/news/nvidia-dgx-spark-arrives-for-worlds-ai-developers.
  35. [41]NVIDIA. NVIDIA GB300 NVL72. Official product page. Specifications list CPU memory as LPDDR5X and GPU memory as HBM3E, reported as rack totals (17 TB LPDDR5X; 20 TB HBM3E across 72 GPUs), 2026a. URL https://www.nvidia.com/en-us/data-center/gb300-nvl72/.
  36. [44]NVIDIA. Jetson AGX Orin for next-gen robotics. Official product page: 64 GB / 32 GB 256-bit LPDDR5, 204.8 GB/s, 2026d. URL https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/. Accessed October 23, 2026.
  37. [46]NVIDIA. NVIDIA Multi-Instance GPU user guide. Official documentation, 2026f. URL https://docs.nvidia.com/datacenter/tesla/mig-user-guide/. Accessed September 21, 2026.
  38. [47]NVIDIA. CUDA driver API: Green contexts. Official CUDA documentation, 2026g. URL https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GREEN__CONTEXTS.html.
  39. [48]NVIDIA. Hardware overview: NVIDIA DGX Spark user guide. Official documentation, 2026h. URL https://docs.nvidia.com/dgx/dgx-spark/hardware.html.
  40. [49]NVIDIA. System overview: NVIDIA DGX Spark porting guide. Official hardware documentation, 2026i. URL https://docs.nvidia.com/dgx/dgx-spark-porting-guide/latest/overview.html. Last updated October 21, 2026. Accessed October 23, 2026.
  41. [50]Open Navigation LLC. Nav2 Controller Server: Configuration Parameters. Nav2 documentation, 2026. URL https://docs.nav2.org/rolling/configuration_and_development/configuration_guide/core_servers/controller_server/. Accessed September 15, 2026.
  42. [51]Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), pp. 1–22, 2023. doi: 10.1145/3586183.3606763.
  43. [52]Pellizzoni, R., Betti, E., Bak, S., Yao, G., Criswell, J., Caccamo, M., and Kegley, R. A predictable execution model for COTS-based embedded systems. In IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 269–279, 2011. doi: 10.1109/RTAS.2011.33.
  44. [53]Qiao, Y., Yu, S., Anzai, S., Ma, H., Yang, S., Wang, Y., Kim, M., Wu, Y., Zhou, Y., Xing, J., Gonzalez, J. E., Stoica, I., and Xu, H. ConServe: Fine-grained GPU harvesting for LLM online and offline co-serving. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://icml.cc/virtual/2026/poster/62686.
  45. [54]Rana, K., Haviland, J., Garg, S., Abou-Chakra, J., Reid, I., and Sünderhauf, N. SayPlan: Grounding large language models using 3D scene graphs for scalable robot task planning. In Proceedings of the 7th Conference on Robot Learning (CoRL), 2023.
  46. [55]Rublee, E., Rabaud, V., Konolige, K., and Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2564–2571, 2011. doi: 10.1109/ICCV.2011.6126544.
  47. [56]Salzmann, T., Kaufmann, E., Arrizabalaga, J., Pavone, M., Scaramuzza, D., and Ryll, M. Real-time neural MPC: Deep learning model predictive control for quadrotors and agile robotic platforms. IEEE Robotics and Automation Letters, 8(4), 2023. doi: 10.1109/LRA.2023.3246839. URL https://arxiv.org/abs/2203.07747.
  48. [57]Sohal, P., Bechtel, M., Mancuso, R., Yun, H., and Krieger, O. A closer look at Intel Resource Director Technology (RDT). In Proceedings of the 30th International Conference on Real-Time Networks and Systems (RTNS), 2022. doi: 10.1145/3534879.3534882.DOI
  49. [58]Stankovic, J. A., Lu, C., Son, S. H., and Tao, G. The case for feedback control real-time scheduling. In Proceedings of the 11th Euromicro Conference on Real-Time Systems, pp. 11–20. IEEE, 1999. doi: 10.1109/EMRTS.1999.777445. URL https://www.cse.wustl.edu/~lu/papers/ercts99.pdf.
  50. [59]Stellato, B., Banjac, G., Goulart, P., Bemporad, A., and Boyd, S. OSQP: An operator splitting solver for quadratic programs. Mathematical Programming Computation, 12(4):637–672, 2020. doi: 10.1007/s12532-020-00179-2.DOI
  51. [60]Stivers, T., Enfield, N. J., Brown, P., Englert, C., Hayashi, M., Heinemann, T., Hoymann, G., Rossano, F., de Ruiter, J. P., Yoon, K.-E., and Levinson, S. C. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26):10592–10592, 2009. doi: 10.1073/pnas.0903616106.DOI
  52. [62]Xin, T., Shi, X., Dong, M., and Mi, Z. Inference in the shadows: Taming memory bandwidth contention in mobile LLM inference with Sereo. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), pp. 2299–2318. USENIX Association, 2026. URL https://www.usenix.org/conference/osdi26/presentation/xin.
  53. [63]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388.
  54. [64]Yun, H., Yao, G., Pellizzoni, R., Caccamo, M., and Sha, L. MemGuard: Memory bandwidth reservation system for efficient performance isolation in multi-core platforms. In 2013 IEEE 19th Real-Time and Embedded Technology and Applications Symposium (RTAS), pp. 55–64. IEEE, 2013. doi: 10.1109/RTAS.2013.6531079. URL https://www.ittc.ku.edu/~hee chul/papers/memguard-rtas13.pdf.DOI
  55. [65]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2312.07104.

Paper details

vlnc:W_tatgfv8q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex, Anthropic Claude

View statement
Looking for related papers…
Contents