RoboScope Kit: Unified Interfaces for Fast and Scalable Embodied-AI Evaluation

Introduction

Simulation benchmarks evaluate complementary aspects of robot behavior: knowledge transfer in LIBERO [16], bimanual manipulation in RoboTwin [5], language-conditioned task sequences in CALVIN [19], and household manipulation in RoboCasa [21]. Each benchmark brings its own evaluator, observation layout, action convention, and episode bookkeeping, and each policy family brings its own preprocessing and inference loop. Connecting several policies to several benchmarks repeats this integration for every pair. Execution cost compounds the problem: with a StarVLA checkpoint, the official evaluation of 500 LIBERO-Long episodes takes about 50 minutes on one GPU, and running 500 RoboTwin Easy episodes with one simulator and one model on one GPU takes 359.7 minutes. Comparing checkpoints, seeds, or training stages multiplies these costs, and a shared evaluation layer can remove much of this repeated work.

Profiling the official evaluators shows where this time goes (Section 2.2). Simulation occupies CPU cores while one loaded model leaves the GPU mostly idle, so simulators and models are best scaled separately. On RoboTwin, an action-chunk Policy reads one of every 16 rendered observations, so the others need not be rendered. RoboScope Kit addresses the repeated integration with one interface and the execution cost with two acceleration mechanisms, scheduling and runtime optimization.

RoboScope Kit wraps each model once as a Policy, a class that loads the checkpoint and keeps the model’s own preprocessing, and each benchmark once as an adapter around the benchmark’s own evaluator. The Policy also declares what RoboScope Kit may rely on: the observations it reads, the chunk length it returns, and how often it needs a new observation. It keeps each episode’s state separate, so one loaded model can serve many simulators at once. Simulator workers, Policy replicas, scheduling, and result collection run as separate components, so the number of simulators $K$ and the number of loaded models $g$ vary independently. The concurrency search chooses $K$ and $g$ for each workload and host from short probes. Runtime optimization in the Simulation Runtime removes simulation work that the Policy never consumes, such as rendering observations between action-chunk boundaries. Beyond one host, RoboScope Kit partitions the work items deterministically across independent GPU instances, each running its own workers and replicas, and merges their results.

This report makes three contributions:

1. A measurement of where evaluation time goes. On one RTX 4090, the official StarVLA LIBERO-Long evaluator spends 55% of its time on simulation and per-step handling on CPU cores while the GPU averages 14.3% utilization. Adding simulators to one model saturates it at 45.7% utilization, and a second Policy replica raises utilization to 87.7%. On RoboTwin, the official evaluator renders three cameras at every control step, yet the StarVLA Policy reads one observation every 16 steps.

2. A shared evaluation interface. A model is wrapped once as a Policy class and then runs on any benchmark that provides the inputs it needs, without changes to RoboScope Kit. Adapters cover six benchmark families. With a StarVLA checkpoint, a fixed RoboScope Kit configuration reproduces all 500 LIBERO-Long success outcomes of the official StarVLA evaluation.

3. Fast evaluation through scheduling and runtime optimization. Scheduling runs simulator workers and Policy replicas as separate pools, so simulators and loaded models scale independently and several simulators share each model; a time-bounded concurrency search sizes both pools for each workload and host, and deterministic sharding extends an evaluation across GPU instances. For runtime optimization, a Policy can declare that it needs a new observation only once per action chunk; RoboScope Kit then renders once per chunk, and each worker reuses one SAPIEN engine and renderer

across its episodes. On one RTX 4090, the configuration chosen by the concurrency search evaluates 500 LIBERO-Long episodes in 485 s, against 2,990 s for the official StarVLA evaluation (6.16×), with equivalent outcomes, and is 3.85× faster than RLinf and 2.10× faster than vla-eval. On RoboTwin Easy, scheduling and runtime optimization together cut the time for 500 episodes from 359.7 minutes, with one simulator and one model on one GPU, to 14.4 minutes on four GPUs (25.04×).

Background and Motivation

Simulation-Based Policy Evaluation

Simulation benchmarks let researchers compare robot policies on fixed task distributions at a cost and scale that physical trials cannot reach. An evaluation runs a fixed set of episodes. Each episode starts from a benchmark-defined task and initial state, alternates between rendering an observation, querying the policy, and stepping the simulator with the returned action, and ends when the task succeeds or a step limit is reached. The benchmark then scores the episode records, usually as the fraction of successful episodes.

Benchmarks differ at every stage of this path. LIBERO supplies two 256×256 cameras, 16-D proprioception, and a language instruction, and expects a 7-D end-effector delta with a gripper command; RoboTwin supplies three cameras and dual-arm joint state and expects a 14-D absolute joint and gripper target; CALVIN and RoboCasa use further layouts (Appendix § 9). Termination rules, step limits, and scoring differ as well. Two policies, or two evaluation systems, are comparable when they run the same episodes under the same rules, so for every run in this report we record the episode list, checkpoint, software versions, and hardware.

Where Evaluation Time Goes

Simulation, rendering, and inference costs. An episode’s wall time divides among simulator initialization and reset, physics stepping, rendering, observation transfer, Policy inference, and cleanup. These stages use different resources: physics, robot control, and environment logic use CPU cores, rendering uses the GPU’s graphics pipeline, and inference uses GPU compute and memory. Figure 2 traces the official StarVLA evaluator and its policy server on 50 LIBERO-Long episodes on one RTX 4090.

The evaluator runs one simulator and queries the server once per action chunk, so simulation and inference take turns. On CPU cores, simulation steps and the per-step handling of observations and actions take 55% of the evaluator’s time. Most of it is robot control and MuJoCo forward kinematics, computed at each of the 25 physics substeps per control step; physics integration takes 11%. Rendering the two cameras on the GPU takes 3%. Episode setup, reset, and settling take 16%, and waiting for actions 26%. An inference takes 55 ms, of which the GPU executes kernels for 26 ms: the model issues about 3,100 kernel launches per inference, and the GPU waits between them. The server is busy 25% of the time, and the GPU averages 14.3% utilization (Appendix § 11). Neither the CPU cores nor the GPU is kept busy. Since simulators consume CPU cores and Policy replicas consume GPU memory and compute, the two can be scaled separately.

Unconsumed observations in action-chunk policies. Many current policies predict action chunks: one inference returns $H$ actions, which the evaluator executes over the next $H$ control steps before querying the Policy again. The StarVLA RoboTwin checkpoint has $H = 16$. The official RoboTwin evaluator still renders all three cameras at every control step, so 15 of every 16 rendered observations are never read. Rendering only at chunk boundaries keeps physics stepping, action execution, and success checks at the native control frequency, and the Policy receives exactly the observations it reads.

Design Goals

These observations lead to five design goals. (G1) Unified. Integrate each model and each benchmark once, and reuse it with every counterpart. (G2) Faithful. Reproduce the outcomes of each benchmark’s official evaluation, under its own tasks, initial states, termination, and scoring. (G3) Efficient. Keep both CPU cores and GPUs busy, and skip simulation work that changes neither the Policy’s inputs nor the score. (G4) Automatic. Configure each evaluation for its workload and host without manual tuning, and report what this costs. (G5) Scalable. Extend one evaluation from one GPU to several GPUs and instances.

Design

Overview

RoboScope Kit has four parts (Figure 1). A Policy is a class, installed in its model’s own environment, that loads a checkpoint and keeps the model’s preprocessing. A benchmark adapter wraps the benchmark’s native evaluator, which keeps task definitions, initial states, termination, and scoring (G2, faithful). Two runtimes connect the two. The Control Runtime schedules the evaluation. It expands a benchmark suite into a manifest, the ordered list of episode keys (task, episode, seed), and groups the keys into work items. It starts $K$ simulator workers, processes that each own one simulator, and $g$ Policy replicas, processes that each hold one loaded Policy, and collects one episode record per episode key. The Simulation Runtime runs inside each simulator worker: the native evaluator, instrumented by the adapter, steps the simulator, renders observations, and sends them to a Policy replica. Acceleration has two parts. Runtime optimization in the Simulation Runtime removes simulation work the Policy never consumes (§ 3.6). Scheduling in the Control Runtime separates simulation from inference, chooses $K$ and $g$, and shards work across instances (§ 3.3, § 3.4, § 3.5).

An evaluation proceeds in four stages. The Control Runtime first validates that the Policy’s declared capabilities match the benchmark’s contract. It then runs the concurrency search, short probes that choose $K$ and $g$, unless they are fixed. Next it runs the manifest as work items across $K$ workers that send observations to $g$ replicas. Finally it aggregates completed items into benchmark scores and execution metrics.

Evaluation Interface

Observation and action contracts. A model is integrated once as a Policy and a benchmark once as an adapter, and each is reused with every counterpart (G1, unified). Each benchmark specification names its observation features and its action. A feature has a canonical name such as images.head, state.joints, or language.instruction, a kind, a dtype, and an optional shape; the action has a name, a dimension, and a dtype. A Policy declares the features it requires, the suites it supports, the chunk lengths it returns, its action representation, and an observation stride per benchmark. Checkpoint-specific work, including image resizing and cropping, normalization, tokenization, and the transform from model outputs to benchmark actions, stays inside the Policy. Adapters apply returned actions in the benchmark’s native convention and execute action chunks one step at a time at the native control frequency.

Policy lifecycle and episode state. A Policy replica moves through five operations (Table 2). load reads one checkpoint and returns the Policy’s capabilities; reset allocates state for one episode; act maps an observation to an action or action chunk, and its batched form act_batch, which defaults to calling act per observation, serves several at once; end_episode releases that episode’s state; and unload releases the model. Several simulator workers share a replica, so requests from different episodes interleave. Every request carries an episode identifier, and stateful Policies key their histories, caches, and action queues by it. Model weights and other replica-scope resources live from load to unload; episode-scope state lives from reset to end of episode (Figure 8).

Method	Lifecycle transition	Policy behavior
load(context)	Created → Loaded	Load one checkpoint replica and declare capabilities
reset(context)	Loaded → Episode active	Allocate state keyed by episode_id
act(observation)	Episode active → Episode active	Produce one action or action chunk from canonical features
act_batch	Episode active → Episode active	Serve an observation batch with native batching when available
end_episode(id)	Episode active → Loaded	Release state for one episode
unload()	Loaded → Released	Release model and process-wide resources

Interface validation. Before any rollout, the Control Runtime checks that the requested benchmark and suite are among the Policy’s declared capabilities, that every required feature is provided, and that the chunk length and action representation resolve to one contract across all replicas. During rollouts it checks each observation’s fields, dtypes, and fixed dimensions, and each returned action’s numeric type, shape, and dimension.

Separating Simulation and Inference

Each simulator worker is a process that owns one simulator instance and the benchmark’s dependencies. Each Policy replica is a process that loads one model on an assigned GPU in the model’s own environment, so the replicas of one instance can spread over all of its GPUs (G5, scalable). Workers and replicas never share an interpreter, so benchmark and model environments can pin conflicting libraries. The two form independent pools of sizes $K$ and $g$, which can be sized to keep both CPU cores and GPUs busy (G3, efficient); $K8/g2$ denotes eight workers sharing two replicas. Workers send observations to replicas over local request–response connections and receive action chunks, and every message carries the episode identifier. Because workers share replicas, GPU memory holds $g$ copies of the model however many simulators run. All processes of an evaluation belong to one process group, which the Control Runtime stops as a unit at shutdown or on failure.

Concurrency Search

Probe workloads and resource constraints. The concurrency search chooses $K$ and $g$ automatically within a time budget (G4, automatic), and § 5.4 reports its cost next to evaluation time. Concurrency is chosen by running the actual workload. A probe starts $K$ workers and $g$ replicas with the real Policy, runs a short episode set, and measures throughput $q$ in control steps per second, consumed CPU cores $c$, GPU utilization $u$, CPU throttling $t$, and per-replica GPU memory $v$. Throughput is measured in steady state: each worker receives three work items, and the estimate drops each worker’s first item, which includes environment creation and Policy warmup, and the items that finish after the first worker runs out of work. A probe fails when a worker is missing or fails, an out-of-memory error occurs, its episodes do not finish within the time limit, or its success rate falls below 85% of the first probe’s on the episodes both probes ran. Resources bound the candidates: CPU cores bound the worker count, and the GPU memory left after loading replicas bounds the replica count.

Selection rules and search budgets. Algorithm 1, the diagonal search, first adds simulator workers alone and then adds workers and replicas together, a diagonal in the $(K,g)$ plane; Table 1 defines its predicates. The worker phase keeps one replica per GPU and doubles $K$ while a doubling raises throughput by at least 25%, a quarter of linear scaling. The first doubling that gains less marks the knee of the throughput curve, beyond which more workers add little. Workers come first because they cost CPU cores, not GPU memory; past the knee, the replicas limit throughput. The accepted $K$ per replica, $r$, is the number of workers one replica keeps busy. The replica phase then adds one replica per GPU at a time and scales the workers with it, $K = rg$, keeping a step while it raises throughput by at least $\varepsilon$. Both phases stop before CPU, GPU, or

Name	Definition	Setting
$\operatorname{Probe}(K,g)$ fails	A worker is missing or fails, an out-of-memory error occurs, episodes do not finish within $\min(\tau_c,\text{ remaining }\tau)$, or success on the episodes shared with the first probe falls below $(1-\delta)$ times the first probe’s	$\tau_c = 15\,\mathrm{min}$, $\tau= 30\,\mathrm{min}$, $\delta= 0.15$
$\operatorname{WorkerCap}(m^*)$	$K^*+\left\lfloor(C-1-c(m^*))/\hat{c}\right\rfloor$, where $\hat{c}$ is the CPU cost per worker fitted from the probes with the fewest and most workers at $g=G$; before two such probes, $\max(G,\lfloor(C-1)/2\rfloor)$; at most 64	—
$\operatorname{ReplicaCap}$	$\min(16,C,G\lfloor(V-r_v)/v\rfloor)$, with $v$ measured by the first probe	$r_v=2\,\mathrm{GiB}$ reserved per GPU
$\operatorname{FitsMemory}(K,g)$	The busiest GPU’s peak memory, extrapolated from the incumbent with the replica footprint and the fitted memory per worker, stays below $V-r_v$	—
$\operatorname{WithinLimits}(m)$	$c(m)\le C-1$, $u(m)\le u_{\max}$, and $t(m)\le t_{\max}$ when $t$ is measured	$u_{\max}=0.95$, $t_{\max}=0.40$
$\operatorname{Knee}(m,m^*)$	$q(m)\ge\left(1+\kappa\left(K(m)/K^*-1\right)\right)q(m^*)$	$\kappa=0.25$
$\operatorname{Gains}(m,m^*)$	$q(m)\ge(1+\varepsilon)q(m^*)$	$\varepsilon=0.03$

Algorithm 1 Diagonal concurrency search.

Input: Workload $W$, Policy $P$, host with $G$ GPUs Output: Configuration $(K^*, g^*)$, or $\bot$ if the first probe fails

1. $(K^*,g^*,m^*) \leftarrow(G,G,\operatorname{Probe}(G,G))$

2. if $m^*$ fails then return $\bot$

3. while $K^* < \operatorname{WorkerCap}(m^*)$ do ▷ worker phase

4. $K \leftarrow\min(2K^*,\operatorname{WorkerCap}(m^*))$; $m \leftarrow\operatorname{Probe}(K,G)$

5. if $m$ fails or not $\operatorname{WithinLimits}(m) \land\operatorname{Knee}(m,m^*)$ then break

6. $(K^*,m^*) \leftarrow(K,m)$

7. $r \leftarrow K^*/G$

8. while $g^* + G \le\operatorname{ReplicaCap}$ do ▷ replica phase

9. $g \leftarrow g^*+G$; $K \leftarrow\max(K^*,\min(\operatorname{round}(rg),\operatorname{WorkerCap}(m^*)))$

10. if not $\operatorname{FitsMemory}(K,g)$ then break

11. $m \leftarrow\operatorname{Probe}(K,g)$

12. if $m$ fails or not $\operatorname{WithinLimits}(m) \land\operatorname{Gains}(m,m^*)$ then break

13. $(K^*,g^*,m^*) \leftarrow(K,g,m)$

14. return $(K^*,g^*)$

throttling limits are exceeded, and the replica phase skips a step whose predicted GPU memory does not fit. Each probe is limited to $\tau_c$ and the whole search to $\tau$; when a probe fails or the budget runs out, the last accepted configuration stands. Probes that the search measures but does not keep are recorded as rejected with their reason.

Sharding Across Instances

One host bounds the workers and replicas an evaluation can use. RoboScope Kit adds a second scheduling level above each instance’s local $K/g$ Control Runtime, so that one evaluation can span several instances (G5, scalable; Figure 9). Given the ordered work items $W=(w_0,\ldots,w_{M-1})$, shard $i$ of $N$ runs

The shards are disjoint and together form the requested workload, and each work item keeps its canonical episode indices and initial states or goals. Each instance runs its own queue, simulator workers, and Policy replicas and, when enabled, its own concurrency search.

Each shard writes its own result, progress counter, and build identity, and two merge paths combine them. Record-tree merging checks that the episode records cover every work item exactly once and then applies the benchmark’s aggregation to the combined records. Result-only reduction checks that all shards share the benchmark protocol and build identity and then combines per-shard counts and partial aggregates; official metrics that are not additive are recomputed from these counts.

Runtime Optimization

Simulation Runtime reuse and redundant work removal. Runtime optimization removes only work that neither feeds the Policy nor affects the score (G3, efficient), and chunk-boundary observation applies only at the stride the Policy declares. Within a rollout, the RoboTwin Simulation Runtime removes work that repeats across episodes or only validates seeds. It reuses one SAPIEN Engine and Renderer for all episodes a worker runs instead of rebuilding them per episode. It hoists a passive-force computation that the official control loop performs twice per step. The official evaluator validates seeds with an expert rollout. Given a verified seed cache, the runtime skips this check and, for joint-position actions, cuRobo initialization. End-effector actions retain one cuRobo planner per worker. Engine reuse and the control-loop change were enabled in all RoboTwin runs reported here, including the single-GPU baseline; chunk-boundary observation follows the Policy’s declared stride.

Work-item size and setup caching. A RoboTwin work item holds one task’s ten episodes by default. Two-episode items let idle workers take over the remaining episodes of slow tasks, so fewer workers sit idle at the end of a run. RoboTwin also rebuilds a static table of clutter-object metadata whenever an evaluator process starts. The Simulation Runtime can build it once, store it with hashes of the generating source and the table, and load it in later processes; the scene contents and physics are unchanged.

Chunk-boundary observation. A Policy declares an observation stride $s$ for a benchmark. At control step $t$, the evaluator renders a new observation if the Policy runs inference at $t$, which happens every $H$ steps for chunk length $H$, or if $t$ is a multiple of $s$; otherwise it passes the previous observation. With $s=H$, rendering happens once per chunk and every inference receives the observation of its own step. The default stride is 1, which reproduces the official cadence of one render per step. Randomized lighting, the interactive viewer, video logging, and point-cloud observations keep one render per step for any $s$. The StarVLA RoboTwin Policy declares $s=H=16$. The four-GPU stride-16 run matches the single-GPU stride-1 runs on every success outcome and terminal step of the 500 RoboTwin Easy episodes (§ 5.2).

Work Distribution and Failure Handling

Work items and result integrity. The Control Runtime converts the ordered manifest into work items: one task’s episodes for LIBERO, and consecutive per-task episode windows for RoboTwin, interleaved across tasks so that long rollouts spread over workers. Each item keeps the canonical episode indices and the seeds or initial states of its episodes. A worker claims an item atomically, moving it from pending to running, appends terminal records to the item’s partial state as episodes finish, and publishes a completed result when the item ends. Publication is idempotent: an identical repeated result is accepted, and a conflicting result for the same item is an integrity error. Aggregation reads completed items. Before scoring, the Control Runtime checks coverage, exactly one record per requested episode key with no duplicates or unexpected keys, and that each record’s seed matches the manifest. Figure 3 shows the item states and the recovery path.

Failure handling. Worker failures are recovered at work-item granularity (Figure 3). When a simulator worker exits or raises an allocator out-of-memory error, the Control Runtime returns its claimed item to pending, discards the item’s partial records, and starts a replacement worker, which replays the item from its first episode. After an out-of-memory error it also lowers worker concurrency on the affected GPU while the other GPUs continue. Aggregation reads completed items, so a replayed episode contributes one record. A

Policy replica that exits aborts the evaluation; the Control Runtime then stops every process of the evaluation and checks that none remains.

Implementation

Packaging and process layout. RoboScope Kit is distributed as a Python package that defines the Policy API and contains the Control Runtime, the Simulation Runtime, and the benchmark adapters. A Policy is installed in its model’s environment, and a configuration file names the command that starts its replica process. The Control Runtime is installed at a pinned version next to the Policy, with a checksummed benchmark configuration. Benchmark simulators, datasets, and native evaluators come from shared storage when an evaluation starts. The command-line entry point takes a benchmark, suite, checkpoint, and output directory; it runs the concurrency search and the formal evaluation and writes the manifest, probe history, episode records, metrics, and resource samples to the output directory. The Control Runtime, simulator workers, and Policy replicas run as separate processes on one host and communicate over local Unix-domain sockets. For a sharded evaluation, the entry point also takes a shard index and shard count. Our evaluation platform, a cluster service outside RoboScope Kit, runs each evaluation on up to four instances: it launches and polls the instances, collects the shard reports, and publishes the score once every shard has completed. If a shard fails, the evaluation fails and all of its instances are released. Cross-instance sharding comes with a merge tool that checks that the shards are disjoint and cover the workload and then recomputes the official result.

Benchmark adapters. An adapter implements what differs between benchmarks: building work items from the suite’s tasks and episodes, starting a native rollout process for an item, translating the benchmark’s observations into named features and the Policy’s actions into the benchmark’s convention, and parsing native results into episode records. Task initialization, reset, action application, termination, and scoring run in the benchmark’s own evaluator, which the adapter instruments to route inference to a Policy replica and to emit episode records. The catalog covers LIBERO (five suites), LIBERO-Plus, RoboTwin 2.0 (Easy and Hard), CALVIN, RoboCasa GR-1 Tabletop, and Meta-World MT50 [33] (Appendix § 13). The measurements in this report use LIBERO-Long and RoboTwin Easy, and the cross-instance run uses MetaWorld MT50.

Listing 1: The stateless StarVLA Policy, RoboTwin path (condensed).

from roboscope_kit import (ActionChunk, Observation, Policy,
                           PolicyCapabilities, SessionContext, run)

CAMERAS = ("images.head", "images.left_wrist", "images.right_wrist")

class StarVLARoboTwinPolicy(Policy):
    def load(self, context: SessionContext) -> PolicyCapabilities:
        self.model = load_checkpoint(context.checkpoint_path).to("cuda").eval()
        horizon = self.model.action_horizon          # 16 for this checkpoint
        return PolicyCapabilities(
            supported_benchmarks=("robotwin",),
            required_features=CAMERAS + ("language.instruction",),
            supported_action_chunk_sizes=(horizon,),
            observation_strides={"robotwin": horizon})  # render once per chunk

    def act(self, observation: Observation) -> ActionChunk:
        f = observation.features                      # canonical feature names
        example = preprocess([f[c] for c in CAMERAS], f["language.instruction"])
        normalized = self.model.predict_action(example)  # (16, 14), in [-1, 1]
        return ActionChunk(unnormalize(normalized).astype("float32"))

    def unload(self) -> None:
        self.model = None

if __name__ == "__main__":
    run(StarVLARoboTwinPolicy())

Adding a Policy. A new Policy is one class. The class subclasses Policy and implements load and act; reset, end_episode, and unload default to no-ops. load returns the Policy’s capabilities, including required features, chunk lengths, action representation, and observation strides. The class’s script ends with run(policy), which serves it on the socket the Control Runtime provides. The shared Control Runtime and every benchmark adapter are reused unchanged, and the same class serves every benchmark whose features it declares.

What a Policy must add beyond load and act depends on whether it keeps state between inference calls. Listing 1 shows a stateless Policy, condensed from the StarVLA RoboTwin integration: each action chunk depends only on the current observation, so the class keeps only replica-scope state, the model, and declares an observation stride equal to its chunk length, which enables chunk-boundary observation (§ 3.6).

A stateful Policy carries information from one call to the next. The X-VLA Policy [35] is the stateful Policy integrated with RoboScope Kit: the model takes as proprioceptive input the end-effector pose of its own previous action chunk, so each episode carries a pose that every act reads and updates. Listing 2 shows its LIBERO path. One replica serves several simulator workers, and a replica handles one call at a time, so calls from different episodes interleave on the same object. Per-episode state therefore cannot live in instance attributes; the Policy keys it by episode_id, discards stale state in reset, initializes it from the observed state on the episode’s first act, because reset carries no observation, and releases it in end_episode. Because the replica serializes calls, the Policy needs no locking. The X-VLA Policy declares a stride of 1 on every benchmark.

Evaluation

Our evaluation answers the following questions:

• Does RoboScope Kit preserve benchmark outcomes (§ 5.2)?

• How does its evaluation time compare with existing systems (§ 5.3)?

• How much does each design component contribute (§ 5.4)?

• How does it scale with GPUs, workers, and Policy replicas (§ 5.5)?

Listing 2: The stateful X-VLA Policy, LIBERO path (condensed).

import numpy as np
from roboscope_kit import (ActionChunk, EpisodeContext, Observation,
                           Policy, PolicyCapabilities, SessionContext)

class XVLAPolicy(Policy):
    def __init__(self) -> None:
        self.proprio: dict[str, np.ndarray] = {}        # episode_id -> state

    def load(self, context: SessionContext) -> PolicyCapabilities:
        self.model = load_xvla(context.checkpoint_path).to("cuda").eval()
        return PolicyCapabilities(
            supported_benchmarks=("libero",),           # other benchmarks omitted
            required_features=("images.primary", "images.wrist",
                               "state.proprioception", "language.instruction"),
            supported_action_chunk_sizes=(self.model.num_actions,))

    def reset(self, context: EpisodeContext) -> None:
        self.proprio.pop(context.episode_id, None)      # no stale state

    def act(self, observation: Observation) -> ActionChunk:
        f, ep = observation.features, observation.episode_id
        if ep not in self.proprio:                      # first call of episode
            self.proprio[ep] = proprio_from_state(f["state.proprioception"])
        raw = self.model.generate(f["images.primary"], f["images.wrist"],
                                  f["language.instruction"], self.proprio[ep])
        self.proprio[ep][:9] = raw[-1, :9]               # feed back last pose
        return ActionChunk(to_libero_actions(raw))       # (30, 7) float32

    def end_episode(self, episode_id: str) -> None:
        self.proprio.pop(episode_id, None)              # release episode state

Experimental Setup

Workloads and checkpoints. The cross-system comparison uses LIBERO-Long (suite libero_10), 10 tasks with 50 initial states each and seed 7, evaluated with the public StarVLA checkpoint StarVLA/Qwen2.5-VL-OFT-LIBERO-4in1 [3, 25]. The ablation study also uses RoboTwin Easy (suite demo_clean), 50 tasks with 10 episodes each, evaluated with StarVLA/Qwen3-VL-OFT-Robotwin2 [2], whose chunk length is 16. Each workload has a frozen ordered manifest of 500 episode keys, and every run of that workload receives the same keys (Appendix § 10).

Baselines. On LIBERO-Long, we compare RoboScope Kit with three evaluation systems that run the same checkpoint and manifest: the official StarVLA evaluation (its policy server and LIBERO evaluator), vla-eval [8], and RLinf [32]. RLinf was tuned once offline for the host, which took 1,704 s. vla-eval runs the fastest configuration of a 9,780 s search over its shard count, model-server replicas, maximum batch size, and batch wait (Table 4): 24 shards share one server that batches up to 24 observations with a 5 ms wait. RoboScope Kit (Search) runs RoboScope Kit with its diagonal concurrency search (§ 3.4) and a StarVLA Policy that uses the same camera-image preprocessing and gripper processing as the StarVLA client; the search selects $K12/g3$ in all three of its launches after a median 506 s, and the cross-system comparison evaluates this configuration with CUDA MPS off, the default. RoboScope Kit (Search+MPS) runs the same search and evaluation with NVIDIA CUDA MPS on (--cuda-mps), which lets the replicas run their kernels concurrently on the GPU instead of in time slices; the search then selects $K16/g4$ in all three launches after a median 504 s (§ 5.4). The ablation study compares the configurations chosen by the search with fixed configurations.

System	Search space	Selected	Runs
vla-eval	Shards $\{8,\ldots,24\}$, servers $\{1,2,3\}$, maximum batch $\{4,\ldots,24\}$, wait $\{0,5,20\}$ ms; 100-episode probes, then 500-episode runs (Table 5)	24 shards, 1 server, batch 24, wait 5 ms	23
RLinf	Start: 10 environments, 1 pipeline stage; environments $\{4, 8, 12, 16, 20\}$; then 2 and 4 pipeline stages at 20 environments; 30-episode probes	20 environments, 1 pipeline stage, 1 rollout worker (30/30)	8

Hardware and software. All GPUs are 48 GiB RTX 4090s. Every cross-system launch runs on one GPU with 12 CPU cores, and every system runs three launches. The worker and replica sweep and the three diagonal searches run on one RTX 4090 with a 20-core quota. RoboTwin experiments run on one RTX 4090 for the single-GPU baseline and on four RTX 4090s with an 80-core quota. Appendix § 12.§ 2 gives the host of every launch.

Metrics. Success rate is the fraction of successful episodes. Outcome agreement between two runs is the fraction of episode keys with the same success outcome; terminal-step agreement also requires the same terminal step. Evaluation time covers the formal evaluation and excludes process initialization, concurrency search or offline tuning, and cleanup; end-to-end time is the wall time of the whole process. Throughput is policy-controlled environment steps per second of evaluation time, and speedup is the ratio of reference to candidate evaluation time over three launches each, with a 95% $t$ interval (Appendix § 12.§ 2). On LIBERO-Long, $\Delta$ success carries a 95% paired task-cluster bootstrap interval (10 tasks, 20,000 draws). Outcomes are equivalent when the $\Delta$ interval lies within $\pm2$ percentage points.

Outcome Agreement

Agreement with the official evaluation. A fixed $K1/g1$ RoboScope Kit run from the worker and replica sweep matches each of the three official launches on all 500 LIBERO-Long success outcomes (Appendix § 12.§ 1). In the cross-system comparison, the official evaluation scores 90.2% and RoboScope Kit (Search) 89.9%, −0.3 percentage points (95% interval −0.8 to 0.0), with 97.9% of episodes having the official outcome; RoboScope Kit (Search+MPS) also scores 89.9%, −0.3 points (−1.3 to +0.8), with 96.5% agreement (Figure 4). Under the $\pm2$-point margin, both are equivalent to the official evaluation. On RoboTwin Easy, the four-GPU stride-16 run matches each of the three single-GPU $K1/g1$ runs on every success outcome and terminal step, and the four-GPU stride-1 runs at $K4/g4$, $K16/g4$, and $K32/g4$ agree with one another. The four-GPU stride-1 runs score 235/500 against 231/500 for the single-GPU runs, a difference of +0.8 percentage points (95% interval −2.0 to +3.8), with 89.2% of episodes having the same outcome.

Agreement with other evaluation systems. RLinf scores 90.3%, +0.1 percentage points relative to the official evaluation [95% interval −2.6 to +3.3], and shares the official outcome on 88.1% of episodes. vla-eval scores 89.8%, 0.4 points below the official evaluation [95% interval −1.6 to +0.7], with 91.7% agreement. For comparison, repeated launches of the same system agree on 86.2–88.4% of episodes for RLinf, 93.6–95.0% for vla-eval, 97.8–99.0% for RoboScope Kit (Search), 97.4–98.4% for RoboScope Kit (Search+MPS), and 100% for the official evaluation, so RLinf differs from the official evaluation about as much as from its own repeated launches. RLinf samples its actions during evaluation (Appendix § 10.§ 3).

Sources of outcome differences. Within RoboScope Kit, LIBERO-Long runs with 2 to 20 simulator workers differ from a one-worker run on 1 to 28 of 500 success outcomes, while the success count moves by at most ten (Table 18). The differences follow task assignment: they fall in tasks split across workers. Appendix § 12.§ 1 breaks them down by episode and condition.

Configuration	Successes	Evaluation time (s)	Steps/s	GPU util. (%)	Speedup vs. $K1/g1$	Efficiency per worker	Success agreement (%)	Terminal agreement (%)
Search → $K12/g3$	447.7	679.0	202.5	97.0	5.25	0.44	97.1	78.5
$K1/g1$	451	3,559.6	38.4	15.4	1.00	1.00	100.0	100.0
$K2/g1$	452	1,883.8	72.5	29.4	1.89	0.94	99.8	98.4
$K4/g1$	451	1,213.6	112.5	45.7	2.93	0.73	98.4	87.0
$K8/g1$	446	1,220.5	112.4	45.8	2.92	0.36	97.0	85.2
$K12/g1$	446	1,216.1	113.7	46.3	2.93	0.24	97.8	78.6
$K16/g1$	448	1,205.0	114.0	46.8	2.95	0.18	96.6	49.8
$K20/g1$	447	1,221.3	112.0	45.6	2.91	0.15	96.0	42.4
$K4/g2$	453	1,001.8	136.1	59.7	3.55	0.89	98.8	88.0
$K8/g2$	445	749.4	183.6	86.7	4.75	0.59	97.2	86.0
$K12/g2$	449	748.0	183.3	87.7	4.76	0.40	97.2	77.4
$K16/g2$	452	759.8	180.0	85.8	4.68	0.29	96.6	49.4
$K8/g3$	451	713.7	191.2	–	4.99	0.62	98.8	86.8
$K12/g3$	451	691.3	198.5	–	5.15	0.43	97.6	78.4
$K16/g3$	450	694.9	196.9	–	5.12	0.32	95.8	48.6
$K20/g3$	441	693.7	197.7	–	5.13	0.26	94.4	41.8
$K12/g4$	448	676.9	203.8	–	5.26	0.44	97.0	78.0
$K16/g4$	452	670.9	203.8	–	5.31	0.33	96.6	48.6
$K20/g4$	443	674.7	203.4	–	5.28	0.26	95.2	41.8

Performance

Evaluation-time comparison. Appendix § 12.§ 2 lists every launch with its initialization, search, and tuning time.

Results. RoboScope Kit (Search) completes LIBERO-Long in a median 671 s of evaluation time, against 2,990 s for the official StarVLA evaluation, 1,882 s for RLinf, and 1,016 s for vla-eval: 4.45× (95% CI [4.40, 4.50]) faster than StarVLA, 2.78× ([2.59, 2.98]) faster than RLinf, and 1.52× ([1.46, 1.58]) faster than vla-eval at its tuned configuration. Its throughput, 204.6 steps/s, exceeds vla-eval’s 133.5, RLinf’s 74.2, and StarVLA’s 45.8. Its launches take a median 738 s end to end; adding the median 506 s of the diagonal search gives 1,244 s, against 3,013 s for StarVLA, 3,639 s for RLinf, and 10,822 s for vla-eval including their offline tuning. Against the official evaluation, RoboScope Kit (Search) is faster with equivalent outcomes, 2.42× end to end. RoboScope Kit (Search+MPS) completes LIBERO-Long in a median 485 s at 282.5 steps/s: 6.16× ([6.08, 6.23]) faster than StarVLA, 3.85× ([3.58, 4.13]) faster than RLinf, 2.10× ([2.03, 2.18]) faster than vla-eval, and 1.38× ([1.37, 1.40]) faster than RoboScope Kit (Search) on the same host and cores. Including its search, a launch takes a median 997 s, 3.02× faster than StarVLA end to end.

Ablation Study

Separating simulation and inference. Separation lets the Control Runtime add a Policy replica without adding simulators. On one RTX 4090 with twelve workers, a second replica shortens LIBERO-Long evaluation from 1,216 s to 748 s (1.63×) and raises mean GPU utilization from 46.3% to 87.7% (Table 18). The twelve workers share two model copies, and peak GPU memory is 25.0 GiB.

Concurrency search and its cost. On one RTX 4090 with a 20-core quota, the host of the worker and replica sweep, the diagonal search selects $K12/g3$ in all three launches after a median 506 s of search (Table 18 and Table 13). The worker phase reaches the knee at $K4/g1$: doubling to $K8/g1$ gains less than 25%. The replica phase then keeps $K8/g2$ and $K12/g3$ and rejects $K16/g4$, which gains only 2.1–2.4%, under the 3% threshold. The selected configuration evaluates LIBERO-Long in a median 679 s, 1.10× faster than $K12/g2$, the fastest fixed configuration with at most two replicas, and 5.25× faster than fixed $K1/g1$, or 3.01× including the search. Its steady-state probes of $K12/g3$ measure within 1.2% of the formal throughput of the same launch. Figure 7a places the search path on the grid search: the grid evaluates eighteen configurations in 5.4 h, while the search probes seven in 8.4 min. Its choice evaluates within 1.2% of the fastest grid configuration, $K16/g4$ (670.9 s), inside the 3% gain below which the search stops adding replicas.

Workload	$K/g$	Search (s)	Eval. (s)	E2e (s)	Successes	Peak GPU mem. (GiB)
LIBERO-Long	12/3	506	679	1,193	448/500	33.6
	12/3	504	675	1,187	447/500	33.6
	12/3	507	680	1,194	448/500	33.5
LIBERO-Long, CUDA MPS, 12 pinned cores	16/4	504	485	997	452/500	44.7
	16/4	505	487	1,000	447/500	44.5
	16/4	502	483	994	449/500	44.4
RoboTwin Easy (100 episodes)	8/1	538	343	884	52/100	41.0
	8/1	561	351	915	52/100	40.7
	8/1	616	351	986	52/100	41.5

Concurrency search under CUDA MPS. With CUDA MPS on, on the cross-system host with 12 pinned cores, a further replica keeps paying off (Figure 7b): the search keeps $K16/g4$, which adds 8–9% over $K12/g3$, and stops because $K20/g5$ is predicted to exceed GPU memory. It selects $K16/g4$ in all three launches after a median 504 s; this is the RoboScope Kit (Search+MPS) row of § 5.3, 1.38× faster in evaluation than $K12/g3$ with MPS on the same host and cores. A grid of eighteen fixed configurations under MPS, $g$ from 1 to 4 (5.0 h of evaluation, Table 14), confirms the choice: $K16/g4$ is its fastest configuration at 482.5 s, ahead of $K20/g4$ (483.2 s) and $K12/g4$ (489.2 s), and every configuration with fewer replicas is at least 5.9% slower.

$K/g$	Successes	Eval. (s)	E2e (s)	Steps/s	CPU cores	GPU util. mean (%)	Peak GPU mem. (GiB)
1/1	451	3,655.4	3,676.2	37.4	1.2	15.0	9.6
2/1	453	1,908.3	1,932.5	71.6	2.2	29.5	10.5
4/1	450	1,177.9	1,204.5	115.9	3.4	48.5	11.5
8/1	451	1,166.5	1,190.5	117.2	3.5	49.5	13.5
12/1	448	1,171.4	1,192.2	117.6	3.6	49.8	16.1
16/1	451	1,186.3	1,208.9	115.5	3.5	49.0	18.0
20/1	444	1,170.3	1,193.6	117.2	3.6	49.6	19.6
4/2	451	992.0	1,033.2	138.2	4.1	53.8	20.4
8/2	450	637.3	693.1	214.2	6.6	79.3	22.3
12/2	449	623.4	672.1	220.5	6.9	81.3	25.2
16/2	451	621.2	666.5	219.8	6.9	81.2	27.0
8/3	449	581.2	638.8	234.8	7.5	84.6	31.1
12/3	448	518.9	588.2	265.6	9.1	92.7	33.6
16/3	447	513.7	582.7	266.7	9.1	92.9	35.1
20/3	450	511.2	575.7	266.8	9.1	92.8	37.4
12/4	446	489.2	574.4	281.9	10.2	95.4	42.8
16/4	450	482.5	573.7	284.3	10.6	96.0	44.4
20/4	442	483.2	575.1	284.9	10.5	95.7	46.3

Runtime optimization. A controlled ladder on one RTX 4090 adds the RoboTwin Simulation Runtime changes one at a time, with a Policy that holds the current state (Figure 5a). Engine and Renderer reuse raises throughput from 8.88 to 9.31 steps/s. For the joint-position Policy, skipping cuRobo initialization reduced total setup time for three episodes from 45.2 s to 1.4 s and raised throughput from 9.31 to 11.63 steps/s. Observing at 16-step chunk boundaries reduces rendered observations from 2,100 to 132 and raises throughput 2.51×, to 29.23 steps/s, and hoisting the duplicate passive force reaches 32.38 steps/s, 3.65× the official-like path. With the real StarVLA Policy at $K16/g4$ on four GPUs, stride 16 together with two-episode work items and the clutter-metadata cache (§ 3.6) is 4.58× faster than stride 1 (Table 16).

Variant	Added Simulation Runtime behavior	Observations	Setup (s)	Wall (s)	Steps/s
00 official-like	Official-like path	2,100	46.9	237	8.88
01 shared context	Reuse Engine and Renderer	2,100	45.2	226	9.31
02 planner warmup	Skip cuRobo initialization	2,100	1.4	180	11.63
03 chunk boundary	Observe at 16-step chunk boundaries	132	1.4	71.9	29.23
04 passive force	Hoist duplicate passive force	132	1.7	64.8	32.38

Change	Configuration	Runs	Eval. time (s)	Factor
Baseline	$K1/g1$, one GPU, stride 1	3	21,582	–
Four GPUs	$K4/g4$, four GPUs, 80 cores	3	5,934	3.64×
Workers	$K16/g4$, same hardware	1	3,951	1.50×
Stride and run setup	$K16/g4$, stride 16	1	862	4.58×
Total				25.04×

Optimized four-GPU RoboTwin run. The fastest RoboTwin Easy configuration combines four RTX 4090 GPUs and an 80-core quota, $K16/g4$, stride 16, two-episode work items, and the clutter-metadata cache. It evaluates the 500 episodes in 862 s, against 21,582 s for the single-GPU $K1/g1$ baseline: 25.04× faster, and 6.26× fewer GPU-hours. All 500 success outcomes and terminal steps match each of the three baseline runs. Figure 5b follows the measured configurations from the baseline to this run. Four GPUs with one worker and one replica each give 3.64×; sixteen workers on the same four GPUs give a further 1.50×; stride 16 with the run setup above gives 4.58×.

Scalability

Worker and Policy-replica scaling. Figure 7a and Table 18 sweep $K$ and $g$ on one RTX 4090 with a 20-core quota. With one replica, throughput rises from 38.4 steps/s at $K1$ to 112.5 at $K4$ and stays at 112–114 up to $K20$, while GPU utilization stays between 45.6% and 46.8%: the replica is the bottleneck. A second replica moves saturation to $K8$, raises throughput to 183.6 steps/s and utilization to 86.7%, and reaches 4.76× over $K1/g1$ at $K12/g2$; a third and a fourth replica add 8% and a further 3%, to 5.31× at $K16/g4$. Per-worker efficiency falls from 0.94 at $K2$ to 0.15 at $K20$ with one replica. Success-outcome agreement with $K1/g1$ is 99.8% at $K2$ and 94.4–98.8% at larger $K$, and terminal-step agreement falls from 98.4% to 41.8% as more workers share each task’s episodes.

Where the bottleneck moves. A stage-timed profile of 50 LIBERO-Long episodes shows why (Figure 6, Table 6). Four workers saturate one replica: it is busy 85% of the time, requests wait 48 ms for it, and workers spend 42% of their time waiting for actions. A second replica on the same GPU overlaps these launch-bound inferences and roughly doubles GPU utilization. With twelve workers sharing two replicas, inference again sets the pace: requests wait 221 ms, and each inference stretches to 84 ms under contention. Adding simulators moves the bottleneck to inference, and adding replicas moves it to the GPU, so the right numbers of workers and replicas depend on each other and on the workload and host.

	$K1/g1$	$K4/g1$	$K12/g2$
Evaluation time (s)	374.1	130.2	82.7
Robot control and forward kinematics	37.1	28.6	16.0
Physics integration	10.4	7.9	4.4
Observable updates and step bookkeeping	2.9	2.3	1.3
Rendering	2.9	3.1	3.3
Episode reset	11.9	9.7	5.7
Task environment creation	3.2	4.1	6.0
Settling steps	1.8	1.4	0.9
Waiting for actions	29.4	41.7	60.0
Episode RPCs and idle	0.5	1.3	2.4
Mean GPU utilization (%)	15.4	42.3	80.3
Replica busy time (% per replica)	26	85	85
Request wait for a replica (ms)	0	48	221
Inference time (ms)	55.6	63.5	84.3

GPU scaling. With one simulator worker and one Policy replica per GPU, four RTX 4090s evaluate the 500 RoboTwin Easy episodes in 5,934 s against 21,582 s on one GPU: 3.64× faster, a parallel efficiency of 0.91 (three runs each; Table 16).

Cross-instance execution. On our evaluation platform, four concurrent instances complete the full MetaWorld MT50 protocol, 2,500 episodes, and the platform merges their shared reports into one result.

Related Work

Benchmarks and evaluation scope. Manipulation benchmarks differ in what a successful episode establishes. LIBERO organizes tasks to study knowledge transfer [16], and LIBERO-Plus tests robustness under systematic perturbations [11]. CALVIN evaluates sequences of language-conditioned tasks [19]; RoboTwin emphasizes bimanual manipulation and scalable task generation [5, 20]; and RoboCasa provides diverse household scenes and tasks [21], including the GR-1 Tabletop collection released with GR00T N1 [22].

Evaluation pipelines and reproducibility. Native evaluators couple task selection and reset with model inference, rollout termination, and score aggregation. OpenVLA supplies benchmark evaluation scripts, and StarVLA unifies evaluation interfaces within its model-development codebase [13, 25]. vla-eval evaluates vision-language-action models across benchmarks with batch-parallel evaluation [8], and RLinf, an infrastructure for reinforcement learning of embodied models, runs parallel environment evaluation [32]. XPolicyLab specifies common observation, action, and trajectory schemas with a minimal policy adapter and separates policy inference from environment execution in a client/server architecture, so that $N$ policies and $M$ environments need $O(N + M)$ integrations [31]; RoboDojo evaluates such policies with heterogeneous parallel simulation in Isaac Sim [6]. RoboScope Kit shares the integrate-once goal and adds what evaluation speed needs: separately sized pools of simulator workers and Policy replicas, a measured concurrency search, and runtime optimization declared through the Policy’s observation stride. SimplerEnv relates simulated to real-robot performance [15]. Gymnasium standardizes the environment boundary [28]; RoboScope Kit standardizes the complementary model boundary and keeps benchmark-specific rollout logic in adapters. Findings on the sensitivity of robot imitation learning [18] and the uncertainty of few-run reinforcement-learning evaluation [1, 12] motivate RoboScope Kit’s frozen manifests, archived configurations, paired episode records, and separate reporting of evaluation and search costs.

Simulation and execution efficiency. MuJoCo, robosuite, and SAPIEN supply physics and task infrastructure for manipulation [27, 30, 36]. Isaac Gym parallelizes simulation on the GPU, and ManiSkill3 parallelizes both simulation and rendering on the GPU [17, 26]; EnvPool accelerates supported environments with a parallel execution engine [29]. RoboScope Kit coordinates separate simulator workers and Policy replicas around existing benchmark runtimes, which complements these engines. Distributed reinforcement-learning systems separate environment stepping from learning [9], and SEED RL and Sample Factory move inference into batched accelerator workers [10, 24]. RoboScope Kit applies this separation to evaluation, where the checkpoint is fixed and the objective is complete, reproducible episode records rather than sample throughput for learning. Its chunk-boundary observation removes rendering that an action-chunk Policy never reads.

Policies and the model–benchmark contract. Generalist and vision-language-action policies vary in sensory inputs, action representations, and inference procedures [4, 13, 23]. Diffusion Policy and ACT predict action sequences executed with receding-horizon or chunked control [7, 34], and OpenVLA-OFT studies speed and success under changes to action decoding and adaptation [14]. These choices determine when a policy needs observations, what state it keeps across steps, and how its outputs must be interpreted. RoboScope Kit’s Policy boundary therefore keeps checkpoint-specific processing with the model, declares chunk lengths and observation strides, and keys state by episode.

Discussion

Reporting search and tuning time beside evaluation time shows when automation pays off: RoboScope Kit’s 506-second search adds 75% to its LIBERO-Long evaluation time and leaves it 2.42× faster than the official evaluation end to end. On short workloads the search weighs more: on a 100-episode RoboTwin Easy subset it takes about 60% of each launch (Table 13).

Conclusion

Profiling the official evaluators shows that simulation occupies CPU cores while one loaded model leaves the GPU mostly idle, and that an action-chunk Policy reads only one of every 16 RoboTwin renders. RoboScope Kit integrates each model once as a Policy and each benchmark once as an adapter around its own evaluator; a fixed configuration matches all 500 official LIBERO-Long outcomes. Scheduling separates simulators from Policy replicas and sizes both by measurement: on one GPU, a second replica lets twelve workers reach 4.76× the throughput of one worker and one replica, and with the twelve workers and three replicas selected by the concurrency search, LIBERO-Long evaluation runs 4.45× faster than the official evaluation with equivalent outcomes, or 6.16× with CUDA MPS, where the search selects sixteen workers and four replicas. Four GPUs scale RoboTwin Easy at a parallel efficiency of 0.91, and together with runtime optimization, which renders only the observations a chunked Policy reads, bring 500 episodes from 359.7 to 14.4 minutes, matching the single-GPU baseline on every success outcome and terminal step. Deterministic sharding extends an evaluation across GPU instances.

Policy and Benchmark Contracts

Public Policy Lifecycle

Figure 8 shows the lifecycle of one replica, and Table 2 lists the lifecycle methods. The Control Runtime invokes load once per replica process. Several simulator workers interleave episodes through one replica, so stateful Policies use the episode identifier as their state key.

Evaluation Data Types

FeatureSpec canonical name, image/state/language kind, required flag, description, dtype, and optional shape.

ActionSpec canonical action name, description, optional dimension, and dtype.

BenchmarkSpec benchmark, suite, protocol version, ordered features, and action spec.

SessionContext session identifier, checkpoint path, benchmark spec, and metadata.

EpisodeContext episode identifier, task identifier, and benchmark metadata.

Observation episode identifier, step index, named features, and optional timestamp.

ActionChunk one action vector or a two-dimensional action sequence.

PolicyCapabilities supported benchmarks and suites, required features, maximum batch, supported chunk lengths, action representations, and observation strides.

The Control Runtime validates that declared action representations and strides refer to supported benchmarks and resolves one execution contract per benchmark, which it records with the run.

Benchmark Contracts

Table 3 summarizes the observation and action contracts.

Benchmark	Protocol	Required canonical features	Action contract
LIBERO	libero-v1	images.primary 256 × 256 × 3 uint8; images.wrist; 16-D float32 proprioception; instruction	7-D float32 end-effector delta plus gripper
LIBERO-Plus	libero-plus-v1	Same named contract as LIBERO	7-D float32 end-effector delta plus gripper
RoboTwin	robotwin2-v1	Head, left-wrist, and right-wrist RGB; dual-arm joint state; instruction	14-D float32 absolute dual-arm joint and gripper target
RoboCasa	robocasa-v1	Primary RGB; optional wrist RGB; GR-1 state; instruction	29-D float32 arms, hands, and waist target
CALVIN	calvin-v1	Static RGB; gripper RGB; proprioception; instruction	7-D float32 relative end-effector and gripper action
MetaWorld	metaworld-v1	images.primary 480 × 480 × 3; 4-D end-effector position and gripper aperture; instruction	4-D float32 end-effector delta and gripper command

Work Queue State Machine

A work item carries a unique ordinal and identifier, a grouping hint, a benchmark payload, and the expected episode-record count. After completing an item, a worker first claims a pending item with the same grouping hint. LIBERO groups items by task, so a worker keeps its task environment instead of rebuilding it; each RoboTwin item has its own group, so an idle worker takes the next window of any task. Queue directories encode state: pending holds available items written by the Control Runtime; running holds atomically claimed items named with worker ownership; partial holds append-only records for a running multi-episode item; and completed holds idempotent item results used by aggregation. Completion validates the item and group identifiers and the expected record count. A repeated completion with identical content is accepted; differing content raises an integrity error. Worker recovery moves each unfinished claim back to pending and clears its partial state.

Recorded Evaluation Artifacts

Episode records contain the episode key, success, steps, elapsed time, and benchmark metadata. Metrics record inference batches, queue wait, prediction time, transport time, errors, and per-replica summaries. GPU samples give utilization and memory per device; CPU samples give consumed cores and cgroup throttling ratios. Run-level artifacts preserve software revisions, checkpoint and manifest hashes, the selected $K/g$, the probe history, queue state, results, and analysis outputs.

Experimental Details

Workloads and Checkpoints

LIBERO-Long uses task IDs 0–9, initial states 0–49, seed 7, ten settling steps, and EGL rendering, with the published 30K-step checkpoint StarVLA/Qwen2.5-VL-OFT-LIBERO-4in1. RoboTwin Easy uses RoboTwin 2.0 demo_clean with 50 tasks and episodes 0–9 per task, a validated seed cache, and StarVLA/Qwen3-VL-OFT-Robotwin2. The RoboTwin checkpoint carries future_action_window_size=15, which StarVLA schema 0.21 maps to action_horizon=16.

StarVLA Revisions

LIBERO-Long runs follow one of two StarVLA revisions with the same checkpoint and manifest. The September 2026 runs follow 7e4a473 (30 June 2026), the upstream head at the time: the official reference run and the fixed runs of Table 12. The October 2026 runs follow eaa51c4 (24 February 2026), the revision vla-eval v0.7.0 pins: the cross-system comparison, the worker and replica sweep of Table 18, and the profiles of § 2.2. The official evaluation and vla-eval run the code of the batch’s revision. RoboScope Kit builds the model from StarVLA 7e4a473 in both batches, and its Policy applies the LIBERO client steps of the batch’s revision. The two LIBERO clients differ in three steps: image resizing (PIL bilinear versus OpenCV area interpolation), action unnormalization (the server’s training pipeline versus min/max in the client), and the point at which the gripper command is binarized (after versus before unnormalization). A fixed $K1/g1$ RoboScope Kit run matches the batch’s official evaluation on all 500 success outcomes: 439/500 under 7e4a473 and 451/500 under eaa51c4.

Workload	$K/g$	$n$	Success/run	Eval. time (s)	Steps/s
LIBERO-Long	1/1	2	439/500	3,069 ± 54	45.49 ± 0.81
RoboTwin Easy	1/1	3	231/500	21,582 ± 389	9.79 ± 0.17
RoboTwin Easy	4/4	2	235/500	5,646 ± 193	36.86 ± 1.26

Baseline Configurations and Tuning

The official StarVLA evaluation runs its policy server and LIBERO evaluator from StarVLA commit eaa51c4, with one environment and one model. vla-eval runs release v0.7.0, which pins the same commit behind its model server, and RLinf runs its main branch of 22 October 2026 with two local changes that align its action unnormalization and gripper convention with the StarVLA checkpoint. All systems share the LIBERO-Long manifest, the checkpoint file, a chunk length of 8, and the timing boundary of § 5.1.

RLinf was tuned once offline (Table 4). Each probe evaluates 30 episodes, initial states 0–2 of every task, within a 30-minute search budget. A probe is eligible if it covers all 30 episodes and has at least as many successes as the starting configuration; RLinf keeps the eligible probe with the shortest 500-episode time projected from its probe epochs. The starting configuration and the 20-environment configuration both scored 30, and the latter was selected. Tuning took 1,704.3 s.

Sampled actions in RLinf’s StarVLA evaluation. RLinf’s evaluation rollout samples StarVLA’s actions instead of taking their mean. Its rollout worker forwards the evaluation mode only to an allow-list of model types that excludes StarVLA, so the StarVLA policy runs in its default training mode, which always enables sampling regardless of temperature_eval. The OFT action head then draws each normalized action from a Gaussian around the predicted mean with standard deviation $\exp(-2.5) \approx0.082$, on actions normalized to [−1, 1]. The LIBERO-Long RLinf launches in this report sample in this way and were not rerun. Their sampled actions are consistent with the lower agreement between repeated RLinf launches (86.2–88.4%) than between repeated launches of RoboScope Kit (Search) (97.8–99.0%). No configuration value disables the sampling, and the evaluation mode reaches every other model type in the allow-list, so this is likely a bug in RLinf’s StarVLA integration; it is present in RLinf’s main branch as of 30 October 2026. Deterministic evaluation needs RLinf to forward the evaluation mode to the StarVLA policy, a one-line change to the rollout worker. We leave the LIBERO-Long launches as run, and the RoboTwin comparison applies this change so that RLinf evaluates the mean action, as the other systems do.

vla-eval was tuned over its shard count, the number of model servers on the GPU, each server’s maximum batch size, and its batch wait (Table 5). With $q$ servers, shard $s$ connects to server $s \bmod q$. Probes evaluate 100 episodes, initial states 0–9 of every task, and are ranked by steady-state throughput: policy steps per second from 30 s after the shards start until the first shard finishes. The first stage crosses 8 to 20 shards with one and two servers, each batching up to the number of shards it serves with a 5 ms wait. Throughput All runs use the LIBERO-Long checkpoint, a chunk length of 8, and 12 pinned cores. Steady steps/s and busy cores are averages from 30 s after the shards start until the first shard finishes. Batch is each server’s maximum batch size.

Stage	Shards	Servers	Batch	Wait (ms)	Episodes	Successes	Evaluation (s)	Steady steps/s	Busy cores
0	10	2	10	5	500	451	1,215.9	–	–
1	10	1	10	5	100	90	317.1	107.9	9.4
1	10	2	5	5	100	91	278.5	110.4	9.8
1	8	1	8	5	100	88	355.2	86.0	7.7
1	8	2	4	5	100	87	356.6	88.2	8.0
1	12	1	12	5	100	88	318.9	123.6	10.7
1	12	2	6	5	100	88	295.5	126.7	11.2
1	16	1	16	5	100	90	235.6	133.6	11.3
1	16	2	8	5	100	89	255.2	133.0	11.7
1	20	1	20	5	100	89	233.7	135.8	11.5
1	20	2	10	5	100	92	226.4	133.4	11.8
2	24	1	24	5	100	89	245.8	136.8	11.6
2	24	2	12	5	100	85	259.5	133.9	11.9
2	14	1	14	5	100	90	266.7	131.7	11.1
2	16	1	16	0	100	89	259.6	133.8	11.3
2	16	1	16	20	100	89	250.8	133.6	11.3
2	20	1	20	0	100	90	229.9	135.8	11.5
2	20	1	20	20	100	92	226.6	134.4	11.5
2	20	1	8	5	100	92	227.3	134.1	11.5
2	20	3	7	5	100	90	227.4	132.5	11.9
3	20	1	20	5	500	445	1,026.7	142.5	11.5
3	24	1	24	5	500	450	1,017.7	141.8	11.6
3	20	2	10	5	500	444	1,038.9	141.6	11.8

rises from 86.0 steps/s at 8 shards to 133.0–135.8 steps/s at 16 and 20, where 11.3 to 11.8 of the 12 cores are busy; a second server changes throughput by at most 2.6%. The second stage varies the shard count, wait, maximum batch, and server count around that point, and every probe falls within 131.7–136.8 steps/s. The third stage runs all 500 episodes at three configurations of that range and keeps the shortest evaluation time: 24 shards on one server, with batches of up to 24 and a 5 ms wait, at 1,017.7 s. Together with one earlier 500-episode run at 10 shards and two servers, the 23 tuning runs took 9,780.1 s.

Evaluation-Time Profile

StarVLA evaluator. Figure 2 profiles the StarVLA policy server and LIBERO evaluator (revision eaa51c4, bf16) with the settings of the official LIBERO-Long evaluation (no video logging, one BLAS thread) on LIBERO-Long, the first five initial states of each task (50 episodes), seed 7, on one RTX 4090 with a 20-core quota; shares are means of three timed runs. A profiling wrapper instruments the evaluator’s environment creation, reset, initial-state loading, step, and close, the client’s action request, and the server’s request handling, recording monotonic start and end times; each step also accumulates MuJoCo physics time and records each rendering call. nvidia-smi samples GPU utilization at 1 Hz, and kernel time and launches per inference come from an Nsight Systems trace of the server over ten episodes.

RoboScope Kit configurations. Table 6 gives the stage-timed RoboScope Kit runs behind § 5.5. The workload is the same 50 episodes on the same host, using the checkpoint and Policy of the cross-system comparison. Each configuration runs three times. Stage timing records monotonic start and end times in every process and marks the same ranges with NVTX; per control step it accumulates MuJoCo physics, robot control and forward kinematics, observable updates, and rendering. Nsight Systems traces CUDA kernels, NVTX ranges, and OS calls; py-spy samples Python stacks in every process; PyTorch Profiler counts operators and kernel launches per inference; NVML sampling gives GPU utilization.

Table 7 splits one inference at $K1/g1$. Wall times come from a run that synchronizes the GPU at each stage boundary; kernel time is the union of CUDA kernel intervals inside each stage, measured under Nsight Systems. The model runs one forward pass of Qwen2.5-VL-3B and an MLP action head, with no autoregressive decoding.

Stage	Wall (ms)	Kernel time (ms)	GPU busy (%)	Kernels
Image processing and tokenization	4.7	0.0	0	0
Vision encoder	23.9	9.7	41	1,486
Language model	24.6	14.7	60	1,641
Action head	0.6	0.4	–	30
Whole inference	57.7	26.1	45	3,220

Each request from a worker to a replica carries about 525 KB; the worker encodes it in 2.6–2.7 ms and the replica decodes it in 2.1–2.4 ms. py-spy places the simulation hot spots in MuJoCo forward kinematics, the physics step, the operational-space controller, and model compilation from XML at episode reset.

Extended Results

Outcome Agreement

Extends § 5.2.

Official reference evaluations. The three official LIBERO-Long launches each score 451/500 (90.2%) with identical success outcomes; per-task success counts are 48, 50, 46, 46, 49, 41, 42, 30, and 49. The fixed $K1/g1$ RoboScope Kit run of the worker and replica sweep matches all 500 success outcomes of each launch, and its terminal steps match on 500, 499, and 499 episodes. In the September batch (Appendix § 10.§ 2), the official evaluation scores 439/500 (87.8%) and a fixed $K1/g1$ RoboScope Kit run matches all 500 success outcomes; per-task success counts are 50, 49, 50, 48, 42, 50, 37, 36, 30, and 47.

Per-episode outcome differences. On RoboTwin Easy, the four-GPU stride-16 run matches each of the three single-GPU $K1/g1$ runs on every success outcome and terminal step, and the four-GPU stride-1 runs agree with one another across K4 to K32 and 48 and 80 cores. On LIBERO-Long, Table 18 gives the success-outcome and terminal-step agreement of every worker and replica configuration with $K1/g1$.

Performance

Extends § 5.3.

Per-run timings and repeats. StarVLA and RLinf ran on the first host with a 12-core CPU quota, launched in interleaved order on the same GPU (Table 10); the speedup of RLinf over StarVLA is the geometric mean of the three paired ratios, with a 95% $t$ interval over the log ratios. RoboScope Kit (Search) ran on another RTX 4090 of the first host, and vla-eval on the second host, both with all processes pinned to 12 cores; their speedups are ratios of geometric-mean evaluation times with a 95% Welch $t$ interval on the log times. The worker and replica sweep and the concurrency searches ran on the second host with a 20-core quota. Success is the mean over launches; time and throughput are medians.

Order	System	Launch	Successes	Evaluation (s)	End-to-end (s)	Steps/s
1	StarVLA	1	451	2,989.9	3,116.5	45.8
2	RLinf	1	453	1,881.7	1,934.8	74.2
3	RLinf	2	452	1,907.5	1,955.3	72.0
4	StarVLA	2	451	2,968.0	2,983.5	46.1
5	RLinf	3	450	1,806.7	1,857.8	77.3
6	StarVLA	3	451	2,997.9	3,012.5	45.6

Table 8 lists the values behind Figure 4, Table 9 the RoboScope Kit launches at $K12/g3$ with their CPU and GPU use, Table 10 the StarVLA and RLinf launches, and Table 11 the vla-eval launches and RoboScope Kit at a fixed $K8/g2$.

Launch	MPS	Successes	Search (s)	Eval. (s)	E2e (s)	Steps/s	CPU cores	GPU util. mean (%)	Peak GPU mem. (GiB)
Search 1, second host	Off	448	506.0	679.0	1,192.6	202.5	7.8	97.0	33.6
Search 2, second host	Off	447	504.1	675.2	1,187.0	204.6	7.8	97.2	33.6
Search 3, second host	Off	448	506.9	679.7	1,193.9	202.5	7.8	96.8	33.5
Fixed 1, first host	Off	449	–	671.2	737.6	204.9	7.7	96.8	33.8
Fixed 2, first host	Off	448	–	673.5	739.1	204.6	7.8	96.6	33.7
Fixed 3, first host	Off	451	–	669.3	731.0	204.6	7.8	96.6	33.7
Search+MPS 1, first host	On	452	503.9	484.6	997.1	281.8	10.6	95.0	44.7
Search+MPS 2, first host	On	447	504.8	486.9	1,000.5	282.5	10.6	95.6	44.5
Search+MPS 3, first host	On	449	502.2	483.4	994.2	283.3	10.5	95.8	44.4

Order	System	Successes	Evaluation (s)	End-to-end (s)	Steps/s
1	RoboScope Kit $K8/g2$	450	762.2	805.5	179.1
2	RoboScope Kit $K8/g2$	451	744.4	785.5	183.4
3	RoboScope Kit $K8/g2$	445	756.4	804.4	181.3
4	RoboScope Kit $K8/g2$	448	754.6	841.5	181.8
5	RoboScope Kit $K8/g2$	449	745.8	789.2	183.2
6	vla-eval	446	1,007.0	1,023.6	135.1
7	vla-eval	451	1,016.2	1,041.8	133.5
8	vla-eval	450	1,036.0	1,055.1	132.8

Ablation Study

Extends § 5.4.

The three searches are the LIBERO-Long launches of Table 13: one RTX 4090, a 20-core quota, RoboScope Kit at commit 12f7927; each searches and then evaluates the selected $K12/g3$. The fixed launches evaluate $K12/g3$ without search on the cross-system host with all processes pinned to 12 cores, using the fixed-configuration runner of an earlier RoboScope Kit build; they are the RoboScope Kit (Search) row of Table 8. The Search+MPS launches run the diagonal search with CUDA MPS on the cross-system host with 12 pinned cores, select $K16/g4$, and are the RoboScope Kit (Search+MPS) row. CPU cores are the mean cores the container consumed during the formal evaluation, from cgroup CPU time; CPU throttling stays below 0.7% of the quota. GPU utilization is the mean of 1 Hz NVIDIA samples during the formal evaluation, and peaks at 100% in every launch. Every launch passes the LIBERO-Long audit; the MPS launches observe four MPS clients. The StarVLA, RLinf, and vla-eval launches did not record CPU or GPU use; Appendix C profiles the StarVLA evaluator separately.

System	Success (%)	$\Delta$ success vs. StarVLA, pp [95% CI]	Agreement with StarVLA (%)	Evaluation time (s)	Throughput (steps/s)	Speedup vs. StarVLA
StarVLA	90.2	reference		2,990	45.8	1.00
vla-eval	89.8	$-0.4$ [$-1.6$, $+0.7$]	91.7	1,016	133.5	2.93
RLinf	90.3	$+0.1$ [$-2.6$, $+3.3$]	88.1	1,882	74.2	1.60
RoboScope Kit (Search)	89.9	$-0.3$ [$-0.8$, $0.0$]	97.9	671	204.6	4.45
RoboScope Kit (Search+MPS)	89.9	$-0.3$ [$-1.3$, $+0.8$]	96.5	485	282.5	6.16

End-to-end is the monotonic wall time of the whole process. RLinf was tuned once offline (1,704.3 s); including that tuning, its median end-to-end time is 3,639 s.

vla-eval runs the configuration selected in Table 5; its search took 9,780.1 s, and including it, median end-to-end time is 10,822 s. Every launch passes the LIBERO-Long audit: 500 unique episodes with the frozen initial states, seed 7, and native outcomes equal to the observer’s.

Fixed-configuration runs. Seven initial runs use fixed configurations: two LIBERO-Long $K1/g1$ runs on a 12-core host, three RoboTwin Easy $K1/g1$ runs on one GPU, and two RoboTwin Easy $K4/g4$ runs on four GPUs with a 48-core quota. Both LIBERO-Long runs score 87.8% with identical success outcomes. RoboTwin Easy scores 46.2% in all three single-GPU runs and 47.0% in both four-GPU runs, with success outcomes and terminal steps identical within each configuration (Table 12).

Concurrency search launches. The concurrency search runs on one RTX 4090 with a 20-core quota, without CPU pinning or CUDA MPS, three times per workload. Each launch is one process that searches and then evaluates at the selected $K/g$. LIBERO-Long uses the frozen 500-episode manifest; the RoboTwin Easy subset takes the first two episodes of each of the 50 tasks, with observation stride 16 and a 60-minute search budget. An audit checks that the formal $K/g$ equals the selection, that no worker fails or runs out of memory, and that the episode records cover exactly the manifest; on LIBERO-Long, an observer additionally records every formal episode and checks its initial state, outcome, and step count against the manifest and the native records. All six launches pass. In every LIBERO-Long search without MPS, the worker phase reaches the knee at $K4/g1$: doubling to $K8/g1$ gains only 0.4–9.6%, under 25%. The replica phase then stops at $K12/g3$: $K16/g4$ gains only 2.1–2.4%, under 3%. Three further LIBERO-Long launches run the search with CUDA MPS on the cross-system host, with all processes pinned to 12 cores; all pass the same audit. Their worker phase also reaches the knee at $K4/g1$ ($K8/g1$ gains 7.6–8.6%), their replica phase keeps $K16/g4$ (8.2–9.4% over $K12/g3$), and $K20/g5$ is skipped because its predicted memory, 54.7–55.2 GiB, exceeds the 46 GiB usable on the GPU. Table 14 lists the fixed-configuration grid under MPS; its $K16/g4$ launch evaluates in 482.5 s with 450 successes, within 0.5% of the searched launches. In every RoboTwin search, the $K16/g1$ probe runs out of GPU memory, so the search keeps $K8/g1$; the three RoboTwin launches match on every success outcome and terminal step.

Controlled Simulation Runtime ablations. The ladder runs one RoboTwin task with three seeds on one RTX 4090; each variant adds one Simulation Runtime change to the previous one and runs once, with a Policy that holds the current joint state in place of a model.

Optimized RoboTwin configurations. The run uses $K16/g4$, two-episode work items, observation stride 16, and the clutter-metadata cache. Against each of the three single-GPU $K1/g1$ runs, all 500 episode keys, success outcomes, and terminal steps match. Its evaluation time is 6.88× lower than the fixed $K4/g4$ mean.

Each launch evaluates the 500 frozen episodes with the fixed-configuration runner of the earlier RoboScope Kit build used for the RoboScope Kit (Search) row, and passes the LIBERO-Long audit. The $g = 1$ and $K4/g2$ launches ran later on a second instance of the same configuration, with the same runner and pinning. The diagonal search under MPS selects $K16/g4$ (Table 13). CPU cores and GPU utilization are means during the formal evaluation.

Scalability

Extends § 5.5.

Scaling measurements. Each configuration in Table 18 evaluates the full LIBERO-Long manifest once on the 20-core host, with the same checkpoint and Policy as the cross-system comparison. Table 18 varies simulator workers $K$ and Policy replicas $g$ on one GPU.

Measurement	Value
Evaluation time	862 s
Evaluation phase (metrics recorder)	855 s
Policy loading	65.8 s
Runtime preparation	0.2 s
End-to-end time	928 s
Successes/episodes	231/500
Total action steps	211,220
Out-of-memory events/worker failures	0/0

One RTX 4090 and a 20-core CPU quota on a second host. Same checkpoint, 500 episodes, and Policy as Table 8. The first row is the diagonal concurrency search, three launches with RoboScope Kit at commit 12f7927 (median search 506 s, excluded from evaluation time): successes are the mean, time, throughput and GPU utilization the median, speedup the geometric mean of the per-launch ratios, and agreement the mean over launches. GPU utilization is the mean sampled utilization over the run. The $g = 3$ and $g = 4$ launches ran later on a second instance of the same configuration, with the same runner and environment; their GPU utilization was not sampled, and their speedup and agreement are against the same $K1/g1$ launch. Efficiency is speedup divided by $K$. Agreement pairs each run with $K1/g1$ on identical episodes: success outcome only, or success outcome and terminal step.

Benchmark Compatibility

Supported Benchmark Configurations

The PyPI package supplies the Policy API, the Control Runtime, the Simulation Runtime, and the adapters; each benchmark supplies simulator packages, datasets, manifests, render dependencies, and shared-storage assets.

Failure Classification

Run records distinguish contract mismatches, Policy failures, Simulation Runtime failures, resource limits, record-integrity failures, and infrastructure errors, each tied to the stage that raised it. Worker exits and allocator out-of-memory errors map to requeue and replay; Policy-replica exits and integrity errors abort the evaluation.

Benchmark	Suites	Default work item	External benchmark assets
LIBERO	libero_spatial, libero_object, libero_goal, libero_10, libero_90	Episodes of one task	Pinned Python environment, simulator, dataset, initial states
LIBERO-Plus	libero_spatial (2,402 tasks), libero_object (2,518), libero_goal (2,591), libero_10 (2,519)	Episodes of one variant	Pinned LIBERO-Plus environment, simulator archive, dataset, initial states
RoboTwin	demo_clean, demo_randomized; 50 tasks	One task episode window	RoboTwin source and environment, checkpoint, validated seed cache
RoboCasa	tabletop; 24 GR-1 tasks	One task with multiple episodes	GR-1 simulator and task source, dataset, EGL environment
CALVIN	task_D_D, task_ABC_D, task_ABCD_D	Consecutive sequences	CALVIN source, dataset, config, official 1,000-sequence manifest
MetaWorld	mt50; 50 tasks	One task with all its episodes	Pinned MT50 environment and protocol artifact