Valency Hub
alphaFeedback
PDF

WorksComputer Science

Know When to Escalate: Critic-Guided Adaptive Execution for Cloud-Local Embodied Agents

Wei Zhang, Lirong Xiang, Charles Fleming

Published Fresh — added to the Hub within the last 30 daysv119 min read

AI contributions
Not recorded for this paper
View details

Abstract

Recent advances in foundation-model agents have enabled increasingly capable embodied systems for long-horizon planning and interaction. However, embodied agents require fast, closed-loop decisions as they continuously perceive and act on a changing environment. Deploying strong cloud agents leaves the platform idle for seconds between consecutive actions, while lightweight local models respond quickly but are more vulnerable to errors. Existing hierarchical architectures typically assign models fixed roles, whereas conventional model routers make stateless per-request decisions. We present CARE, a critic-guided adaptive execution framework that dynamically determines when to escalate control from a local executor to a stronger cloud model. A lightweight critic model monitors execution reliability and triggers escalation when local execution becomes unreliable; once escalated, control persists until the agent recovers, avoiding step-wise switching. To support such control transfer, a shared handoff memory maintains execution context and task-level progress across executors, enabling the model to intervene without restarting from scratch. Across diverse embodied benchmarks, CARE improves task success by 6.3-18.3 percentage points over the strongest evaluated collaboration baseline. CARE further approaches or exceeds Cloud-only success while reducing median time-between-action latency by 28.8-82.6%.

Introduction

Executing long-horizon robotic tasks requires embodied agents to repeatedly interpret observations, select grounded actions, and update their decisions based on action outcomes. Recent advances in generative foundation models have strengthened these capabilities [1, 2, 3, 4], enabling such models to coordinate perception, decision making, and embodied skills as general-purpose reasoning modules [5, 6, 7, 8]. Because model inference lies directly within the perception–action loop, high inference latency can leave the agent acting on stale observations, potentially causing failure in time-sensitive interactions. To balance strong reasoning with responsive closed-loop interaction, recent hierarchical and slow–fast architectures pair powerful foundation models with lightweight local executors or specialized controllers [9, 10].

Despite these advances, long-horizon embodied execution remains challenging because each action affects what the agent observes and must do next. Consider an agent midway through a rearrangement task. An earlier action has displaced the target object, causing the next grasp to fail even though the resulting observation appears similar to the previous one. If the agent overlooks this feedback, it may repeat the same action, lose track of task progress, and compound earlier errors [9, 11]. Execution difficulty also changes along the trajectory.

A lightweight executor may handle routine steps well but struggle after ambiguous observations, failed actions, or deviations from the expected state. The appropriate executor may therefore change as interaction unfolds.

This challenge is particularly pronounced in cloud–local embodied systems, where execution must continuously balance model capability against interaction latency. As shown in Figure 1, local models provide responsive interaction but may struggle in difficult states, whereas cloud models offer stronger reasoning at the cost of seconds per decision [1, 2, 10]. Existing hierarchical systems typically assign these models fixed planner–executor roles, implicitly assuming that the designated executor remains appropriate throughout the task. Replanning methods can revise plans or actions in response to feedback, but generally do so without reconsidering which model should execute the subsequent trajectory [7, 12, 13]. Per-query model routing likewise selects an executor independently for each request, without explicitly modeling how executor suitability evolves within an embodied trajectory [14].

Image not extracted

Figure 1. No fixed assignment of executors fits a whole task. The same rearrangement instance executed by (a) the local model only, (b) the cloud model only, and (c) CARE. By adapting the executor as interaction unfolds, CARE completes the task while retaining responsive execution.

We present CARE, a critic-guided adaptive execution framework for cloud–local embodied agents. Rather than routing individual actions independently, CARE uses a critic model to assess whether local execution should continue under the current interaction state and selectively escalates execution to the cloud when local execution becomes unreliable. Importantly, escalation is treated as a stateful control transfer rather than a one-step model selection: cloud intervention persists through the current recovery, avoiding premature handbacks. Once the critic supports local execution again, control returns to the local executor to recover low-latency interaction. Meanwhile, a shared handoff memory preserves observations, completed progress, and failure context across model handoffs, allowing both executors to maintain a consistent execution state. By combining critic-guided control transfer with stateful handoff memory, CARE avoids both persistent cloud invocation and myopic per-action routing while adapting execution authority to the evolving difficulty of the task.

Our main contributions are summarized as follows:

  • We introduce a new formulation of cloud–local embodied collaboration as a stateful adaptive execution problem and propose a novel critic-guided execution framework that dynamically determines when execution should be escalated and when local control can be safely resumed.

  • We develop CARE, a novel stateful execution mechanism that combines persistent cloud intervention with shared execution memory, enabling reliable control transfer while preserving execution continuity across model handoffs.

  • We conduct a comprehensive evaluation of CARE across three long-horizon embodied benchmarks spanning diverse observation modalities and execution interfaces. CARE improves task success by 6.3–18.3 percentage points over the strongest evaluated collaboration baseline while reducing median time-between-action latency by 28.8–82.6%.

Related Work

Long-horizon embodied agents

Generative foundation models are increasingly used as high-level planners for embodied agents. Early work studies text-based interactive environments [5, 15] using explicit world knowledge [16], reflective reasoning [13, 17], and plan construction [18]; with vision-language models, planning has extended to visually grounded interaction through scene graph representations [19], memory-augmented visual reasoning [20], and multimodal plans that guide downstream control policies [8]. Other work trains open-source planners by reinforcement learning [21] or supervised finetuning [22]. Another parallel line of work makes interaction latency a first-class design constraint by organizing execution hierarchically, pairing a computationally intensive foundation model with faster specialized models or policies. This hierarchy is instantiated through program or skill generation [6], intermediate representation prediction [23], slow–fast reasoning and control [2], and semantic guidance to downstream policies [24], while the execution models themselves have grown more capable through large-scale vision-language-action pretraining [1, 25]. Both lines strengthen what each model contributes, but they fix the assignment in advance: planning improves the instructions an executor receives, and hierarchy fixes which model executes them. We instead study the runtime question left open by both—which model should hold execution authority as a plan is carried out, and when that authority should change.

Embodied execution monitoring

Outside embodied settings, cascades invoke a strong model only when a weak model’s output fails a reliability check, using signals such as answer consistency [26, 27] or few-shot self-verification [28], while routers predict the target model before generation [14]. These methods decide over a single self-contained input: the escalated model answers once, and the decision does not affect what is asked next. Embodied execution violates both properties—control may later return to the local executor, and an incorrect action changes the state at which later decisions are made—so escalation becomes a stateful control problem rather than a routing problem. Within embodied systems, failure-recovery methods use execution feedback to diagnose failures and generate corrective plans [29, 30]. Their primary output is an explanation or a revised plan. Our critic instead controls a different decision variable: which of two executors holds execution authority, and when that authority should change.

Method

Figure Figure 2 illustrates CARE along a single execution trajectory. The agent begins under local control. When a pick action repeatedly fails and progress stalls, CARE changes which executor proposes the next action while leaving the perception–action loop unchanged. A critic assesses whether local execution should continue based on the current observation, local proposal, and execution history. Control transfers to the cloud when needed and remains there until the critic supports returning to local execution. Throughout these switches, shared handoff memory preserves task progress, allowing the incoming executor to continue from the current state rather than restart. We first formalize stateful cloud–local execution (§III-A), then describe the critic-guided transfer mechanism (§III-B) and shared handoff memory (§III-C).

Image not extracted

Figure 2. Critic-guided control transfer over an execution trajectory. Declining local-continuation confidence triggers persistent cloud takeover, while a higher return threshold restores local control after recovery.

Problem Formulation

We consider cloud–local collaboration at the embodied decision layer, where models select grounded actions while low-level motor control is handled by the underlying platform. The agent separates one-time task planning from closed-loop execution. Given a task instruction GG and initial observation o0o_0, a planner πP\pi_P first produces a high-level plan P=πP(G,o0)P = \pi_P(G, o_0), which provides task-level guidance throughout the episode. The resulting plan is carried out by two executors: a lightweight local model πL\pi_L and a high-capability cloud model πC\pi_C. Both operate at the same level of abstraction and output navigation or manipulation actions, but exhibit a capability–latency asymmetry: the cloud executor generally provides stronger reasoning at substantially higher inference latency. At interaction step tt, both executors have access to a shared execution context:

xt=(G,P,ot,Mt),(1)x_t = (G, P, o_t, M_t), \tag*{(1)}

where GG denotes the task instruction, oto_t the current observation, and MtM_t the shared execution memory accumulated along the trajectory.

Let zt∈{L,C}z_t \in\{L, C\} denote the executor in control at step tt. The executed action is then denoted as at=πzt(xt)a_t = \pi_{z_t}(x_t). Executing ata_t yields the next observation ot+1o_{t+1} together with an execution outcome ete_t, which captures environment feedback from the interaction. The control state ztz_t evolves over time according to the critic-guided transfer mechanism introduced in §III-B.

Our objective is to approach Cloud-only task success while reducing closed-loop response time, particularly the delay between consecutive embodied actions. Because each executed action changes the subsequent observation and task state, executor selection is coupled across the trajectory rather than made independently for each request.

Critic-Guided Control Transfer

Assessing Local Execution. The central challenge is to determine when the local executor should retain control and when execution should be escalated to the cloud. This judgment requires more than the current observation, as execution reliability depends on recent progress, prior outcomes, and the trajectory leading to the current state. We therefore employ a large language model πϕ\pi_\phi as the critic, that, conditioned on the shared context xtx_t and a local proposal a^tL\hat{a}_t^L, assesses whether local execution remains reliable enough to continue and assigns one of two control labels, local or cloud. Let

stm=log⁡Pϕ(y=m∣xt,a^tL),m∈{L,C},(2)s_t^m = \log P_\phi\left(y=m \mid x_t,\hat{a}_t^L\right), \qquad m \in\{L,C\}, \tag*{(2)}

denote the log-likelihood assigned to each candidate label. Their relative likelihood is normalized into a reliability-aware confidence score

ct=exp⁡(stL)exp⁡(stL)+exp⁡(stC).(3)c_t = \frac{\exp\left(s_t^L\right)}{\exp\left(s_t^L\right)+\exp\left(s_t^C\right)}. \tag*{(3)}

A larger ctc_t indicates stronger evidence for retaining or returning control to the local executor. Further details on the critic model and implementation setting can be found in §IV-A.

Stateful Control Transfer. The critic score provides an instantaneous estimate of whether local execution should continue, while the control state determines how this signal affects the ongoing trajectory. We use two asymmetric thresholds to govern escalation and return:

zt={C,zt−1=L∧ct≤θesc,L,zt−1=C∧ct≥θret,zt−1,otherwise.(4)z_t = \begin{cases} C, & z_{t-1}=L \land c_t \le\theta_{\mathrm{esc}},\\ L, & z_{t-1}=C \land c_t \ge\theta_{\mathrm{ret}},\\ z_{t-1}, & \text{otherwise}. \end{cases} \tag*{(4)}

The thresholds satisfy θesc<θret\theta_{\mathrm{esc}} < \theta_{\mathrm{ret}}, introducing hysteresis between escalation and return. Once escalated, the cloud executor retains control across subsequent actions until local reliability recovers above θret\theta_{\mathrm{ret}}, avoiding frequent handoffs during multi-step recovery. While cloud control is active, the critic continues to evaluate local proposals to determine when local execution can safely resume.

Speculative Execution. During local execution, cloud action generation may additionally be launched speculatively alongside critic evaluation. If local control is retained, the cloud

Algorithm 1 Speculative Critic-Guided Adaptive Execution
Require: Instruction $G$, plan $P$, initial observation $o_0$, executors $\pi_L,\pi_C$, critic $\pi_\phi$, parameters $\theta_{\mathrm{esc}},\theta_{\mathrm{ret}}$
1: $z \leftarrow L$, $M \leftarrow \varnothing$
2: for each interaction step $t$ do
3: $x_t \leftarrow (G,P,o_t,M)$
4: if $t$ is selected for critic evaluation then
   // Assess and transfer control
5: $r_t^C \leftarrow \operatorname{ASYNC}(\pi_C,x_t)$ ▷ speculative backup
   when $z=L$
6: $\hat{a}_t^L \leftarrow \pi_L(x_t)$
7: $c_t \leftarrow \operatorname{CRITICCONFIDENCE}(x_t,\hat{a}_t^L)$
8: $z \leftarrow \operatorname{UPDATESTATE}(z,c_t)$ ▷ {xref}`Eq. 4`
   // Commit the proposal selected by the new state
9: if $z=C$ then
10: $a_t \leftarrow \operatorname{AWAIT}(r_t^C)$
11: else
12: $a_t \leftarrow \hat{a}_t^L$
13: $\operatorname{DISCARD}(r_t^C)$
14: end if
15: else
16: $a_t \leftarrow \pi_z(x_t)$ ▷ retain the current executor
17: end if
18: $(o_{t+1},e_t) \leftarrow \operatorname{EXECUTE}(a_t)$
19: $M \leftarrow \operatorname{UPDATEMEMORY}(M,(o_t,a_t,e_t,o_{t+1}))$
20: end for

proposal is discarded; if escalation is triggered, the proposal is directly reused, reducing the latency of control transfer without changing the critic score or switching policy. In practice, the critic is invoked every kk steps, where kk is a per-benchmark setting reported in §IV-A; this adds only modest overhead across the P5–P95 latency percentiles (§IV-D).

Shared Handoff Memory

Control transfer creates a state-continuity problem that neither executor can solve alone. An incoming executor observes the current state, but not which subgoals are already complete, which actions have already failed, or how the current state was reached. Lacking this context, it may repeat resolved work or re-attempt a known failure — precisely the behavior escalation is meant to correct. We therefore externalize execution context into a shared handoff memory MtM_t that is owned by the controller rather than by either executor, and is accessible to both executors and the critic.

Execution Context. The shared memory captures execution information at two temporal scales:

Mt=(Mtobs,Mtepi),(5)M_t = \left(M_t^{\mathrm{obs}}, M_t^{\mathrm{epi}}\right), \tag*{(5)}

where MtobsM_t^{\mathrm{obs}} maintains a bounded window of recent observation–action–outcome transitions, providing fine-grained context for how the current state was reached. In contrast, MtepiM_t^{\mathrm{epi}} stores a compact episode-level summary of verified progress, persistent failures, and other execution-relevant information that should remain available beyond the recent window. The two are complementary rather than redundant: a bounded window alone would discard early progress over long horizons, while a summary alone would lose the local context needed to interpret a recent failure. This decomposition preserves both without retaining the full interaction history.

Memory Updates. Updates require no additional model call, so memory contributes no inference latency or cost. After each executed action, the transition is normalized and appended to the window; only state supported by the resulting observation or public execution feedback is promoted to MtepiM_t^{\mathrm{epi}}, where newer verified state supersedes incompatible older records and unresolved failures persist until an observation indicates recovery. Unexecuted proposals, model reasoning traces, and privileged simulator state are never stored: memory grants no information unavailable to a single-executor baseline, only continuity across a change of executor.

Handoff-Consistent Access. At a control transfer, the same serialized memory is exposed to πL\pi_L, πC\pi_C, and πϕ\pi_\phi; it is neither reset nor regenerated for the incoming owner. A handoff therefore changes only the control state ztz_t, leaving the execution context intact, so the incoming executor resumes from the existing task state rather than reconstructing prior progress. This access is bidirectional. For example, if the local executor opens a cabinet but repeatedly fails to grasp the target, the cloud executor receives both the verified open state in MtepiM_t^{\mathrm{epi}} and the recent failure sequence in MtobsM_t^{\mathrm{obs}}. It can therefore begin with recovery rather than repeat completed navigation and manipulation. Actions performed during cloud control update the same memory before control returns, allowing the local executor to continue from the post-recovery state. Section IV-D evaluates how each component affects task recovery and execution continuity.

Evaluation

Experimental Setup

Models and implementation. We use Claude Sonnet 4.6 [31] as the cloud planner and executor and a single zero-shot Qwen3.6-35B-A3B [32] checkpoint as both the local executor and the critic, served with SGLang on H200 GPUs and differing only in prompt and output interface. The critic evaluation interval is k=1k=1 on Habitat and k=3k=3 on EmbodiedBench and VLABench. Generation uses temperature 0 throughout, with a fixed seed for the local models. Escalation and return thresholds are selected per benchmark from a sweep over (θesc,θret)(\theta_{\mathrm{esc}},\theta_{\mathrm{ret}}). §IV-C reports the full sweep on Habitat and shows that performance is stable across a broad band of settings. All methods use the same models and decoding settings and differ only in how these models are coordinated during execution. §IV-D additionally evaluates heterogeneous executor–critic pairings with Qwen3-VL-8B [33] and InternVL3.5-14B [34].

Baselines. We compare against two classes of methods. Single-executor baselines include Local-only and Cloud-only, together with their ReAct-augmented variants, Local-ReAct and Cloud-ReAct [17]. Collaboration baselines cover complementary coordination strategies: Hi Robot* [35] represents a fixed cloud-planner–local-executor hierarchy, RouteLLM* [14] represents model routing during execution, and AdaPlanner [12] represents feedback-triggered cloud replanning with local execution. Hi Robot* and RouteLLM* denote benchmark-specific adaptations of the original methods. “Local-only” and “Cloud-only” refer to the executor rather than to the planning model. All baselines receive the same memory representation; for methods that transfer requests between executors (RouteLLM*), the memory is passed intact to the receiving model, identical to CARE.

MethodHabitatEmbodiedBenchVLABench
SR ↑ (%)CRR ↓ (%)E2E ↓ (s)TBA ↓ (s)SR ↑ (%)CRR ↓ (%)E2E ↓ (s)TBA ↓ (s)SR ↑ (%)CRR ↓ (%)E2E ↓ (s)TBA ↓ (s)
Single-executor baselines
Local-only64.2–10.560.7840.0–11.250.8571.2–16.360.53
Local-ReAct60.8–22.991.5858.8–17.321.1967.5–17.280.85
Cloud-only91.7100.049.624.6180.0100.040.074.4185.0100.023.672.47
Cloud-ReAct88.3100.053.526.1480.0100.077.476.8882.5100.028.174.44
Collaboration baselines
RouteLLM*55.827.432.180.7953.827.033.031.1473.811.718.670.54
Hi Robot*65.04.015.540.7752.59.018.220.9476.225.624.480.58
AdaPlanner74.22.221.661.2658.818.218.190.9781.237.325.423.90
CARE92.524.432.360.8075.034.437.283.1487.533.522.621.31
vs. Cloud-only+0.8–75.6–34.8%–82.6%–5.0–65.6–7.0%–28.8%+2.5–66.5–4.4%–47.0%

Table 1. Reliability–efficiency performance across long-horizon embodied benchmarks. The italic row reports CARE relative to Cloud-only (points for SR and CRR, percent for E2E and TBA). CARE achieves the strongest reliability–latency trade-off among collaboration methods, and is the only one to exceed 70% success on all three benchmarks.

Benchmarks. We evaluate on 120 Habitat [36] rearrangement instances, 80 instances from EmbodiedBench [37], and 80 VLABench [38] instances, covering mobile manipulation, long-horizon household interaction, and tabletop manipulation, respectively. The three benchmarks expose substantially different execution interfaces, including grounded PDDL-style actions, discrete action IDs, and structured macro skills, that allow us to test whether the control mechanism generalizes across different action abstractions and embodiment settings. All methods are evaluated on the same instances.

Metrics. We report task success rate (SR), median end-to-end episode latency (E2E), median time between actions (TBA), average trajectory length (Steps), and cloud request ratio (CRR). E2E measures wall-clock time from task issuance to episode termination, while TBA captures closed-loop interaction responsiveness. We define CRR =NC/(NC+NL)= N_C/(N_C + N_L), where NCN_C and NLN_L count cloud and local executor requests, respectively. All cloud executor requests, including speculative requests, are included in NCN_C. Critic calls are excluded from CRR and reported separately in the latency breakdown analysis (§IV-D).

Overall Performance

CARE maintains task success while improving closed-loop responsiveness. Table I reports task completion and closed-loop response times across the three benchmarks. Relative to Local-only, CARE improves success rate by 16.3–35.0 percentage points. It further exceeds Cloud-only success on Habitat and VLABench, while remaining within 5.0 points on EmbodiedBench. The slight gains over Cloud-only can arise from differences in execution trajectories: CARE may reach a different environment state before invoking the cloud executor, while continuous cloud execution may remain trapped in an ineffective action pattern. Meanwhile, CARE reduces median TBA relative to Cloud-only by 28.8–82.6% and lowers median E2E latency across all three benchmarks. These reductions allow the agent to act more responsively without substantially sacrificing task completion.

CARE achieves high task success with selective cloud use. Figure 3 plots success against cloud request ratio, revealing a different aspect of the comparison. Cloud-only and Cloud-ReAct occupy the far-right edge at 100% CRR, whereas CARE uses the cloud for only 24.4–34.4% of executor decisions and remains on the empirical non-dominated frontier in all three panels. Hi Robot, RouteLLM, and AdaPlanner use comparable or fewer cloud requests but consistently lie below CARE in task success. This pattern suggests that more cloud requests alone do not guarantee higher task success; CARE’s advantage comes from using them at execution states where stronger reasoning is most useful, as examined in §IV-C.

Image not extracted

Figure 3. Success rate versus cloud request ratio across three embodied benchmarks. Cloud request ratio denotes the fraction of executor-model requests generated by the cloud model. Dashed lines indicate the empirical non-dominated frontier among the evaluated methods. CARE approaches or exceeds Cloud-only success with substantially fewer cloud-executed decisions.

CARE shifts toward cloud assistance when local execution is less capable. EmbodiedBench illustrates how this balance changes with the local executor. Local-only achieves 40.0% success, the lowest among the three benchmarks. Accordingly, CARE uses a more aggressive escalation threshold and invokes the cloud more frequently. Its action latency therefore moves away from the local but remains below Cloud-only, while recovering substantially more task success than Local-only. Thus, CARE does not enforce a fixed cloud-use target: its responsiveness depends on how much of the trajectory can be handled locally.

Does CARE Know When to Escalate?

We evaluate the title claim from two perspectives: whether critic-guided escalation improves task success under matched cloud usage, and how control transfers unfold over complete episodes.

Timely escalation improves recovery from local failures. To isolate cases in which cloud intervention can change the outcome, we evaluate Habitat instances on which Local-only fails. We sweep (θesc,θret)(\theta_{\mathrm{esc}}, \theta_{\mathrm{ret}}) to vary CARE’s cloud request ratio (CRR) and compare it with Random and Periodic schedules under matched cloud usage (Figure 4a). Under a tight ∼20%\sim20\% CRR, all three methods achieve similar success, indicating that selective routing provides little additional benefit when the intervention budget is too restrictive. Beyond this regime, CARE converts additional cloud budget into substantially stronger recovery: over the 30–50% CRR range, its success rises from 88.4% to 93.0%, while the state-independent schedules remain between 74.4% and 79.0%. Increasing cloud usage alone does not close this gap: Periodic requires over 70% CRR to reach 93.0%, while Random remains below it even near 80% CRR. CARE itself saturates at 93.0% around 50% CRR, showing diminishing returns from more aggressive thresholds. Thus, successful recovery requires sufficient cloud capacity, but CARE’s advantage comes from allocating that capacity to the execution states where intervention is most useful.

Image not extracted

Figure 4. Understanding adaptive escalation. (a) CARE achieves higher success than matched-budget random and periodic escalation baselines. (b) Episode-level control pathways: 43% of episodes never escalate and 93% of those succeed; among escalated episodes, returned ones succeed at the same 93% rate. (c) Most takeovers span multiple decisions, while most escalated episodes contain only one takeover.

CARE retains local control when possible and sustains cloud control when needed. Figure 4b shows that 43% of episodes complete without escalation, of which 93% succeed, indicating that CARE avoids unnecessary cloud intervention when local execution is sufficient. Episodes returned to local control achieve the same 93% success rate, suggesting that local execution is resumed without sacrificing reliability. Meanwhile, 30% of episodes remain under cloud control until termination, and Figure 4c shows that 81% of takeovers span multiple decisions. Together, these results show that CARE produces state-dependent control transfers rather than independent per-step model choices.

System Analysis and Ablation Studies

CARE reduces task execution time across the latency distribution. All methods issue the same one-time planning call, so Figure 5 reports execution-phase latency with this shared cost excluded. CARE reduces execution latency across the P5–P95 range relative to Cloud-only, indicating that its average-case gains do not come at the expense of tail latency. The reduction comes from replacing cloud execution with lower-latency local execution rather than from shortening trajectories. Critic evaluation contributes only 0.15–2.45 s across the measured percentiles, and speculative control transfer overlaps cloud generation with critic evaluation so that escalation avoids a serial cloud round trip: removing speculation costs 0.05 s at P50 and 2.32 s at P95. CARE therefore improves the latency distribution without introducing substantial control overhead.

Image not extracted

Figure 5. Task execution latency breakdown on Habitat. Across the P5, P50, and P95 episodes, CARE reduces latency by 1.8–7.1× relative to Cloud-only, while critic evaluation and speculative execution add only modest overhead.

Shared handoff memory preserves task progress across control transfers. As shown in Table II, we resume execution from recorded control-transfer points in Habitat, sampled from states where the critic triggered escalation and therefore representative of the difficult states at which handoffs actually occur. Holding the environment state and execution history fixed, we vary only the memory available to the incoming executor. Removing memory reduces success from 100% to 25% and more than doubles both trajectory length and cloud usage, indicating a substantial loss of progress continuity. Episodic memory alone recovers most of the success (90%), while observation memory alone recovers less (75%); combining them restores full success and further reduces execution cost. The two components therefore degrade in the distinguishable ways predicted in §III-C: verified task progress is what an incoming executor most needs, and recent action–outcome context supplies the complementary local detail.

Incoming MemorySR (%) ↑Steps ↓E2E (s) ↓Cloud ↓
Full100.012.1538.83109
Episodic only90.016.0561.80152
Observation only75.015.8063.18144
None25.025.1589.52245

Table II. Effect of handoff memory at matched Habitat execution states (N=20N = 20). Variants differ only in the memory available to the incoming executor.

CARE remains effective across executor and critic choices. To assess whether CARE relies on a particular executor–critic pairing, we replace one component at a time while keeping the other fixed. Table III shows that the default pairing achieves the best success rate, trajectory length, and end-to-end latency. Nevertheless, all substituted configurations retain 83.3–88.3% success, indicating that CARE remains effective across different model families rather than depending on a single pairing. Executor substitutions incur larger increases in trajectory length and latency, whereas critic substitutions preserve execution efficiency more closely. In particular, the

VariantModelSR (%) ↑Steps ↓E2E (s) ↓
DefaultQwen3.6-35B-A3B92.510.4332.36
ExecutorQwen3-VL-8B85.013.3251.18
ExecutorInternVL3.5-14B88.315.2064.83
CriticQwen3-VL-8B86.711.4340.55
CriticInternVL3.5-14B83.311.5745.25

Table III. Executor and critic substitutions on Habitat. The default uses Qwen3.6-35B-A3B for both components; each remaining row replaces one component.

8B critic retains 86.7% success with substantially lower degradation than the executor replacements, suggesting that critic capacity can be reduced when deployment cost is prioritized.

Limitations and Future Work

CARE operates at the embodied decision layer above the underlying low-level controller; however, its behavior under sensing noise, network variability, and hardware execution failures remains unverified. Accordingly, the reported latencies characterize decision-layer responsiveness rather than end-to-end physical robot response. Evaluating CARE on a physical platform is therefore an important next step, particularly because real network conditions may affect the latency and availability of cloud assistance. Beyond physical deployment, systematically characterizing how model scale, task difficulty, and action abstraction shape the boundary between local and cloud competence remains an important future direction.

Conclusion

We presented CARE, an adaptive execution framework for cloud–local embodied agents. CARE evaluates local proposals against the current observation and execution history, transfers control to a stronger cloud executor when local execution deteriorates, and preserves task state across model handoffs. Across Habitat, EmbodiedBench, and VLabench, CARE improves task success by 6.3–18.3 percentage points over the strongest collaboration baselines while reducing median time-between-action latency by 28.8–82.6% relative to Cloud-only. Matched-budget comparisons further show that these gains arise from selecting useful intervention states rather than simply increasing cloud usage. Overall, executor selection should adapt with the embodied trajectory rather than remain fixed throughout a task.

References

  1. [1]M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi et al., “Open-source vision-language-action model,” arXiv preprint arXiv:2406.09246, 2024.arxiv.org/abs/2406.09246
  2. [2]S. Liu, H. Yuan, M. Hu, Y. Li, Y. Chen, S. Liu, Z. Lu, and J. Jia, “Rlgpt: Integrating reinforcement learning and code-as-policy,” Advances in Neural Information Processing Systems, vol. 37, pp. 28 430–28 459, 2024.DOI
  3. [3]Y. Mu, Q. Zhang, M. Hu, W. Wang, M. Ding, J. Jin, B. Wang, J. Dai, Y. Qiao, and P. Luo, “Embodiedpt: Vision-language pre-training via embodied chain of thought,” Advances in neural information processing systems, vol. 36, pp. 25 081–25 094, 2023.arxiv.org/abs/2305.15021
  4. [4]Q. Lv, H. Li, X. Deng, R. Shao, M. Y. Wang, and L. Nie, “Robomp2: A robotic multimodal perception-planning framework with multimodal large language models,” arXiv preprint arXiv:2404.04929, 2024.
  5. [5]M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman et al., “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022.arxiv.org/abs/2204.01691
  6. [6]S. Ahn, W. Choi, J. Lee, J. Park, and H. Woo, “Towards reliable code-as-policies: A neuro-symbolic framework for embodied task planning,” Advances in Neural Information Processing Systems, vol. 38, pp. 75 428–75 459, 2026.arxiv.org/abs/2510.21302
  7. [7]W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar et al., “Inner monologue: Embodied reasoning through planning with language models,” arXiv preprint arXiv:2207.05608, 2022.arxiv.org/abs/2207.05608
  8. [8]Z. Lei, C. Liu, Y. Xiong, M. Xiong, Y. Ding, Z. Zhang, W. Li, and S. Chen, “Towards long-horizon embodied agents with tool-aligned vision-language-action models,” arXiv preprint arXiv:2605.13119, 2026.arxiv.org/abs/2605.13119
  9. [9]M. Chang, G. Chhablani, A. Clegg, M. Dallaire Cote, R. Desai, M. Hlavac, V. Karashchuk, J. Krantz, R. Mottaghi, P. Parashar et al., “Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 65 205–65 268.arxiv.org/abs/2411.00081
  10. [10]Z. Li, X. Xu, Z. Xu, S. Lim, and H. Zhao, “Larm: Large auto-regressive model for long-horizon embodied intelligence,” arXiv preprint arXiv:2405.17424, 2024.arxiv.org/abs/2405.17424
  11. [11]M. Li, S. Zhao, Q. Wang, K. Wang, Y. Zhou, S. Srivastava, C. Gokmen, T. Lee, L. E. Li, R. Zhang et al., “Embodied agent interface: Benchmarking llms for embodied decision making,” Advances in Neural Information Processing Systems, vol. 37, pp. 100 428–100 534, 2024.arxiv.org/abs/2410.07166
  12. [12]H. Sun, Y. Zhuang, L. Kong, B. Dai, and C. Zhang, “Adaplanner: Adaptive planning from feedback with language models,” Advances in neural information processing systems, vol. 36, pp. 58 202–58 245, 2023.arxiv.org/abs/2305.16653
  13. [13]Y. Feng, J. Han, Z. Yang, X. Yue, S. Levine, and J. Luo, “Reflective planning: Vision-language models for multi-stage long-horizon robotic manipulation,” arXiv preprint arXiv:2502.16707, 2025.arxiv.org/abs/2502.16707
  14. [14]I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. Kadous, and I. Stoica, “Routellm: Learning to route llms from preference data,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 34 433–34 448.arxiv.org/abs/2406.18665
  15. [15]M. Shridhara, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht, “Alfworld: Aligning text and embodied environments for interactive learning,” arXiv preprint arXiv:2010.03768, 2020.
  16. [16]S. Qiao, R. Fang, N. Zhang, Y. Zhu, X. Chen, S. Deng, Y. Jiang, P. Xie, F. Huang, and H. Chen, “Agent planning with world knowledge model,” Advances in Neural Information Processing Systems, vol. 37, pp. 114 843–114 871, 2024.arxiv.org/abs/2405.14205
  17. [17]S. Yao, J. Zhao, Y. Du, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022.arxiv.org/abs/2210.03629
  18. [18]J. Singh, W. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” arXiv preprint arXiv:2209.11302, 2022.
  19. [19]Z. Dai, A. Ashgrivaskasi, T. Duong, S. Lin, M.-E. Tzes, G. Pappas, and N. Atanasov, “Optimal scene graph planning with large language model guidance,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 14 062–14 069.arxiv.org/abs/2309.09182
  20. [20]H. Shi, B. Xie, Y. Liu, L. Sun, F. Liu, T. Wang, E. Zhou, H. Fan, X. Zhang, and G. Huang, “Memoryvla: Perceptual-cognitive memory in vision-language-action models for robotic manipulation,” arXiv preprint arXiv:2508.19236, 2025.arxiv.org/abs/2508.19236
  21. [21]Z. Bo, Y. Hu, J. Ma, M. Zhou, J. Yin, Y. Kang, Y. Liu, T. Wu, D. Xiang, and H. Chen, “Reinforced embodied planning with verifiable reward for real-world robotic manipulation,” arXiv preprint arXiv:2509.25852, 2025.arxiv.org/abs/2509.25852
  22. [22]W. Zhang, M. Wang, G. Liu, H. Xu, Y. Jiang, Y. Shen, G. Hou, Z. Zheng, H. Zhang, X. Li et al., “Embodied-reasoner: Synergizing visual search, reasoning, and action for embodied interactive tasks,” in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, pp. 41 178–41 207.
  23. [23]K. Zhang, P. Ren, B. Lin, J. Lin, S. Ma, H. Xu, and X. Liang, “Pivot: Primitive-driven waypoint-aware world model for robotic manipulation,” Advances in Neural Information Processing Systems, vol. 37, pp. 54 105–54 136, 2024.arxiv.org/abs/2410.10394
  24. [24]B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid et al., “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” in Conference on Robot Learning. PMLR, 2023, pp. 2165–2183.arxiv.org/abs/2307.15818
  25. [25]P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai et al., “π_0.5: a vision-language-action model with open-world generalization,” arXiv preprint arXiv:2504.16054, 2025.arxiv.org/abs/2504.16054
  26. [26]L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” arXiv preprint arXiv:2305.05176, 2023.arxiv.org/abs/2305.05176
  27. [27]M. Yue, J. Zhao, M. Zhang, L. Du, and Z. Yao, “Large language model cascades with mixture of thought representations for cost-efficient reasoning,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 21 691–21 728.arxiv.org/abs/2310.03094
  28. [28]P. Aggarwal, A. Madaan, A. Anand, S. P. Potaraju, S. Mishra, P. Zhou, A. Gupta, D. Rajagopal, K. Pappaguthu, Y. Yang et al., “Automix: Automatically mixing language models,” arXiv preprint arXiv:2310.12963, 2023.arxiv.org/abs/2310.12963
  29. [29]Z. Liu, A. Bahety, and S. Song, “Reflect: Summarizing robot experiences for failure explanation and correction,” arXiv preprint arXiv:2306.15724, 2023.arxiv.org/abs/2306.15724
  30. [30]Y. Guo, Y.-J. Wang, L. Zha, and J. Chen, “Doremi: Grounding language model by detecting and recovering from plan-execution misalignment,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 12 124–12 131.arxiv.org/abs/2307.00329
  31. [31]Anthropic, “Claude sonnet 4.6 system card,” https://www.anthropic.com/system-cards, Feb. 2026, version released February 2026.
  32. [32]Qwen, “Qwen3.6-35B-A3B: Agentic coding power, now open to all,” April 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.6-35b-a3b
  33. [33]Q. Team, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388
  34. [34]W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao et al., “Internvl3.5: Advancing open-source multi-modal models in versatility, reasoning, and efficiency,” arXiv preprint arXiv:2508.18265, 2025.arxiv.org/abs/2508.18265
  35. [35]L. X. Shi, B. Ichter, M. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai et al., “Hi robot: Open-ended instruction following with hierarchical vision-language-action models,” arXiv preprint arXiv:2502.19417, 2025.
  36. [36]M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, K. Koltun, J. Malik et al., “Habitat: A platform for embodied ai research,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2019, pp. 9338–9346.arxiv.org/abs/1904.01201
  37. [37]R. Yang, H. Chen, Z. Zhang, M. Zhao, Q. Qian, K. Wang, Q. Wang, T. V. Koripella, M. Movahedi, M. Li et al., “Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents,” arXiv preprint arXiv:2502.09560, 2025.arxiv.org/abs/2502.09560
  38. [38]S. Zhang, Z. Xu, P. Liu, X. Yu, Y. Li, Q. Gao, Z. Fei, Z. Yin, Z. Wu, Y.-G. Jiang et al., “Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks,” in 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025, pp. 11 142–11 152.arxiv.org/abs/2412.18194

Paper details

Publication

vlnc:W_5m5qrfhy
v1 · latestHistory (1)

published · Fresh — added to the Hub within the last 30 days

added ·

Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

CC BY-SA 4.0

Original source

Taxonomy

  • Computer Science

Authors

AI contributions

Not recorded for this paper · View statement →

Looking for related papers…
Contents