WorksComputer Science
Learning Robot Metareasoning: Adaptive Computation Budgets for Real-Time Control
Abstract
Planning while moving faces an historical trade-off between deliberation and reactivity. A planner running concurrently with execution needs time to compute, so the robot must commit to an open-loop action chunk for that duration: a longer commitment buys a better plan at the cost of feedback frequency. Existing controllers fix this action commitment, and consequently the computation budget, before deployment. However, the optimal computation time is highly state-dependent (due to factors such as unmodeled dynamics, safety constraints, and marginal optimisation gains), i.e. dynamic, therefore offline static tuning risks leaving systems blind during critical maneuvers or unable to find global effective solutions. We cast the identification of the optimal computation budget as a semi-Markov decision process (SMDP) in which a single decision defines both the robot action commitments and the time allocated to the solver. To solve this, we train a reinforcement-learning policy conditioned on the current state, the executing plan, and inexpensive planner statistics. Across ground-robot navigation, agile quadrotor flight, and contact-rich manipulation, the learned policy changes its budget with the demands of the scenario, discovering efficient reasoning patterns that outperform fixed-budget baselines. https://lorenzomagnino.github.io/learning-robot-metareasoning-review/
Introduction
Deciding how much to think is as hard as the thinking itself (e.g., deciding what to do) [1]. Biological agents dynamically arbitrate their cognitive resources, continuously balancing rapid, reactive responses against slower, deliberate reasoning based on environmental uncertainty and temporal constraints [1], [2]. This adaptive allocation represents a fundamental speed-optimality trade-off where extended deliberation is penalized by elapsed time [3], [4]. The value of computation (VOC) formally characterizes this trade-off between the marginal gain in decision optimality and the cost of deliberation [5], [6]. Traditional metareasoning frameworks assume a frozen world state during planning, requiring an explicitly defined cost function to penalize extended computation, a paradigm we refer to as the economic value of computation. Embodied autonomous agents, conversely, operate in continuous time where this assumption fails. Because the robot and its environment evolve concurrently with the planner’s execution, heuristic timing penalties are unnecessary. Instead, the physical dynamics inherently penalize computational delay through unmitigated state drift,
thereby hindering high-frequency reactivity and reducing safety margins. We introduce this dynamic trade-off as the physical value of computation, formalized in Sec. III. As standard for computational efficiency, robots typically plan in receding-horizon: the system optimizes a trajectory over a finite horizon, executes the initial segment, and subsequently replans from the evolved state. Model predictive control (MPC) [7], [8] commits to a single control step, resolving the problem at every control period, providing high-rate feedback around a nominal trajectory.
Alternatively, action-chunking architectures generate a sequence of actions to execute in open-loop while the next plan is produced. This commitment can improve motion smoothness and sample efficiency [9]–[11], yet fundamentally limits the system’s ability to react to high-frequency disturbances or environmental changes. In both paradigms, the solver’s computational budget and execution horizon are statically defined prior to deployment. Consequently, the robot lacks a principled mechanism to dynamically arbitrate when a state warrants extended deliberation for trajectory optimality versus when it necessitates rapid, reactive feedback (Figure 1). Identifying an optimal static computation budget cannot be resolved via offline tuning. It is fundamentally a state-dependent quantity, dictated by the unmodelled dynamics, the proximity to state-space constraints, and the marginal optimality gained through additional solver iterations. Moreover, the budget choice cannot be assessed from the current state alone: its consequences propagate to future states throughout the episode, where they may influence later decisions and outcomes.

Figure 1. Robot Metareasoning: different states require different computation budgets. Robots must adapt their computation budget based on the situation that they are facing: (s) deliberate longer to search for an optimal path, (s′) react rapidly to noisy moving obstacles, or (s″) balance planning and reactivity. Across real-world simulated experiments, we show that adapting the computation budget across states within an episode is a fundamental capability for embodied intelligence.
We address these challenges by formulating adaptive com-
This work is supported by European Research Council (ERC) Project 949940 (gAIA) and ARL DCIST CRA W911NF-17-2-0181. We gratefully acknowledge their support. putation budgets for real-time control as a semi-MDP over decision epochs. At each epoch, one decision sets both the open-loop execution horizon and the parallel computational resources allocated to solver. We then learn this decision from experience with an RL policy (Figure 2).

Figure 2. Method Overview. At each decision epoch, the state, the previous plan and its statistics inform the budget policy , which selects the next execution horizon and thereby allocates the solver’s budget. This lets the robot decide how much to deliberate under environmental uncertainty and temporal constraints. The new plan takes over, the state and statistics are refreshed, and the loop repeats.
Our contributions are the following:
(i) Semi-MDP formulation. We cast variable-computation adaptive chunking as a semi-MDP, in which the robot chooses how long to commit to a plan before replanning, and the solver budget follows from that choice.
(ii) Adaptive policy through experience. We propose an RL policy that adapts the computation budget online, letting the controller trade reactivity against deeper solver optimisation on the fly.
(iii) Insights from experiments. Across different real-world and simulated environments, we show that planner statistics are strong signals to the policy and that adaptive computation budgets match or outperform fixed-budget strategies across several robotic systems.
Related Work
We review the work most closely related to our setting.
**a) Real-time and adaptive action chunking: ** Because a chunking policy emits several actions at once, a mechanism is required to cover the gap while the next chunk is generated. RTC [12] keeps execution smooth across that gap by conditioning the new chunk on the actions still running, and DEFLECT [13] instead trains the policy to tolerate acting on observations that have gone stale; both leave the chunk length fixed. [14] makes that length adaptive, while [15] precomputes several candidate chunks and follows whichever best matches the realised state. Unlike our approach, none of these methods couples the execution length with the solver’s computation budget, so a larger commitment cannot be leveraged to compute a better plan.
**b) Adaptive receding-horizon MPC: ** Event- and self-triggered MPC frameworks dynamically determine when to replan rather than solving at every control step, preserving feasibility and stability of the system [16]. Alternative approaches dictate the replanning interval based strictly on safety, defining it as the maximum duration the current trajectory can be executed while a collision-free fallback remains available [17]. However, these triggering mechanisms are designed fundamentally to conserve computation subject to a baseline guarantee, treating the underlying optimizer as an oracle whose solution quality does not improve with additional allocated time.
**c) Metareasoning and test-time computation: ** Metareasoning prices computation by the improvement in the decision quality it produces, net of the cost of running it [5], [6]. This principle underlies anytime algorithms, deliberation scheduling [18], [19], and learned computation budget [20]. In these conventional formulations, the world is treated as frozen while the agent deliberates, so the cost of thinking is an explicit penalty chosen by the designer to trade against the expected benefit.
**d) Learning planning budgets: ** Closest to our work, [21] learns how long to deliberate in real time and casts the decision as a semi-MDP over budgeted options. Their setting, however, is restricted to discrete game examples and therefore does not expose uncontrolled variation that arises when acting in the real world. Furthermore, their method requires a separate reflex policy that acts in the open-loop phase, an assumption that our framework does not need.
Preliminaries
Given a Markov Decision Process (MDP) defined by the tuple , let be the state space, be the action space. We define as a transition kernel (dynamics model) yielding transition probabilities . Moreover, let and be, respectively, the one-step and the goal-conditioned reward functions. At any time the set of goals will be denoted by . We work throughout in discrete time with continuous state: where the index advances at the fixed control period .
The objective of is to determine a sequence of actions (control inputs), that drives the system into a desired state, solving the following infinite-horizon goal-conditioned control problem.
with is the indicator function equal to 1 if the state has reached the goal (and 0 otherwise), so that the terminal-style reward is collected only when the goal is reached. To solve online in a receding horizon, we utilize a monotonic-improving optimizer, . A defining characteristic of such a solver is that its solution monotonically improves with the computational budget it is allocated. However, allocating more computation time introduces a temporal cost: the physical system must continue to operate while the solver thinks. To afford a computational budget of control periods, the solver must therefore commit to generating a sequence of actions (a chunk) of length , where . We define this solver as a mapping , where is the space of action sequences. By committing to this extended sequence, the robot executes the chunk open-loop, explicitly buying the required time window for the solver to concurrently compute the subsequent plan. While classic MPC formulations restrict this mapping to a single step (), our formulation extends to leverage the solver’s monotonic improvement at the expense of delayed reactivity. We evaluate this framework using Model Predictive Path Integral (MPPI) and the Cross Entropy Method (CEM), two standard sampling-based receding horizon solvers, though our method accommodates any monotonic-improving optimizer.
Two fundamental difficulties arise when solving online with a receding horizon time-monotone solver whose computation time is non-negligible and proportional to the action chunk length :
(C1) Open-loop execution creates vulnerability to un-modelled dynamics and disturbances.
When the solver commits to an action chunk of length , the system effectively operates without feedback for timesteps. During this interval, stochasticity, external disturbances, or discrepancies in the transition model cause the true system state to diverge from the solver’s predicted trajectory, leading to compounding errors or potential constraint violations.
(C2) Finding the strict trade-off between plan optimality and system reactivity is a non-trivial problem.
Because the solver’s computational budget scales with the chunk length , the problem formulation inherently couples how deeply the solver reasons with how frequently the system observes. A small enables rapid feedback to counter the disturbances described in (C1), but heavily restricts the computational budget, resulting in myopic and suboptimal planning. Conversely, a large allocates sufficient budget for deep, long-horizon deliberation but forces the system to remain open-loop for extended periods. A static therefore forces a rigid compromise between optimality and robustness, failing to exploit the fact that different states in the MDP require different levels of deliberation versus reaction. We take up these challenges by adopting real-time chunking, which makes (C1) explicit rather than assuming it away, and by adapting the commitment length (and consequently budget of the solver) online through an RL policy learnt from experience, which turns the design constant of (C2) into a decision.

Figure 3. Approaches. (Left) Real-time MPC commits to one step (), whereas action chunking (right) executes a fixed block of actions.
Adaptive Computation Budgets as a semi-MDP {#sec:III}
We formalise the problem as a semi-MDP (SMDP) [22] over decision epochs , embedded in the underlying MDP (whose environment time is ). At epoch , as the computation budget scales with from Section II, a budget-policy emits
where is the simplex over . The output conditioned on the underlying MDP and fixes the next decision epoch at (see Figure 3). During each epoch , the agent accumulates the discounted step-level reward of its committed chunk:
By chaining these epoch rewards, we recover the original infinite-horizon objective (). The resulting semi-MDP objective, , mirrors the original control problem but discounts bootstrapping by the option’s duration instead of a single step:
and , where denotes the transition kernel induced by the budget policy and planner on the underlying MDP . The construction is only meaningful if optimising when and how to think optimises the problem we started from, rather than a surrogate of it. Let , and are bounded and .
Remark 1 (Equivalence of objectives). Under the aforementioned assumptions, for every budget policy and initial state , the expected semi-MDP return equals the expected step-level return:
Consequently, maximizing the chunk-level semi-MDP objective directly maximizes the underlying MDP objective .
Justification. This equivalence is a direct consequence of evaluating temporally extended actions within the Options framework [22]. By construction, the epoch intervals form a strict, disjoint partition of the environment time (). Because the intra-epoch return is defined as the exact discounted sum of step-level rewards over , summing over all epochs pathwise seamlessly recovers the infinite-horizon return. Taking the expectation under the closed-loop trajectory distribution yields (4).
Remark 2 (relation to hierarchical RL). the construction is formally an SMDP over options [22], but the high-level action is a duration, not a behaviour: every budget executes the same chunk from the same fixed , so there is no option library to discover and no low-level non-stationarity.
The physical value of computation is an action value. The semi-MDP does not only state the problem: it collapses the informal notion of physical value of computation as value inherited from the physics of the world, into a one-line definition. As an ”action” (in the SMDP sense) is a budget , the action-value function represents how much a stretch of deliberation is worth: . From here, we define the relative value of computation as the difference:
with . Therefore, the optimal budget is defined as , which (2) learns from input signal . Optimizing , we aim at the optimal relative return gain from the trivial solution .
Bounded Divergence under Adaptive Commitment
A natural question is how far open-loop execution can drift from the solver’s prediction as the commitment grows. For general solvers , we bound the return deviation caused by open-loop execution under model mismatch, under the following assumption.
Assumption 1. The step-level reward is -Lipschitz continuous with respect to the state. Furthermore, the true transition kernel and the solver’s internal nominal model exhibit a bounded single-step mismatch in the -Wasserstein distance, , and the nominal dynamics are -Lipschitz continuous in the state space under .
Theorem 1 (Bounded Open-Loop Deviation). Under Assumption 1, let be the initial prediction error defined below. For any acceptable degradation , there exists a maximum open-loop commitment horizon such that executing any action chunk of length keeps the expected cumulative open-loop return deviation bounded by :
Proof. Let denote the expected state divergence at step of the chunk. Because the solver computes the current plan during the execution of the previous budget , the new chunk begins from a predicted state and inherits a non-zero initial divergence . By Assumption 1 and the Kantorovich–Rubinstein duality, the expected error propagates recursively as . Unrolling this yields . The left-hand side of (6) is therefore bounded by
, and is nondecreasing in because it is a sum of nonnegative terms. Hence, for , the set is a nonempty initial segment of . Its supremum satisfies the claim. □
Theorem 1 informs the design of the action space . The lower bound gives the most frequent feedback and is admissible whenever the initial projection error is acceptable. The upper bound should respect both the mismatch tolerance and the inherited initial divergence, .
Optimization and Training
The policy conditions on a context that is deliberately richer than the environment state, because the quantity being predicted (how much deliberation the current situation affords) depends on the plan in flight and on the solver’s own confidence, not only on where the agent is:
The observation is the state at the invocation, is the previous budget; the chunk to be executed , flattened over its steps, is the plan the agent will run open-loop while thinks.
Solver statistics: The last entry is a fixed-dimensional, budget-invariant summary of the search that produced . It is computed from the rollout bundle and the latest execution feedback, giving the policy information about planning difficulty without exposing a variable-length set of trajectories or requiring an additional planner call. With denoting the softmax weight of sampled rollout among candidates, the effective sample size measures how concentrated the search is: low values indicate that a few rollouts dominate the update, whereas high values indicate that many rollouts contribute comparably, as in a flat cost landscape or a search that has already contracted around a solution. The time-to-infeasibility is the earliest predicted step at which a rollout leaves the feasible set and therefore measures the plan’s remaining safety margin. Finally, the dynamics-model mismatch is the mean displacement between predicted and realised positions of the
objects intersected by the rollout tube, providing an online measure of model reliability. is the term that distinguishes this formulation from a state-conditioned schedule: it lets the policy condition on how hard the problem currently is for the solver, which is observable to the planner but not to the state.
Optimisation
We optimise with PPO over decision epochs. Each transition in the buffer is one epoch , with accumulated as in (III) and advantages estimated by a duration-aware GAE,
so that both the bootstrap and the eligibility decay follow elapsed environment time rather than epoch count. The actor and critic share an encoder over and the actor emits a categorical distribution over .
Proposition 1 (Monotonic Improvement). Assuming exact advantages and exact maximisation of the PPO surrogate at each iteration, updates of monotonically improve and, by Remark 1, .
Proof. Let , with the remaining mass sent to an absorbing zero-reward state. Then is a standard -discounted MDP with identical values, since . PPO is an instance of mirror learning [23], whose updates monotonically improve such MDPs; Remark 1 transfers the improvement to .
Experiments
Our experimental evaluation is designed to validate the efficacy of adaptive computation budgets across diverse robotic domains. Specifically, we seek to answer the following research questions:
Q1 (Performance): Does dynamic adaptation of the computational budget and execution horizon consistently outperform fixed-budget baselines in dynamic and constrained environments?
Q2 (Adaptation Mechanism): Can the learned policy correctly identify state-dependent requirements for deliberation versus reactivity, and how critical are internal planner statistics to this arbitration?
Q3 (Boundary Behavior): How does the adaptive policy behave in non-smooth cost landscapes where the optimal commitment horizon lacks strong state-dependent variation?
To quantitatively evaluate these questions, we report two primary metrics: (1) task success rate, the fraction of evaluation episodes that satisfy the task-specific goal criterion in goal-conditioned scenarios; and (2) mean total return, the undiscounted sum of environment rewards over an episode, averaged over all evaluation episodes, including failures. Higher values are better for both metrics. For navigation tasks, we additionally measure harshness, a dimensionless measure combining path tortuosity and motion jerk, . Here, is the travelled path length, and is the straight-line distance between the robot’s initial and final positions. The normalized jerk measures how abruptly acceleration changes, using the accumulated squared jerk normalized by episode duration and path length. Lower harshness indicates a more direct and smoother trajectory.
We also present the optimality regret of a fixed-commitment baseline at each decision boundary along its trajectory as the relative value of computation with respect to the optimal budget-action at state :
Thus, measures the estimated return lost by choosing the fixed commitment instead of the best available commitment under (lower values are better).
We deploy our framework across three distinct scenarios, systematically chosen to stress the computation-reactivity trade-off from different physical dimensions: holonomic ground navigation, deployed in the real-world; quadrotor with suspended cable payload; and contact-rich planar manipulation (3D Push-T). The properties of each system are described in the related paragraphs.
Implementation Details. The hardware and software infrastructure is standardized across experiments to ensure consistent solver performance. The underlying control and communication stack is orchestrated via ROS 2. Hardware experiments are conducted on a DJI RoboMaster platform [24] for the holonomic ground navigation. Simulated environments utilize MuJoCo [25] for the quadrotor dynamics and the Push-T library using a [26] for planar manipulation. To achieve real-time, high-frequency planning, the predictive model and the Model Predictive Path Integral (MPPI) solver are implemented entirely in JAX and executed via MuJoCo MJX [25]. All neural network inference and hardware-accelerated planning are executed on a single NVIDIA A100 GPU (40GB VRAM), ensuring the solver’s wall-clock latency strictly dictates the physical temporal cost of computation. The control period is s for navigation, s for cable-payload flight, and s for PushT; a commitment of therefore corresponds to seconds of open-loop execution. The budget policy is a two-layer tanh MLP (128 units per layer), with a categorical actor and scalar critic. PPO uses , learning rate , minibatches of 256, four epochs, clip 0.2, and gradient clipping at 0.5. We train for 2,500 updates with 2,048 transitions per update. In the experiments, MPPI scales its prediction horizon linearly with and scales its samples linearly for navigation and as for cable flight; CEM on PushT scales both horizon and population linearly.
Holonomic 2D navigation
In this navigation example, because the dynamics are simple, failures arise strictly from failing to balance the deep search required for global optimality against the rapid feedback needed to avoid dynamic obstacles. We consider three scenarios with distinct obstacle geometries and dynamics: a fixed wall with three stochastic obstacles (“barrier”, Figure 1, right), one crossing robot with three stochastic obstacles (“merging lane”, Figure 5), and five laterally moving obstacles (“highway”, Figure 1, centre). The environment reward is , where and are the robot and goal positions. Success or collision terminates the episode.

Figure 5. Real-world Merging Lane scenario. Top: temporal composites on the RoboMaster platform under our learned budget and fixed commitments ; each panel overlays successive robot poses from one episode. Bottom: the commitment our policy selects along its episode, against the three fixed budgets. The budget collapses to while the robot merges past the crossing obstacle (red band) and while it threads the close passage between the static obstacles.

Figure 8. Dynamic commitment during cable-payload flight. Four stages from one episode show the learned policy increasing the commitment for transit and reducing it near each target.
Aggregate and per-scenario results are shown in Figure 4 and Table I. The adaptive policy achieves the strongest overall performance across success rate, total return, and harshness. In the merging lane scenario (Figure 5), the policy uses a larger budget early in the episode to identify a safe route, then reduces the commitment to increase feedback while passing the crossing robot and nearby stochastic obstacles. Around s, it approaches the reactive limit . Figure 6 plots the optimality regret, , (8) of fixed commitment at each decision boundary.

Figure 4. Navigation Results, pooled over the three scenarios (300 randomised episodes). Our learned budget policy (left of the dashed rule) against every fixed commitment (blue). The two right-hand panels condition on successful episodes and therefore flatter the low-success baselines. The solid and dashed outlines mark the best fixed budget under success rate () and under total return (): no single fixed is best at both, while the learned budget dominates both.
Figure 6. Optimality Regret in the Merging Lane scenario along three fixed commitment trajectories. Lines are means and standard deviations over 10 episodes. The tint marks Phase 1, before the moving obstacle has been passed, which ends at a different step for each commitment. Dotted rules mark the mean collision step for each fixed strategy.
| Scenario | Strategy | Success ↑ | Total return ↑ | Harshness ↓ |
| Barrier | Ours | 0.97 | −435 | 7.37 |
| Barrier | (S) | 0.73 | −1167 | 7.28 |
| Barrier | (R) | 0.12 | −1523 | 7.42 |
| Barrier | (H) | 0.63 | −1400 | 7.55 |
| Merging Lane | Ours | 1.00 | −385 | 7.02 |
| Merging Lane | (S) | 0.63 | −1462 | 7.17 |
| Merging Lane | (R) | 1.00 | −460 | 9.68 |
| Merging Lane | (H) | 0.61 | −1466 | 7.46 |
| Highway | Ours | 0.99 | −534 | 9.05 |
| Highway | (S) | 0.92 | −719 | 8.84 |
| Highway | (R) | 0.78 | −1265 | 11.73 |
| Highway | (H) | 0.08 | −2988 | 9.26 |
Table I. Navigation results (100 episodes per scenario). We compare our policy with the same three fixed budgets selected from the pooled results in Figure 4: (best Success rate), (best total Return), and (best Harshness). The baselines remain fixed across scenarios; total return is averaged over all episodes.
Sim to Real. The RoboMaster evaluation uses the same learned policy and budget-to-planner mapping as simulation, with only the physical robot, sensing, and ROS 2 execution stack changed. The close agreement with the simulated navigation behaviour indicates a small simulation-to-real gap for this task.
✓ Planner statistics guide adaptation. The solver summary is what tells the policy when to spend and when to react. The ablation in Figure 7 shows that removing this feedback lowers success from 98% to 86%; feedback alone reaches 74% success, showing that state and solver signals are complementary.

Figure 7. Holonomic 2D navigation: planner statistics ablation. Ablation of the policy input, pooled over the three scenarios (300 episodes per variant). FULL uses ; PS ONLY uses the planner statistics alone; FULL-PS uses everything except . The full input achieves the highest success rate and total return, while removing planner statistics drops success by 12 percentage points.
Quadrotor with a suspended cable payload
The quadrotor carries a cable-suspended load through a two-goal slalom. The stiff, underactuated dynamics punish staleness via open-loop system divergence rather than environmental changes. This requires the system to balance deep, computationally expensive sampling for stable transit against highly reactive short-horizon commitments near targets. Its dense reward is . For distance and braking-aware target velocity , and
. The remaining terms reward Gaussian hover accuracy, penalize action/body rates, and set and ; intermediate settling adds 250 plus up to 100 for precision. Here is the distance to the active target, is the quadrotor translational velocity, is the braking-aware target velocity, and is the payload’s cable angular velocity. A robot trial is successful only when the system reaches the first goal, remains within the goal and stability tolerances for the required settling window, and then reaches and stabilises at the final goal under the same criterion. Our policy succeeds in 80% of trials, while every fixed commitment fails, and it discovers a two-stage computation pattern: it uses larger budgets during stable transit and shorter commitments near the targets, achieving the highest return among the fixed strategies (Figure 9). Furthermore, Figure 9 confirms that planner statistics help the policy achieving the best performance.

Figure 9. Cable-payload slalom return. Mean total return over 20 episodes per method. The left group compares the full policy (gold) with ablations without planner statistics (orange) and using planner statistics only (salmon); fixed commitments are shown in blue.
Contact-rich planar manipulation
In Push-T [10], a planar pusher slides a free T-shaped block onto a target using the hydrax model [26] and CEM in MuJoCo MJX. With position error , angular error , pusher–block distance , and normalized action , its environment reward is . The budget policy receives the decrease in the bracketed pose cost across its chunk. Changing contacts makes plans stale and tests local budget sensitivity; results are in Figure 10.

Figure 10. Boundary behaviour on the 3D PushT environment. Left: robot-arm manipulation setup. Right: terminal position and orientation errors after 600 steps, reported as mean sample standard deviation over 16 matched validation episodes. Our policy selects .
It can be seen that the policy collapses to everywhere, matching the best fixed baseline. We attribute this to the nominal setting: planning and execution share exact dynamics, the system is fully observed and disturbance-free, and the objective does not reward computational savings or earlier arrival, leaving little incentive for frequent feedback.
| [cm] | [rad] | |
| 1 | ||
| 2 | ||
| 3 | ||
| 4 | ||
| 5 | ||
| 6 | ||
| 7 | ||
| 8 | ||
| 9 | ||
| 10 | ||
| Ours |
Table 1.
Conclusion
Robot metareasoning is a fundamental capability of embodied intelligence: an agent must decide how much to deliberate under environmental uncertainty and temporal constraints. We introduce a semi-MDP framework for robot metareasoning that learns adaptive computation budgets for real-time control. By leveraging planner statistics, the policy dynamically balances plan execution length and planner computation time based on plan quality and uncertainty. Across three environments with distinct sources of difficulty, the policy adapts its budget to the demands of the task. Our results show that online adaptation is highly beneficial when exploitable budget variation exists. It discovers longer commitments for global route, smooth transit and shorter ones near dynamic obstacles, critical passage and, in cable-payload flight, near targets and disturbances. In contact-rich planar manipulation, the learned policy converges to a constant budget matching the best fixed baseline, indicating that adaptation is unnecessary when the nominal task provides little exploitable variation. Future work will explore richer, high-dimensional planner statistics, learned world models, and more expressive planning algorithms.
References
References
- [1]A. R. Otto and N. D. Daw, “The opportunity cost of time modulates cognitive effort,” Neuropsychologia, vol. 123, pp. 92–105, 2019.DOI
- [2]N. D. Daw, Y. Niv, and P. Dayan, “Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control,” Nature Neuroscience, vol. 8, no. 12, pp. 1704–1711, 2005.pubmed.ncbi.nlm.nih.gov/16286932
- [3]M. Keramati, A. Dezfouli, and P. Piray, “Speed/accuracy trade-off between the habitual and the goal-directed processes,” PLoS Computational Biology, vol. 7, no. 5, p. e1002055, 2011.DOI
- [4]A. Shenhav, S. Musslick, F. Lieder, W. Kool, T. L. Griffiths, J. D. Cohen, and M. M. Botvinick, “Toward a rational and mechanistic account of mental effort,” Annual Review of Neuroscience, vol. 40, pp. 99–124, 2017.DOI
- [5]S. Russell and E. Wefald, “Principles of metareasoning,” Artificial Intelligence, vol. 49, no. 1–3, pp. 361–395, 1991.DOI
- [6]N. Hay, S. Russell, D. Tolpin, and S. E. Shimony, “Selecting computations: Theory and applications,” in Proc. Conf. Uncertainty in Artificial Intelligence (UAI), 2012.
- [7]G. Williams, N. W. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou, “Information theoretic MPC for model-based reinforcement learning,” in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2017, pp. 1714–1721.DOI
- [8]H. Bharadwaj, K. Xie, and F. Shkurti, “Model-predictive control via cross-entropy and gradient-based optimization,” in Proceedings of the Conference on Learning for Dynamics and Control, 2020, pp. 277–286.
- [9]T. Z. Zhao, W. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proc. Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2304.13705
- [10]C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proc. Robotics: Science and Systems (RSS), 2023.
- [11]K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky, “π₀: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024.arxiv.org/abs/2410.24164
- [12]K. Black, M. Y. Galliker, and S. Levine, “Real-time execution of action chunking flow policies,” in Advances in Neural Information Processing Systems (NeurIPS), 2025, arXiv:2506.07339.arxiv.org/abs/2506.07339
- [13]Y. Zhu, Y. Chen, Z. Yang, Y. Hu, and X. Chen, “DEFLECT: Temporal counterfactual preference learning for delay-robust asynchronous VLAs,” arXiv preprint arXiv:2605.19294, 2026.arxiv.org/abs/2605.19294
- [14]Y. Zhao, M. Bogdanovic, A. Sohal, L. Tao, K. Darvish, A. Aspuru-Guzik, F. Shkurti, and A. Garg, “Dynamic execution horizon prediction for chunk-based robot policies,” arXiv preprint arXiv:2606.11408, 2026.arxiv.org/abs/2606.11408
- [15]W. Chen, K. Zhang, C. Lin, Z. Zhang, Y. She, Y. Liu, R. A. Yeh, S. Mou, and Y. Gu, “DREAM-Chunk: Reactive action chunking with latent world model,” arXiv preprint arXiv:2606.18589, 2026.arxiv.org/abs/2606.18589
- [16]Z. Sun, L. Dai, K. Liu, D. V. Dimarogonas, and Y. Xia, “Robust self-triggered mpc with adaptive prediction horizon for perturbed nonlinear systems,” IEEE Transactions on Automatic Control, vol. 64, no. 11, pp. 4780–4787, 2019.DOI
- [17]S. Magdici, Z. Ye, and M. Althoff, “Determining the maximum time horizon for vehicles to safely follow a trajectory,” in Proc. IEEE Int. Conf. Intelligent Transportation Systems (ITSC), 2017, pp. 1–7.
- [18]T. L. Dean and M. Boddy, “An analysis of time-dependent planning,” in Proc. AAAI Conf. Artificial Intelligence, 1988, pp. 49–54.
- [19]S. Zilberstein, “Using anytime algorithms in intelligent systems,” AI Magazine, vol. 17, no. 3, pp. 73–83, 1996.DOI
- [20]F. Callaway, S. Gul, P. M. Krueger, T. L. Griffiths, and F. Lieder, “Learning to select computations,” in Proc. Conf. Uncertainty in Artificial Intelligence (UAI), 2018.
- [21]A. Muppidi, F. Darwish, D. Cope, J. F. Henriques, and J. N. Foerster, “Finding the time to think: Learning planning budgets in real-time RL,” arXiv preprint arXiv:2606.26463, 2026.arxiv.org/abs/2606.26463
- [22]R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” Artificial intelligence, vol. 112, no. 1-2, pp. 181–211, 1999.DOI
- [23]J. G. Kuba, C. Schroeder de Witt, and J. Foerster, “Mirror learning: A unifying framework of policy optimisation,” in Proc. Int. Conf. Machine Learning (ICML), ser. PMLR, vol. 162, 2022.arxiv.org/abs/2201.02373
- [24]J. Blumenkamp, A. Shankar, M. Bettini, J. Bird, and A. Prorok, “The cambridge robomaster: An agile multi-robot research platform,” in International Symposium on Distributed Autonomous Robotic Systems. Springer, 2024, pp. 439–456.arxiv.org/abs/2405.02198
- [25]E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033.DOI
- [26]V. Kurtz, “Hydrax: Sampling-based model predictive control on gpu with jax and mujoco mjx,” 2024, https://github.com/vincekurtz/hydrax.
