betaFeedback

WorksComputer Science

VARP: A Visual Adaptive Rearrangement Planner for Non-Monotone Object Rearrangement Planning

Abstract

Object rearrangement planning in cluttered environments is often non-monotone, requiring objects to be temporarily relocated before reaching their final destinations. Although search-based planners can solve such tasks, they incur substantial computational cost and require expensive replanning when the environment changes. We propose the Visual Adaptive Rearrangement Planner (VARP), which learns non-monotone rearrangement directly from visual state representations generated from expert demonstrations. VARP predicts pick-and-place actions from the current and goal visual state representations, and incorporates an Action Refinement Strategy (ARS) to improve action feasibility and a Progress Enforcement Strategy (PES) to mitigate progress stagnation. By continuously updating actions from the latest visual state representations, VARP adapts online to disturbances without replanning from scratch or performing online combinatorial search. In challenging rearrangement benchmarks, VARP achieves a 98.3% success rate with only 0.4 s planning time per task, 97.5% success under execution-time disturbances, and 94.5% under continuous severe disturbances, while generalizing effectively to unseen object counts. These results demonstrate that VARP provides a scalable and robust solution for non-monotone object rearrangement.

Introduction

Object rearrangement planning is a fundamental capability for robotic manipulation, with applications in warehouse automation, household assistance, manufacturing, and logistics [1]. Given an initial object arrangement and a desired target configuration, the robot must determine a sequence of prehensile pick-and-place actions that transform the scene while avoiding collisions and satisfying task constraints [2]. In cluttered environments, achieving the target arrangement often requires non-monotone planning, where objects must be temporarily relocated to an intermediate buffer region before they can be placed at their final destinations. Such non-monotone rearrangement problems are known to be NP-hard in general [3], making the task particularly challenging as scene complexity increases.

Classical rearrangement planners typically formulate the problem as a search over object relocation sequences and rely on task-and-motion planning, sampling-based methods [4], or tree search algorithms [5] to identify feasible solutions. While these approaches can produce high-quality plans, they often incur substantial computational cost due to repeated collision checking, dependency reasoning, and motion planning. Furthermore, when the environment changes during execution, planners frequently require expensive replanning procedures to recover from disturbances, limiting their applicability in dynamic settings.

Recent advances in imitation learning and vision-based manipulation have motivated the use of learning-based policies for rearrangement tasks [6]. By learning an observation-to-action mapping directly from expert demonstrations, neural policies can amortize planning effort and generate actions with significantly lower inference latency [7]. However, CNN-based or Transformer-based policies approximate the observation-to-action mapping from finite data and may produce errors under perceptual uncertainty or distribution shift. Unlike search-based planners which explicitly reason about state transitions and action feasibility, existing learning-based approaches often struggle in cluttered non-monotone environments, where small prediction errors may lead to infeasible actions, self-relocations, or repetitive behaviors that prevent task completion [8]. Achieving the efficiency of learned policies while retaining the robustness of classical planning thus remains an open challenge.

To this end, we propose a Visual Adaptive Rearrangement Planner (VARP) for non-monotone object rearrangement planning. VARP combines a goal-conditioned visual policy that efficiently predicts rearrangement actions from visual state representations with two lightweight inference-time reasoning mechanisms that improve action feasibility and long-horizon execution reliability. Executed in a closed-loop manner, VARP continuously updates its decisions based on the latest scene configuration, enabling robust adaptation to execution-time disturbances without expensive online replanning or combinatorial search. Extensive experiments demonstrate that VARP achieves superior success rates and substantially lower planning latency than state-of-the-art search-based planners while exhibiting strong robustness to environmental disturbances and generalization to out-of-distribution scenarios. Overall, the main contributions of this work are summarized as follows:

  • We propose VARP, a visual adaptive rearrangement planner that integrates a learning-based goal-conditioned visual policy with lightweight inference-time reasoning for efficient and reliable non-monotone object rearrangement planning.

  • We introduce two lightweight inference-time reasoning mechanisms: the Action Refinement Strategy (ARS), which performs diversity-aware candidate exploration and feasibility validation, and the Progress Enforcement Strategy (PES), which performs short-horizon rollout reasoning to mitigate progress stagnation without expensive replanning.

  • We conduct extensive evaluations on standard rearrangement, disturbance recovery, out-of-distribution, and real-world scenarios, demonstrating that VARP achieves high success rates, short planning time, and robust performance across diverse evaluation settings.

Related Work

Object rearrangement planning: Object rearrangement planning has been widely studied in robotics, where the goal is to move multiple objects from an initial configuration to a target configuration while avoiding collisions and respecting robot motion constraints [4, 9, 10, 11, 12]. Classical approaches typically formulate rearrangement as a combinatorial search problem and rely on task-and-motion planning, sampling-based planning, or tree search to reason over object dependencies and intermediate placements [13, 14]. These methods can provide strong planning performance in cluttered scenes; however, their computational cost often increases rapidly with the number of objects and the complexity of the required non-monotone reasoning [15, 16]. In contrast, VARP learns a goal-conditioned visual policy from expert demonstrations and performs fast closed-loop inference without solving a new combinatorial search problem from scratch at test time.

Learning-based rearrangement policies: Recent learning-based approaches have investigated neural network-based policies that predict pick-and-place actions directly from visual observations [17, 18, 19]. Other works further incorporate language, spatial relations, or multimodal prompts for goal-conditioned and semantic rearrangement [20, 21, 22]. These approaches reduce online planning cost by amortizing decision making into a learned policy, but they often struggle in cluttered non-monotone settings where intermediate relocations are required and small action errors can lead to invalid or unproductive behaviors. Unlike prior learning-based approaches that rely solely on feed-forward policy inference, VARP tightly couples a learned goal-conditioned visual policy with lightweight inference-time reasoning, enabling adaptive closed-loop planning without combinatorial search or expensive replanning.

Closed-loop and reactive rearrangement: Closed-loop rearrangement is important as execution may deviate from the planned trajectory due to perception errors, actuation uncertainty, or external disturbances [23]. Prior work has explored reactive manipulation through online replanning, task-and-motion replanning, and model-predictive planning frameworks that repeatedly recompute actions based on updated observations [24, 25]. While these approaches can improve execution robustness, their computational cost can become substantial in cluttered rearrangement settings. In contrast, VARP uses visual feedback at every step and directly predicts the next pick-and-place action from the current and goal visual state representations.

Problem Formulation

We consider a non-monotone object rearrangement planning problem in a confined workspace W⊂R3\mathcal{W} \subset\mathbb{R}^{3} containing nn geometrically identical but uniquely identifiable movable cylindrical objects O={o1,…,on}\mathcal{O} = \{o_{1}, \ldots, o_{n}\}. The workspace is modeled as a cabinet-like enclosure with a single front opening and enclosed boundaries on all other sides. Its inner surface dimensions along the XX and YY axes are denoted by (dx,dy)(d_{x}, d_{y}), respectively. Inside the workspace, each object i∈{1,…,n}i \in\{1, \ldots, n\} occupies a collision-free placement location poi=(xoi,yoi)p_{o_{i}} = (x_{o_{i}}, y_{o_{i}}), where poi∈R2p_{o_{i}} \in\mathbb{R}^{2}. At any time step tt, the placement locations of all objects define an object placement configuration st={po1t,…,pont}∈Ss^{t} = \{p^{t}_{o_{1}}, \ldots, p^{t}_{o_{n}}\} \in\mathcal{S}, with S\mathcal{S} denoting the configuration space and st[oi]=pois^{t}[o_{i}] = p_{o_{i}} indicating the object oio_{i} occupies location poip_{o_{i}}. Finally, the volume occupied by all objects at time tt is written as V(st)⊂R3V(s^{t}) \subset\mathbb{R}^{3}.

A robot manipulator R\mathcal{R} equipped with a parallel-jaw gripper is placed in front of the workspace opening at location pmp_{m} in order to perform prehensile object relocation actions. A rearrangement action at time step tt is defined as a composite pick-and-place action:

at=(apkt,aplt)(1)a^{t} = (a^{t}_{\mathrm{pk}}, a^{t}_{\mathrm{pl}}) \tag*{(1)}

where apkta^{t}_{\mathrm{pk}} selects and grasps an object oio_{i} from its current placement st[oi]s^{t}[o_{i}], and aplta^{t}_{\mathrm{pl}} relocates the grasped object to a target placement st+1[oi]s^{t+1}[o_{i}]. The action induces the state transition:

st+1=f(st,at)(2)s^{t+1} = f(s^{t}, a^{t}) \tag*{(2)}

The dd-degree-of-freedom robot configuration space is denoted by Q⊂Rd\mathcal{Q} \subset\mathbb{R}^{d}. During a rearrangement action ata^{t} manipulating object oio_i, the robot follows a motion trajectory τ(at)={q0t,…,qkt}\tau(a^{t}) = \{q_{0}^{t}, \ldots, q_{k}^{t}\}, where each qjt∈Qq_{j}^{t} \in\mathcal{Q} denotes an intermediate robot configuration. The corresponding swept volume of the robot is denoted by V(τ(at))⊂R3V(\tau(a^{t})) \subset\mathbb{R}^{3}. The action ata^{t} is considered feasible under the current configuration sts^{t} if the corresponding robot trajectory is collision-free with respect to all non-manipulated objects:

V(τ(at))∩V({st[oj]∣oj∈O,oj≠oi})=∅(3)V(\tau(a^{t})) \cap V\left(\{s^{t}[o_j] \mid o_j \in\mathcal{O}, o_j \ne o_i\}\right) = \varnothing \tag*{(3)}

We formulate the object rearrangement task as a deterministic goal-conditioned planning problem defined by the tuple

P=(S,A,f,s0,sg)(4)\mathcal{P} = (\mathcal{S}, \mathcal{A}, f, s^{0}, s^{g}) \tag*{(4)}

where S\mathcal{S} denotes the object configuration space, A\mathcal{A} denotes the space of feasible pick-and-place actions, f:S×A→Sf : \mathcal{S} \times\mathcal{A} \to\mathcal{S} is the deterministic state transition function, and s0s^{0} and sgs^{g} denote the initial and goal configurations, respectively. The objective is to develop a planner that sequentially selects actions at∈Aa^{t} \in\mathcal{A} to transform the current configuration sts^{t} toward the goal configuration sgs^{g}. The resulting action sequence T={a0,…,aT}\mathcal{T} = \{a^{0}, \ldots, a^{T}\} constitutes a feasible rearrangement plan that transforms the workspace from s0s^{0} to sgs^{g}.

Method

Method Overview

To solve this problem, we propose the Visual Adaptive Rearrangement Planner (VARP), a goal-conditioned closed-loop planner that generates one rearrangement action at a time using visual feedback from the environment. VARP consists of a learned goal-conditioned visual policy augmented with two inference-time reasoning modules: the Action Refinement Strategy (ARS) and the Progress Enforcement Strategy (PES), as illustrated in Figure 2. Given visual cues rendered from the current state sts^{t} and goal state sgs^{g}, the visual policy first outputs pick and place heatmaps. ARS then transforms the raw policy predictions into feasible rearrangement actions by promoting action diversity, enforcing motion feasibility, and rejecting ineffective self-placement actions. PES further evaluates the selected action through a short-horizon rollout probe and revises it when progress stagnation is detected. The resulting action is executed, producing a new state st+1s^{t+1}, and the process repeats until the goal configuration is reached.

Figure 2. Architecture of the proposed Visual Adaptive Rearrangement Planner (VARP). A goal-conditioned visual policy first predicts pick and place heatmaps, which are subsequently refined by the Action Refinement Strategy (ARS) and Progress Enforcement Strategy (PES) into executable rearrangement actions through lightweight inference-time reasoning.

Goal-Conditioned Visual Policy

Rather than solving the rearrangement problem through explicit symbolic search, we learn a goal-conditioned visual policy that predicts the next rearrangement action directly from visual representations. The underlying state is the object placement configuration sts^{t}, while the planner receives a rendered visual state representation of the configuration. We assume the current object configuration is provided by an external perception or object-tracking system. VARP operates on rendered visual state representations generated from these estimated object configurations rather than directly on raw RGB images. Specifically, given the current object configuration sts^{t} and the goal configuration sgs^{g}, we obtain their corresponding visual representations through a rendering function Φ(⋅)\Phi(\cdot):

It=Φ(st),Ig=Φ(sg)(5)I^{t} = \Phi(s^{t}), \qquad I^{g} = \Phi(s^{g}) \tag*{(5)}

where ItI^{t} and IgI^{g} denote the current and goal visual state representations, respectively. The rendering function Φ(⋅)\Phi(\cdot) converts the object configuration into a fixed-resolution visual state representation while preserving the spatial relationships among objects. Geometrically identical objects are rendered using unique fixed colors, allowing the visual policy to distinguish individual objects.

Then, the learned visual policy is defined as

a^t=πθ(It,Ig)(6)\hat{a}^{t} = \pi_{\theta}(I^{t}, I^{g}) \tag*{(6)}

where πθ\pi_{\theta} is a visual policy model by a neural network with learnable parameters θ\theta, and a^t\hat{a}^{t} denotes the pick-and-place action predicted by the planner, consisting of a pick action a^pkt\hat{a}^{t}_{\mathrm{pk}} and a place action a^plt\hat{a}^{t}_{\mathrm{pl}} to initiate state transition from sts^{t} to st+1s^{t+1} (c.f. Eq. 1 and Eq. 2). The proposed policy generates pick-and-place actions by predicting spatial heatmaps over a discretized workspace, enabling workspace-wide reasoning while preserving spatial structure.

We implement the policy end-to-end using an encoder-decoder structured neural network Ψθ\Psi_{\theta} that predicts both pick and place spatial heatmaps. To enable a single network to perform both pick and place prediction, we introduce two conditioning inputs in addition to current and goal visual representations (ItI^{t} and IgI^{g}): an object-conditioning map GtG^{t} and a binary phase indicator map ϕt\phi^{t}. Both GtG^{t} and ϕt\phi^{t} are single-channel image-sized maps with the same spatial dimensions as the visual representations. During pick prediction, no object has yet been selected. Therefore, a zero conditioning map Gpkt=0G^{t}_{\mathrm{pk}} = 0 and phase indicator map ϕpkt=0\phi^{t}_{\mathrm{pk}} = 0 are provided to the network. During place prediction, an object-specific conditioning map is constructed from the selected object and the phase indicator is set to ϕplt=1\phi^{t}_{\mathrm{pl}} = 1. These conditioning signals allow the same network to distinguish between the two prediction stages while sharing the same parameters.

For the pick action prediction, given the current ItI^{t} and the goal visual representations IgI^{g}, the neural network Ψθ\Psi_{\theta} predicts a pick heatmap Hpkt∈Rm×mH^{t}_{\mathrm{pk}} \in\mathbb{R}^{m \times m}, where each cell encodes the likelihood of selecting the corresponding workspace location.

Hpkt=Ψθ(It,Ig,Gpkt,ϕpkt)(7)H^{t}_{\mathrm{pk}} = \Psi_{\theta}(I^{t}, I^{g}, G^{t}_{\mathrm{pk}}, \phi^{t}_{\mathrm{pk}}) \tag*{(7)}

A pick action location apkta^{t}_{\mathrm{pk}} is subsequently extracted from HpktH^{t}_{\mathrm{pk}} using the action selection procedure described in § 4.D.

Since the predicted pick location may not exactly coincide with an object center, it is associated with the nearest object oko_k in the current state:

ok=arg min⁡oi∈O∥apkt−st[oi]∥(8)o_k = \underset{o_i \in\mathcal{O}}{\operatorname{arg\,min}} \left\|a^{t}_{\mathrm{pk}} - s^{t}[o_i]\right\| \tag*{(8)}

This association converts the spatial pick prediction into a concrete object identity, which is subsequently used to condition the place action prediction. Next, given the selected object oko_k, its current location s[ok]ts^{t}_{[o_k]} is encoded into an object-specific conditioning map. Specifically, we construct a Gaussian indicator map

Gt=Γ(s[ok]t)(9)G^{t} = \Gamma(s^{t}_{[o_k]}) \tag*{(9)}

where Γ(⋅)\Gamma(\cdot) denotes the Gaussian indicator map centered at the current location of the selected object oko_k. This indicator map explicitly identifies the selected object and provides object-aware context for the subsequent place action prediction. After the object-specific conditioning map is constructed, the network performs a second inference pass to predict a place heatmap Hplt∈Rm×mH^{t}_{\mathrm{pl}} \in\mathbb{R}^{m \times m}, where each cell encodes the likelihood of placing the selected object at the corresponding workspace location.

Hplt=Ψθ(It,Ig,Gt,ϕplt)(10)H^{t}_{\mathrm{pl}} = \Psi_{\theta}(I^{t}, I^{g}, G^{t}, \phi^{t}_{\mathrm{pl}}) \tag*{(10)}

A place action location aplta^{t}_{\mathrm{pl}} is similarly extracted from HpltH^{t}_{\mathrm{pl}} using the same action selection procedure in § 4.D. Although Equations (7) and (10) are written as separate pick and place prediction passes for clarity, both predictions are generated using the same network parameters θ\theta. The distinction arises solely from the object-conditioning map and phase indicator provided as network inputs, allowing a single network to perform both pick and place prediction while sharing the same parameters.

Action Refinement Strategy (ARS)

The learned visual policy predicts probabilistic pick-and-place heatmaps over the workspace, where high-confidence predictions are not necessarily executable in cluttered non-monotone rearrangement tasks. To convert these predictions into executable actions, we propose the Action Refinement Strategy (ARS), an inference-time action selection framework comprising three sequential stages: diversity-promoting candidate exploration, feasibility validation, and self-placement rejection.

Rather than committing to the highest-scoring heatmap peak, ARS first extracts the top-KK candidate locations. Their confidence values are normalized into a multinomial distribution, from which candidate actions are sampled to produce a prioritized evaluation order. This preserves the preference for high-confidence predictions while allowing alternative candidates to be considered when the highest-scoring prediction is infeasible.

The sampled candidate actions are then validated before execution using the Linear Motion Planner (LMP) [10], which employs a rectangular swept-volume approximation for computationally efficient feasibility checking. Specifically, LMP verifies grasp accessibility for pick actions and rearrangement feasibility for place actions according to the constraints described in § 3. Invalid candidates are discarded until a feasible action is found. Finally, self-placement actions are rejected by temporarily excluding the selected object and recomputing the pick and place predictions. Together, these stages transform probabilistic heatmap predictions into feasible and effective rearrangement actions without additional training or expensive online replanning.

Progress Enforcement Strategy (PES)

To mitigate repetitive relocation loops and improve long-horizon execution, we propose the Progress Enforcement Strategy (PES), an inference-time reasoning module that performs a short-horizon rollout probe to evaluate whether the action selected by ARS is likely to sustain meaningful progress toward the goal configuration. While ARS ensures that the selected action is feasible, learning-based visual policies may still repeatedly select the same object, leading to little progress toward the target arrangement.

Starting from the current state, PES virtually executes the action proposed by ARS and rolls the policy forward for several decision steps using the same goal-conditioned policy and ARS action selection. If the rollout repeatedly selects the same object, the current action is classified as leading to progress stagnation. The selected object is then temporarily excluded from the candidate set, allowing ARS to select an alternative object for relocation. The resulting action is executed directly without performing an additional rollout probe.

Unlike ARS, which improves immediate action feasibility, PES evaluates the short-horizon consequences of candidate actions before execution to prevent repetitive relocation loops. Together, ARS and PES improve both immediate action quality and long-horizon rearrangement performance without additional training or expensive online replanning.

Visual Adaptive Rearrangement Planner (VARP)

Algorithm 1: Visual Adaptive Rearrangement Planner
Data: s0s^{0}, sgs^{g}, Ψθ\Psi_{\theta}, Tmax⁡T_{\max}
Result: T={a0,a1,…}\mathcal{T} = \{a^{0}, a^{1}, \ldots\} ▷ Final Action Sequence
1 T←∅\mathcal{T} \leftarrow\varnothing, t←0t \leftarrow0
2 while st≠sgs^{t} \ne s^{g} and t<Tmax⁡t < T_{\max} do
3 It←Φ(st)I^{t} \leftarrow\Phi(s^{t}), Ig←Φ(sg)I^{g} \leftarrow\Phi(s^{g}) ▷ Render Visual States
4 Hpkt←Ψθ(It,Ig,Gpkt,0)H_{\mathrm{pk}}^{t} \leftarrow\Psi_{\theta}(I^{t}, I^{g}, G_{\mathrm{pk}}^{t}, 0) ▷ Pick Heatmap
5 a^pkt←ARSpk(Hpkt,st)\hat{a}_{\mathrm{pk}}^{t} \leftarrow\mathrm{ARS}_{\mathrm{pk}}(H_{\mathrm{pk}}^{t}, s^{t}) ▷ Action Refinement
6 ok←arg⁡min⁡oi∈O∥a^pkt−st[oi]∥o_{k} \leftarrow\arg\min_{o_{i}\in O}\lVert\hat{a}_{\mathrm{pk}}^{t}-s^{t}[o_{i}]\rVert ▷ Association
7 a^pkt←st[ok]\hat{a}_{\mathrm{pk}}^{t} \leftarrow s^{t}[o_{k}] ▷ Predicted Pick Action
8 Gpkt←Γ(st[ok])G_{\mathrm{pk}}^{t} \leftarrow\Gamma(s^{t}[o_{k}]) ▷ Pick Object Indicator
9 Hplt←Ψθ(It,Ig,Gt,1)H_{\mathrm{pl}}^{t} \leftarrow\Psi_{\theta}(I^{t}, I^{g}, G^{t}, 1) ▷ Place Heatmap
10 a^plt←ARSpl(Hplt,st,ok)\hat{a}_{\mathrm{pl}}^{t} \leftarrow\mathrm{ARS}_{\mathrm{pl}}(H_{\mathrm{pl}}^{t}, s^{t}, o_{k})
11 a^t←(a^pkt,a^plt)\hat{a}^{t} \leftarrow(\hat{a}_{\mathrm{pk}}^{t}, \hat{a}_{\mathrm{pl}}^{t}) ▷ Proposed Action
12 at←PES(a^t)a^{t} \leftarrow\mathrm{PES}(\hat{a}^{t}) ▷ Progress Enforcement
13 T.add(at)\mathcal{T}.\mathrm{add}(a^{t}) ▷ Append to Final Sequence
14 st+1←f(st,at)s^{t+1} \leftarrow f(s^{t}, a^{t}) ▷ New State Transition
15 t←t+1t \leftarrow t+1 ▷ Time Step Update
16 return T\mathcal{T} if st=sgs^{t}=s^{g} else ∅\varnothing

Table 1.

Algorithm 1 summarizes the proposed Visual Adaptive Rearrangement Planner (VARP), which integrates the goal-conditioned visual policy, Action Refinement Strategy (ARS), and Progress Enforcement Strategy (PES) into a unified closed-loop framework.

Given the initial and goal states, VARP first predicts a pick heatmap using the goal-conditioned visual policy (Lines 3–4). ARS then performs candidate exploration and feasibility validation to select a manipulation object (Lines 5–8). Conditioned on the selected object, the policy predicts a place heatmap, and ARS is applied again to obtain a feasible placement location (Lines 9–11).

The resulting pick-and-place action is evaluated by PES through a short-horizon rollout probe to detect progress stagnation (Line 12). If stagnation is detected, the selected object is excluded and ARS generates an alternative action, which is accepted without further PES evaluation.

The validated action is appended to the solution sequence and executed to obtain the next state (Lines 13–15). This process repeats until the goal configuration is reached or the maximum execution horizon is exceeded (Line 2), after which VARP returns the generated action sequence T={a0,a1,…}\mathcal{T} = \{a^{0}, a^{1}, \ldots\} (Line 16).

Expert Demonstration Generation

The proposed visual policy is trained on demonstrations generated by the Multi-Stage Monte Carlo Tree Search (MS-MCTS) rearrangement planner [10] in confined 20×2020 \times20 planar workspaces containing 4 to 8 cylindrical objects with radius 1. MS-MCTS formulates rearrangement as a 2D planning problem on the workspace plane and addresses non-monotone rearrangement by decomposing object dependencies into subgoals and jointly reasoning over object relocations and robot actions. We adopt the same 2D formulation for visual policy learning.

For each rearrangement instance, MS-MCTS generates a successful trajectory from the initial configuration s0s^{0} to the goal configuration sgs^{g}. The trajectory is decomposed into state-action pairs (st,at)(s^{t}, a^{t}) and rendered into training tuples (It,Ig,at)(I^{t}, I^{g}, a^{t}). Aggregating all successful trajectories forms the demonstration dataset for supervised policy learning.

Policy Learning

Given the MS-MCTS demonstration dataset, each expert action at=(apkt,aplt)a^{t} = (a_{\mathrm{pk}}^{t}, a_{\mathrm{pl}}^{t}) is converted into target pick and place cells on the workspace grid. The network is trained via supervised imitation learning using the joint objective

L=αLpk+(1−α)Lpl(11)L = \alpha L_{\mathrm{pk}} + (1-\alpha)L_{\mathrm{pl}} \tag*{(11)}

where LpkL_{\mathrm{pk}} and LplL_{\mathrm{pl}} are cross-entropy losses for the pick and place predictions, while α\alpha is the hyperparameter for adjusting loss weights.

VARP uses an encoder-decoder neural network to learn dense spatial action distributions, preserving object spatial relationships rather than directly regressing object coordinates. This representation naturally captures multiple plausible manipulation actions, enabling the downstream ARS and PES modules to refine actions and enforce execution progress at inference time.

Implementation Details

The goal-conditioned visual policy uses a U-Net-based [26] encoder-decoder architecture with 4.2M trainable parameters. The network receives an 8-channel input consisting of the current and goal RGB visual state representations, a pick-indicator channel, and a phase channel. A shared network predicts both pick and place heatmaps over the discretized workspace.

The policy is trained by supervised imitation learning on MS-MCTS demonstrations. Expert pick and place actions are discretized onto the 20×2020 \times20 workspace grid and used as supervision targets. Training uses AdamW [27] with a learning rate of 3×10−43 \times10^{-4}, batch size 128, and a combined pick-place cross-entropy loss with label smoothing [28] (α=0.5\alpha= 0.5). Teacher forcing is applied during place prediction.

During inference, ARS performs top-5 candidate exploration, feasibility validation, and self-placement rejection, while PES uses a three-step rollout probe to detect progress stagnation. All experiments use a maximum execution budget of 5NN actions, where NN is the number of objects.

The final dataset contains 100,012 demonstrations (1M state-action transitions), requiring 34.7 hours to generate on an Intel Xeon Gold 6242R CPU. Training on a single NVIDIA Tesla V100 GPU required 6.5 hours for 10 epochs.

Experiments

Experiment Setup

We evaluate VARP on non-monotone object rearrangement tasks in a synthetic discretized confined workspace of size 20×2020 \times20 units. Each instance contains NN cylindrical objects with radius of 1 unit, where N∈{4,5,6,7,8}N \in\{4, 5, 6, 7, 8\}, representing increasing levels of task difficulty. All methods use the same Linear Motion Planner (LMP) [10] for motion generation to ensure a fair comparison of high-level planning performance. Initial and goal configurations are sampled to be collision-free. Due to the confined and cluttered workspace, many instances require intermediate object relocations before all objects can be placed at their target locations.

We first compare VARP against three representative baselines: MS-MCTS [10], a search-based expert planner; PERTS (CIRS) [15], a perturbation-based theoretically complete rearrangement planner; and the Transporter Network [17], a learning-based manipulation policy, on a randomly generated standard rearrangement benchmark. We then evaluate VARP on a disturbance recovery benchmark against the expert planner (MS-MCTS), followed by ablation studies, out-of-distribution generalization, robustness, and real-world evaluations.

Evaluation Metrics: The following metrics are used for quantitative evaluation:

  • Success Rate (SR) (%): The percentage of rearrangement instances that reach the goal configuration within the evaluation budget, where search-based methods are given a 60 s planning budget and learning-based methods are allowed up to 5NN actions where NN is the number of objects in the scene.

  • Planning Time (PT) (s): The average end-to-end computation time required to generate a rearrangement solution before execution, measured in seconds.

  • Number of Steps (NS): The total number of pick-and-place actions executed to complete the task.

  • Object Displacement Distance (DD): The cumulative distance traveled by all manipulated objects throughout the rearrangement process.

Standard Rearrangement Benchmark

The standard benchmark consists of 2000 rearrangement instances generated by randomly sampling collision-free initial and goal configurations in the confined 20×20 workspace with only the front opening, consisting of 400

instances for each object count from 4 to 8. The constrained workspace frequently requires intermediate object relocations during planning.

Table 1 summarizes the results on the 2,000-instance benchmark. VARP achieves the highest success rate among all methods while requiring only 0.40 s of planning time on average, providing over a 20× speedup compared with the expert. These results demonstrate that VARP effectively handles the long-horizon object dependencies required for confined rearrangement. Although MS-MCTS typically produces slightly shorter rearrangement sequences through explicit online search, it incurs substantially higher computational cost and solves fewer instances. In contrast, VARP prioritizes robust task completion with only a modest increase in relocation actions. PERTS struggles with the complex dependencies and intermediate relocations of confined rearrangement, while the Transporter Network performs substantially worse due to its limited long-horizon reasoning capability.

MethodSR ↑PT ↓NS ↓DD ↓
VARP (Ours)98.250.40 ± 0.2613.59 ± 7.0888.99 ± 39.04
MS-MCTS89.508.67 ± 11.309.46 ± 3.5176.06 ± 28.60
PERTS68.302.20 ± 6.8120.47 ± 49.58351.12 ± 406.53
Transporter21.850.06 ± 0.037.31 ± 2.8456.45 ± 19.32

Table 1. Performance on the standard rearrangement benchmark containing 2,000 evaluation instances. VARP outperforms all baselines in success rate while maintaining sub-second planning time.

Disturbance Recovery Benchmark

To evaluate the adaptive capability of VARP, we introduce a reactive disturbance benchmark. Both VARP and the expert planner first execute three identical relocation actions from a shared initial state. A disturbance is then introduced by randomly relocating 2 to 5 objects to collision-free positions, with 50 instances per disturbance level (200 tasks in total). Following the disturbance, the expert planner is allowed a single replanning step, while VARP continues execution using closed-loop visual feedback.

Table II reports the results, where kk denotes the number of disturbed objects. VARP achieves a 97.5% success rate, outperforming the expert by 6.5% while reducing the average planning time from 9.91s to 0.33s. The improvement is statistically significant (paired McNemar’s test [29], χ2=8.47\chi^2 = 8.47, p=3.6×10−3p = 3.6 \times10^{-3}). As the disturbance level increases, VARP consistently widens its performance advantage, demonstrating strong robustness to execution-time disturbances without repeated replanning.

Ablation Studies

Table III evaluates the contributions of ARS, PES, and training data scale on the same standard rearrangement benchmark mentioned in Section V-B. Removing both ARS and PES reduces the success rate from 98.25% to 36.75%, indicating that the learned policy alone is insufficient for reliable execution in cluttered non-monotone rearrangement tasks. Adding ARS increases the success rate to 96.60% by enforcing feasibility-aware action selection and eliminating most invalid or self-defeating actions. PES further improves performance to 98.25%, with the largest gains observed on dense scenes, e.g., 91.5% to 95.5% on 8-object instances, demonstrating its effectiveness in mitigating residual progress stagnation behaviors. Finally, increasing the training set from 40k to 100k demonstrations improves overall success from 97.45% to 98.25%, with the most noticeable benefit occurring on the most challenging high-cluttered instances.

VariantSR ↑\uparrow (N=45678Avg.)PT ↓\downarrowNS ↓\downarrowDD ↓\downarrow
VARP (ARS + PES)100.099.2098.8097.8095.5098.250.40 ±\pm 0.2613.59 ±\pm 7.0888.99 ±\pm 39.04
VARP w/o PES (ARS only)99.5098.2097.5096.2091.5096.600.21 ±\pm 0.1213.58 ±\pm 7.0188.92 ±\pm 39.26
VARP w/o (ARS & PES)70.8051.5036.2017.2008.0036.750.09 ±\pm 0.048.30 ±\pm 3.3062.87 ±\pm 22.79
VARP w/ 40k training Data99.8099.8098.5097.5091.8097.450.40 ±\pm 0.2513.79 ±\pm 7.0989.14 ±\pm 38.43

Table III. Ablation study of the inference-time reasoning modules. Both ARS and PES consistently improve the planner’s performance, while increasing the amount of expert demonstrations further enhances the success rate.

KKSR ↑\uparrow (Ours / Exp)PT ↓\downarrow (Ours / Exp)NS ↓\downarrow (Ours / Exp)DD ↓\downarrow (Ours / Exp)
2100.0 / 98.00.19 ±\pm 0.07 / 2.90 ±\pm 2.357.26 ±\pm 2.06 / 6.90 ±\pm 1.5352.24 ±\pm 11.18 / 52.32 ±\pm 11.33
396.0 / 98.00.25 ±\pm 0.12 / 7.33 ±\pm 10.449.10 ±\pm 2.60 / 8.39 ±\pm 1.8763.65 ±\pm 18.58 / 61.88 ±\pm 15.41
4100.0 / 90.00.36 ±\pm 0.13 / 13.71 ±\pm 13.2212.68 ±\pm 3.50 / 11.27 ±\pm 2.6586.05 ±\pm 25.11 / 83.02 ±\pm 23.94
594.0 / 78.00.54 ±\pm 0.22 / 17.58 ±\pm 12.1718.38 ±\pm 5.78 / 13.82 ±\pm 2.37117.68 ±\pm 27.64 / 105.83 ±\pm 21.71
ALL97.5 / 91.00.33 ±\pm 0.20 / 9.91 ±\pm 11.7011.78 ±\pm 5.63 / 9.86 ±\pm 3.3679.49 ±\pm 32.83 / 73.95 ±\pm 27.28

Table II. Performance on the disturbance recovery benchmark. VARP achieves higher overall success rates and substantially lower planning times than the expert planner (Exp) under execution-time disturbances.

Generalization and Robustness Evaluation

Beyond benchmark performance, we further evaluate the generalization and robustness of VARP. Specifically, we investigate the ability of the learned policy to handle problem settings that differ from the training distribution and assess its sensitivity to visual perturbations in the environment. These experiments provide additional insight into the scalability and practical reliability of the proposed framework.

  1. Continuous Disturbance Stress Test: To evaluate robustness under persistent environmental changes, we introduce a continuous disturbance benchmark in which two objects are randomly relocated every two relocation actions throughout execution. This setting requires the planner to repeatedly adapt while continuing to progress toward the target arrangement.

Table IV-(a) reports the results. VARP achieves an overall success rate of 94.5%, outperforming the expert planner by 32% while requiring substantially less planning time. The performance gap further increases with task complexity, demonstrating that VARP remains robust under sustained execution-time disturbances.

  1. Out-of-Distribution Generalization: To evaluate the scalability and generalization capability of VARP, we train the policy on rearrangement tasks containing 4 to 8 objects and evaluate it on unseen scenes containing 2, 3, 9, and 10 objects without additional retraining or finetuning.

As shown in Table IV-(b), VARP achieves a 100% success rate on 2 and 3 objects scenes and maintains success rates of 91% and 64% on 9 and 10 objects scenes, respectively. These results demonstrate that VARP generalizes effectively beyond the training distribution to both simpler and more complex rearrangement tasks.

NNSR ↑\uparrow (Ours / Exp)PT ↓\downarrow (Ours / Exp)NS ↓\downarrow (Ours / Exp)
4100.0 / 100.00.20 / 3.166.75 / 6.08
5100.0 / 97.50.37 / 7.5511.05 / 10.15
697.5 / 67.50.51 / 23.4615.92 / 19.70
797.5 / 42.50.69 / 45.9621.00 / 26.24
877.5 / 5.01.22 / 139.9134.26 / 60.00
ALL94.5 / 62.50.57 / 16.9317.01 / 13.90
NNSRPTNS
2100.00.062.51
3100.00.104.16
991.00.9228.40
1064.01.1635.48
ALL88.80.4915.55

Table IV. Robustness and out-of-distribution generalization results. VARP remains robust under continuous disturbances and generalizes to unseen object counts beyond the training distribution. (a) Continuous Disturbance Stress Test. (b) Out-Of-Distribution Generalization.

Real-World Experiments

We validate the proposed VARP on a Trossen ViperX-300S 6-DOF robot arm in a confined workspace measuring 52 cm ×\times 40 cm with cylindrical objects of diameter 6 cm. The robot accesses the workspace exclusively through the front opening, matching the manipulation protocol used in simulation. VARP is trained from expert demonstrations generated using the real robot workspace geometry with 4 and 5 objects.

We then evaluate VARP on 11 real-world rearrangement instances spanning three scenarios: (1) standard in-distribution tasks with 4 and 5 objects, (2) out-ofdistribution tasks containing 3 and 6 objects, and (3) disturbance recovery tasks, where objects are manually displaced during execution. For each instance, the robot is initialized to a randomly generated start configuration and executes VARP in a closed-loop manner until the target arrangement is reached.

VARP successfully solves all 11 real-world instances. One challenging four-object in-place 90° rotation example is shown in Figure 1. The policy performs real-time inference, requiring only 0.0187 s per decision and 0.136 s per instance on average. Despite being trained only on 4- and 5-object demonstrations, VARP generalizes to both smaller (3) and larger (6) object sets and consistently recovers from execution-time disturbances. These results demonstrate effective sim-to-real transfer with fast closed-loop execution, strong generalization, and robust disturbance recovery. Additional examples are provided in the supplementary video.

Real-world execution of VARP on a challenging non-monotone four-object rearrangement task

Figure 1. Real-world execution of VARP on a challenging non-monotone four-object rearrangement task requiring an in-place 90-degree rotation. Our method continuously updates and predicts the next pick-and-place action based on the current and goal configurations until the goal arrangement is achieved, requiring only 0.118 s of planning time for the entire task.

Conclusion

This paper presented VARP, a visual adaptive rearrangement planner for non-monotone object rearrangement planning in confined environments. By combining a goal-conditioned visual policy with the proposed Action Refinement Strategy (ARS) and Progress Enforcement Strategy (PES), VARP transforms and refines raw visual policy predictions into feasible and progress-oriented rearrangement actions without additional training. Extensive simulation and real-world experiments demonstrate that VARP achieves high success rates while substantially reducing planning time compared with the state-of-the-art baselines. Furthermore, VARP exhibits strong robustness to execution-time disturbances and generalizes beyond the training distribution to unseen object counts. Future work will extend VARP to more complex manipulation settings, including multi-arm planning problems and in unstructured environments.

References

  1. [1]S. D. Han, N. M. Stiffler, A. Krontiris, K. E. Bekris, and J. Yu, “High-quality tabletop rearrangement with overhead grasps: Hardness results and fast methods,” arXiv preprint arXiv:2505.09180, 2017.
  2. [2]Z. Pan, A. Zeng, Y. Li, J. Yu, and K. Hauser, “Algorithms and systems for manipulating multiple objects,” IEEE Transactions on Robotics, vol. 39, no. 1, pp. 2–20, 2022.DOI
  3. [3]G. Wilfong, “Motion planning in the presence of movable obstacles,” in Proceedings of the fourth annual symposium on Computational geometry, 1988, pp. 279–288.
  4. [4]J. E. King, M. Cognetti, and S. Srinivasa, “Rearrangement planning using object-centric and robot-centric action spaces,” in IEEE International Conference on Robotics and Automation (ICRA), 2016.DOI
  5. [5]A. Goyal, A. Mousavian, C. Paxton, Y.-W. Chao, B. Okorn, J. Deng, and D. Fox, “IFOR: Iterative flow minimization for robotic object rearrangement,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2202.00732
  6. [6]W. Yuan, J. A. Stork, D. Kragic, M. Y. Wang, and K. Hang, “Rearrangement with nonprehensile manipulation using deep reinforcement learning,” in 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 270–277.DOI
  7. [7]A. H. Qureshi, A. Mousavian, C. Paxton, M. C. Yip, and D. Fox, “NeRP: Neural rearrangement planning for unknown objects,” in Robotics: Science and Systems (RSS), 2021.arxiv.org/abs/2106.01352
  8. [8]B. Tang and G. S. Sukhatme, “Selective object rearrangement in clutter,” in Conference on Robot Learning. PMLR, 2023, pp. 1001–1010.
  9. [9]M. Stilman and J. J. Kuffner, “Planning among movable obstacles with artificial constraints,” The International Journal of Robotics Research (IJRR), vol. 27, no. 11–12, pp. 1295–1307, 2008.
  10. [10]H. Ren and A. H. Qureshi, “Multi-stage monte carlo tree search for non-monotone object rearrangement planning in narrow confined environments,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 12078–12085.arxiv.org/abs/2305.17175
  11. [11]A. Krontiris and K. E. Bekris, “Dealing with difficult instances of object rearrangement,” in Robotics: Science and Systems (RSS), 2015.DOI
  12. [12]H. Ren, J. Kim, A. Tharmathiran, and A. H. Qureshi, “Multi-agent monte carlo tree search for makespan-efficient object rearrangement in cluttered spaces,” arXiv preprint arXiv:2602.02411, 2026.arxiv.org/abs/2602.02411
  13. [13]Y. Labbé, S. Zagoruyko, I. Kalaytavkh, I. Laptev, J. Carpentier, M. Aubry, and J. Sivic, “Monte-carlo tree search for efficient visually guided rearrangement planning,” IEEE Robotics and Automation Letters (RA-L), vol. 5, no. 2, pp. 3715–3722, 2020.
  14. [14]W. Chen, H. Ren, and A. H. Qureshi, “Language-guided active sensing of confined, cluttered environments via object rearrangement planning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 11 224–11 230.arxiv.org/abs/2402.02308
  15. [15]R. Wang, Y. Miao, and K. E. Bekris, “Efficient and high-quality prehensile rearrangement in cluttered and confined spaces,” in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 1968–1975.arxiv.org/abs/2110.02814
  16. [16]J. Kim, H. Ren, and A. H. Qureshi, “Integrating active sensing and rearrangement planning for efficient object retrieval from unknown, confined, cluttered environments,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 12 572–12 578.arxiv.org/abs/2411.11733
  17. [17]A. Zeng, P. Florence, J. Thompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani et al., “Transporter networks: Rearranging the visual world for robotic manipulation,” in Conference on Robot Learning. PMLR, 2021, pp. 726–747.arxiv.org/abs/2010.14406
  18. [18]M. Shridhar, L. Manuelli, and D. Fox, “Multimodal manipulation: What and where pathways for robotic manipulation,” in Conference on Robot Learning (CoRL), 2021.arxiv.org/abs/2109.12098
  19. [19]H. Ren and A. H. Qureshi, “Neural rearrangement planning for object retrieval from confined spaces perceivable by robot’s in-hand rgb-d sensor,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 15 388–15 394.arxiv.org/abs/2402.06976
  20. [20]W. Liu, C. Paxton, T. Hermans, and D. Fox, “StructFormer: Learning spatial structure for language-guided semantic rearrangement of novel objects,” in IEEE International Conference on Robotics and Automation (ICRA), 2022.arxiv.org/abs/2110.10189
  21. [21]M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” in Conference on Robot Learning (CoRL), 2022.arxiv.org/abs/2209.05451
  22. [22]Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandakumar, Y. Zhu, and L. Fan, “VIMA: General robot manipulation with multimodal prompts,” in International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2210.03094
  23. [23]D. Kappler, F. Meier, J. Issac, J. Mainprice, C. G. Cifuentes, M. Wüthrich, V. Berenz, S. Schaal, N. Ratliff, and J. Bohg, “Real-time perception meets reactive motion generation,” IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 1864–1871, 2018.arxiv.org/abs/1703.03512
  24. [24]J. Liu, W. Zeng, R. Urtasun, and F. Yumer, “Deep structured reactive planning,” in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 4897–4904.arxiv.org/abs/2101.06832
  25. [25]C. Bowen and R. Alterovitz, “Closed-loop global motion planning for reactive execution of learned tasks,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014, pp. 1754–1760.
  26. [26]O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2015, pp. 234–241.arxiv.org/abs/1505.04597
  27. [27]I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.arxiv.org/abs/1711.05101
  28. [28]C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.arxiv.org/abs/1512.00567
  29. [29]Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947.pubmed.ncbi.nlm.nih.gov/20254758

Paper details

vlnc:W_mteww7vs
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents