betaFeedback

WorksComputer Science

Inhabitant: An Agent Framework for Long-Horizon Human-Scene Interaction with Spatial Control and Quantitative Feedback

Abstract

Recent work on human-scene interaction (HSI) generation moves from single interactions to long-horizon interaction, where a language-model agent plans and executes a sequence of interactions in a 3D scene under one instruction. However, existing agents struggle in challenging everyday scenarios (e.g., picking up a toy and throwing it into a basket) because they lack spatial control over the motion and quantitative feedback on its execution, and they often fail to complete the interaction sequence. To address these gaps, we present Inhabitant, a training-free agent framework in which the agent specifies each motion through positions, paths and poses, measures how the executed interaction deviates from its intent, and revises the control specification accordingly. For an extensive evaluation in such scenarios, we build Inhabit, a benchmark of 75 household tasks in 31 furniture layouts of an apartment, in which every task chains one to three goals on movable objects and is carried out by the whole body, excluding fingers. On Inhabit, Inhabitant completes 98.7% of the tasks with Claude Opus 5.5, and 93.3% and 82.7% with Claude Sonnet 5.5 and Gemini 4 Argon, respectively. Ablations confirm that without spatial control the agent fails most tasks, and without quantitative feedback its motion becomes less precise. Code is available on the project page.

Project page: https://ryenhails.github.io/Inhabitant/

Introduction

Human-scene interaction (HSI) generation, which synthesizes how a person moves through a scene and acts on its objects, is a core problem for computer graphics and embodied AI [7, 8, 29, 38]. Early work typically focuses on isolated, short-horizon interactions: a single action involving one target object, such as sitting down on a chair, standing up from a chair, or picking up an object [14, 18, 25, 47, 61, 64]. Recent work has shifted towards a more challenging setting of long-horizon interaction, where a language-model agent plans and executes a sequence of interactions in a 3D scene under one instruction, for example, “put the cube in the basket, kick the ball under the table and put the cushion back on the sofa”. These scenes are often simulated environments that respond physically to the body [1, 12, 37, 43, 51, 52, 60].

However, existing agents often fail to complete such a sequence when every interaction must be executed through full-body motion (Figure 1a). In the example above, the agent first tries to pick up the cube, but the hand does not reach close enough to grasp it and closes a few centimetres away. The agent cannot correct this failure: a text prompt cannot specify the required hand position precisely, and rendered frames do not quantify the spatial error. We investigate such failures by adapting three representative agents to these scenarios (§ 5.2) and attribute them to two causes, both concerning the interface between the agent and the generated interaction. First, the control is mostly high-level: the agent conditions the human motion only on a skill with one parameter [37], a text description of each step [49, 60], or a few contact or goal positions [12, 52]. As a result, it cannot specify the motion at the level of precise positions, paths and poses. Second, the feedback is absent or qualitative: most agents plan the interaction once and never observe its outcome [43, 51, 52], including the few that write poses for a motion generator [1, 30]. Other agents observe only rendered frames [37, 60], which may show that an interaction failed but cannot indicate precisely how far the executed interaction deviates from its intent.

Figure 1. One step of a household task: picking up a cube from the counter. (a) Existing agents control the motion through a text prompt or a skill and receive no feedback or only rendered frames; after the hand misses the cube, the agent can only reword the prompt, and the hand misses again. (b) Inhabitant measures how far the hand stopped from the cube (quantitative feedback), fixes where the right hand must be at a given frame (spatial control), and grasps the cube on the next try. The frames in (a) come from our agent restricted to text prompts.

We present Inhabitant, a training-free agent framework that turns long-horizon human-scene interaction into a closed-loop spatial control problem (Figure 1b). The key novelty is to expose two interfaces that existing agents lack: spatially grounded control over generated whole-body motion and quantitative feedback on its physical execution. For control, the agent directly writes the control specification of a pretrained motion generator, Kimodo [32], and specifies the motion by positions, paths and poses rather than only high-level skills or text prompts. For feedback, each motion is executed in a scene simulator with half-physics [36]; the agent then queries measurements of the body and the scene, such as positions, distances and contacts, to quantify how the executed interaction deviates from its intent. It revises the control specification using the measured deviation. The two interfaces close a loop: the agent proposes a motion, executes it, measures the outcome, revises the specification and commits the result as a short motion segment only when the measurements match its intent. By composing such verified segments, Inhabitant builds a complete interaction sequence segment by segment.

For an extensive evaluation in such challenging everyday scenarios, we build Inhabitant (Figure 3), a new benchmark of 75 household tasks in 31 furniture layouts of the ReplicaCAD apartment [39, 56], simulated with half-physics [36] in line with HumanCLAW [37]. Each task states one to three goals, such as placing an object, throwing it or moving it with a foot; ten tasks additionally combine a goal with a body posture, such as sitting, lying or kneeling. The goals are designed to require coordinated use of the hands, the feet and whole-body posture. Inhabit differs from previous benchmarks in two respects. First, HSI benchmarks typically evaluate a single interaction with a static target at a time, such as walking to a piece of furniture and sitting on it [37, 47]. In contrast, Inhabit chains multiple interactions with movable objects under a single instruction, so the outcome of one interaction, including its errors, becomes the initial condition for the next. Second, embodied household benchmarks also contain long-horizon tasks, but agents usually solve them by calling predefined skills such as navigating, picking and placing [3, 56]. In Inhabit, every step must instead be realized through generated whole-body motion, excluding fingers, so the benchmark tests whether the body can reach, hold and move an object through physically executed motion. A success rule judges the physical outcome, while collisions, disturbed objects and foot skate measure the quality of the motion.

Figure 3. Overview of Inhabit. (a) Half-physics simulation: the hollow body shows the generated motion before execution, and the solid body shows the executed motion after scene interaction. (b) The five goal types, illustrated on segments from an Inhabitant episode. (c) Example tasks from each tier in the apartment, showing the body path, each object’s start position and its target. (d) Success rule for the high-tier task in (c), with one condition per goal. (e) The full set of 75 tasks: each circle represents one task, with one sector per goal colored as in (b), alongside three of the 31 furniture layouts.

On Inhabit, Inhabitant completes 74 of the 75 tasks (98.7%) with Claude Opus 5.5 as the agent, and 93.3% and 82.7% of them with Claude Sonnet 5.5 and Gemini 4 Ar-gon, respectively. By comparison, the best of three existing agents adapted to Inhabit completes only 26.7% with Claude Opus 5.5. Ablations show that without spatial control the agent fails most tasks despite retaining all feedback, and without quantitative feedback its motion becomes less precise and the agent less reliably recognizes when a task has been completed.

Related Work

Motion for human-scene interaction. Learned HSI methods synthesize interactions in 3D scenes from interaction data, either kinematically [7, 13, 14, 18, 38, 45, 47, 48, 58, 61, 62, 64] or with physics simulation [8, 25, 40, 44, 46, 50, 54, 55, 59]. These methods are effective for the interaction types and object categories represented in their training data, but are less suited to open-ended long-horizon tasks in which an agent must repeatedly adapt motion to the evolving physical state of a scene. A complementary line of work trains general motion generators without a scene, but allows users to specify spatial constraints in addition to text, such as target positions, paths or poses. Examples include OmniControl [53], Kimodo [32], and related controllable motion generators [2, 4, 15, 16, 19, 26, 27, 42, 65]. Recent systems also let a language model write such constraints, usually once, for a motion generator [6, 23, 66]. Inhabitant builds on this capability without training, but writes the constraints repeatedly and revises them from measurements of the executed motion.

Language agents for human-scene interaction. Agents that direct a body in a scene differ in what the language model writes: a plan for trained controllers or expert models [5, 43, 51], such as UniHSI [52] and World2HOSI [20]; atomic skills, such as HumanCLAW [37]; text for a text-to-motion model, such as Motion-Agent [49] and InteractAgent [60]; or spatial targets for a motion generator [63], such as GPT-Connect [30], BiBo [12] and GEMS [1]. Among them, InteractAgent is closest in spirit because it revises motion after observing rendered outcomes, but the revision is made through text and relies on qualitative visual feedback. Inhabitant instead revises spatial constraints from metric measurements of the execution.

Embodied household tasks. Household benchmarks pose long-horizon tasks with movable objects, such as PARTNR [3] and EmbodiedBench [56], which build on earlier simulated household environments [17, 21, 28, 29, 33, 34, 35, 39]. Language agents typically solve these tasks in a perception-action loop with feedback from the environment [9, 10, 11, 22, 31, 57]. The closest case is FAEA [41], which refines a manipulation program over repeated attempts in simulation. However, these systems generally act through skills, primitives or end-effector commands, including for the humanoid avatars of Habitat 3.0. Inhabitant keeps the long-horizon household setting but requires every step to be realized by generated whole-body motion.

Inhabitant

Inhabitant is a training-free agent framework with three components. The agent is connected to the other two components by two interfaces, which are the core of the framework: spatial control, through which it specifies the motion for the generator by positions, paths and poses, and quantitative feedback, through which the simulator reports the executed result as measurements. We first state the problem and then describe the components one by one.

Problem Formulation

An interaction sequence is a fixed-frame-rate sequence of states Σ=(s0,…,sN)\Sigma= (s_0, \ldots, s_N), where each state sts_t comprises the poses of the body, all objects and the articulated furniture at frame tt. Here, NN denotes the number of frames. The agent does not act frame by frame but through segments: a segment τk=(stk−1,…,stk)\tau_k = (s_{t_{k-1}}, \ldots, s_{t_k}) is a contiguous, few-second portion of the sequence that is generated and executed as a unit. The full sequence is the concatenation of KK segments, Σ=(τ1,…,τK)\Sigma= (\tau_1, \ldots, \tau_K).

The input is a natural-language instruction II and the initial state s0s_0. Task completion is evaluated by a success rule GG, which checks the states of the sequence and returns 1 for success and 0 otherwise. The rule is task-specific, defined together with its instruction (§ 4), and hidden from the agent. The objective is to produce a successful interaction sequence Σ\Sigma such that G(Σ)=1G(\Sigma) = 1.

Agent

The agent π\pi is a language model that builds the interaction sequence segment by segment. The prompt of the agent contains the instruction II, the state description ϕ(s0)\phi(s_0) of the initial state, the documentation of the generator, the dimensions of the simulated body, and the request for person-like motion that neither collides with the scene nor slides its feet (Supp. A.2). At each step it issues one tool call aa [57],

a=π(I,ϕ(s),H)∈{TRY⁡(c,e), MEASURE⁡(q), COMMIT⁡, ROLLBACK⁡(j)},(1)a = \pi(I, \phi(s), H) \in\{\operatorname{TRY}(c,e),\ \operatorname{MEASURE}(q),\ \operatorname{COMMIT},\ \operatorname{ROLLBACK}(j)\}, \tag*{(1)}

where the state description ϕ(s)\phi(s) lists, in text, the positions and sizes of the body and of the objects in the state $s where the preceding segment ended. The agent can re-read the ϕ(s)\phi(s) at any time. HH stores previous tool calls and their results. The agent chooses both the call type and its arguments. The framework prescribes neither the order of the calls nor their content. Next we define the four call types.

  • TRY⁡(c,e)\operatorname{TRY}(c,e) requests a motion for the next segment. The control specification cc states what the body should do, for how long, and where specified joints should be at specified frames; This is how the agent exerts spatial control (§ 3.3). The interaction events ee state which hand grasps or releases which object, and when. The simulator executes the generated motion from the state at the end of the last segment, and the call returns qualitative feedback σ(τ)\sigma(\tau) on the resulting candidate segment τ\tau, i.e., the sequence of states that the execution produces (§ 3.4). A candidate segment is not added to the sequence until it is committed; a subsequent TRY replaces it, allowing the agent to revise a motion by trying again from the same state.

  • MEASURE⁡(q)\operatorname{MEASURE}(q) asks the simulator for a measurement and returns its value M(q,τ)M(q,\tau). The agent chooses the query qq:

Figure 2. Overview of Inhabitant. Left: the agent π\pi controls a pretrained motion generator G\mathcal{G} by writing a control specification cc (spatial control) and interaction events ee. The scene simulator E\mathcal{E} executes the motion, and the agent queries measurements M(q,τ)M(q,\tau) of the candidate segment τ\tau (quantitative feedback) before it tries again, commits or rolls back (§ 3.2). Right: one episode with Claude Opus 5.5, 13 committed segments out of 37 tries. Solid bodies: ends of committed segments, numbered in order; hollow bodies: discarded candidates; insets: three of the interactions; check mark: the success rule is satisfied.

what quantity to measure, between which entities, and at which frame or interval. For example, it may ask for the distance between a hand and an object at a given frame of the last try. These measurements are the quantitative feedback, and the agent receives only the measurements it asks for. The available queries, which may refer to an earlier segment or to the current state, are given in § 3.4.

  • COMMIT appends the candidate segment of the last try to the interaction sequence, τk+1=τ\tau_{k+1}=\tau, and returns the state description of the new state.

  • ROLLBACK(jj) restores the state stjs_{t_j} at the end of an earlier segment jj, discards all later committed segments, and returns the state description of the restored state.

These calls build the interaction sequence in a loop; pseudocode is given in the supplement (Algorithm 1). The loop ends when the success rule is satisfied after a commit, when the agent stops issuing calls, or when the budget of calls to the language model is used up. The sequence Σ\Sigma built up to that point is returned.

Motion Generator: Spatial Control

The motion generator G\mathcal{G} provides the spatial control interface to the agent. We adopt Kimodo [32], a pretrained generator of whole-body motion conditioned on text and spatial constraints. For each TRY call, Kimodo converts the control specification into a motion

m=G(c;h),(2)m = \mathcal{G}(c; h), \tag*{(2)}

where mm is a sequence of whole-body poses at the frame rate of the sequence, conditioned on the motion hh at the end of the preceding segment.

The control specification cc is written by the agent in the input format of Kimodo, which is documented in its prompt. It has three parts. First, a timeline of text prompts with their durations, generated one after another. Second, a set of spatial constraints K\mathcal{K}, through which the agent exerts spatial control: each fixes a set of specified joints at a specified frame to a target position or pose. Kimodo supports three types of constraints: root waypoints, which fix the position of the root on the ground and optionally its heading, end-effector targets, which fix a hand or a foot, and full-body keyframes, which fix all joints. Third, the sampling parameters, namely the random seed and the guidance weights for text and constraints. The agent uses no intermediate representation between the agent and Kimodo. Since a full-body keyframe is written as joint rotations, the agent can ask by forward kinematics where the joints of a pose would be, for example a hand, before it uses the pose in a try. When K\mathcal{K} is not available, the agent can only choose the text prompts, their durations and the seed.

The condition hh in Eq. (2) is the final frames of the executed body in the preceding segment, converted to the body of the generator and imposed as a full-body constraint before the first frame of the new motion.

The body of the generator and the simulated body differ in their bone lengths. Before execution, a map ρ\rho transfers the generated motion to the simulated skeleton while keeping the height of the feet above the floor, and then solves inverse kinematics for the joint rotations. The prompt states the dimensions of the simulated body, so that targets are specified for the body that is executed.

Scene Simulator: Feedback

The scene simulator E\mathcal{E} executes each try and returns qualitative and quantitative feedback on it.

For a try of segment k+1k+1, the simulator starts from the state at the end of segment kk and executes the retargeted motion together with its events.

τ=E(stk,ρ(m),e).(3)\tau= \mathcal{E}(s_{t_k}, \rho(m), e). \tag*{(3)}

The result is a candidate segment τ\tau. During execution, the body may deviate from the generated motion mm because it is blocked by the scene, while objects move only through contact with the body.

As in household benchmarks [39, 56], grasping is abstracted by a rule on the distance between the palm and the object (§ 4). The set ee contains grasps and releases events. A grasp event specifies a hand, an object and a time interval, and attaches the object to the hand if the grasp rule is met at any frame of the interval. A release event specifies a hand and a frame, and detaches the object held by that hand at that frame.

After each try, the agent automatically receives qualitative feedback o(τ)o(\tau) on the candidate segment: the success check, which says whether the success rule GG is satisfied, and a set of rendered frames (Table 1).

Qualitative feedback o(τ)o(\tau), after every try
Checkwhether the success rule GG is satisfied by the sequence extended by the candidate segment
Frameseight frames of the candidate segment, each from two third-person cameras: one follows the body, one views the body and the target of the motion from the side
Quantitative feedback M(q,τ)M(q,\tau), on a query
Body(i) position of a body part; (ii) distance from a body part to an object, a surface or a handle; (iii) contacts of the body; (iv) pose that the body executed at a frame
Scene(v) position of an object; (vi) distance between two objects or between an object and a region; (vii) extent of an object or a piece of furniture

Table 1. The four kinds of feedback that the agent receives on a candidate segment.

Quantitative feedback is returned only when the agent issues a MEASURE(qq) call. Here, qq is a measurement query: it specifies the quantity to measure, the entities involved, and the frame or interval of the candidate segment to which the measurement applies. The measurement operator returns the corresponding numeric value M(q,τ)M(q,\tau). A measurement is a body measurement if its query involves a body part, and a scene measurement otherwise. Tab. 1 lists the available measurement types. The measurements use the same coordinates as the control specification. The agent can therefore converts a measurement into a constraint directly, for example by moving a hand target by the measured distance to an object, or by reusing an executed pose as a keyframe.

Inhabit

Inhabit is built on the 31 furniture layouts of the ReplicaCAD apartment used in EB-Habitat [39, 56]. Its scene simulator E\mathcal{E} uses half-physics [36, 37] (Figure 3a): the body follows the generated motion kinematically but is stopped by furniture and walls, objects respond to the body under rigid-body physics, and grasping and releasing are decided by rules (Supp. B.1).

Inhabit contains 75 household tasks. A task (I,s0,G)(I,s_0,G) consists of an instruction, an initial state and a success rule (§ 3.1). Each instruction states one or more goals from five types (Figure 3b): placing an object, throwing it from at least 1 m away, moving it with a foot without grasping it, opening or closing a drawer or the fridge, and taking a posture. The tasks are organized into four tiers (Figure 3c, Figure 3e): 30 low tasks with one goal, 20 mid tasks with two goals, 15 high tasks with three goals, and 10 posture tasks that combine a goal with a posture.

Each goal gives one condition on the states of the interaction sequence, and the success rule GG of a task is the conjunction of its conditions (Figure 3d). All conditions are listed in Supp. B.2.

An episode is one run of an agent framework on one of these tasks. It ends as described in § 3.2, with a budget of 150 calls to the language model, and counts as a success only if the final returned sequence satisfies the success rule.

On the interaction sequence that an episode returns, we report eight metrics in three groups (Supp. B.3). For task completion, Success is the fraction of episodes whose final state satisfies the success rule, and Sub-goal is the mean fraction of success-rule conditions satisfied at the end. For motion quality, Collisions counts body contacts with furniture and walls, excluding feet on the floor and hands on a grasped object; Disturbed counts non-target objects that end more than 5 cm from their initial positions; and Skate is the foot-skate ratio of the sequence [15]. For cost, Tries counts the motions generated and executed in an episode, Calls counts the calls to the language model, and Tokens reports the size of the history at the end of the episode.

Experiments

Experimental Setup

All experiments run on Inhabit, follow the episode protocol in § 4 and report the metrics defined in § 4. The comparisons in § 5.2 use all 75 tasks, and the ablations in § 5.3 to § 5.5 use an ablation set of 20 tasks to reduce cost. It is selected to cover all four tiers (9/4/3/4) and all goal types (Supp. B.2). In the ablation tables, success and missed grasps are reported as percentages, and the other columns are means per episode.

Unless stated otherwise, Inhabitant uses Kimodo [32] as the motion generator and Claude Opus 5.5 as the language model. We also run it with Claude Sonnet 5.5, Gemini 4 Argon, gpt-6-luna and the open-weight GLM-5.3-Flash, accessed through their APIs. A single prompt is used across tasks, language models and settings; each ablation only removes the description of the component that it turns off.

We compare against three agentic HSI baselines: InteractAgent† [60], which controls a text-to-motion generator with text prompts only; HumanCLAW† [37], which calls a fixed set of atomic skills; and GEMS† [1], which submits one plan and receives no execution feedback. To evaluate them on Inhabit, we adapt each baseline with minimal changes: each baseline keeps its own control interface and loop, while sharing our language model (Claude Opus 5.5), scene simulator, state description, interaction events, success rule and budget of calls (Supp. C.2).

Main Results

With Claude Opus 5.5, Inhabitant satisfies the success rule on 74 of the 75 tasks, whereas the best baseline using the same language model reaches only 26.7% success (Table 2). The gap widens with task length: the baselines solve some low tier tasks but almost none of the mid- and high-tier tasks.

MethodLanguage modelAlllowmidhighpostureSub-goalCollisionsDisturbedSkate (%)TriesCallsTokens
InteractAgent† [60]Claude Opus 5.526.753.310.00.020.034.5105.02.40.952.8133123k
HumanCLAW† [37]Claude Opus 5.520.033.315.06.710.026.842.10.83.611.91291k
GEMS† [1]Claude Opus 5.56.713.35.00.00.013.511.91.05.33.8798k
InhabitantGLM-5.3-Flash44.080.010.00.070.053.236.81.75.938.2112221k
Inhabitantgpt-6-luna30.740.025.06.750.041.445.31.55.446.4122143k
InhabitantGemini 4 Argon82.7100.085.046.780.084.34.00.55.812.573107k
InhabitantClaude Sonnet 5.593.396.790.086.7100.094.917.91.46.223.669127k
InhabitantClaude Opus 5.598.7100.0100.0100.090.099.69.31.04.717.15199k

Table 2. Results on the 75 tasks of Inhabit. †: baseline adapted to Inhabit. Bold: best success rate; for motion quality and cost, best among the runs of Inhabitant only, since the baselines stop early and return shorter sequences.

Each baseline fails at the interface it lacks. Since all baselines share the same language model, lacking a specific interface causes each to fail with a distinct cost pattern. InteractAgent† has feedback but no spatial control: it can observe what went wrong but can only reword its text prompts. It therefore needs three times as many tries as Inhabitant and most failures use up the budget. HumanCLAW† is limited to a fixed set of skills that cannot place a hand precisely enough to interact with an object; the agent recognizes this and stops by itself in every failure, after about a dozen calls. GEMS† has spatial control but no feedback: the errors of its single plan accumulate, most of its grasps miss, and it succeeds only on tasks with loose targets such as kicking or throwing. Figure 4 shows these failure modes on one task.

Figure 4. Inhabitant and the three baselines on one task with the same language model; chips show which of our two interfaces each agent has and what it has in place of a missing one. Red dots: where the throws of InteractAgent† land; yellow circle: the 10 cm radius of the grasp rule; inset: a missed grasp of GEMS†. More examples are in the video on the project page.

The interfaces matter more than the choice of the language model. Across five language models, the success of Inhabitant follows model capability: the three stronger models solve at least 82.7% of the tasks, and the two weaker ones solve less than half, mostly in the low tier and posture tasks. Nevertheless, both weaker models with Inhabitant’s interfaces outperform every baseline with Claude Opus 5.5. The weaker models gain less from each revision: after a missed grasp, a revised try of Claude Opus 5.5 succeeds twice as often as one of the weaker models (Supp. D.2). As a result, weaker models need more tries, collide more often, and mostly fail by using up the budget. Inhabitant is also cheaper per solved task: $1.7 in API fees with Claude Opus 5.5, against $4.8 to $21.1 for the baselines (Supp. D.1).

Spatial Control

To ablate spatial control, Table 3 starts from text prompts alone and adds the three types of spatial constraints one at a time, with the feedback unchanged. Without spatial constraints, the agent fails on most tasks, whether it controls motion through text prompts or atomic skills. With text prompts, it spends four times as many tries as the full system.

Control specificationSuccessCollisionsMissed graspsTriesCalls
Atomic skills30.08.46812.632
Text prompts only20.07.27361.0134
+ root waypoints90.09.66227.969
+ end-effector targets95.07.74517.451
+ full-body keyframes (Inhabitant)95.07.84014.847

Table 3. Granularity of spatial control on the ablation set with Claude Opus 5.5.

The three types of constraints help at different levels. Root waypoints account for most of the success gain, from 20.0% to 90.0%: they decide where the body walks and stops, which every task with several steps needs first. End-effector targets and full-body keyframes then add precision rather than success. They solve only one more task, but missed grasps, the share of the grasp events of all tries that attach no object, fall by a third and the tries per episode by half. The tasks that fail with root waypoints alone are reaches in which the palm stops short of the object (Supp. D.3).

Finally, we examine how different language models use the three types of constraints when all of them are available. In the episodes of Table 2, we count segments whose control specification contains an end-effector target or a full-body keyframe. Claude Opus 5.5 uses one in more than a third of its segments, while GLM-5.3-Flash and gpt-6-luna rarely use them and rely mostly on root waypoints. These two language models also miss grasps more often (Supp. D.2).

Feedback

To ablate feedback, Table 4 keeps the full control specification and removes parts of the feedback: first the two kinds of measurements (rows 1 to 4), then the success check (rows 5 and 6). Without measurement (row 1), the agent solves only two tasks fewer than with all feedback (row 4), but it acts far less precisely: it needs about 50% more tries, and produces roughly twice as many body collisions. The rendered frames show that a try failed but not by how much, so the revisions are coarse.

#CheckFramesBodySceneSuccessCollisionsMissed graspsTriesCalls
1✓85.014.65822.754
2✓✓95.010.94719.364
3✓✓✓95.012.46122.059
4✓✓✓✓95.07.84014.847
5✓✓✓85.09.54719.860
6✓70.011.15321.654

Table 4. Granularity of feedback on the ablation set with Claude Opus 5.5 and all three types of spatial constraints. Check, Frames, Body and Scene are the four kinds of feedback of Table 1.

Body and scene measurements complement each other. Either kind alone (rows 2 and 3) is enough to reach the success rate of the full feedback, but only the two together reduces the tries and the collisions. Missed grasps fall with the body measurements, which localizes the hand, but not with the scene measurements.

The success check tells the agent after every try whether the task is solved. Rows 5 and 6 remove this check, requiring the agent to decide when to stop. With measurements, success drops from 95.0% to 85.0% (rows 4 and 5); with rendered frames alone, it drops from 85.0% to 70.0% (rows 1 and 6). The typical failure is premature stopping before the task is actually solved. On all 75 tasks without the success check, Inhabitant still achieves 86.7% success, and nine of ten episodes in which it stops by itself are successful (Supp. D.4). Thus, quantitative measurements allow the agent to verify the outcome by itself in most episodes.

Further Analysis

In Inhabitant, the agent may revise a try using feedback as many times as needed before committing it (§ 3.2). To assess the value of this revision loop, we compare Inhabitant with two settings that remove it (Table 5). In Single-Try, each try is committed as soon as it is executed, so the agent still receives feedback but can only correct a mistake in later segments (similar to ReAct [57]). In Open-Loop, the agent writes all control specifications at once, and they are executed without any feedback. Open-Loop solves no task, since nothing corrects the deviations of the executed motion from the plan. Single-Try solves four tasks fewer than Inhabitant, and its body collides about four times as often, because every imprecise try stays in the interaction sequence. Revising a try before committing it therefore helps success and, more importantly, keeps the interaction sequence clean.

SettingSuccessCollisionsTriesCalls
Inhabitant95.07.814.847
Single-Try75.030.413.236
Open-Loop0.010.94.22

Table 5. The execution loop on the ablation set with Claude Opus 5.5.

The scene simulator moves the body kinematically, without balance or torques. To test whether the interaction sequences are nevertheless physically plausible, we track those of Inhabitant with Claude Opus 5.5 with the pre-trained physics controller PHC+ [24] on a flat floor, in the parts where the body is not supported by furniture. The controller follows 94% of these parts without falling, and half of its failures are in tasks that end in a kneeling or lying posture (Suppl. D.6).

Conclusion and Future Work

We presented Inhabitant, a training-free agent framework for long-horizon human-scene interaction that gives the agent what existing agents lack: spatial control over the motion and quantitative feedback on its execution. To evaluate such interaction, we built Inhabit, a benchmark of 75 household tasks in which every step is carried out by the motion of the whole body and a success rule decides on the physical outcome. On Inhabit, Inhabitant completes 98.7% of the tasks with Claude Opus 5.5, where existing agentic HSI baselines complete at most 26.7% with the same language model, and the ablations confirm that both interfaces are needed: without spatial control the agent fails most tasks, and without quantitative feedback its motion is less precise.

Inhabitant has four main limitations. First, it requires a scene simulator in which a motion can be tried repeatedly, making it best suited for offline uses such as animation and data generation; an online setting, where every executed motion is irreversible, are better reflected by the Single-Try results in § 5.5. Second, the body is driven kinematically, without balance, torques or fingers, and grasping is rule-based; a pretrained physics controller already follows most of the interaction sequences (§ 5.5), and driving the body with such a controller is future work. Third, the agent is given the state description of the scene and a success check after each try; it still solves most tasks without the check (§ 5.4), and replacing the given state by perception is left for future work. Fourth, because of budget constraints, each task is run once and the ablations use a 20-task subset. Moreover, the baselines are our adaptations. Future evaluations should include repeated runs and comparisons with the original implementations.

LLM Usage Statement

LLMs are an object of study in this work: the agent of Inhabitant is an LLM that is used without training. We evaluate Claude Opus 5.5, Claude Sonnet 5.5, Gemini 4 Argon, gpt-6-luna and the open-weight GLM-5.3-Flash as this agent, and Claude Opus 5.5 as the agent of the adapted baselines (§ 5.1). In addition, we used Claude as an assistant for writing and debugging the code of the agent, the benchmark and the evaluation, for launching and monitoring experiment runs and computing the reported statistics, for surveying related work, and for drafting and editing parts of the manuscript text and figures. The research questions, method design, benchmark tasks, experimental protocol and conclusions are the authors’ own. All LLM-assisted code, text, figures and citations were reviewed and verified by the authors, and all reported results come from our own runs.

Details of Inhabitant

This section supplements § 3.

The loop of the agent

Algorithm 1 gives the loop of § 3.2 as pseudocode.

Algorithm 1 Building an interaction sequence with Inhabitant

Require: instruction II, initial state s0s_0, budget BB

Ensure: interaction sequence Σ=(τ1,…,τK)\Sigma= (\tau_1,\ldots,\tau_K)

1: s←s0s \leftarrow s_0, Σ←()\Sigma\leftarrow(), τ←∅\tau\leftarrow\varnothing, H←∅H \leftarrow\varnothing 2: for n=1,…,Bn = 1,\ldots,B do ▷ one call per step 3: a←π(I,ϕ(s),H)a \leftarrow\pi(I,\phi(s),H) 4: if a=TRY⁡(c,e)a = \operatorname{TRY}(c,e) then ▷ spatial control 5: τ←ES(ρ(G(c;h)),e)\tau\leftarrow\mathcal{E}_S(\rho(\mathcal{G}(c;h)),e) ▷ hh: end of Σ\Sigma 6: r←o(τ)r \leftarrow o(\tau) ▷ qualitative feedback 7: else if a=MEASURE⁡(q)a = \operatorname{MEASURE}(q) then 8: r←M(q,τ)r \leftarrow M(q,\tau) ▷ quantitative feedback 9: else if a=COMMIT⁡a = \operatorname{COMMIT} and τ≠∅\tau\ne\varnothing then 10: append τ\tau to Σ\Sigma, s←s \leftarrow last state of τ\tau, τ←∅\tau\leftarrow\varnothing 11: if G(Σ)=1\mathcal{G}(\Sigma) = 1 then return Σ\Sigma 12: end if 13: r←ϕ(s)r \leftarrow\phi(s) 14: else if a=ROLLBACK⁡(j)a = \operatorname{ROLLBACK}(j) then 15: keep τ1,…,τj\tau_1,\ldots,\tau_j in Σ\Sigma, s←stjs \leftarrow s_{t_j}, τ←∅\tau\leftarrow\varnothing 16: r←ϕ(s)r \leftarrow\phi(s) 17: else return Σ\Sigma ▷ the agent stops 18: end if 19: H←H∪{(a,r)}H \leftarrow H \cup\{(a,r)\} 20: end for 21: return Σ\Sigma

Prompt and tool definitions

The prompt of the agent (§ 3.2) consists of a system prompt and a first message with the instruction and the state description ϕ(s0)\phi(s_0) of the initial state. The system prompt has two parts. The first part, written by us, is given verbatim in Figure 5 and Figure 6: it describes the inputs of the motion generator, the task of the agent, the tools, the map, the rule of the interaction events and the dimensions of the simulated body. The second part is the public documentation of Kimodo, copied unchanged: its pages on the constraints, on their JSON format (coordinate space and units, constraint types, examples), on the generation parameters and on best practices (text prompting, constraints, limitations), about 11,000 characters. The system prompt is the same for every task and every language model, and it names no object, no piece of furniture and no task.

System prompt of Inhabitant, part 1 of 2

Figure 5. System prompt of Inhabitant, part 1 of 2: the inputs of the motion generator, the task and the tools. The text is verbatim, with the symbols for the angle and the degree spelled out. It is followed by Figure 6 and by the documentation of Kimodo.

System prompt of Inhabitant, part 2 of 2

Figure 6. System prompt of Inhabitant, part 2 of 2: the map, the rule of the interaction events and the simulated body.

Table 6 lists the tools that implement the four calls of Eq. (1), and Table 7 the queries of a measurement. A measurement may contain several queries, and a query refers to the last try or, with an index, to a committed segment. Besides the four calls, the agent can request the state description of the current state at any time. Figure 7 shows a try, a measurement and a commit of one episode as the agent issues them and as they are answered.

CallTool and argumentsResult
TRY⁡(c,e)\operatorname{TRY}(c,e)generate segment: text prompts, durations, constraints, seed, post-processing, guidance weights (c)(c) and interaction events (e)(e)success check and eight rendered frames of the try
MEASURE⁡(q)\operatorname{MEASURE}(q)inspect: a list of queriesone value or time series per query
COMMIT⁡\operatorname{COMMIT}commitstate description within 2 m of the body
ROLLBACK⁡(j)\operatorname{ROLLBACK}(j)retry-from: index of a committed segmentstate description
–get_statefull state description

Table 6. The tools of Inhabitant and the calls of Eq. (1) that they implement.

QueryKindReturns
wherebody, sceneposition of a body part or an object, now, at a time of the try or every 0.1 s; for an object also the surface or hand it is on, for a handle whether it is open
distancebody, scenedistance between two entities (body parts, objects, surfaces, furniture) every 0.1 s, with the direction at the closest moment
contactsbodycontacts of the body with the scene during the try
posebodypose that the body executed at a time of the try, in the format of a full-body keyframe
extentscenecorners and heights of an object, a surface or a piece of furniture
fk–joint positions of a pose written as a full-body keyframe, before it is used in a try (§ 3.3)

Table 7. The queries of a measurement and the kind of quantitative feedback of Table 1 that they give.

Excerpt of an episode of Inhabitant with tool-call panels and state description

Figure 7. Excerpt of an episode of Inhabitant with Claude Opus 5.5 on the task “Could you tidy up the toy airplane? It goes in the cardboard box by the TV stand.” The agent requests a try for its second segment with a root waypoint and a grasp event, measures where the airplane is after the try, and commits the candidate segment. Blue boxes are the calls of the agent and grey boxes the results that it receives. The state description is cut.

Choice of motion generator

We compare Kimodo with OmniControl [53], a text-to-motion model with spatial control of joints, and with Kimodo given only text prompts, as a plain text-to-motion model. The control specifications are those of the segments that Claude Opus 5.5 committed in the main runs. Over all 526 committed segments, 8.7% contain a full-body keyframe, 1.3% several text prompts and 65.8% a heading of the root, none of which OmniControl takes as input. In addition, 85.7% of the segments continue from the executed end of the previous one, while OmniControl starts every motion from its own canonical first frame. To compare the generators on what both can express, we sample 120 committed segments without full-body keyframes or several text prompts, 60 with only root waypoints and 60 with end-effector targets. Every control specification is generated once from a standing start with the same seed. OmniControl receives the same text prompt and duration, the pelvis position on the ground for each root waypoint, and the 3D positions of the constrained joints for each endeffector target, computed with Kimodo’s kinematics from the pose in the control specification. We measure the errors at the constrained frames in the frame of the segment, after resampling OmniControl from 20 to 30 frames per second.

Kimodo reaches the end-effector targets almost exactly and the root waypoints within a few centimetres (Table 8). OmniControl reaches the median root waypoint within 2 cm, but one in ten misses by more than 25 cm, and 81% of its end-effector targets are within 10 cm. It cannot set the heading, which the agent uses in two thirds of its segments to face an object or a piece of furniture. It also takes about 25 times as long per segment, which would add about half an hour of generation to an episode of 18 tries. Text prompts

GeneratorRoot (m)End effector (m)Within 10 cmHeading (°)Time (s)
Kimodo, full control specification0.0350.005100%1.53.7
OmniControl0.0850.05781%39.995.4
Kimodo, text prompts only1.8860.7080%52.93.6

Table 8. Generators on 120 control specifications of the agent: mean error of the root at its waypoints (214 targets) and of the constrained joints of the end-effector targets (133 targets), share of end-effector targets reached within 10 cm, mean heading error at the waypoints that set a heading, and generation time per segment on a V100. OmniControl has no heading input.

alone miss the targets by metres, as with InteractAgent† in the main paper. Foot skate of the generated motion is higher for Kimodo (19.7% of frames, 27.7% in segments with only root waypoints) than for OmniControl (13.0%) and for text prompts only (13.6%), so we do not claim better motion quality for Kimodo. The comparison has limits. All motions start from a standing pose although the agent wrote the control specifications for the executed end of the previous segment, OmniControl’s HumanML3D body differs from Kimodo’s SMPL-X body by a few centimetres, and the motions are not executed in the simulator.

Details of Inhabit

This section supplements § 4.

Simulation

A grasp event succeeds if the palm centre, 5.4 cm beyond the wrist, stays within 10 cm of the object for 0.1 s in its time window. A release event lets go of the object with the velocity of the hand, so that an object can be thrown.

A drawer or a fridge door follows the hand that holds its handle. The body is driven kinematically, without a balance or torque model, and execution is not exactly repeatable: identical inputs can differ by a few centimetres, and every execution is taken as it happened.

The executed body drifts backward. On 286 tries of early development episodes we compared the pelvis of the motion as it is placed in the scene with the pelvis as executed. In the 108 tries in which the body should stay in place, the executed pelvis ends a median of 8.6 cm behind its target along the facing of the body, and the lag grows steadily at about 3 cm/s (1.8 cm after 0.5 s, 6.4 cm after 2 s). Both feet move back with the pelvis, so the whole body slides on the floor, and the drift is faster when the trunk leans forward (about 4 cm/s). The transfer to the simulated skeleton and the inverse kinematics contribute less than 2 mm. We keep the simulation as released and do not correct the drift: the agent measures where the body ended and compensates for it in its next try.

To quantify the repeatability, we take all pairs of consecutive tries in runs of Inhabitant with the five language models on all 75 tasks that start from the same state with an identical control specification and identical interaction events (257 pairs over the five language models). The final pelvis positions of a pair differ by a median of 0.3 cm, by less than 1 cm in 81% of the pairs and by less than 5 cm in 93%.

The 75 tasks

A throw goal counts only if the object is released with the wrist at least 1 m (horizontally) from the target region, and a goal of moving an object with a foot only if the object was never grasped. Goals of EB-Habitat’s 300 episodes need only grasp and release events, whereas ours also require throwing, kicking, pushing and postures. Table 10 defines the conditions of the success rule, and

ConditionHolds if
onthe object rests inside the region of the target surface and is not held
inthe centre of the object is inside the target box or basket, wherever that container is now, and the object is not held
insidethe object is inside the fridge and is not held
openthe fridge door is opened by more than 1.07 rad, or the drawer is pulled out by more than 0.35 m
closedthe joint of the door or drawer is within 0.15 of its closed position
throwthe object ends on or in the target, and the palm was at least 1 m (horizontally) from the target when the object left the hand
kickthe object ends at the target after a foot touched it, and it is not in a hand from that contact on
pushthe object ends at the target, was never held, and the body touched it
sitfor 1 s, the pelvis is over the surface and 0.05 to 0.35 m above it, with the trunk within 45∘45^\circ of vertical
liefor 1 s, the trunk is more than 60∘60^\circ from vertical, with the pelvis and the head over the surface and the pelvis at most 0.35 m above it
kneelfor 1 s, a knee is on the floor and the pelvis is 0.25 to 0.8 m above it
avoidthe body never touches the named object
togetherthe two objects are held at the same time, one in each hand
EBthe goal of the original EB-Habitat episode holds

Table 10. The conditions of the success rules. The target of a kick or a push can be the floor under a piece of furniture, where the object must lie less than 0.15 m above the floor.

TierTaskswith throwby footEB original
low (1 goal)30553
mid (2 goals)20343
high (3 goals)15032
posture10––0
total758128

Table 9. Composition of the tasks: number of tasks that contain a throw goal or a goal of moving an object with a foot, and original EB-Habitat episodes. All other goals are place goals. The 10 posture tasks are sitting (4), lying (2) and kneeling (4). Goal counts of EB originals follow EB’s own goal.

Table 11 to Table 13 list the instruction and the conditions of every task. The 20 tasks of the ablation set, marked in these tables, cover all four tiers (9 low, 4 mid, 3 high and 4 posture tasks) and all five goal types.

TaskInstructionConditions
low_01Could you tidy up the toy airplane? It goes in the cardboard box by the TV stand.in
low_02Put the lego in the cardboard box next to the black table.in
low_03Put the toy airplane in the cardboard box next to the black table.in
low_04The towel is on the floor. Put it back on the sofa.on
low_05There’s a towel lying on the floor - could you put it back on the sofa?on
low_06Pick up the remote control from the floor and put it on the TV stand.on
low_07*Could you put the can away on the top shelf of the tall shelf next to the right counter?on
low_08Put the ball on the top shelf of the tall shelf next to the brown table.on
low_09Could you put the orange away on the bottom shelf of the TV stand?on
low_10Please bring the dirty coffee mug on the brown table to the sink.on
low_11*Bring the coffee mug on the brown table to the sink - careful, don’t touch the vase beside it.on, avoid
low_12*Put the banana in the fridge.inside
low_13Put the apple in the fridge.inside
low_14Please open the fridge.open
low_15The fridge door is open, please close it.closed
low_16*Could you pull the middle drawer in the kitchen open?open
low_17Open the right drawer of the kitchen counter.open
low_18*The fridge door is open because of my mistake. Can you close it, please?EB
low_19Hey, on the sink, I accidentally left my plate, can you bring it to the right counter?EB
low_20Can you bring something to assist in cutting paper at the sink?EB
low_21*Toss the ball into the cardboard box by the TV stand from where you can, at least a metre away.throw
low_22Toss the toy airplane into the basket by the TV stand from where you can, at least a metre away.throw
low_23Throw the ball lying on the floor onto the sofa - from a metre or more.throw
low_24*Toss the cushion onto the sofa from at least a metre away.throw
low_25Throw the towel lying on the floor onto the sofa - from a metre or more.throw
low_26*There’s a ball on the floor - kick it under the black table.kick
low_27Kick the ball on the floor under the black table.kick
low_28Nudge the shoe box under the black table - no need to pick it up, just push it with your foot.push
low_29Nudge the cardboard box under the black table - no need to pick it up, just push it with your foot.push
low_30*Push the shoe on the floor out of the way with your foot (half a metre or more); don’t pick it up.push

Table 11. The 30 low tasks of Inhabit: instruction and conditions of the success rule (Table 10). * marks a task of the ablation set.

TaskInstructionConditions
high.01Clear the sofa: all the things on it go on the top shelf of the tall shelf next to the black table.3×on
high.02Clear the sofa: all the things on it go on the top shelf of the tall shelf next to the brown table.3×on
high.03*Take everything out of the fridge, put it on the black table, and close the fridge again.2×on, closed
high.04Empty the fridge onto the black table, then shut the fridge door.2×on, closed
high.05Put all the things on the black table into the cardboard box beside it.3×in
high.06Clear everything off the black table into the cardboard box next to it.3×in
high.07*Could you tidy up? Put the lego in the basket next to the sofa, kick the ball under the black table, and the cushion on the floor goes back on the sofa.in, kick, on
high.08Could you tidy up? Put the rubriks cube in the basket next to the sofa, kick the ball under the black table, and the cushion on the floor goes back on the sofa.in, kick, on
high.09Could you tidy up? Put the toy airplane in the basket next to the TV stand, kick the ball under the black table, and the cushion on the floor goes back on the sofa.in, kick, on
high.10*Please move the coffee mug, dinner plate and serving bowl on the right counter to the black table.3×on
high.11Set the black table: bring the coffee mug, the dinner plate and the serving bowl from the right counter.3×on
high.12Put the apple and the plum in the fridge and close the door afterwards.2×inside, closed (ordered)
high.13Put the peach and the lemon in the fridge and close the door afterwards.2×inside, closed (ordered)
high.14Move the ball to the brown table, the block to the brown table, and the hammer to the black table.EB
high.15Move the lemon to the left counter, the block to the brown table, and the bowl to the brown table.EB
posture.01*Pick the cushion up off the floor and put it on the sofa; after that, take a seat on the sofa.on, sit (ordered)
posture.02Put the cushion from the floor back on the sofa, then sit down on the sofa for a moment.on, sit (ordered)
posture.03*Pick the cushion up off the floor and put it on the sofa; after that, have a rest lying on the sofa.on, lie (ordered)
posture.04Put the cushion from the floor back on the sofa, then lie down on the sofa for a moment.on, lie (ordered)
posture.05*Bring the coffee mug from the right counter to the black table and sit down on the chair.on, sit (ordered)
posture.06Take the coffee mug on the right counter over to the black table, then have a seat on the chair.on, sit (ordered)
posture.07Get down on your knees and put the cup away on the bottom shelf of the TV stand.on, kneel
posture.08Get down on your knees and put the apple away on the bottom shelf of the TV stand.on, kneel
posture.09*The toy airplane rolled under the black table. Kneel down, get it out and put it in the cardboard box.in, kneel
posture.10The block rolled under the black table. Kneel down, get it out and put it in the cardboard box.in, kneel

Table 13. The 15 high tasks and the 10 posture tasks of Inhabit, as in Table 11.

Metrics

Success is the share of episodes whose final state satisfies the success rule, and sub-goal the mean share of the conditions of the success rule that hold at the end, where a required order between two conditions counts as a condition of its own. Collisions count the contacts of the body with furniture, walls and pillars in the sequence, excluding the feet on the floor and the hands on a grasped object. Disturbed objects are objects that the task does not ask to move and that end more than 5 cm from their start. Skate is the foot-skate ratio of Karunratanakul et al. [15]. It partly reflects a backward drift of the body during execution (about 3 cm/s while standing), so we use it only as supporting evidence. Tries count the tries of an episode, calls the calls to the language model, and tokens the size of the history at the end of the episode, which is comparable across language models. In the ablations we additionally report missed grasps, which take every grasp event of every try of an episode and count the share that does not attach the object, that is, where the palm centre does not stay within 10 cm of the object for 0.1 s in the time window. The share is computed per episode and averaged over the episodes, and an episode without grasp events counts as 0.

Corrections of success rules

After the main runs we corrected three success rules and re-judged all runs of every model, baseline and ablation from their recorded final states, without running anything again. The agents never saw any of these rules, since every try only reports whether the success rule is satisfied. (1) high_15, an original EB-Habitat task, asks for a bowl. Our benchmark renames the ReplicaCAD bowl to “serving bowl” in its text, and the same name was used when EB’s goal looked for an object of category bowl, so the goal could never be met. EB’s goals now use EB’s own categories. Opus 5.5 had completed the task. (2) high_03 and high_04 required both fruits to be on the table before the fridge door closes. Closing the door while still holding the last fruit and then putting it down also completes the instruction, so only the final state is checked now. This changes high_03 of the Single-Try ablation. (3) Kneeling required the trunk to be within 30 degrees of vertical. Kneeling and leaning under a table to fetch a toy failed that test, so a kneel now needs a knee on the floor with the pelvis between 0.25 and 0.8 m above it for one second. This changes body_09 of gpt-6-luna. The records are in the released runs (runs/checks/rejudge_*).

Experimental setup

This section describes how the experiments of § 5.1 are carried out.

Implementation

Language models. Claude Opus 5.5 and Claude Sonnet 5.5 use adaptive thinking at medium effort, Gemini 4 Argon a medium thinking level at temperature 1, gpt-6-luna (version 2026-09-22) medium reasoning effort, and GLM-5.3-Flash high reasoning effort at temperature 1 and top-pp 0.95. Every language model receives the same system prompt and the same tools (Supp. A.2). The history of an episode is never truncated or summarized, so the rendered frames of all tries stay in it; the eight frames of a try are sent as one image, 768 pixels wide.

Motion generator. We use the released Kimodo model for the SMPL-X skeleton (Kimodo-SMPLX-RP-v1) without any training, at 30 frames per second. A segment lasts 0.5 to 8 s. Unless the agent sets them, the seed is 0, post-processing is on, and the guidance weights of the text and of the constraints are both 2, the defaults of Kimodo. After the motion of a segment the simulation runs on for 0.3 s, or 1.5 s after a release event, so that the objects settle.

TaskInstructionConditions
mid_01Could you tidy the toy airplane and the rubiks cube into the basket by the black table?2×in2 \times\mathrm{in}
mid_02Could you tidy the lego and the toy airplane into the cardboard box by the sofa?2×in2 \times\mathrm{in}
mid_03Tidy up: the rubiks cube goes in the basket next to the sofa; throw the toy airplane in from a metre or more.in, throw
mid_04The fridge is open. Put the banana inside and close the door.inside, closed (ordered)
mid_05The fridge is open. Put the peach inside and close the door.inside, closed (ordered)
mid_06*Take both the coffee mug and the serving bowl on the black table to the sink in one go - one in each hand.2×on2 \times\mathrm{on}, together
mid_07Take both the coffee mug and the serving bowl on the brown table to the sink in one go - one in each hand.2×on2 \times\mathrm{on}, together
mid_08Tidy the rubiks cube into the cardboard box next to the sofa, and push the shoe on the floor out of the way with your foot (at least half a metre).in, push
mid_09Put the lego in the cardboard box by the sofa. There’s a shoe lying in the way - nudge it aside with your foot, half a metre or more, without picking it up.in, push
mid_10Clear the coffee mug and the dinner plate from the brown table into the sink.2×on2 \times\mathrm{on}
mid_11Please take the dirty coffee mug and dinner plate on the black table to the sink - without touching the vase next to them.2×on2 \times\mathrm{on}, avoid
mid_12Put the ball on the top shelf of the tall shelf next to the black table and the orange on the bottom shelf of the TV stand.2×on2 \times\mathrm{on}
mid_13The cushion and the towel are on the floor: put the cushion back on the sofa and throw the towel onto it from a metre or more.on, throw
mid_14Two things are on the floor: put the cushion back on the sofa, and toss the towel onto the sofa from at least a metre away.on, throw
mid_15Tidy up: the rubiks cube goes in the cardboard box next to the sofa, and the ball gets kicked back under the black table.in, kick
mid_16*Please tidy up: the rubiks cube in the cardboard box and kick the ball back under the black table.in, kick
mid_17*Could you put the banana away in the left drawer in the kitchen?open, inside (ordered)
mid_18*Transfer all ball to the brown table.EB
mid_19Transfer all lego to the sofa.EB
mid_20Transport all plate and put them on the right counter.EB

Table 12. The 20 mid tasks of Inhabit, as in Table 11. “Ordered” means that the conditions must become true in the order of the instruction.

One NVIDIA V100 GPU hosts the motion generator and the simulations of several episodes. A try of Claude Opus 5.5 takes about 45 s for generation, execution and rendering, and an episode 21 minutes on average, of which 6 are spent in the language model. The episodes of the other language models take longer: 38 minutes for Claude Sonnet 5.5, 42 for Gemini 4 Argon, 90 for gpt-6-1una and 110 for GLM-5.3-Flash.

Budget and run time. An episode has a budget of 150 calls to the language model, and every task is run once.

Ablations. An ablation changes the prompt only where it describes the component that is turned off. For the granularity of spatial control (Table 3), one sentence states which types of constraints are available, and constraints of the other types are rejected; with text prompts only, the constraints and their documentation are removed. For the granularity of feedback (Table 4), the description of the measurements lists only the queries of the remaining kind, or the measurement tool is removed. Without the success check, a try no longer reports whether the success rule is satisfied, and the agent ends the episode with an additional call when it considers the task done. In Single-Try the commit and the rollback are removed, and in Open-Loop the agent has a single call that takes the control specifications of all segments.

Adapted baselines

The three baselines of the main paper are adaptations and do not run the original systems. Each keeps the control interface of the original agent, and all use Claude Opus 5.5, our scene simulator, the state description, the interaction events, the success rule and the budget of 150 calls. InteractAgent† follows the loop of InteractAgent with a text-to-motion generator (MotionLLM): the agent writes only text prompts, and the generator takes no duration, seed or spatial constraint. After each try the agent receives the eight frames, the success check and the state description, and it has no measurement queries. It first writes a plan of steps, may redo each step twice before the segment is committed, and may restart the episode from the beginning up to five times. It differs from the setting with text prompts only of Table 3, which keeps our motion generator and all our feedback. HumanCLAW† calls the skills of HumanCLAW (walk, turn, side step, step back, sit and stop). Each skill is executed at once and cannot be revised, and the agent has no measurement queries. It sees the rendered frames and the state description after each skill. GEMS† may query positions and extents of the initial scene, then submits one plan of segments with text prompts and full-body keyframes for the same motion generator. The plan is executed once and the agent receives no feedback on it.

Additional results

This section gives the details of the results of § 5.2 to § 5.5, in the order of the main paper.

Baselines and language models (Sec. 5.2)

Table 14 shows how the failed episodes of each baseline end and how many of its grasps miss, which supports the three failure modes of § 5.2. InteractAgent† fails on 55 tasks, and 46 of these episodes use up the budget of 150 calls; in the other nine the agent stops by itself after 101 to 149 calls. HumanCLAW† fails on 60 tasks, and in every one of them the agent stops by itself, after 12 calls on average. GEMS† executes its single plan to the end in every episode and fails in 70 of them; four of its five successes contain a throw or a push, whose targets are loose. All three miss most of their grasps, against 48% for Inhabit with the same language model.

BaselineFailed episodesBudget used upStopped by the agentPlan executedMissed grasps
InteractAgent†55469–83%
HumanCLAW†60060–75%
GEMS†70––7068%

Table 14. How the failed episodes of the baselines of Table 2 end, and their missed grasps over all 75 episodes (Supp. B.3). GEMS† has one call for its plan, so its episodes end when the plan is executed.

Table 16 gives the result of every task for the eight runs of Table 2. No baseline solves a high task except HumanCLAW† on one, and 48 of the 75 tasks are solved by none of the three baselines.

TaskIA†HC†GEMS†GLMlunaGeminiSonnetOpusTaskIA†HC†GEMS†GLMlunaGeminiSonnetOpus
low_01✓✓·✓·✓✓✓mid_09····✓✓✓✓
low_02✓··✓·✓✓✓mid_10·····✓✓✓
low_03···✓·✓✓✓mid_11·····✓✓✓
low_04✓✓✓✓✓✓✓✓mid_12······✓✓
low_05✓✓·✓✓✓✓✓mid_13··✓··✓✓✓
low_06·✓·✓✓✓✓✓mid_14·····✓✓✓
low_07*·····✓✓✓mid_15*·····✓✓✓
low_08✓··✓·✓✓✓mid_16·✓···✓✓✓
low_09·····✓✓✓mid_17*·✓·✓✓✓✓✓
low_10···✓✓✓✓✓mid_18*····✓✓✓✓
low_11*✓··✓·✓✓✓mid_19·····✓·✓
low_12*✓··✓·✓✓✓mid_20····✓✓✓✓
low_13·····✓✓✓high_01······✓✓
low_14✓··✓✓✓✓✓high_02·····✓·✓
low_15✓··✓✓✓✓✓high_03*······✓✓
low_16*✓✓·✓✓✓✓✓high_04·····✓✓✓
low_17✓✓·✓✓✓✓✓high_05·····✓✓✓
low_18*✓··✓·✓✓✓high_06······✓✓
low_19✓··✓·✓·✓high_07*·····✓✓✓
low_20·····✓✓✓high_08·✓···✓·✓
low_21*·····✓✓✓high_09······✓✓
low_22···✓·✓✓✓high_10*·····✓✓✓
low_23✓·✓✓✓✓✓✓high_11·····✓✓✓
low_24*···✓✓✓✓✓high_12······✓✓
low_25·····✓✓✓high_13······✓✓
low_26*···✓·✓✓✓high_14····✓·✓✓
low_27·✓·✓·✓✓✓high_15······✓✓
low_28·✓✓✓·✓✓✓posture_01*✓✓·✓✓✓✓✓
low_29✓✓·✓✓✓✓✓posture_02✓··✓·✓✓✓
low_30*✓✓✓✓✓✓✓✓posture_03*···✓✓✓✓·
mid_01·····✓✓✓posture_04···✓✓✓✓✓
mid_02·····✓✓✓posture_05*···✓·✓✓✓
mid_03·······✓posture_06···✓✓✓✓✓
mid_04·····✓✓✓posture_07·····✓✓✓
mid_05✓·····✓✓posture_08······✓✓
mid_06*·····✓✓✓posture_09*···✓✓✓✓✓
mid_07·····✓✓✓posture_10······✓✓
mid_08✓✓·✓✓✓✓✓solved201553323627074

Table 16. Per-task results of the runs of Table 2: ✓ marks a success. IA†, HC† and GEMS† are the three baselines with Claude Opus 5.5, and the other columns are Inhabitant with the five language models. * marks a task of the ablation set. The last row counts the solved tasks out of 75.

Table 15 gives the cost of the runs of Table 2 in API fees. For every call of the language model, we take the tokens that the provider bills (input, input read from the prompt cache, and output including reasoning) and multiply them by its list prices. With Claude Opus 5.5, an episode of Inhabitant costs $1.70 and a solved task $1.72. The baselines with the same language model cost $0.96 to $2.56 per episode but solve few tasks, so a solved task costs $4.80 to $21.08. Among the three stronger language models, Claude Sonnet 5.5 is the cheapest per solved task ($1.26). gpt-6-luna costs less per solved task ($0.82) because its list price is forty times lower, but it solves only 23 tasks. GLM-5.3-Flash has open weights and no list price. The GPU that runs the motion generator and the scene simulator is not included; one V100 is enough for both.

MethodLanguage modelInputCachedOutputEpisodeSolved task
InteractAgent†Claude Opus 5.56.3M98%39k2.569.62
HumanCLAW†Claude Opus 5.51.5M95%16k0.964.80
GEMS†Claude Opus 5.51.3M92%32k1.4121.08
InhabitantGLM-5.3-Flash15.4M0%106k––
Inhabitantgpt-6-luna10.5M87%51k0.250.82
InhabitantGemini 4 Argon5.3M89%31k1.942.35
InhabitantClaude Sonnet 5.55.0M98%42k1.181.26
InhabitantClaude Opus 5.53.2M97%30k1.701.72

Table 15. Cost in API fees of the runs of Table 2.

How the agent uses control and feedback (Secs. 5.2 and 5.3)

Table 17 is computed from the episodes of the main table. Most constraints are root waypoints, which decide where the body walks and stops and how it goes around furniture. More than half of the segments of Opus 5.5 (54%) are committed only after one or more revisions, 2.4 tries per segment on average, and Opus 5.5 issues a measurement after 93% of its tries, for example where the hand stopped relative to the object. Its revisions edit the constraints that shape the motion: only 6% of them change nothing but the seed. InteractAgent† can only reword its text prompts: it misses 83% of its grasps, and after a miss its next try succeeds in 4% of the cases.

Opus 5.5Sonnet 5.5GeminiGLMlunaInteractAgent†
Segments with a spatial constraint97%96%100%90%90%–
with an end-effector target or a full-body keyframe37%26%49%17%4%–
Committed at the first try46%52%48%46%36%32%
Tries followed by a measurement93%99%98%80%95%–
Revisions that change only the seed6%7%0%1%1%–
Missed grasps48%66%23%73%79%83%
Next grasp succeeds after a miss,
end-effector target or keyframe revised47%27%44%24%24%–
end-effector target or keyframe unchanged21%24%n/a9%10%4%
Failed episodes1513425255
budget used up0211404546
ended before the budget is used up132279

Table 17. How the agent uses the spatial constraints and the feedback in the episodes of the main table, for Inhabitant with the five language models and for InteractAgent†, which has text prompts only and neither measurement queries nor a seed. Revisions are pairs of consecutive tries for the same segment. The rows on grasps take every try whose grasp failed and ask whether the grasp of the next try for the same segment succeeds, split by whether the agent changed an end-effector target or a full-body keyframe in between. The last rows show how the failed episodes end. –: does not apply; n/a: no such case.

All five language models measure after most of their tries (80% to 99%), so the weaker ones do not lack feedback; they differ in how they use it. GLM-5.3-Flash and gpt-6-luna rarely write an end-effector target or a full-body keyframe (17% and 4% of their segments), miss three quarters of their grasps, and after a miss a revised try succeeds in 24% of the cases, against 47% for Claude Opus 5.5. Gemini 4 Argon uses these constraints most often (49%) and misses the fewest grasps. The failed episodes of the weaker language models mostly use up the budget: 40 of 42 for GLM-5.3-Flash and 45 of 52 for gpt-6-luna.

Spatial control (Sec. 5.3)

On the ablation set, the two failures with root waypoints alone (Table 3) are reaches into the fridge and across a counter, where the palm stayed 0.1 to 0.2 m short of the objects, and an end-effector target on a foot lets the agent tap a ball under a table where the kicks from text prompts passed over it. Tab. 18 gives the result of every task of the ablation set for the five settings.

Feedback (Sec. 5.4)

Without body measurements (Table 4) the agent estimates the hand from the joint rotations it wrote and, in one episode, needs 20 tries for a single grasp in the fridge. Without scene measurements it cannot follow a thrown object and must infer where it went from the final state. Tab. 19 gives the result of every task of the ablation set for the six settings.

Without the check the agent fails on ten tasks. In seven of them it declared the task done although the success rule was not satisfied, most of them throws and final postures, and in the other three it used up the budget. Of the 72 episodes that it ended as done, 65 are successes. On the ablation set (Table 4), the two extra failures with all measurements are tasks the agent declared done: it sat on the front edge of a chair, and it carried a ball to the target by hand where the ball had to be kicked. With the rendered frames only, three of the six failures are again tasks declared done: a posture that does not count as lying, a toss released closer than the required metre, and a three-part tidy-up with one part not done.

The loop with other control specifications (Sec. 5.5)

With text prompts only, Single-Try solves 8 tasks and the full loop 4: without the option to revise, the agent commits imprecise segments and corrects them with further segments, at the cost of 178 collisions per task, while with the

TaskSkillsText+ root+ end eff.+ keyfr.
low_07··✓✓✓
low_11··✓✓✓
low_12··✓✓✓
low_16✓✓✓✓✓
low_18··✓✓✓
low_21··✓·✓
low_24·✓✓✓✓
low_26✓·✓✓✓
low_30✓✓✓✓✓
mid_06··✓✓✓
mid_15··✓✓✓
mid_17✓·✓✓✓
mid_18··✓✓✓
high_03···✓✓
high_07✓·✓✓✓
high_10···✓✓
posture_01✓·✓✓✓
posture_03·✓✓✓·
posture_05··✓✓✓
posture_09··✓✓✓
solved64181919

Table 18. Per-task results of the settings of Table 3 on the ablation set, in the order of its rows: atomic skills, text prompts only, and the three types of spatial constraints added one at a time. ✓ marks a success.

Task123456
low_07·✓✓✓✓·
low_11✓✓✓✓✓✓
low_12✓✓✓✓✓✓
low_16✓✓✓✓✓✓
low_18✓✓✓✓✓✓
low_21✓✓✓✓✓·
low_24✓✓✓✓✓·
low_26✓✓✓✓✓✓
low_30✓✓✓✓✓✓
mid_06✓✓✓✓✓✓
mid_15·✓✓✓·✓
mid_17✓✓✓✓✓✓
mid_18✓✓✓✓✓✓
high_03·✓✓✓✓·
high_07✓·✓✓··
high_10✓✓✓✓✓✓
posture_01✓✓✓✓✓✓
posture_03✓✓✓·✓·
posture_05✓✓·✓·✓
posture_09✓✓✓✓✓✓
solved171919191714

Table 19. Per-task results of the six settings of Table 4 on the ablation set, numbered as its rows. ✓ marks a success.

Success checkAlllowmidhighpostureSub-goalCollisionsDisturbedCallsTokens
with98.7100.0100.0100.090.099.69.31.05199k
without86.796.785.080.070.092.08.20.758128k

Table 20. Inhabitant with Claude Opus 5.5 and all measurements on the 75 tasks of Inhabit, with and without the success check. Success rates and sub-goals in %, the other columns are means per episode as in Table 2. Without the check the agent ends the episode itself, and the episode is judged on its final state.

full loop it spends the budget on retrying seeds. With HumanCLAW’s skills both solve 6 tasks.

Control specificationFull loopSingle-TryOpen-Loop
Full (Inhabitant)19 (7.8)15 (30.4)0 (10.9)
Text prompts only4 (7.2)8 (178.3)0 (23.2)
HumanCLAW skills6 (8.4)6 (23.9)1 (20.4)
Full, no measurements17 (14.6)

Table 21. All ten settings on the ablation set with Claude Opus 5.5: tasks solved out of 20 (mean collisions per task).

Figure 8. Success against collisions per committed segment for the ten settings; marker area is the mean number of tool calls per task.

Tracking the motions with a physics controller

Half-physics has no balance model (§ 4), so we test whether a physically simulated humanoid with balance and torques can follow the interaction sequences. We track the interaction sequence of each of the 75 episodes of Claude Opus 5.5 with the pretrained PHC+ controller [24] in Isaac Gym, on a flat floor without furniture or objects. Parts where the body is supported by furniture cannot be tracked without the furniture, so we remove every interval in which a body part other than the feet touches furniture for at least 0.3 s or a foot stands on it, with 0.5 s on each side, and track each remaining window of at least 1.5 s separately, starting from the pose and velocity of its first frame. This gives 171 windows covering 73% of the frames. We leave out the first 0.5 s of each window and count a window as failed when the mean joint error exceeds 0.5 m, the criterion of PHC.

The controller follows 161 of the 171 windows without failing, at a mean joint error of 54 mm (Table 22). The ten failures fall in ten different tasks. Five of them are among the 13 windows of the tasks that end in a posture, where the body kneels or lies down, and low postures also have the largest error. In the tasks without a posture only 5 of 158 windows fail. The test has limits. Without furniture and objects it leaves out the supported parts of the motion and the weight of carried objects, the reference is the body as exe- @page 22/23 starts_mid=0 ends_mid=1

Windows (frames covered)171 (73%)
Tracked without failure161 (94%)
Global MPJPE53.7 mm
Root-relative MPJPE40.5 mm
Acceleration error6.0 mm/frame{sup}2`
Global MPJPE, walking52.8 mm
Global MPJPE, low postures110.8 mm
Global MPJPE, other47.6 mm

Table 22. PHC+ tracking of the interaction sequences of Claude Opus 5.5 on all 75 tasks, without furniture and objects. Low postures are frames with the head below 1.2 m: bending, squatting, kneeling and lying.

TaskSkillsText+ root+ end eff.+ keyfr.
low_07··✓✓✓
low_11··✓✓✓
low_12··✓✓✓
low_16✓✓✓✓✓
low_18··✓✓✓
low_21··✓·✓
low_24·✓✓✓✓
low_26✓·✓✓✓
low_30✓✓✓✓✓
mid_06··✓✓✓
mid_15··✓✓✓
mid_17✓·✓✓✓
mid_18··✓✓✓
high_03···✓✓
high_07✓·✓✓✓
high_10···✓✓
posture_01✓·✓✓✓
posture_03·✓✓✓·
posture_05··✓✓✓
posture_09··✓✓✓
solved64181919

Table 18 (PDF p. 22). Per-task results of the settings of Table 3 on the ablation set, in the order of its rows: atomic skills, text prompts only, and the three types of spatial constraints added one at a time. ✓ marks a success.

Task123456
low_07·✓✓✓✓·
low_11✓✓✓✓✓✓
low_12✓✓✓✓✓✓
low_16✓✓✓✓✓✓
low_18✓✓✓✓✓✓
low_21✓✓✓✓✓·
low_24✓✓✓✓✓·
low_26✓✓✓✓✓✓
low_30✓✓✓✓✓✓
mid_06✓✓✓✓✓✓
mid_15·✓✓✓·✓
mid_17✓✓✓✓✓✓
mid_18✓✓✓✓✓✓
high_03·✓✓✓✓·
high_07✓·✓✓··
high_10✓✓✓✓✓✓
posture_01✓✓✓✓✓✓
posture_03✓✓✓·✓·
posture_05✓✓·✓·✓
posture_09✓✓✓✓✓✓
solved171919191714

Table 19 (PDF p. 22). Per-task results of the six settings of Table 4 on the ablation set, numbered as its rows. ✓ marks a success.

Success checkAlllowmidhighpostureSub-goalCollisionsDisturbedCallsTokens
with98.7100.0100.0100.090.099.69.31.05199k
without86.796.785.080.070.092.08.20.758128k

Table 20 (PDF p. 22). Inhabitant with Claude Opus 5.5 and all measurements on the 75 tasks of Inhabit, with and without the success check. Success rates and sub-goals in %, the other columns are means per episode as in Table 2. Without the check the agent ends the episode itself, and the episode is judged on its final state.

full loop it spends the budget on retrying seeds. With HumanCLAW’s skills both solve 6 tasks.

Control specificationFull loopSingle-TryOpen-Loop
Full (Inhabitant)19 (7.8)15 (30.4)0 (10.9)
Text prompts only4 (7.2)8 (178.3)0 (23.2)
HumanCLAW skills6 (8.4)6 (23.9)1 (20.4)
Full, no measurements17 (14.6)

Table 21 (PDF p. 22). All ten settings on the ablation set with Claude Opus 5.5: tasks solved out of 20 (mean collisions per task).

Figure 8 (PDF p. 22). Success against collisions per committed segment for the ten settings; marker area is the mean number of tool calls per task.

Tracking the motions with a physics controller

Half-physics has no balance model (§ 4), so we test whether a physically simulated humanoid with balance and torques can follow the interaction sequences. We track the interaction sequence of each of the 75 episodes of Claude Opus 5.5 with the pretrained PHC+ controller [24] in Isaac Gym, on a flat floor without furniture or objects. Parts where the body is supported by furniture cannot be tracked without the furniture, so we remove every interval in which a body part other than the feet touches furniture for at least 0.3 s or a foot stands on it, with 0.5 s on each side, and track each remaining window of at least 1.5 s separately, starting from the pose and velocity of its first frame. This gives 171 windows covering 73% of the frames. We leave out the first 0.5 s of each window and count a window as failed when the mean joint error exceeds 0.5 m, the criterion of PHC.

The controller follows 161 of the 171 windows without failing, at a mean joint error of 54 mm (Table 22). The ten failures fall in ten different tasks. Five of them are among the 13 windows of the tasks that end in a posture, where the body kneels or lies down, and low postures also have the largest error. In the tasks without a posture only 5 of 158 windows fail. The test has limits. Without furniture and objects it leaves out the supported parts of the motion and the weight of carried objects, the reference is the body as executed in half-physics, and the numbers are not comparable to tracking results on motion capture data.

Video

The video on the project page compares Inhabitant and the three baselines, and all agents in it use the same language model. It first summarises which of the two interfaces each agent has. It then plays three tasks with the four agents side by side, so that the differences in the motion are visible: the baselines bump into furniture and walls, miss their grasps or stop, while Inhabitant completes the task. Next it follows the loop of § 3.2 on one task: the tries of a segment are shown one after another with the measurement that the agent requested and the change that it made to the control specification before the next try. The video then shows one long-horizon task from start to end with the number of tries of every segment, and more tasks of the benchmark, including throwing, moving objects with a foot, drawers, the fridge and postures.

References

  1. [1]Chongyu Bao, Haokai Yang, Yuhan Wang, Zhaochong An, Kunpeng Liu, and Xiaolan Liu. Generative embodied multiple behavior control systems for human-like agents. arXiv preprint arXiv:2609.22691, 2026. 1, 3, 6arxiv.org/abs/2609.22691
  2. [2]German Barquero, Sergio Escalera, and Cristina Palmero. Seamless human motion composition with blended positional encodings. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 457–469. IEEE, 2024. 3arxiv.org/abs/2402.15509
  3. [3]Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations, pages 65205–65268, 2025. 2, 3arxiv.org/abs/2411.00081
  4. [4]Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel Van de Panne. Flexible motion in-betweening with diffusion models. In ACM SIGGRAPH 2024 conference papers, pages 1–9, 2024. 3arxiv.org/abs/2405.11126
  5. [5]Zekai Deng, Ye Shi, Kaiyang Ji, Lan Xu, Shaoli Huang, and Jingya Wang. Human-object interaction via automatically designed llm-guided motion policy. In International Conference on Learning Representations, pages 100691–100718, 2026. 3arxiv.org/abs/2503.18349
  6. [6]Purvi Goel, Kuan-Chieh Wang, C Karen Liu, and Kayvon Fatahalian. Iterative motion editing with natural language. In ACM SIGGRAPH 2024 Conference Papers, pages 1–9, 2024. 3DOI
  7. [7]Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11374–11384, 2021. 1, 2arxiv.org/abs/2108.08284
  8. [8]Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scene interactions. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–9, 2023. 1, 2arxiv.org/abs/2302.00883
  9. [9]Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Thompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. 3arxiv.org/abs/2207.05608
  10. [10]Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.arxiv.org/abs/2307.05973
  11. [11]Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024. 3arxiv.org/abs/2409.01652
  12. [12]Yingzhao Jian, Zhong’an Wang, Yi Yang, and Hehe Fan. Endowing gpt-4 with a humanoid body: Building the bridge between off-the-shelf vlms and the physical world. In International Conference on Learning Representations, pages 134007–134065, 2026. 1, 3arxiv.org/abs/2511.00041
  13. [13]Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, and Yixin Zhu. Autonomous character-scene interaction synthesis from text instruction. In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024. 2arxiv.org/abs/2410.03187
  14. [14]Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1737–1747. IEEE, 2024. 1, 2arxiv.org/abs/2403.08629
  15. [15]Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2151–2162. IEEE, 2023. 3, 5, 17arxiv.org/abs/2305.12577
  16. [16]Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1334–1345. IEEE, 2024. 3arxiv.org/abs/2312.11994
  17. [17]Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80–93. PMLR, 2023. 3arxiv.org/abs/2403.09227
  18. [18]Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Controllable human-object interaction synthesis. In European Conference on Computer Vision, pages 54–72. Springer, 2024.arxiv.org/abs/2312.03913
  19. [19]Jiefeng Li, Jinkun Cao, Haotian Zhang, Davis Rempe, Jan Kautz, Umar Iqbal, and Ye Yuan. Genmo: A generalist model for human motion. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11766–11776. IEEE, 2025.arxiv.org/abs/2505.01425
  20. [20]Jiushu Li, Jiaoran Wang, Yuming Liu, Zhiyang Dou, Yuan Liu, Yuexin Ma, and Cheng Lin. World2hosi: A multi-expert harness for long-horizon human-object-scene interaction generation. arXiv preprint arXiv:2610.27725, 2026.
  21. [21]Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li E Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking llms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428–100534, 2024.
  22. [22]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International conference on robotics and automation (ICRA), pages 9493–9500. IEEE, 2023.
  23. [23]Hanchao Liu, Xiaohang Zhan, Shaoli Huang, Tai-Jiang Mu, and Ying Shan. Programmable motion generation for open-set motion control tasks. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 139–1408. IEEE, 2024.
  24. [24]Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895–10904, 2023.arxiv.org/abs/2305.06456
  25. [25]Liang Pan, Zeshi Yang, Zhiyang Dou, Wenjia Wang, Buzhen Huang, Bo Dai, Taku Komura, and Jingbo Wang. Tokehsi: Unified synthesis of physical human-scene interactions through task tokenization. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5379–5391. IEEE, 2025.
  26. [26]Mathis Petrovich, Ori Litany, Umar Iqbal, Michael J Black, Gul Varol, Xue Bin Peng, and Davis Rempe. Multi-track timeline control for text-driven 3d human motion generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1911–1921, 2024.
  27. [27]Ekkasit Pinyouantanapong, Muhammad Saleem, Korrawe Karunratanakul, Pu Wang, Hongfei Xue, Chen Chen, Chuan Guo, Junli Cao, Jian Ren, and Sergey Tul yakov. Maskcontrol: Spatio-temporal control for masked motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9955–9965, 2025.
  28. [28]Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In 2018 IEEE/cvf conference on computer vision and pattern recognition, pages 8494–8502. IEEE, 2018.arxiv.org/abs/1806.07011
  29. [29]Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars, and robots. In International Conference on Learning Representations, pages 15306–15336, 2024.arxiv.org/abs/2310.13724
  30. [30]Haoxuan Qu, Ziyan Guo, and Jun Liu. Gpt-connect: Interaction between text-driven human motion generator and 3d scenes in a training-free manner. arXiv preprint arXiv:2403.14947, 2024.arxiv.org/abs/2403.14947
  31. [31]Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. arXiv preprint arXiv:2307.06135, 2023.
  32. [32]Davis Rempe, Mathis Petrovich, Ye Yuan, Haotian Zhang, Xue Bin Peng, Yifeng Jiang, Tingwu Wang, Umar Iqbal, David Minor, Michael de Ruyter, et al. Kimodo: Scaling controllable human motion generation. arXiv preprint arXiv:2603.15546, 2026.
  33. [33]Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9338–9346. IEEE, 2019.arxiv.org/abs/1904.01201
  34. [34]Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 10737–10746. IEEE, 2020.
  35. [35]Arth Shukla, Stone Tao, and Hao Su. Maniskill-hab: A benchmark for low-level manipulation in home rearrangement tasks. In International Conference on Learning Representations, pages 15288–15317, 2025.
  36. [36]Li Siyao, Yao Feng, Omid Taheri, Chen Change Loy, and Michael J Black. Half-physics: Enabling kinematic 3d human model with physical interactions. arXiv preprint arXiv:2507.23778, 2025.
  37. [37]Li Siyao, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li, Chengcheng Tang, Po-Chen Wu, Ivan Shugurov, Lingni Ma, et al. Humanclaw: Can vision-language models act through a body? arXiv preprint arXiv:2607.27180, 2026.
  38. [38]Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions. ACM Transactions on Graphics, 38(6):178, 2019.DOI
  39. [39]Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing systems, 34:251–266, 2021.arxiv.org/abs/2106.14405
  40. [40]Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics-based character control through masked motion inpainting. ACM Transactions On Graphics (TOG), 43(6):1–21, 2024.
  41. [41]Brian Y Tsui, Alan Y Fang, and Tiffany J Hwu. Demonstration-free robotic control via llm agents. arXiv preprint arXiv:2601.20334, 2026. 3
  42. [42]Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. Tlcontrol: Trajectory and language control for human motion synthesis. In European Conference on Computer Vision, pages 37–54. Springer, 2024. 3arxiv.org/abs/2311.17135
  43. [43]Wenjia Wang, Liang Pan, Zhiyang Dou, Jidong Mei, Zhouyingcheng Liao, Yuke Lou, Yifan Wu, Lei Yang, Jingbo Wang, and Taku Komura. Sims: Simulating stylized human-scene interactions with retrieval-augmented script generation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14117–14127. IEEE, 2025. 1, 3arxiv.org/abs/2411.19921
  44. [44]Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, and Lei Zhang. Physoi: Physics-based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04393, 2023. 2arxiv.org/abs/2312.04393
  45. [45]Yuan Wang, Yali Li, Xiang Li, and Shengjin Wang. Hsigpt: A general-purpose large scene-motion-language model for human scene interaction. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7147–7157. IEEE, 2025. 2DOI
  46. [46]Yinhuai Wang, Qihan Zhao, Runyi Yu, Hok Wai Tsui, Ailing Zeng, Jing Lin, Zhengyi Luo, Jiwen Yu, Xiu Li, Qifeng Chen, et al. Skillmimic: Learning basketball interaction skills from demonstrations. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17540–17549. IEEE, 2025. 2arxiv.org/abs/2408.15270
  47. [47]Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Humanise: Language-conditioned human motion generation in 3d scenes. Advances in Neural Information Processing Systems, 35:14959–14971, 2022. 1, 2arxiv.org/abs/2210.09729
  48. [48]Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say, interact as you can: Language-guided human motion generation with scene affordance. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 433–444. IEEE, 2024. 2arxiv.org/abs/2403.18036
  49. [49]Qi Wu, Yubo Zhao, Yifan Wang, Xinhang Liu, Yu-Wing Tai, and Chi-Keung Tang. Motion-agent: A conversational framework for human motion generation with llms. In International Conference on Learning Representations, pages 48023–48043, 2025. 1, 3arxiv.org/abs/2405.17013
  50. [50]Yan Wu, Korrawe Karunratanakul, Zhengyi Luo, and Siyu Tang. Uniphys: Unified planner and controller with diffusion for flexible physics-based character control. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13214–13224. IEEE, 2025. 2arxiv.org/abs/2504.12540
  51. [51]Zhen Wu, Jiaman Li, Pei Xu, and C Karen Liu. Human-object interaction from human-level instructions. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11176–11186. IEEE, 2025. 1, 3arxiv.org/abs/2406.17840
  52. [52]Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. In International Conference on Learning Representations, pages 24450–24461, 2024. 1, 3arxiv.org/abs/2309.07918
  53. [53]Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. In International Conference on Learning Representations, pages 28176–28194, 2024. 3, 13arxiv.org/abs/2310.08580
  54. [54]Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, and Liang-Yan Gui. Intermi: Towards universal whole-body control for physics-based human-object interactions. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12266–12277. IEEE, 2025. 2arxiv.org/abs/2502.20390
  55. [55]Xinyu Xu, Yizheng Zhang, Yong-Lu Li, Lei Han, and Cewu Lu. Humanvla: Towards vision-language directed object rearrangement by physical humanoid. Advances in Neural Information Processing Systems, 37:18633–18659, 2024. 2arxiv.org/abs/2406.19972
  56. [56]Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Kopella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560, 2025. 2, 3, 5arxiv.org/abs/2502.09560
  57. [57]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. 3, 8arxiv.org/abs/2210.03629
  58. [58]Hongwei Yi, Justus Thies, Michael J Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. In European Conference on Computer Vision, pages 246–263. Springer, 2024. 2arxiv.org/abs/2404.10685
  59. [59]Haozhuo Zhang, Jingkai Sun, Michele Caprio, Angelo Cangelosi, Jian Tang, Shanhang Zhao, Qiang Zhang, and Wei Pan. Lhm-humanoid: Long-horizon human motion control for continuous object transport in cluttered scenes. arXiv preprint arXiv:2508.16943, 2025. 2arxiv.org/abs/2508.16943
  60. [60]Tao Zhang, Senhe Zhang, Zeyu Zhang, Yujia Zhang, and Dong Gong. Interactagent: Agentic human motion interaction with memory-augmented reflection. arXiv preprint arXiv:2610.25649, 2026. 1, 3, 6arxiv.org/abs/2610.25649
  61. [61]Xiaohan Zhang, Yanran Lai Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll. Couch: Towards controllable human-chair interactions. In European Conference on Computer Vision, pages 518–535. Springer, 2022. 1, 2arxiv.org/abs/2205.00541
  62. [62]Yan Zhang and Siyu Tang. The wanderings of odysseus in 3d scenes. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20449–20459. IEEE, 2022. 2arxiv.org/abs/2112.09251
  63. [63]Yangsong Zhang, Abdul Ahad Butt, Gül Varol, and Ivan Laptev. Interpose: Learning to generate human-object interactions from large-scale web videos. In 2026 International Conference on 3D Vision (3DV), pages 750–759. IEEE, 2026. 3
  64. [64]Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14692–14703. IEEE, 2023.
  65. [65]Kaifeng Zhao, Gen Li, and Siyu Tang. Dartcontrol: A diffusion-based autoregressive motion model for real-time text-driven motion control. In International Conference on Learning Representations, pages 23569–23592, 2025.arxiv.org/abs/2410.05260
  66. [66]Jiakun Zheng, Ting Xiao, Shiqin Cao, Xinran Li, Zhe Wang, and Chenjia Bai. Re²MoGen: Open-vocabulary motion generation via llm reasoning and physics-aware refinement. arXiv preprint arXiv:2604.17807, 2026.arxiv.org/abs/2604.17807

Paper details

vlnc:W_3bad9eaw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude

View statement
Looking for related papers…
Contents