RynnUnify: Towards Unified Embodied Brain

https://alibaba-damo-academy.github.io/RynnUnify.github.io

https://github.com/alibaba-damo-academy/RynnUnify

https://huggingface.co/collections/Alibaba-DAMO-Academy/RynnUnify

Introduction

Introduction

Developing general robot brains [10, 21] requires models that understand the physical world, anticipate interaction outcomes, and translate task intentions into action. Across interdependent operations, a robot must identify subgoals, predict scene changes, and determine how they affect subsequent decisions. These demands motivate unified embodied models that coordinate understanding, world prediction, and action generation [33, 23].

Existing unified multimodal models (UMMs) have made encouraging progress in jointly modeling understanding and generation [33, 23, 11]. Their coordination requires strong embodied understanding to ground task intentions, reason about physical interactions, and interpret predicted outcomes. Sustaining this interaction over long horizons remains challenging as scene changes and prediction errors accumulate, while static visual predictions can overlook intermediate dynamics. Addressing these challenges requires interleaved reasoning [17] that connects embodied understanding with temporally coherent video prediction.

To this end, we introduce RynnUnify, a 16B unified embodied model built upon the Mixture-of-Transformers (MoT) architecture [24] of Cosmos3 [33], comprising an 8B autoregressive reasoner and an 8B diffusion generator. Centered on interleaved video-language reasoning, RynnUnify combines strong embodied understanding with video prediction to support understanding-guided long-horizon visual planning: the reasoner guides successive video predictions, whose outcomes provide context for subsequent subtask planning. To explore generation-enhanced embodied understanding, we further post-train an additional 8B agent from RynnUnify’s reasoner through reinforcement learning with final-answer feedback. The agent dynamically chooses between direct answering and invoking the frozen RynnUnify model to generate videos, which it then interprets to answer questions about unobserved future states and interaction outcomes. To bring the UMM’s understanding and world-prediction capabilities to robot control, we also extend the full RynnUnify model to a World-Action Model (WAM) through latent planning and joint video-action training.

We perform extensive pretraining on general multimodal understanding, spatial reasoning, physical prediction, and affordance grounding, complemented by subtask planning, world-modeling descriptions, and question answering with intermediate video context. This supervision strengthens foundational embodied understanding while preparing the reasoner to formulate task-relevant visual predictions and interpret their outcomes. Generation training covers long-horizon egocentric activities, robot manipulation, and embodied reasoning videos. We introduce scene memory and self-forcing [18] to improve visual consistency and robustness during long-horizon generation.

Dedicated benchmarks remain lacking for assessing sustained visual planning and the contribution of video prediction to future-state reasoning. We therefore introduce Rynn-VP and Rynn-GEU. Rynn-VP evaluates whether visual plans remain consistent with the task and preceding operations over long horizons. Rynn-GEU focuses on action consequences, physical changes, and spatial relationships after motion or viewpoint changes, whose answers concern states not directly observable in the input. Its direct setting tests whether models can infer these states without explicit video generation, while its generation-assisted setting tests whether predicted videos improve answers to the same questions.

Extensive evaluations demonstrate state-of-the-art embodied understanding among the evaluated UMMs on public benchmarks, together with strong video generation performance. RynnUnify also demonstrates sustained planning on Rynn-VP and strong direct anticipatory reasoning on Rynn-GEU, with further GEU improvements through its agentic extension. For robot control, the WAM extension achieves a 51.5% success rate on 20 selected composite RoboCasa365 [29] tasks. Experiments on AStriBot S1 further demonstrate effective multi-step manipulation involving body reorientation, precise object handling, and history-dependent execution.

Figure 1 summarizes RynnUnify’s capabilities and representative evaluation results.

Image not extracted

Our main contributions are summarized as follows:

• We introduce RynnUnify, a unified embodied model that combines strong embodied understanding with interleaved video-language reasoning for long-horizon visual planning.

• We post-train an agent from RynnUnify’s reasoner to explore generation-enhanced embodied understanding, demonstrating the utility of video predictions for reasoning about unobserved future states.

• We extend the full RynnUnify UMM to a WAM through latent planning and joint video-action training, and validate its capabilities in simulated and real-robot manipulation.

• We introduce Rynn-VP and Rynn-GEU for long-horizon visual planning and anticipatory embodied reasoning, complemented by extensive evaluations on public benchmarks that establish RynnUnify’s state-of-the-art embodied understanding among the compared UMMs.

Model Architecture

As shown in Figure 2, RynnUnify adopts the Mixture-of-Transformers (MoT) [24] architecture of Cosmos3 [33]. The model comprises an autoregressive reasoner for multimodal understanding and language generation, and a diffusion generator for predicting continuous visual latents through rectified flow [25]. The two branches use separate Transformer parameters and interact through joint attention, allowing the generator to attend to representations produced by the reasoner. We retain the reasoner architecture unchanged and extend the generator’s visual conditioning to incorporate cross-chunk scene memory and support first-and-last-frame-conditioned generation. These extensions provide persistent scene references and continuous interaction context for visual predictions across successive subtasks.

Image not extracted

Reasoner

We retain the autoregressive reasoner of Cosmos3 [33] without architectural modifications. The reasoner processes multimodal observations and language context to support embodied understanding, spatial grounding, and task planning. For visual planning, it takes the task goal, initial and current observations, and preceding subtask history as input, and predicts the next executable subtask together with a detailed description of its anticipated visual evolution and an estimated duration. These outputs provide the generator with structured textual conditions specifying the intended interaction and its temporal extent. During multi-step rollout, the final frame of each generated clip becomes the updated observation for subsequent reasoning, connecting subtask planning with the visual consequences of preceding predictions.

Visual tokenization and input organization.

For video generation, we encode visual inputs using a causal video VAE with spatial and temporal compression factors of 16 and 4, respectively. The resulting latents are spatially patchified into visual tokens. Both scene-memory frames and the main video follow this VAE pathway and are processed by the generator branch.

Each scene-memory frame is encoded independently, producing a group of visual tokens that represents its spatial content. Four reference frames therefore correspond to four groups of tokens, whose individual lengths depend on the frame resolutions. When scene memory is enabled, these groups are placed after the reasoner’s textual context and before the main video tokens:

where $S_{\mathrm{AR}}$ denotes the textual conditioning tokens processed by the reasoner, $Z_{\mathrm{mem}}^{(i)}$ denotes the visual tokens of the $i$-th reference frame, and $z_t$ denotes the main video tokens at denoising time $t$. Both $Z_{\mathrm{mem}}$ and $z_t$ belong to the generator’s visual sequence. The memory groups retain their boundaries as independently encoded visual items; their placement before the main video is a concatenation of token groups, rather than an insertion of additional frames into the main video’s temporal axis. For generation tasks without scene memory, $Z_{\mathrm{mem}}$ is omitted.

At each denoising step, the generator processes the memory and main video tokens through its existing attention layers while attending to the reasoner’s textual representations. The main video can therefore access the scene references directly, without introducing a separate memory encoder or additional cross-attention modules.

Cross-chunk scene memory.

Long-horizon planning requires the model to retain visual information beyond the current observation, particularly when objects or scene regions leave the field of view. We maintain four reference frames comprising a persistent initial observation, two keyframes selected across earlier history, and a reference associated with the most recent step. This selection preserves both broader scene coverage and recent task context. When the history is insufficient, additional frames are selected from available historical clips; at the first step, the initial observation is repeated to fill the reference slots. During rollout, the initial observation remains fixed while the historical references are updated from generated videos.

We complement these sparse references with a continuous video prefix from the preceding chunk. Unlike the independently encoded memory frames, this prefix occupies the beginning of the main video’s temporal sequence. The preceding tail and the current segment form a continuous sequence, with the shared boundary frame included only once. The reference frames serve as visual anchors for scene appearance and layout, while the temporal prefix preserves recent motion and interaction context. Together, they form cross-chunk scene memory for successive visual predictions.

Generation modes and fixed conditions.

We support three generation modes through different arrangements of visual conditions within the main video sequence. In image-to-video (I2V) generation, the first latent frame is fixed to the encoded starting observation, and the model predicts the subsequent video. In first-and-last-frame-to-video (FL2V) generation, both the first and final latent frames are fixed, providing boundary conditions for generating the intervening transition. In cross-chunk video continuation, the initial latent frames are fixed to a continuous prefix from the preceding video’s tail, allowing the model to predict the next segment from recent motion and interaction context. During multi-step rollout, this prefix is extracted from the previously generated video.

Across these modes, the scene-memory latents and the conditioning latents within the main video remain fixed throughout denoising. The generator updates the remaining video positions while attending to the textual context and visual conditions. During training, only the positions to be predicted contribute to the generation loss. The same conditioning interface accommodates observed and self-generated visual inputs, supporting the self-forcing strategies used during generator fine-tuning.

Textual and temporal conditioning.

The generator receives structured textual conditions containing the subtask instruction, a detailed description of the anticipated visual evolution, and the requested duration. These conditions connect task-level reasoning with the content and temporal extent of each video prediction. We support variable-length video sequences of the VAE-compatible form $4n + 1$ frames and retain the backbone’s three-dimensional rotary position encoding and frame-rate modulation to represent spatial and temporal coordinates. For cross-chunk continuation, the requested duration describes the new operation and excludes the historical prefix. The reasoner can therefore specify what should happen next and for how long, while the generator predicts the corresponding visual changes using the scene memory and temporal conditions.

Data

Table 1 summarizes the data used to fine-tune the reasoner and generator.

Category	Subtask / Mode	Data Sources	Samples / Clips	Duration (h)
Understanding Data
General Understanding	General	RynnBrain [10]	∼2.0M (10% subset)	–
Visual Planning	Human	RynnBrain-Planning [10], HiFi-UMI [2], OpenAoE [22]	550,000	–
Visual Planning	Robot	AgiBot-Beta [1], RoboMIND [42], RoboCOIN [43], RoboTwin [27], HIW-500 [5]	600,000	–
Visual Planning		RoboCasa365 [29]	86,800	–
Embodied Perception and Reasoning	Reasoning	RynnUnify-Spatial, RynnUnify-Phys, RynnUnify-ActionReasoning, RynnUnify-Mess	153,195	–
Embodied Perception and Reasoning	Perception	RynnUnify-Affordance, RynnUnify-Area, RynnUnify-Trajectory	43,405	–
Video Generation Data
Visual Planning	TI2V	RynnBrain-Planning [10], HiFi-UMI [2], OpenAoE [22], HIW-500 [5], AgiBot-Beta [1], RoboMIND [42], RoboCOIN [43], RoboTwin [27]	806,685	966.3
Visual Planning	TI2V	RoboCasa365 [29]	80,258	146.4
Embodied Perception and Reasoning	FL2V	RynnUnify-Spatial	117,921	146.0
Embodied Perception and Reasoning	TI2V	RynnUnify-Spatial, RynnUnify-Phys, RynnUnify-ActionReasoning, RynnUnify-Mess, RynnUnify-Affordance, RynnUnify-Area, RynnUnify-Trajectory	78,594	111.4
Total Video Generation Data			1,083,458	1,370.1

General Understanding Data

To maintain broad embodied understanding and reasoning capabilities, we use a 10% subset of the RynnBrain pretraining corpus [21] as the foundation of our understanding data. Reasoning about embodied tasks requires the model to interpret task intentions, recognize objects and their properties, understand spatial relationships, and track changes across successive observations. These capabilities are particularly important for long-horizon visual planning, where each subtask must be grounded in the current scene and considered in relation to subsequent operations. They also support the interaction between understanding and video generation: the model needs to identify which aspects of the scene are relevant to a task, formulate the corresponding visual prediction, and interpret the predicted changes to inform further reasoning.

The data cover general image and video understanding, visual question answering, temporal reasoning, and instruction following, together with embodied cognition tasks including object understanding, spatial reasoning, counting, and egocentric task understanding. Spatially grounded supervision further covers object, area, and affordance localization, trajectory prediction, and manipulation planning, connecting language instructions with explicit locations and interaction targets. We combine this foundational supervision with our visual planning and embodied perception and reasoning data to develop interleaved video–language reasoning while retaining a broad range of understanding capabilities. This combination is intended to support both direct responses to embodied questions and the interpretation of visual evidence produced during task-oriented video prediction.

Visual Planning Data

Long-horizon visual planning requires the reasoner to determine successive subtasks from an evolving scene and the generator to predict the visual consequences of their execution. Developing these complementary capabilities requires supervision that connects task intentions, intermediate decisions, and continuous scene changes across multiple operations. We therefore curate human and robot manipulation demonstrations and organize them into temporally ordered sequences of subtask instructions and corresponding video clips, supporting the fine-tuning of both the reasoner and the generator.

Data collection and annotation. Our data sources include RynnBrain-Planning [10], HiFi-UMI [2], OpenAoE [22], HIW-500, AgiBot-Beta [1], RoboMIND [42], RoboCOIN [43], RoboTwin [27], and RoboCasa365 [29], covering human and UMI demonstrations, real-robot interactions, and simulated manipulation. We additionally collect egocentric loco-manipulation videos in indoor environments as part of our own RynnUfy data collection. Participants perform step-by-step object interactions using hand motions that imitate robotic manipulation, with each operation explicitly demonstrated in the recorded video. These recordings capture how successive operations change object states and spatial relationships throughout a task.

For datasets with explicit subtask annotations, we use the available instructions and temporal boundaries to associate each operation with its corresponding visual observations. For demonstrations containing extended multi-step activities but lacking explicit subtask decomposition, we use Qwen3.6-27B to perform temporal segmentation and relabeling. Each resulting segment is paired with a semantically meaningful subtask instruction describing the operation performed. We aim to keep video clips between 2 and 10 seconds whenever possible, yielding manageable training segments while preserving the semantic completeness of each operation and the temporal evolution of the interaction. We then use GPT-5.6-Terra to rewrite each subtask instruction into a structured visual prompt grounded in the corresponding clip, describing the scene configuration, relevant objects, interaction progression, and resulting state changes. These prompts supervise the reasoner’s visual-prompt formulation and serve as textual conditions for training the generator on the corresponding video clips, establishing a shared conditioning format between planning and visual prediction.

Agentic data filtering. Training data from heterogeneous sources vary in quality and coverage. We combine quality filtering with capability-guided selection to preserve the generator’s general capabilities while addressing task-specific weaknesses. An open-source MLLM scores individual clips and complete trajectories based on visual quality, operation completeness, and instruction–video alignment, and low-scoring samples are removed. We further use GPT-5.6-Terra to analyze videos generated by the base model and identify capability gaps associated with particular scenarios or tasks. We select examples covering similar tasks and scenes from the filtered training pool and combine them with randomly sampled data to retain broad coverage.

Interleaved sample construction. To construct long-horizon reasoning examples, we preserve the original task goal and temporal ordering of the demonstrations, interleaving visual observations with subtask instructions, structured visual prompts, and operation durations. This organization trains the reasoner to identify the next subtask from the task goal and interaction history, describe its anticipated visual evolution, and estimate its duration. The aligned prompt–video pairs provide supervision for the generator, while the connections between consecutive clips support video continuation across subtask boundaries. Together, these training examples connect task decomposition, visual prompt formulation, and continuous video prediction throughout a multi-step task.

Generation Enhanced Understanding (GEU) Data

GEU training examples are organized as interleaved conversations: given an observation and a question or task instruction, the assistant first produces a world-modeling instruction, receives the corresponding video as visual context, and then generates a reasoning response ending with an answer or spatial output.

RynnUnify-Phys

We construct RynnUnify-Phys from the rendered videos and simulation annotations of PhysInOne [53] to connect physical future prediction with reasoning over visual evidence. Each selected sequence is split into an observed prefix and a future continuation, with question–answer pairs derived from object trajectories and event annotations in the continuation. These questions probe how objects move and interact over time, covering displacement and final speed, the timing of contact and settling, and the objects and support relations involved in subsequent interactions. To couple these prediction targets with visual continuation, we organize each SFT example as an interleaved sequence of the observed video and question, a world-modeling instruction, the ground-truth continuation video, and a textual reasoning response ending with the answer. This structure pairs the future visual evidence with the reasoning it supports.

A central consideration in constructing the world-modeling instructions is to direct generation toward the physical interaction relevant to the question while leaving its outcome to be predicted. We therefore use GPT-5.6-Luna [34] to rewrite the world-modeling instructions using timestamped frames from the observed prefix. The reasoner is instructed to ground each request in the relevant objects, visible motion, and support or contact relations, and to ask for continuation through the interaction without specifying how it resolves. The rewritten instructions undergo format checks and a separate input-conditioned audit for factual grounding, interaction coverage, and future-outcome leakage, with review status retained for unresolved cases. The resulting interleaved examples connect question-conditioned physical continuation with subsequent reasoning over visual evidence. We finally use 10,028 examples for supervised fine-tuning and 1,553 numerical questions for reinforcement learning.

RynnUnify-Mess

Physical reasoning in cluttered environments requires anticipating how a local intervention propagates through interactions among objects. Moving a supporting object, for example, may cause other objects to slide, rotate, or fall, producing a sequence of dependent changes. We construct RynnUnify-Mess to support learning these physical dependencies and predicting how a scene evolves under a specified intervention.

We first construct cluttered tabletop scenes in RoboTwin [8] with interacting and partially occluded objects. Starting from a stable configuration, we apply controlled interventions, such as pulling or removing an object, and simulate the resulting dynamics. Throughout each rollout, we record object poses, contact and support relations, and subsequent events, including support loss, falling, and changes in visibility. These records provide structured supervision for the effects of an intervention on other objects in the scene.

To obtain continuous video supervision, we first use GPT-5.6-Terra [35] to rewrite the video-generation prompts, explicitly describing the scene configuration, the requested intervention, and the expected physical consequences for other objects. We then condition MiniMax-H3 on these rewritten prompts and the simulated initial and final frames in the FL2V setting to generate the corresponding interaction videos. The rewritten prompts are also retained as world-modeling annotations for subsequent training. Using the recorded simulator states and events, we automatically construct questions and initial reference answers about which objects are affected and how their positions, orientations, or visibility change. This process connects the intervention description, the visual evolution of the scene, and the reasoning required to interpret its consequences.

We combine automatic motion and visibility criteria with GPT-based post-filtering to improve data quality. The former prioritize visually discernible events and suppress negligible peripheral motions and weak visibility changes. The latter assess whether the generated video faithfully depicts the requested intervention with plausible physical interactions, rejecting videos with artifacts such as teleportation, severe interpenetration, or unrelated instability. For retained videos, we additionally check reference answers against the visible outcomes and correct inconsistent descriptions. Each resulting training example combines an initial observation and intervention question with a video-generation prompt, an intermediate video rollout, and an answer, connecting physical prediction with the interpretation of its visual consequences.

RynnUnify-Spatial

RynnUnify-Spatial covers position, direction, distance, and size reasoning through category-specific QA construction and a shared video-generation and reasoning-annotation pipeline. We construct the position subset from ScanNet++ [47] using camera intrinsics and extrinsics, object 3D bounding boxes, and per-view visibility measured by visible proportions and pixel counts. We consolidate semantically similar object labels and filter categories unsuitable for position reasoning. Rule-based templates construct four question types: object–camera position, camera–camera position, camera motion, and object–object position. All four types use reference objects visible in both images as common cross-view references and apply question-specific geometric or visibility constraints. Object–camera questions select an object that is absent or only slightly visible in the queried view but clearly visible in the other. Camera–camera questions ask for one camera’s position in the other’s reference frame. Camera-motion questions constrain camera displacement and isolate a dominant yaw or pitch change to distinguish left, right, upward, and downward rotation. Object–object questions relate objects across the two views using either a specified camera frame or a horizontal compass frame defined by a reference object pair. Answers are computed from scene geometry, with balanced sampling across question types, reference-view configurations, answers, and scenes. The position subset contains 100,000 examples: 99,200 for training and 800 for testing, with 200 test examples per question type. For direction, distance, and size, we sample RynnBrain’s video-based spatial perception data [10] and reformulate them as single- or dual-frame QA examples. Qwen3.8-27B extracts keyframes, assesses visual overlap and semantic relevance between candidate frames, and scores question-answering difficulty. These scores guide filtering and sampling across the three reasoning dimensions.

Across all four categories, initial video-generation prompts are constructed from question-type-specific templates. For position examples, object–camera and camera–camera prompts start from the view defining the reference frame, camera-motion prompts follow the queried motion direction, and object–object prompts randomly use either transition order. Direction examples follow the same view-transition strategy, while size and distance prompts guide the camera to move closer to the target object. Following MiniMax-H3’s prompt-expansion guidance, GPT-5.6-Luna [34] expands the initial prompts using the original images. The expanded prompts and image pairs condition MiniMax-H3 [26] to generate intermediate visual context in FL2V mode, while a portion of the size examples use single-image I2V generation. GPT-5.6-Luna then receives the original visual inputs, question and answer options, video-generation prompt, generated video, and correct answer to produce a chain-of-thought annotation. Each training example links a spatial question, a world-modeling prompt, an intermediate video, and a reasoning response ending with the answer. The resulting data supervise the reasoner’s prompt formulation and video-conditioned spatial reasoning, while providing aligned visual inputs, prompts, and videos for generator training in the corresponding mode.

RynnUnify-ActionReasoning

We construct RynnUnify-ActionReasoning for embodied planning and predictive world modeling from the physics-aware planning demonstrations in RynnBrain [10], including AgiBot [1], Galaxea [20], RoboCasa [28], RoboCOIN [43], RoboMIND [42], RoboTwin [27], SoftFold [51], and self-captured data. Episodes are divided into adjacent observed–context and future-continuation clips of 2–6 seconds. Inspired by ERQA [15], the dataset covers seven capabilities: action-primitive selection, translation and reach, action-intent recognition, push direction, rotation and orientation, metric action, and affordance and feasibility. Generation is conditioned on the target category to improve coverage of underrepresented rotation, metric, and affordance questions; push-direction examples are restricted to visually unambiguous pushing transitions.

For each clip pair, GPT-5.6-Luna [34] uses frames near the observed–continuation boundary to generate a category-specific planning question, a world-modeling instruction, reasoning, and an answer grounded in the visible transition. After selection, world-modeling instructions are refined using the paired frames and question to add motion detail while preserving the target action and excluding unsupported properties, absolute measurements, and hidden states. Each SFT example interleaves the observed video and question, the world-modeling instruction, the ground-truth continuation video, and a reasoning response ending with the answer. Quality control combines independent multimodal judging for action-primitive, translation, intent, and push questions with applicability scoring for rotation, metric, and affordance questions. The former checks visual grounding, category alignment, answer correctness, reasoning consistency, and instruction–video agreement; the latter assesses whether the transition provides sufficient evidence for the requested reasoning. We retain the top 2,000 examples per category, yielding 14,000 examples split into 13,300 training and 700 test examples (1,900 and 100 per category). A balanced 200-example subset supports efficient evaluation.

RynnUnify-Affordance and Area

We construct RynnUnify-Affordance and RynnUnify-Area from a subset of RynnBrain-Planning [10], which decomposes long-horizon manipulation tasks into temporally ordered atomic subtasks. Each subtask combines a language instruction with spatial annotations, including target object bounding boxes, affordance points, and placement area points. We use its streaming format, in which successive visual observations are interleaved with these spatially grounded subtask instructions.

Using the subtask boundaries and their associated affordance and area annotations, we segment the original videos into individual manipulation clips and retain the corresponding instructions and spatial labels. For RynnUnify-Affordance, we construct task-conditioned questions that ask the model to identify an interaction point on the target object. For RynnUnify-Area, the questions require localizing the target region specified by the subtask, such as the destination for object placement. To support subsequent video-generation training, we further use GPT-5.6-Terra [35] to produce world-modeling descriptions from the subtask video clips, detailing the scene configuration, the performed operation, and the resulting state changes. These descriptions provide textual conditions for generating the corresponding interaction videos. Each resulting sample associates a visual observation and task instruction with a world-modeling description, an execution clip, and a spatial annotation, linking video prediction with task-conditioned localization. This process yields 20,039 affordance samples and 13,774 area samples, totaling 33,813 training examples.

RynnUnify-Trajectory

We construct RynnUnify-Trajectory from RH20T [13] manipulation videos to connect task-conditioned trajectory prediction with visual evidence of action execution. Each example pairs the initial frame of a manipulation clip with a task instruction and asks the model to predict the corresponding two-dimensional trajectory in image space. The target trajectory is represented as an ordered sequence of at most ten points, with coordinates normalized to the range of 0–1000. This representation provides spatial supervision for the motion associated with the requested interaction.

To support interleaved reasoning and video generation, we organize each example as an initial observation and instruction, a world-modeling description, the corresponding execution video, and a textual reasoning response ending with the trajectory. The execution clips are at most eight seconds long and sampled at 24 frames per second. We initially use Qwen-VL-Max to generate world-modeling descriptions from the clean initial frame, describing the scene and intended motion. We further rewrite these descriptions using four frames from the execution clip as visual evidence. The rewritten descriptions anchor the scene to the initial observation and specify the progression of the interaction through shot descriptions and an integer-valued target duration. These descriptions provide textual conditions for generating the aligned execution videos, while the videos supply intermediate visual evidence for subsequent trajectory prediction.

We exclude examples whose projected trajectories have a total path length below 20 pixels, including motions that occur primarily in depth and provide little visible displacement in the image plane. We also validate the answer format, coordinate range, and number of trajectory points, and check sample identifiers for uniqueness. The resulting dataset contains 9,892 examples, split into 9,592 training examples and 300 test examples with no overlapping sample identifiers. Each retained example therefore links a task-conditioned visual prediction with an explicit spatial description of the resulting motion.

Training

Figure 3 summarizes the two-stage pretraining procedure: we first fine-tune the reasoner, then freeze it and fine-tune the generator.

Image not extracted

Stage 1: Reasoner Fine-tuning

Training Objectives

We fine-tune the reasoner inherited from Cosmos3 [33] to develop four complementary capabilities: spatio-temporal understanding, physical-world grounding, subtask planning, and video-prompt formulation. Following RynnBrain [10, 21], video-based understanding serves two purposes: maintaining broad embodied understanding and reasoning capabilities, and preparing the model to extract and reason over task-relevant information from predicted videos. The latter provides a foundation for subsequent generation-enhanced embodied understanding (GEU), where video predictions serve as intermediate visual evidence for reasoning about spatial relationships, physical interactions, and their consequences before producing an answer. Physical grounding connects these capabilities with explicit objects, interaction regions, and trajectories. We further train the reasoner to determine executable subtasks from the current observation and task history, and to formulate detailed video-generation prompts. Motivated by our experience with MiniMax-H3 [26], we explicitly supervise prompt formulation to provide the generator with descriptions of both the current scene and its anticipated visual evolution.

Training Format

To preserve broad embodied understanding and physical grounding, we follow the multimodal training format of RynnBrain [10], an embodied foundation model that jointly learns spatio-temporal understanding, spatial localization, and planning. Images and videos are represented as ordered frame sequences, while assistant responses contain natural language and, when required, spatial coordinates for objects, interaction regions, and trajectories. This shared format supports reasoning over temporal observations while grounding the outputs in explicit elements of the physical scene.

Long-horizon visual planning requires the reasoner to determine the next operation from task progress and provide the generator with a clear description of what should happen during its execution. We therefore jointly supervise subtask prediction and video-prompt formulation. Given the task goal, initial and current observations, and preceding subtask history, the reasoner predicts an executable subplan together with a structured visual prompt describing the relevant scene context, object motions, and anticipated state changes. The subplan specifies the intended operation, while the visual prompt translates that intention into detailed conditions for video generation. We additionally supervise the operation duration to specify the temporal extent of the prediction:

USER [context; no loss]:
[HIGH-LEVEL TASK] Task instruction.
[INITIAL STATE] Initial observation.
[PREVIOUS STEPS] Subtask history from the three preceding steps.
[CURRENT OBSERVATION] Current observation.

ASSISTANT [supervised]:
<subplan>Next executable subtask.</subplan>
<world_modeling>
Detailed scene context and expected visual evolution during the operation.
</world_modeling>
<duration>Estimated duration in seconds.</duration>

Generation-enhanced embodied understanding requires the reasoner to formulate a task-relevant visual prediction and interpret the temporal evidence it provides. To supervise both abilities, we organize these examples into conversations that connect prompt formulation with video-conditioned reasoning. Given an observation and a question, the assistant first writes a video-generation prompt specifying the scene and the visual evolution to be predicted. The corresponding ground-truth video is then supplied as a user turn, followed by the assistant’s reasoning and answer. The first assistant turn teaches the model to express the visual prediction needed for the question, while the second teaches it to extract and reason over evidence in the provided video. This supervision prepares the reasoner to use video predictions as intermediate context during subsequent generation-enhanced embodied understanding:

USER [context; no loss]:
[VISUAL INPUT] Images or video frames.
[QUESTION] Embodied question.

ASSISTANT [supervised]:
<world_modeling>
Scene context and the requested visual prediction.
</world_modeling>
<duration>Estimated duration in seconds.</duration>

USER [context; no loss]:
[VIDEO] Corresponding ground-truth video frames.

ASSISTANT [supervised]:
<thinking>Reasoning based on the video.</thinking>
<answer>Final answer.</answer>

Optimization

We optimize the reasoner using a standard autoregressive loss over assistant response tokens:

where $s$ denotes the multimodal conversation, $\theta_R$ denotes the reasoner parameters, and $\mathcal{I}_{R}$ indexes the assistant response tokens. Both prompt-generation and answer-generation turns are supervised; system and user turns, including their visual inputs, are excluded from the loss. Only the reasoner is optimized in this stage. The reasoner SFT hyperparameters are summarized in Table 8.

Parameter	Reasoner SFT	Generator SFT	WAM Post-training
			Stage I	Stage II	Stage III
Optimizer	AdamW	AdamW	AdamW	AdamW	AdamW
Learning rate	$2 \times10^{-6}$	$2 \times10^{-5}$	$1 \times10^{-6}$	$2 \times10^{-5}$	$1 \times10^{-5}$
Global batch size	1024	128	1024	256	256
Warmup ratio	0.03	0.1	0.05	0.1	0.1
Gradient clipping norm	1.0	0.3	1.0	1.0	1.0

Stage 2: Generator Fine-tuning

Following reasoner fine-tuning, we freeze the reasoner and optimize the generator to translate structured visual prompts and scene context into video predictions. Beyond modeling individual operations, this stage aims to maintain scene consistency across successive subtasks and improve robustness to imperfect visual conditions during long-horizon rollout. We jointly train complementary generation tasks with cross-chunk scene memory and self-forcing strategies.

Training Tasks

We combine human and robot manipulation videos with videos for embodied perception and reasoning to jointly train image-to-video generation (I2V), first-and-last-frame-to-video generation (FL2V), and cross-chunk video continuation. These tasks learn to predict subsequent changes from an observation, generate transitions between specified visual states, and extend an interaction from preceding visual history, respectively. Each planning clip is paired with its annotated subtask instruction, a detailed visual prompt describing the anticipated scene evolution, and the operation duration. These aligned text–video pairs connect the structured descriptions learned by the reasoner with the visual predictions expected from the generator.

Training with Cross-Chunk Scene Memory

For planning examples, we construct visual conditions from the initial observation and preceding clips within the same episode, following § 3. Four independently encoded reference frames, comprising the initial observation and selected historical frames, provide persistent scene information before the main video tokens. Cross-chunk continuation additionally uses a continuous prefix extracted from the preceding clip’s tail at the beginning of the main video sequence. These conditions provide both sparse references to earlier scene appearance and layout and continuous context for recent motion and interactions.

During training, we sample from admissible prefix lengths of $k \in\{2,4,8\}$ latent frames. Under the causal VAE’s temporal compression scheme, these correspond to the last $1 + 4(k - 1)$ video frames of the preceding clip, yielding prefixes of 5, 13, or 29 frames. We concatenate the selected prefix with the current clip, including the shared boundary time step only once. The combined main video contains at most 241 frames, and the prefix occupies at most half of its latent frames to retain sufficient prediction supervision. The textual duration describes only the current operation and excludes the historical prefix. At inference, we use the final 13 frames of the preceding generated clip as the prefix; if the clip is too short, we perform I2V generation from its final frame while retaining the scene-memory conditions.

Self-forcing Training

During long-horizon rollout, the starting observation and recent visual history of each subsequent clip depend on preceding predictions. Training exclusively with ground-truth conditions therefore leaves the generator unprepared for visual errors that accumulate across successive predictions. Inspired by Self Forcing [18], we employ three complementary strategies to vary the visual conditions encountered during training while retaining ground-truth supervision for the continuation.

Condition degradation. For selected I2V examples, we perturb the starting condition frame in pixel space before VAE encoding. Specifically, we add Gaussian noise with a randomly sampled strength and optionally apply downsampling–upsampling blur. These perturbations expose the generator to imperfect starting observations and encourage robust prediction when the conditioning image contains noise or loses visual detail.

Offline endpoint replacement. We generate preceding video clips offline using a fixed generator checkpoint and cache their predictions. The cache is prepared once and reused throughout training without refreshing it as the generator parameters change. For examples with an available cached prediction, we replace the current clip’s ground-truth starting observation with the final frame of the corresponding generated clip. This trains the generator to continue from its own predicted scene states. Examples without an available cached prediction retain their original observations.

Joint history and endpoint replacement. To further expose the generator to self-generated visual history, we extend replacement from the starting frame to the preceding videos used for condition construction. For examples using this strategy, cached generated clips replace the corresponding ground-truth historical videos, and the final frame of the same generated predecessor supplies the current starting observation. Scene-memory references and temporal prefixes are then constructed using the procedures in § 3, preserving the existing visual input organization. This ensures that the recent history and its resulting starting observation originate from the same prediction, allowing the model to learn continuation from generated historical context.

Across these strategies, the ground-truth initial observation remains a persistent scene reference, and the current clip’s ground-truth continuation remains the supervision target. Scene memory supplies visual information retained across steps, while self-forcing trains the generator to use this information under imperfect conditions.

Optimization

We retain the flow-matching objective of the Cosmos3 generator [33], applying supervision only to the main video positions that require prediction:

where $\theta_G$ denotes the generator parameters, $z_t$ the main video latent sequence at flow time $t$, $\mathbf{c}$ the textual and visual conditions, and $v^{\star}_t$ the target flow velocity determined by the ground-truth video latents and sampled noise. The set $\Omega$ indexes the main video positions to be predicted. Fixed conditioning positions and independently encoded scene-memory items are excluded from the loss. We normalize over active prediction positions and average only over visual items containing prediction targets, preventing conditioning inputs from diluting the supervision. Condition degradation and offline replacement modify the conditioning inputs while preserving the ground-truth continuation objective.

For training the generator, we use AdamW with a peak learning rate of $2\times10^{-5}$, $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-6}$, and no weight decay. We train for 8,000 optimization steps, linearly increasing the learning rate from zero over the first 800 steps, followed by cosine decay, and clip the global gradient norm to 0.3. For self-forcing training, we apply pixel-space condition degradation to scene-memory-conditioned planning I2V examples with a probability of 0.3. For each selected example, we perturb the frames of the main video clip before VAE encoding, leaving all prediction targets unchanged. With pixel values normalized to $[-1,1]$, we add Gaussian noise with standard deviation $\sigma\sim U(0,0.15)$. Independently, with a probability of 0.5, we apply downsampling–upsampling blur by sampling a spatial scale $s\sim U(0.5,1.0)$, downsampling the perturbed frame, and restoring its original resolution using bilinear interpolation. The resulting pixel values are clipped to $[-1,1]$.

Stage 3: Agentic Reinforcement Learning

To learn when and how visual generation supports reasoning, we propose Branch-Conditional Policy Optimization (BCPO), an agentic reinforcement learning framework that jointly optimizes reasoning behavior and generator invocation (Figure 4). BCPO combines two complementary learning signals: within-branch reward comparisons improve reasoning along each interaction path, while between-branch reward differences teach the reasoner to select the path that benefits the task. This formulation connects the decision to generate visual evidence with the ability to request and interpret it.

Image not extracted

Given a question and its visual observations $x$, the reasoner chooses between direct reasoning ($D$) and generation-assisted reasoning ($G$). The direct branch produces a reasoning response and final answer from the original input. The generation-assisted branch follows a request–observe–answer interaction: the reasoner specifies a world-modeling instruction and duration, receives a video from the frozen generator, and incorporates the returned visual evidence into its response. Task rewards supervise the resulting trajectories, updating the reasoner’s generation requests and answers while keeping the generator fixed. BCPO explores both branches for each routable question, sampling four direct and four generation-assisted trajectories. Within each branch, relative rewards identify effective reasoning behaviors under the same interaction path. Across branches, the difference in mean reward provides a learning target for generator invocation. The reasoner therefore learns both how to reason along each path and how to anticipate the benefit of generation from the original input. At inference time, it autonomously selects a branch and executes the corresponding reasoning process.

Optimization Objective

Within-branch optimization. Based on a sequence-level GSPO [50] objective, BCPO optimizes direct and generation-assisted trajectories in separate groups. For trajectory $i$ in branch $z$, the normalized advantage is

where $r_{z,i}$ is the trajectory reward, and $\mu_z$ and $\sigma_z$ are the reward mean and standard deviation within that branch. This comparison favors successful reasoning behaviors under the same interaction path. In the generation-assisted branch, the request and subsequent response share the trajectory-level feedback, allowing both to be optimized toward final answer quality.

Let $\rho_{z,i}$ denote the sequence-level importance ratio, computed as the exponential of the mean token log-probability difference between the current and rollout policies. The policy objective is

The objective supervises the reasoner-generated request and response tokens, with generated videos serving as observations. Sequence-level clipping controls policy updates across trajectories of different lengths.

Learning when to generate. We train invocation decisions from the mean reward difference between the two paths. For each question, we construct a soft target for generator invocation:

where $\operatorname{sg}$ denotes stop-gradient and $T_{\mathrm{route}}$ controls target smoothness. A higher generation-assisted reward increases the target invocation probability, while similar branch rewards yield a balanced preference.

The reasoner predicts its invocation probability from the two route-token logits under the original input. We fit this prediction to $q_G(x)$ using a masked binary cross-entropy loss $\mathcal{L}_{\mathrm{route}}$, applying supervision to valid branch comparisons with informative reward feedback. This teaches the reasoner to anticipate the benefit of generation from the question and observations available before tool use.

Joint optimization. The complete objective combines branch-specific reasoning, invocation learning, and policy regularization:

Here, $\mathcal{L}_{\mathrm{KL}}$ regularizes the policy against the frozen initial reasoner, and $\mathcal{H}$ is policy entropy. Within-branch optimization improves how the reasoner acts along each path, while routing supervision determines which path to select. Detailed loss definitions and hyperparameters are provided in Appendix § 17.

Task-Grounded Rewards

Both paths share the same task-grounded answer objective, making their reward difference directly relevant to invocation learning. For routable examples, we combine interaction-protocol validity with task-specific answer quality:

where $k$ denotes the task, and $\hat{y}$ and $y^\star$ are the predicted and reference answers. For numerical and geometric tasks, a normalized error $e_k$ is mapped to a continuous reward:

Numerical physical and spatial questions use relative error; affordance and area localization use symmetric nearest-point distance; and trajectory prediction uses discrete Fréchet distance normalized by reference arc length. The shared shaping function provides graded feedback for numerical precision and geometric alignment. Discrete and textual answers use option matching, normalized answer matching, or task-specific semantic judgments. Mess uses a level-based assessment of predicted interaction outcomes. The complete reward definitions are given in Appendix § 17.

Training Data and Recipe

We train on 22,007 questions spanning embodied perception, physical prediction, spatial reasoning, interaction outcomes, and localization, with 883 questions for in-training validation. Each update uses 64 questions and eight rollouts per question, yielding 512 trajectories. Routable questions use four direct and four generation-assisted rollouts, while native multi-image examples use eight direct rollouts. Training responses are sampled at temperature 1.0 with a limit of 2,048 generated tokens per reasoner turn. We initialize the agent from RynnUnify’s fine-tuned 8B reasoner and optimize the agent for one epoch using Adam at a constant learning rate of $10^{-6}$, keeping the generator and reference policy fixed. We use Qwen3.8-27B-FP8 [37] for task-specific semantic judgments, alongside deterministic scorers for structured answers. Validation is performed every ten updates with greedy decoding and autonomous D/G selection. Full training settings and interaction protocols are provided in Appendix § 17.

Post-training for the World-Action Model (WAM)

We extend RynnUnify to a world-action model (WAM) for robot control. We train the model in three stages.

Model Architecture

Figure 5 illustrates the WAM architecture. Building on RynnUnify, the reasoner takes the task instruction, current RGB observations, and visual history as input and produces a continuous subtask latent. The generator uses this latent to predict future video, while an action head predicts robot actions from the generator’s video features.

Image not extracted

Visual memory. At step $t$, the reasoner receives the instruction $\ell$, current observations $\mathcal{O}_t$, and visual history $\mathcal{H}_t$. The current observations contain left, wrist, and right camera views. For RoboCasa365 [29], the history includes one left-view image from the start of the episode, up to five earlier left-view images, and up to three earlier wrist images sampled at different time intervals. Together with the current views, the reasoner uses at most 12 images.

Subtask-conditioned video prediction. The reasoner produces a continuous subtask latent $h_t^{\mathrm{plan}}$ from the instruction and visual context. This latent conditions the generator through layerwise attention. Following RynnUnify, reasoner tokens use causal attention, while generation tokens attend to both reasoner and generation tokens. The current camera views serve as a fixed visual condition, and the generator predicts the future video latents.

Video features and action head. We perform a single video denoising step, then use another generator forward pass to collect video hidden states from all 36 layers. An 18-block action head reads these features through cross-attention. Each block attends to two adjacent generator layers, with action tokens as queries and video features as keys and values. The head uses flow matching to predict the velocity field for an action chunk $A_t \in\mathbb{R}^{48 \times12}$.

Training

Training data. We initialize the model from pretrained RynnUnify. Robot adaptation uses approximately 1,600 hours of atomic-task trajectories and 177 hours of composite-task trajectories from RoboCasa365 [29]. The data include task instructions, three synchronized RGB views, robot actions, and subtask and skill annotations. We train Stages I and II on atomic tasks and then repeat them on composite tasks, followed by Stage III on composite tasks. For real-robot adaptation, we initialize from the atomic-task model and train the generator and action head on real-robot demonstrations while keeping the reasoner frozen.

Training samples are constructed according to each stage. Stage I uses task instructions, current observations, visual history, and subtask annotations. For Stage II, each sample contains the current multi-view frame, 48 future frames, and 48 aligned actions at 20 Hz, covering 2.4 seconds. Stage III uses the current observations, future actions, and relative camera poses over the same horizon.

Stage I: Reasoner training. We train the reasoner’s language layers using subtask and skill annotations, with the visual encoder and generator frozen. We use token-level cross-entropy on the annotated response. This stage trains the reasoner to identify the current subtask from the instruction, current observations, and visual history.

Stage II: Generator training. We freeze the reasoner and train the generator using its subtask latent as a condition. Video and native action tokens interact within the shared generator. We retain the video and action flow-matching losses of the base model, with equal weights. The video loss follows § 5.2.

Stage III: Action-head training. We jointly train the generator and the new action head while keeping the reasoner and the generator’s original action output layer frozen. The action head uses flow matching conditioned on the video features. Its loss also updates the generator through the video denoising step and feature readout.

We also train an auxiliary camera-pose head to predict future camera poses relative to the current observation. The Stage III objective is

For an action chunk $A$, we sample Gaussian noise $\epsilon^{a}$ and form $A_{\sigma} = (1-\sigma)A + \sigma\epsilon^{a}$. The action head predicts its flow velocity from $\mathbf{A}_{\sigma}$ and the video features $\mathbf{H}$:

For camera $c$, let $\mathbf{T}_{t,k,c}=\mathbf{C}_{t,c}^{-1}\mathbf{C}_{t+k,c}$ denote its pose at future step $k$ relative to the current frame. We measure pose error using

where $s_{\mathrm{T}}=0.25\ \mathrm{m}$. The pose and cross-camera losses are

Here, $\widehat{\mathbf{E}}_{t,k,cd}$ and $\mathbf{E}_{t,k,cd}$ are the relative transforms between camera pair $(c,d)$, formed from the predicted and ground-truth camera poses, respectively, using their known relation at the current frame. All losses are averaged over valid targets; the action loss uses two independent noise–time samples per training example.

Inference. At each decision step, we extract the video features and reuse them during action sampling. The action head takes 16 Euler steps to produce a 48-step action chunk, of which the first 16 actions are returned for execution. The camera-pose head is used only in training.

Rynn-VP and Rynn-GEU Benchmarks

Reciprocal interaction between visual generation and understanding can help embodied models anticipate task-relevant scene evolution and use predicted outcomes to guide reasoning and planning. Despite extensive exploration of this interaction in general-purpose unified multimodal models [38, 54], dedicated benchmarks in embodied settings remain limited; we therefore introduce Rynn-VP and Rynn-GEU, summarized in Figure 6.

Image not extracted

Rynn-VP: Long-Horizon Visual Planning

Efficiently evaluating embodied planning remains challenging: a high-level task requires interdependent operations that transform an initial scene into a goal-satisfying state, with each operation changing the conditions for subsequent decisions. Next-step prediction measures local decisions but does not establish full-trajectory validity [9, 19]. Evaluating complete plans in simulators or with human controllers [10, 14] requires environment setup and repeated execution, which can constrain evaluation scale and efficiency. We view visual planning as a native capability of embodied unified models that interleaves task-advancing subtask reasoning with coherent video prediction, using predicted scene changes to inform subsequent planning.

To evaluate this capability, we introduce Rynn-VP, comprising 200 manually collected egocentric task scenarios. Given an initial image and task instruction, a model generates a video depicting task execution or an ordered sequence of step images. Our simulator-free protocol assesses task progress, operation dependencies, scene continuity, and physical plausibility from these generated rollouts. Reference demonstrations of successful execution and timestamped subtask annotations are reserved for evaluation, specifying task goals and required process evidence while allowing alternative valid execution sequences.

We formalize the scoring criteria and verification procedures as a reusable evaluation Skill, executed by GPT-6-sol within the Codex harness. The evaluator first examines the reference demonstration and annotations to identify task goals, relevant entities, and dependencies between subtasks. It then inspects the generated visual sequence to assess task progress, intermediate actions, and scene-state transitions. A dedicated hallucination audit further examines object lifecycles, physical interactions, and scene consistency, identifying unsupported object appearance or disappearance, contradictory state changes, physically implausible interactions, and unexplained changes in scene layout. The audit distinguishes clear visual inconsistencies from occlusion or insufficient evidence, and its findings inform the calibration of the corresponding planning and visual realization scores. The resulting structured judgments undergo deterministic score aggregation. Manual review of the evaluation procedure is described in Appendix § 16.

Rynn-VP measures performance along three dimensions:

• Planning Intelligence (PI): Evaluates executable goal achievement, step validity, dependency satisfaction, and plan completeness based on the generated visual evidence.

• Visual Realization (VR): Evaluates visible goal completion and action evidence, together with entity consistency, state continuity, consistency across successive video segments, and physical feasibility and plausibility.

• Visual Quality (VQ): Characterizes perceptual quality and the clarity of task-relevant visual evidence.

All scores are reported on a scale of 0–100. The overall benchmark score combines the calibrated PI and VR scores with VQ:

The evaluation recognizes valid partial progress, and results are reported with the output modality identified to distinguish videos from ordered step images. Detailed scoring rubrics, submetric weights, evidence-sampling procedures, and calibration rules are provided in Appendix § 16.

Rynn-GEU: Embodied Understanding with Generated Visual Context

Embodied understanding requires anticipating how actions, physical interactions, and viewpoint changes affect states that are not directly observable in the input. We introduce Rynn-GEU to evaluate this anticipatory reasoning and investigate whether explicit video prediction improves it. The benchmark comprises ten subsets covering spatial relationships (position, direction, distance, and size), physical prediction, chain reactions in cluttered scenes, action reasoning, affordance grounding, area grounding, and trajectory prediction.

Data construction. We follow the construction pipelines of the corresponding RynnUnify-GEU training data, with explicit separation between training and evaluation scenes to limit scene memorization. Mess uses RoboTwin2.0 [8] scenes absent from training, while Spatial, Trajectory, and Action Reasoning exclude scenes that occur in or closely resemble the training data. Phys is constructed from PhysInOne [53] using distinct assets and scenes for training and testing. Affordance and Area are constructed from AgiBot-Beta [1], providing a different source from the RynnBrain-Planning data [10] used for training.

Future-oriented sample selection. To emphasize questions that benefit from reasoning about subsequent states, we further filter the constructed examples using GPT-5.6-Luna [34]. For each candidate, we compare its answers under the same initial observations and question, with and without access to the reference future video. We primarily retain examples answered well with future visual evidence but poorly without it, while also including a smaller portion answered correctly without the reference video. This process emphasizes questions sensitive to future visual evidence while retaining variation in direct-reasoning difficulty. Because model responses are stochastic, the comparison serves as a selection heuristic rather than a definitive label of whether explicit generation is necessary.

Evaluation metrics. We use task-specific metrics on a common 0–100 scale. Action Reasoning uses choice accuracy, while categorical Phys questions use normalized exact matching. For Spatial Position, Direction, and Size, GPT-5.6-Luna compares predictions with reference answers, assigning 100 only when the answer is both correct and complete, and 0 otherwise.

For Spatial Distance, we convert predictions to meters and compute

where $\hat{d}$ and $d > 0$ are the predicted and reference distances. For numerical Phys questions, we validate the predicted value and its unit compatibility, then compute

where $\hat{y}$ and $y \ne0$ are the predicted and reference physical quantities. For either metric, a zero-valued reference receives 100 only when the prediction is also zero.

Affordance and Area evaluate the spatial agreement between predicted and reference point sets:

where $P$ and $Q$ are the predicted and reference point sets in normalized image coordinates, and $d_C$ is their symmetric mean nearest-neighbor distance. Trajectory evaluates both path geometry and point ordering:

where $\hat{\mathcal{T}}$ and $\mathcal{T}$ are the predicted and reference trajectories after coordinate normalization and arc-length resampling, and $d_F$ denotes their discrete Fréchet distance.

For Mess, GPT-5.6-Luna assigns 0, 25, 50, 75, or 100 points according to the correctness and coverage of predicted effects on non-manipulated objects, including displacement, rotation, and support or contact changes. Each subset reports its mean per-example score.

Experiments

Embodied Understanding Capability

Foundational Embodied Understanding

Evaluation protocol. We evaluate embodied cognition on VSI-Bench [45], MMSI-Bench [46], MindCube [40], ERQA [15], RoboSpatial [39], and EmbSpatial [12], and localization on RefSpatial-Bench [52], ShareRobot (Affordance and Trajectory) [19], and Where2Place [48] in Table 2. All methods answer directly from benchmark inputs. RynnUnify uses only its foundational reasoner, without invoking the generator or the additional RL-trained agent. We report benchmark-specific metrics and their unweighted mean after 0–100 normalization.

Model	VSI-Bench	MMSI-Bench	Mind Cube	ERQA	RoboSpatial	EmbSpatial	RefSpatial-Bench	ShareRobot Affordance	ShareRobot Trajectory↓	Where2Place	Avg.
Open-source Understanding Models
Qwen3-VL-8B-Instruct [4]	60.3	29.8	36.8	44.8	58.2	78.5	53.4	37.9	0.37	64.7	52.7
Qwen3.5-9B [36]	61.0	13.4	33.1	41.5	54.1	76.9	37.3	7.1	0.53	48.8	42.0
RynnBrain-8B [10]	71.0	39.6	56.6	46.8	73.1	80.4	59.2	44.7	0.35	75.5	61.2
MiMo-Embodied-7B [16]	48.5	30.2	43.1	46.8	70.3	76.2	48.0	35.6	0.41	63.6	52.1
Embodied-R1.5-8B [49]	56.1	29.8	27.9	46.0	69.7	78.1	54.2	25.2	0.29	74.0	53.2
Closed-source Understanding Models
GPT-5.6-Terra [35]	58.9	36.9	62.6	56.5	57.2	81.1	31.4	31.3	0.29	39.3	52.6
Claude-Sonnet-5 [3]	37.3	28.1	49.6	46.2	41.3	74.3	15.1	2.9	0.47	3.3	35.1
Unified Multimodal Models
BAGEL [11]	34.5	31.0	34.7	39.5	41.0	67.0	8.6	3.2	0.88	8.8	28.0
RxBrain [23]	43.5	35.8	47.5	42.8	60.5	82.3	34.1	11.7	0.49	39.8	44.9
Cosmos 3 Nano [33]	54.9	36.2	40.5	46.0	62.1	81.4	53.1	28.0	0.34	64.0	53.2
RynnUnify (Ours)	67.1	35.9	44.2	43.8	72.8	83.0	60.3	41.8	0.27	56.9	57.9

Main results. RynnUnify achieves the highest average score among the evaluated UMMs (Table 2), reaching 57.9 compared with 53.2 for Cosmos 3 Nano and leading on six of ten metrics. It also compares favorably with general-purpose multimodal models, demonstrating strong embodied cognition and localization capabilities across spatial understanding, interaction grounding, and trajectory prediction. On VSI-Bench and RoboSpatial, it outperforms Cosmos 3 Nano by 12.2 and 10.7 points, respectively, showing stronger spatial understanding across complementary settings. For localization, ShareRobot Affordance improves from 28.0 to 41.8, indicating more accurate grounding of task-relevant interaction points. RynnUnify also achieves the best results across all compared models on EmbSpatial (83.0), RefSpatial-Bench (60.3), and ShareRobot Trajectory (0.27, lower is better). These complementary capabilities provide a foundation for interleaved video–language reasoning: understanding scene geometry and interaction targets supports subtask planning and world-model prompt formulation, while spatial and temporal reasoning helps interpret the outcomes of subsequent video predictions. Additional embodied cognition and localization examples are provided in Figure 11 and Figure 12 in the appendix.

Image not extracted

01 Affordance

Question: Pick up the air-column wrap to the right of the cardboard box.

Answer: (650, 523)

02 Affordance

Question: Dip the tea bag into the clear glass cup and lift it.

Answer: (571, 646)

03 Target area

Question: Place the held shampoo bottle into the red canvas tote bag.

Answer: (505, 390), (532, 454), (552, 400), (525, 425), (552, 431)

04 Trajectory

Question: Move an object from one box to another.

Answer: (380, 356), (450, 556), (461, 656), (438, 475), (419, 350)

05 Target area

Question: Place the purple bowl in the dishwasher with the left arm.

Answer: (417, 723), (456, 700), (430, 756), (456, 740), (430, 706)

06 Trajectory

Question: Put the knife on the cutting board.

Answer: (375, 344), (389, 431), (395, 517), (395, 599), (395, 669)

Embodied Reasoning with Agentic Visual Generation

Main results. Table 3 evaluates embodied reasoning both without explicit video generation and with generated visual context. On Rynn-GEU, RynnUnify achieves the strongest direct-answering performance among the compared UMMs, with an average score of 44.0. This highlights its intrinsic capacity for anticipatory embodied reasoning: inferring action consequences, physical changes, and spatial relationships beyond what is directly observable in the input. These results support the motivation for our foundational embodied pretraining, which develops the spatial, physical, and action-related knowledge needed to reason about unobserved states. This predictive understanding also provides a basis for formulating task-relevant video prompts and interpreting generated visual evidence.

Model	MMSI-Bench	Mind Cube	ERQA	Avg.	Traj.	Aff.	Area	Pos.	Dist.	Size	Dir.	Traj. Reas.	Physics	Mess	Avg.
Open-source Understanding Models
Qwen3-VL-8B-Instruct [4]	29.8	36.8	44.8	37.1	67.3	62.3	31.7	27.5	39.6	52.5	20.0	45.5	15.0	7.3	36.9
Qwen3.5-9B [36]	13.4	33.1	41.5	29.3	73.8	47.5	17.5	42.0	9.2	57.5	17.0	45.0	7.2	15.8	33.2
RynnBrain-8B [10]	39.6†	56.6	46.8	47.7	79.2	60.5	28.1	8.0	43.5	63.0†	34.5†	40.0	23.0	9.5	38.9
MiMo-Embodied-7B [16]	30.2	43.1	46.8	40.0	64.8	16.0	15.1	34.0	45.0	53.5	13.0	47.5	5.3	18.2	31.2
Embodied-R1.5-8B [49]	29.8	27.9	46.0	34.6	77.9	63.5	35.6	28.0	43.1	52.5	19.5	44.0	21.9	5.3	39.1
Closed-source Understanding Models
GPT-5.6-Terra [35]	36.9	62.6†	56.5†	52.0†	77.6	53.2	42.9	44.0†	46.5	60.0	24.0	56.0	20.0	26.3	45.1
Claude-Sonnet-5 [3]	28.1	49.6	46.2	41.3	70.9	20.6	16.0	31.5	47.5	58.0	22.2	63.0†	24.0	27.1	38.1
Unified Multimodal Models
BAGEL [11]	31.0	34.7	39.5	35.1	4.4	26.9	3.0	11.5	15.0	41.0	7.0	8.0	0.0	2.0	11.9
+ Interleaved Thinking	30.7	39.6	44.8	38.4	1.4	22.8	3.5	18.5	10.8	33.5	5.5	10.0	0.5	1.9	10.8
RxBrain [23]	35.8	47.5	42.8	42.0	42.8	44.9	27.0	24.5	40.2	44.5	21.5	33.5	13.7	21.0	31.4
+ Interleaved Thinking	31.8	45.3	37.0	38.0	33.3	46.0	27.0	24.0	37.9	42.5	20.5	26.0	9.6	19.5	28.6
Cosmos 3 Nano [33]	36.2	40.5	46.0	40.9	76.8	61.3	31.8	27.0	39.5	42.3	17.5	40.5	20.0	16.8	37.3
+ Interleaved Thinking	30.1	36.4	41.4	36.0	73.7	53.1	23.9	32.0	28.4	39.1	21.7	40.0	4.5	19.0	33.5
RynnUnify	35.9	44.2	43.8	41.3	78.9	65.7	35.9	38.5	52.9†	57.0	22.0	57.5	22.6	8.5	44.0
+ Agentic Thinking	36.3	52.6	44.8	44.6	81.5†	67.0†	66.2†	36.0	51.8	56.5	25.5	61.0	25.0†	29.1†	50.0†

The agentic extension further increases the Rynn-GEU average to 50.0, the highest among all compared methods, exceeding GPT-5.6-Terra by 4.9 points. It improves seven of ten tasks over direct answering and outperforms the other UMM rollout variants on all ten. The gains span perception-level grounding, spatial reasoning, and physical prediction, with particularly large improvements on Area (35.9 to 66.2) and Mess (8.5 to 29.1). Improvements also extend to all three public reasoning benchmarks, where the average rises from 41.3 to 44.6, including an 8.4-point gain on MindCube. Together, these results demonstrate the value of combining a strong embodied reasoning foundation with agentic use of video prediction.

Effect of generated visual context.

Generated visual context has different effects across models and benchmarks. On the public benchmarks, BAGEL’s interleaved rollout increases its average from 35.1 to 38.4, with MindCube improving from 34.7 to 39.6 and ERQA from 39.5 to 44.8, while MMSI-Bench decreases slightly from 31.0 to 30.7. RxBrain instead declines on all three benchmarks, from 35.8 to 31.8 on MMSI-Bench, 47.5 to 45.3 on MindCube, and 42.8 to 37.0 on ERQA, reducing its average from 42.0 to 38.0. RynnUnify’s agentic rollout improves all three scores and reaches the highest public-benchmark average among the evaluated UMMs and their reasoning variants, at 44.6 compared with 38.4 for BAGEL and 38.0 for RxBrain. On ERQA, RynnUnify and BAGEL both score 44.8 at the reported precision; RynnUnify’s stronger aggregate performance reflects higher MMSI-Bench and MindCube scores.

On Rynn-GEU, interleaved rollouts reduce the average scores of BAGEL, RxBrain, and Cosmos 3 Nano from 11.9 to 10.8, 31.4 to 28.6, and 37.3 to 33.5, respectively. Together with BAGEL’s gains on the public benchmarks, these results indicate that the usefulness of generated visual context depends on the model and the task. A plausible explanation for the declines is that inaccurate dynamics, inconsistent geometry, or task-irrelevant predictions introduce misleading evidence into subsequent reasoning. We hypothesize that RynnUnify’s gains reflect three complementary factors. First, selective generation preserves a direct-answering path instead of requiring a rollout for every question. Second, stronger embodied understanding supports more robust interpretation of imperfect videos, allowing the reasoner to extract useful cues while remaining grounded in the original observations. Third, precise world-model prompt formulation helps focus video prediction on the interactions and state changes relevant to the question, improving the usefulness of generated evidence. Nevertheless, position, distance, and size scores decrease slightly after agentic rollout, suggesting that fine-grained geometric judgments remain sensitive to distortions in predicted scenes.

Case Study

Figure 7 illustrates direct and generation-assisted reasoning across public benchmarks and Rynn-GEU. In panel (a), the model answers from the input observations, covering object-state recognition in ERQA and viewpoint reasoning in MindCube. In panel (b), the agent formulates a task-specific video prompt, inspects the generated sequence, and returns a textual or spatial answer. The ERQA example relates the motion of a yellow pepper to the location of the target pot. The FL2V examples use transitions between given views to reason about camera rotation in MMSI-Bench and the direction of a doorway in Rynn-GEU.

Image not extracted

The Rynn-GEU Area and Trajectory examples connect generated visual context with spatial predictions in the original observation. For brush storage, the agent returns points identifying the target placement region. For tap control and speaker-volume adjustment, it returns ordered gripper trajectories. The predicted points and paths are overlaid on crops of the input image, linking the anticipated interaction to locations in the observed scene. Together, these examples illustrate how task-focused video generation supplies intermediate context for action reasoning, viewpoint reasoning, and spatial grounding, while retaining a direct-answering path. Additional generation-assisted reasoning examples are provided in Figure 13 in the appendix.

Image not extracted

Interleaved Visual Planning

Evaluation Settings

Evaluation protocol. Evaluation is conducted on Rynn-VP, which contains 200 egocentric task scenarios, each initialized with a scene image and a high-level task instruction. The interleaved planning setting alternates between subtask reasoning and visual generation, using predicted scene states to inform subsequent planning. Rollouts terminate when the model indicates task completion or reaches a maximum of 15 steps. We follow the scoring protocol in § 7.1 and report PI, VR, VQ, and the overall score. Reference videos and subtask annotations are used only for evaluation. Videos are generated at 480p and 24 fps. For video generation models, we use a default duration of 4 seconds per step. We enable the dynamic duration prediction mechanism of RynnUnify to adapt each generated segment to the planned subtask.

Compared methods. We compare RynnUnify with BAGEL [11], RxBrain [23], Cosmos 3 Nano [33], and a composed system combining GPT-5.6 Terra [35] with MiniMax-H3 [26]. In our evaluation, BAGEL and RxBrain produce ordered step images, whereas the other methods generate videos.

Main Results

RynnUnify achieves the highest PI, VR, and overall scores among the evaluated UMMs (Table 4), improving the overall score from 45.05 to 54.80 over Cosmos 3 Nano. PI increases from 30.50 to 45.66 (+15.16), indicating stronger progress toward task goals and coordination of interdependent operations. VR increases from 54.58 to 60.54 (+5.96), indicating more consistent and physically plausible visual realization. These gains accompany a slightly lower VQ score (77.53 vs. 79.69), showing that the overall improvement is driven by planning and visual realization.

Long-horizon visual planning and generation quality on Rynn-VP. We report Planning Intelligence (PI), Visual Realization (VR), and Visual Quality (VQ), all on a 0–100 scale. Overall is computed as $0.50 \times\mathrm{PI}+0.40 \times\mathrm{VR}+0.10 \times\mathrm{VQ}$. Bold indicates the best result among UMMs. Higher is better for all metrics.

Method	Output	PI	VR	VQ	Overall
Composed Video Generation System
GPT-5.6 Terra + MiniMax-H3	Video	59.82	69.56	79.73	65.71
Unified Multimodal Models
BAGEL	Step images	9.46	24.36	58.50	20.32
RxBrain	Step images	1.28	11.35	28.15	8.00
Cosmos 3 Nano	Video	30.50	54.58	79.69	45.05
RynnUnify	Video	45.66	60.54	77.53	54.80

Relative to the Cosmos3 backbone, RynnUnify combines embodied fine-tuning with cross-chunk scene memory, self-forcing training, and subtask-conditioned duration prediction. Scene memory retains initial and historical scene references together with recent motion context, providing persistent cues for object identity, scene layout, and interaction state across successive clips. Self-forcing exposes the generator to perturbed observations and cached generated endpoints and histories during training, preparing it to continue from the imperfect predictions encountered during inference. Duration prediction adapts each generated segment to the temporal requirements of the planned operation. These extensions target complementary challenges in long-horizon rollout: retaining task-relevant scene information and continuing coherently as prediction errors accumulate. The stronger PI and VR results are consistent with this design goal, demonstrating improved task progression and visual consistency at the system level.

Visualization

Figure 8 presents qualitative comparisons on Toy Storage and Pillowcase Folding, illustrating task progression and visual consistency across successive predictions. RynnUnify interleaves subtask reasoning with video generation, using the predicted scene to inform subsequent planning. In Toy Storage, it progresses from manipulating one toy to collecting subsequent toys, while maintaining the backpack and sofa as consistent spatial references. In Pillowcase Folding, it generates successive fabric manipulations that progressively produce a compact folded configuration. These examples cover complementary challenges: tracking multiple objects across repeated transfers and maintaining coherent object transformations during deformable-object manipulation.

Image not extracted

The baseline sequences exhibit several characteristic difficulties. Some show repeated interactions with limited visible task progression, whereas others introduce abrupt changes in object appearance, scale, or background layout. These differences highlight that plausible individual frames alone are insufficient for long-horizon visual planning: successive predictions must preserve relevant scene state while advancing the task. RynnUnify exhibits more coherent task progression in these examples, illustrating how interleaved reasoning and video prediction can connect individual operations into a sustained plan. The reference demonstrations provide examples of successful execution; effective predictions need not reproduce their exact motions or intermediate configurations. Additional long-horizon visual planning cases are provided in Figure 14 in the appendix.

Image not extracted

Evaluation of the Post-trained WAM

Evaluation Settings on Simulation Benchmarks

Benchmark and evaluation protocol. We evaluate RynnUnify on the Composite-Seen and Composite-Unseen splits of RoboCasa365 [29]. We also report results on 20 composite mobile-manipulation tasks, referred to as Selected-20, including PackIdenticalLunches, SeparateFreezerRack, and DeliverStraw. These tasks involve base movement and multiple manipulation steps under changing camera views. We run 10 episodes per task on Selected-20, for a total of 200 episodes, and report task success rates. Results for the full composite splits and Selected-20 are given in Table 5.

Method	Composite Seen	Composite Unseen	Selected 20 tasks
$\pi_0$ [6]	6.1	1.1	3.0
$\pi_{0.5}$ [7]	7.1	1.2	2.0
GR00T N1.5 [30]	14.8	2.7	7.5
GR00T N1.6 [31]	9.4	1.7	6.5
Cosmos 3 [33]	49.6	39.0	45.0
Xiaomi-Robotics-1 [44]	57.1	32.1	46.0
CMA [41]	50.6	44.0	47.0
RynnUnify	52.0	50.0	51.5

Training and policy configuration. We initialize our WAM from pretrained RynnUnify and use approximately 1,600 hours of atomic-task trajectories and 177 hours of composite-task trajectories from RoboCasa365 for robot adaptation. As described in § 6.2, we first train the reasoner with subtask supervision while keeping the generator frozen. We then freeze the reasoner and train the generator with video and native-action supervision. Finally, we jointly train the generator and a separate action head, with auxiliary camera-pose supervision.

At each decision step, the reasoner receives the task instruction and at most 12 RGB images in total. These include the current left, wrist, and right camera views, one left-view image from the beginning of the episode, up to five earlier left-view images, and up to three earlier wrist images. Historical images are selected at fixed time offsets. The generator uses the current observations and the subtask latent from the reasoner. The action head predicts a $48 \times12$ action chunk, of which the first 16 actions are used for execution. The 12-dimensional action space includes planar base translation, base yaw, torso height, end-effector translation and rotation, gripper control, and a control-mode indicator.

Results on RoboCasa365.

As shown in Table 5, RynnUnify achieves success rates of 52.0% on Composite-Seen and 50.0% on Composite-Unseen, ranking second and first, respectively, among the compared methods. On Composite-Unseen, it exceeds CMA and Cosmos3 by 6.0 and 11.0 percentage points. On Selected-20, our WAM achieves the highest success rate of 51.5%, exceeding CMA, Xiaomi-Robotics-1, and Cosmos3 by 4.5, 5.5, and 6.5 percentage points, respectively. These results are relevant to mobile manipulation, where the policy must coordinate base movement and object manipulation as its viewpoint changes.

Evaluation Settings on Real Robot

Experimental setup.

We conduct real-world experiments on an Astribot S1 dual-arm mobile robot in indoor tabletop environments, with the experimental setups shown in Figure 9. We select three representative tasks to assess mobile manipulation, precise object rearrangement, and execution that depends on interaction history.

Image not extracted

Tasks. (1) Turn-and-Grasp. The robot turns toward the target objects and grasps them. This task evaluates coordination between body reorientation and arm manipulation, requiring the model to localize and interact with objects as its viewpoint changes. (2) Block Swapping. The robot exchanges the positions of a red cube and a blue cube. This task requires identifying the two objects, tracking their locations during manipulation, and placing each cube at the other’s initial position. The small size of the cubes places additional demands on visual localization and grasping and placement precision. (3) Bowl $\times3$ then Spoon. The robot picks up and puts down a bowl three times before picking up a spoon. This task evaluates whether the model can track completed action cycles and transition to the next subtask at the correct stage. Since successive cycles can yield similar visual observations, correct execution requires tracking task progress through interaction history.

Evaluation protocol. We conduct 20 trials per task and report the success rate for each task. A trial is considered successful when the robot completes the full instructed task. For the memory task, success requires exactly three bowl pick-up-and-put-down cycles followed by grasping the spoon in the specified order.

Results on the real robot. Figure 10 shows representative executions of the three tasks, complementing the task-level and stage-wise success rates in Table 6. As shown in Table 6, RynnUnify achieves the highest complete-task success rate on all three tasks, averaging 46.7%, compared with 26.7% for GR00T N1.7 and 25.0% for $\pi_{0.5}$. On Turn-and-Grasp, it achieves 55.0% success, demonstrating effective coordination between body reorientation and manipulation under changing viewpoints. On Block Swapping, its 35.0% success rate highlights improved precision in grasping and placing small objects across successive operations. On Bowl ×3 then Spoon, RynnUnify reaches 50.0% success, exceeding the strongest baseline by 10.0 percentage points. Completing three bowl manipulation cycles before grasping the spoon requires tracking progress across visually similar states, providing behavioral evidence consistent with the utility of visual history for repeated, ordered manipulation.

Model	Turn-and-Grasp: Grasp Block 1	Turn-and-Grasp: Place Block 1	Turn-and-Grasp: Grasp Block 2	Turn-and-Grasp: Task SR	Block Swapping: Grasp Block 1	Block Swapping: Place Block 1	Block Swapping: Grasp Block 2	Block Swapping: Task SR	Bowl $\times3$ then Spoon: Grasp Bowl	Bowl $\times3$ then Spoon: Three Cycles	Bowl $\times3$ then Spoon: Grasp Spoon	Bowl $\times3$ then Spoon: Task SR	Avg. SR
$\pi_{0.5}$ [7]	50.0	30.0	30.0	20.0	70.0	45.0	35.0	15.0	80.0	60.0	50.0	40.0	25.0
GR00T N1.7 [32]	55.0	35.0	35.0	25.0	65.0	55.0	30.0	20.0	90.0	55.0	60.0	35.0	26.7
RynnUnify	90.0	75.0	70.0	55.0	85.0	85.0	60.0	35.0	80.0	60.0	60.0	50.0	46.7

Image not extracted

Conclusion and Future Work

We introduced RynnUnify, a unified embodied model that connects scene understanding, video prediction, and robot action through interleaved video–language reasoning. Building on Cosmos3, RynnUnify combines embodied supervision with cross-chunk scene memory and self-forcing training to support visual planning across successive operations. Its agentic extension learns when to invoke video prediction for reasoning about unobserved states, while its WAM extension transfers the model’s understanding and predictive representations to action generation through latent planning and joint video–action training. We also introduced Rynn-VP and Rynn-GEU to evaluate sustained visual planning and anticipatory embodied reasoning. RynnUnify improves the Rynn-VP overall score from 45.05 to 54.80 over Cosmos 3 Nano, and its agentic extension increases the Rynn-GEU average from 44.0 to 50.0. Together with results on public embodied understanding benchmarks and simulated and real-robot manipulation, these findings support the value of coordinating reasoning with task-relevant visual predictions.

Several challenges remain. Generated videos can distort fine-grained geometry, as reflected by the decreases on some spatial tasks after agentic rollout. Moreover, stronger performance in simulator-free visual planning needs to be complemented by broader validation of physical execution across tasks and environments. Future work will improve geometric fidelity and the selection of reliable generated evidence, study how visual planning scores relate to robot execution outcomes, and reduce the computational cost of interleaved generation for closed-loop control.

Detailed Contributions

• Data: Minghao Zhu, Bohan Hou, Yifei Yang, Kun Xiang, Yinzhou Tang, Zhikai Wang, Zixuan Wang, Tianyi Zhang, Kehan Li, Xin Li

• Model Training:

  • Reasoner Training: Kehan Li, Xin Li, Bohan Hou, Minghao Zhu, Yifei Yang

  • Generator Training: Yinzhou Tang, Minghao Zhu, Bohan Hou, Zixuan Wang, Kun Xiang

  • Agentic RL: Kun Xiang, Minghao Zhu, Yinzhou Tang, Zhikai Wang, Bohan Hou

  • WAM Post-Training: Zixuan Wang, Bohan Hou, Tianyi Zhang, Minghao Zhu, Yinzhou Tang

• Infrastructure: Kun Xiang, Kehan Li, Yinzhou Tang, Minghao Zhu, Zhikai Wang, Tianyi Zhang, Xin Li

• Evaluation: Bohan Hou, Yifei Yang, Minghao Zhu, Kun Xiang, Zixuan Wang, Tianyi Zhang, Zhikai Wang

Qualitative Examples

We provide additional examples of embodied cognition, localization, agentic rollout, and long-horizon visual planning to complement the quantitative evaluations in the main paper.

Embodied Cognition

Figure Figure 11 presents six examples from MindCube and ERQA. The cases cover reasoning across viewpoints, camera motion, interaction grounding, and anticipated object motion, illustrating how visual observations are connected to task-relevant textual answers.

Embodied Localization

Figure Figure 12 presents six examples of task-conditioned localization. Affordance questions return an interaction point, target-area questions return a set of placement-region points, and trajectory questions return an ordered sequence of image-space coordinates.

Agentic Rollout

Figure Figure 13 provides four examples of generation-assisted reasoning about camera motion, object relationships, and action outcomes. Each example connects the input question with a task-specific video prompt, intermediate generated frames, and the final answer.

Long-Horizon Visual Planning

Figure Figure 14 provides additional qualitative cases of long-horizon visual planning on Rynn-VP, complementing the examples in the main paper.

Embodied Cognition

Embodied Localization

Agentic Rollout

More Cases

Prompt Details

Evaluation and Inference Prompts

Table Table 7 presents the evaluation prompts and interaction formats used in our experiments. Benchmark prompts use Python f-string notation where applicable; planning and agentic interactions are shown as input and output formats.

Benchmark	Prompt Template
VSI-Bench	1. f"{question}\nAnswer with the option’s letter from the given choices directly."<br>2. f"{question}\n\nAnswer the question with an exact number, which should be accurate to at most two decimal places."
MMSI-Bench	f"{question}"
MindCube	f"Select the best answer to the following multiple-choice question based on the image.\n{question}\nAnswer with the option’s letter from the given choices directly and only give the best option. The best answer is: "
ERQA	f"{question}\nAnswer with the option letter from the given choices directly."
RoboSpatial	1. f"{question}. Pinpoint several points within the space. Your answer should be formatted as a list of tuples, i.e. [(x1, y1), ...], where each tuple contains the x and y coordinates... indicating the normalized pixel locations of the points."<br>2. f"{question} Answer yes or no."
EmbSpatial	f"Select the best answer to the following multiple-choice question based on the image.\nQuestion: {question}\nOptions:\n{option_string}Answer with the option’s letter from the given choices directly and only give the best option.The best answer is:"
RefSpatial-Bench	f"Locate {object_name} in this image. Output the point coordinates in JSON format."
ShareRobot Affordance	f"{question}The coordinates should be between 0 and 1000, indicating the normalized pixel locations of the point."
ShareRobot Trajectory	f"{question}Your answer should be formatted as a list of tuples, i.e. [(x1, y1), (x2, y2), ...], where each tuple contains the x and y coordinates of a point. The coordinates should be between 0 and 1000, indicating the normalized pixel locations of the point."
Where2Place	f"{question}"
Rynn-VP	Input:<br>[HIGH-LEVEL TASK] {task_instruction}\n [INITIAL STATE] {initial_observation}\n[PREVIOUS STEPS] {subtask_history}\n[CURRENT OBSERVATION] {current_observation}<br>Output format:<br><subplan>{next_executable_subtask}</subplan>\n <world_modeling>\n{scene_context_and_expected_visual_evolution}\n </world_modeling>\n<duration>{duration_in_seconds}</duration>
Rynn-GEU Direct	You are a visual reasoning expert. Answer the question from the supplied visual input. Put only the final answer inside an <answer> </answer> block.
Benchmark	Prompt Template (Continued)
Rynn-GEU Agentic rollout	Input: [VISUAL INPUT] {images_or_video_frames}\n [QUESTION] {question} Generation request: <world_modeling>\n {scene_context_and_requested_visual_prediction}\n </world_modeling>\n <duration>{duration_in_seconds}</duration> Returned visual context: [VIDEO] {generated_video_frames} Final response: <thinking>{reasoning}</thinking>\n <answer>{final_answer}</answer>

Evaluation Hyperparameters

Decoding and preprocessing. Unless otherwise specified, sampling is disabled during autoregressive text generation for RynnUnify. For ERQA, RoboSpatial, and ShareRobot, we enable sampling with a temperature of 0.2, top_p of 0.95, and top_k of 50. For image inputs, min_pixels and max_pixels are set to 16 × 32 × 32 and 16384 × 32 × 32, respectively. For RefSpatial-Bench and ShareRobot, we increase min_pixels to 1024 × 32 × 32 to retain finer spatial detail for precise localization. Input videos are sampled at 2 fps, with a maximum of 512 frames per video. If this limit is exceeded, frames are uniformly subsampled to 512. The minimum pixel budget per frame is 16 × 32 × 32, and the total pixel budget per video is capped at 24576 × 32 × 32. These settings apply to video inputs to the reasoner.

Video generation. On Rynn-VP, RynnUnify generates videos at 480p and 24 fps using 50 diffusion steps, a classifier-free guidance scale of 5, and a fixed random seed. Each segment’s duration is specified by the reasoner-predicted <duration> field. Video-generation baselines use a default duration of 4 seconds per step. Rollouts terminate when the model indicates task completion or reaches 15 steps. For GEU agentic reasoning, videos are generated at 480p and 24 fps using 35 diffusion steps, a classifier-free guidance scale of 5, and a fixed random seed. The generation mode depends on the number of input images: i2v for a single image and fl2v for multiple images. The generated video’s duration is specified by the <duration> field in the generation request.

SFT Training Hyperparameters

Table 8 summarizes the optimization settings for reasoner fine-tuning, generator fine-tuning, and WAM post-training.

Detailed Evaluation Protocol for Rynn-VP

Evaluation setup. Rynn-VP evaluates long-horizon visual planning on 200 task instances across six indoor scene categories. Given a natural-language instruction and an initial observation, each system produces a predicted video or an ordered sequence of step images. Reference demonstrations and timestamped annotations are reserved for evaluation. We assess whether the predicted sequence supports an executable plan, faithfully realizes task-relevant actions and effects, and maintains sufficient visual quality.

Task specification. Before evaluating candidates, we compile each task into a shared specification containing the original entities, required goal states, necessary action dependencies, protected scene relationships, and allowed variations. Task requirements include object identity, quantity, required transformations, destination, and necessary support or release conditions. The demonstration serves as a solvability witness: alternative strategies, motions, and orders of independent operations are accepted when they satisfy the instruction. The same specification is reused across compared systems.

Evaluation procedure. Our evaluator uses GPT-6-sol with the Codex harness and a dedicated evaluation skill defining the evidence requirements and scoring rules. Evaluation comprises five logical stages: observation of objects, actions, and state changes; assessment of visual realization and quality; dense auditing of entity continuity and hallucinations; assessment of planning intelligence; and critical review of the preceding judgments. The standard video evidence package contains at most 50 source frames, including the initial observation, up to four reference anchors, and candidate frames covering the sequence, segment boundaries, and terminal state. Dense auditing uses 30 candidate panels organized into five storyboards. Unresolved contradictions are reviewed before scores are finalized.

Scoring criteria.

We report Planning Intelligence (PI), Visual Realization (VR), and Visual Quality (VQ), each on a 0–100 scale. PI evaluates the executable task structure supported by the sequence, while VR evaluates its visual execution. Table 9 defines their component criteria and within-axis weights. PI and VR are computed as weighted averages of their evidence-calibrated component scores. The overall score is

Criterion	Definition	Weight
Planning Intelligence (PI)
PI-1: Executable Goal Achievement	Whether the required goals are achieved through executable actions involving the correct original entities.	50%
PI-2: Step Validity Rate	Whether individual operations preserve entity correspondence and satisfy action preconditions with feasible effects.	20%
PI-3: Dependency and Order	Whether necessary causal and temporal dependencies are respected, while allowing equivalent orders for independent operations.	15%
PI-4: Plan Completeness	Whether the sequence covers the necessary operations and establishes durable progress, rather than merely repeating attempts.	15%
Visual Realization (VR)
VR-1: Visible Goal Achievement	Whether required object identities, quantities, states, spatial relationships, and necessary support or release are visibly established.	20%
VR-2: Action Evidence Fidelity	Whether relevant contact, grasping, transport, manipulation, and release are supported by clear visual evidence.	20%
VR-3: Entity Consistency	Whether original objects retain consistent identities and quantities, without unexplained creation, duplication, disappearance, or replacement.	15%
VR-4: Effect and State Continuity	Whether task-relevant changes have plausible causes, persist when required, and are inherited by subsequent operations.	15%
VR-5: Cross-Clip Coherence	Whether object identities, scene layouts, container states, and ongoing interactions remain consistent across generated segments.	10%
VR-6: Physical Feasibility	Whether depicted actions and configurations are possible given contact, accessibility, support, and containment constraints.	10%
VR-7: Physical Plausibility	Whether motion and deformation exhibit credible responses to gravity, contact, and support.	7%
VR-8: Perceptual Fluency and Quality	Whether visual execution is clear and temporally readable, with limited blur, flicker, ghosting, tearing, or motion discontinuities.	3%

Visual quality. VQ provides a holistic assessment supported by seven diagnostic dimensions: image clarity, task observability, freedom from artifacts, motion readability, temporal stability, freedom from unintended freezing, and camera continuity. These dimensions explain the quality judgment rather than define a fixed arithmetic average. VQ is assessed separately from the low-weight perceptual component VR-8. For ordered step images, quality is judged according to the native image format; the absence of continuous motion is not itself penalized.

Hallucinations and evidence gaps. We distinguish physical execution violations, entity lifecycle violations, and scene consistency violations. These include motion without a plausible cause, unexplained object duplication or replacement, and unsupported changes to containers or scene structure. Confirmation requires examining the affected original entity, its before-and-after states, the action responsible for the change, and plausible alternative explanations. Missing contact in sparsely sampled or occluded evidence is recorded as an evidence gap unless additional observations support a violation. Ordered step images follow the same task and entity criteria, while unshown intermediate actions remain unverified.

Goal verification and partial credit. Complete execution requires sufficient support for all necessary goal conditions. For placement tasks, we check the specified destination and appropriate support or release; continued hand contact alone does not imply that placement is unfinished. For folding tasks, we inspect persistent folds and reduced spatial extent while allowing different valid folding strategies. Valid subgoals retain partial credit when the full task remains incomplete. Repeated attempts do not accumulate additional credit without new progress, and later actions that undo a required state reduce terminal goal achievement. Apparent completion based on a substituted object or an invalid task-critical state transition does not establish completion of the original task.

Human-guided calibration. Ten participants, all holding a master’s degree or higher, reviewed evaluator judgments across the 200 benchmark tasks. Each participant reviewed 30 cases in rotating batches, yielding 300 participant–case review assignments with overlapping coverage. Review focused on task progress, terminal goal satisfaction, hallucination detection and severity, and appropriate partial credit. Their feedback guided iterative refinement of the evaluation skill and review of affected judgments until the resulting assessments aligned with human preferences regarding the correctness and quality of visual task execution.

Reporting. We report mean PI, VR, VQ, and Overall scores across evaluated tasks, using unrounded per-task values for aggregation. Reports retain criterion-level judgments, supporting visual evidence, and unresolved uncertainties. Missing or invalid candidates are marked unevaluable, with evaluation coverage reported explicitly.

Detailed Training Protocol of Agentic RL

Training settings. We train the agent with 22,007 training questions and 883 questions for in-training validation. The agent is initialized from RynnUnify’s 8B reasoner, while the visual-generation model, reference policy, and semantic judge remain frozen. Each update contains 64 questions and eight trajectories per question, yielding 512 trajectories. Routable questions use four direct and four generation-assisted trajectories; native multi-image questions use eight direct trajectories. We optimize the agent for one epoch using Adam with a constant learning rate of $10^{-6}$, shuffling the questions and cyclically padding the final batch to obtain 344 updates. Responses are sampled at temperature 1.0 with at most 2,048 generated tokens per reasoner turn. We set the advantage stabilizer to $10^{-6}$, the lower and upper GSPO clipping thresholds to 0.0003 and 0.0004, the routing temperature to 0.1, and the routing, KL, and entropy coefficients to 0.01, 0.001, and 0.001, respectively. Validation uses greedy decoding and autonomous D/G selection every ten updates, and checkpoints are saved every twenty updates.

Interaction protocols. A direct trajectory answers from the original observations, whereas a generation-assisted trajectory produces a <world_modeling> instruction and a <duration> value, receives the generated video, and then outputs its reasoning and final answer in <thinking> and <answer> blocks. Each trajectory invokes the generator at most once, with a requested duration in $(0, 10]$ seconds. The generator uses the observed conditioning images; for physical prediction from an observed video prefix, it uses the final decoded frame of that prefix. Reference answers and future videos are excluded from the policy context. The generation request and subsequent answer share one sequence-level importance ratio and advantage, while prompts, forced route prefixes, and visual observations are excluded from the action-token loss. Routing supervision is enabled only when a question is routable, all eight trajectories satisfy their output protocols, and at least one branch has a positive mean reward: $m(x)=\mathbf{1}[\operatorname{routable}(x)]\prod_{i=1}^{8}F(\tau_i)\mathbf{1}[\max(\mu_D,\mu_G)>0]$. The masked routing loss is normalized by the full question batch, and its soft targets are detached from the gradient. Questions with a zero routing mask retain their within-branch policy objective.

Reward design. Both branches use the same task-specific answer score, ensuring that their reward difference measures the benefit of generation under a shared criterion. For routable questions, the final reward is $r(\tau)=F(\tau)A_k(\hat{y},y^{\star})$, where $F(\tau)$ is a binary protocol-validity gate. Native multi-image questions use their native answer score directly. Numerical and geometric tasks use the common shaping function

Table 10 specifies the task-dependent errors and discrete scores.

Task	Scoring rule
Multiple choice	Exact match between the extracted prediction and reference option.
Physical quantities and spatial distance	$A_k=C(e_k)$, with $e_k=|\hat{a}-a^{\star}|/|a^{\star}|$ after compatible-unit conversion.
Affordance and area	$A_k=C(e_k)$, with $e_k=\frac{1}{2000}\left[\frac{1}{|P|}\sum_{p\in P}\min_{q\in P^{\star}}\lVert p-q\rVert_2+\frac{1}{|P^{\star}|}\sum_{q\in P^{\star}}\min_{p\in P}\lVert q-p\rVert_2\right]$.
Trajectory	$A_k=C(e_k)$, with $e_k=d_{\mathrm{DF}(\tilde{P},\tilde{P}^{\star})}/L(P^{\star})$, where $d_{\mathrm{DF}}$ is discrete Fréchet distance and $L(P^{\star})=\sum_{j=2}^{|P^{\star}|}\lVert p_j^{\star}-p_{j-1}^{\star}\rVert_2$.
Textual answers	Normalized deterministic matching where applicable, followed by binary semantic judgment for unmatched answers.
Mess	$A_k=\ell/4$, where $\ell\in\{0,1,2,3,4\}$ measures correctness and coverage of predicted interaction effects.

Numerical parsing preserves signs and checks unit compatibility. A zero-valued reference receives full credit only for an exact zero prediction. Affordance and area scoring uses point sets in the 0–1000 image-coordinate system. For trajectory scoring, we parse up to ten ordered points and resample each trajectory to fifteen points at uniform arc-length intervals, producing $\tilde{P}$ and $\tilde{P}^{\star}$. The normalization uses the reference arc length before resampling. A zero-length reference receives full credit only when the Fréchet distance is zero. Invalid numerical or geometric predictions receive zero answer reward.

Semantic judgments use the frozen Qwen3.8-27B-FP8 judge, which assesses correctness and completeness from the question, reference answer, and extracted final answer, excluding the agent’s reasoning trace and generation request. Mess judgments assess effects on the main non-manipulated objects, including motion direction and magnitude, rotation, and support or contact changes. Levels 0–4 indicate no substantive correct effects, a small minority of correct effects, substantial partial correctness, mostly correct effects with limited omissions, and complete correctness, respectively. Malformed or incomplete judgments trigger up to three evaluation attempts. Generator failures are treated as missing feedback rather than zero rewards, distinguishing infrastructure failures from incorrect predictions.